github-starred/TinyTorch
2
0
Fork 0
mirror of https://github.com/MLSysBook/TinyTorch.git synced 2026-03-11 19:13:51 -05:00
Code Issues 7 Packages Projects Releases Wiki Activity

Clean up stale documentation - remove outdated workflow patterns

Browse Source 
- Remove 5 outdated development guides that contradicted clean NBGrader/nbdev architecture
- Update all documentation to reflect assignments/ directory structure
- Remove references to deprecated #| hide approach and old command patterns
- Ensure clean separation: NBGrader for assignments, nbdev for package export
- Update README, Student Guide, and Instructor Guide with current workflows
This commit is contained in:
Vijay Janapa Reddi
2025-07-12 12:36:31 -04:00
parent 83fb269d9f
commit 3d81f76897
24 changed files with 653 additions and 9978 deletions
Download Patch File Download Diff File Expand all files Collapse all files

252
assignments/source/00_setup/00_setup.ipynb Normal file
View File

@@ -0,0 +1,252 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "fc1bae4e",
"metadata": {
"cell_marker": "\"\"\""
},
"source": [
"# Assignment 0: Setup - TinyTorch Development Environment (INSTRUCTOR VERSION)\n",
"\n",
"This is the instructor solution version showing how solutions are filled in."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "876aef2e",
"metadata": {},
"outputs": [],
"source": [
"#| default_exp core.utils"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e9d346d7",
"metadata": {},
"outputs": [],
"source": [
"#| export\n",
"# Required imports for TinyTorch utilities\n",
"import sys\n",
"import platform\n",
"from datetime import datetime\n",
"import os\n",
"from pathlib import Path"
]
},
{
"cell_type": "markdown",
"id": "687b365f",
"metadata": {
"cell_marker": "\"\"\"",
"lines_to_next_cell": 1
},
"source": [
"## Problem 1: Hello Function (10 points)\n",
"\n",
"Write a function that displays a welcome message for TinyTorch."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c8f5870e",
"metadata": {
"lines_to_next_cell": 1,
"nbgrader": {
"grade": false,
"grade_id": "hello_function",
"locked": false,
"schema_version": 3,
"solution": true,
"task": false
}
},
"outputs": [],
"source": [
"#| export\n",
"def hello_tinytorch():\n",
" \"\"\"\n",
" Display a welcome message for TinyTorch.\n",
" \n",
" This function should:\n",
" 1. Try to load ASCII art from 'tinytorch_flame.txt' if it exists\n",
" 2. If the file doesn't exist, display a simple text banner\n",
" 3. Print \"TinyTorch\" and \"Build ML Systems from Scratch!\"\n",
" 4. Handle any exceptions gracefully\n",
" \"\"\"\n",
" ### BEGIN SOLUTION\n",
" # YOUR CODE HERE\n",
" raise NotImplementedError()\n",
" ### END SOLUTION"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3a6d8a5a",
"metadata": {
"lines_to_next_cell": 1,
"nbgrader": {
"grade": false,
"grade_id": "add_function",
"locked": false,
"schema_version": 3,
"solution": true,
"task": false
}
},
"outputs": [],
"source": [
"#| export\n",
"def add_numbers(a, b):\n",
" \"\"\"\n",
" Add two numbers together.\n",
" \n",
" Args:\n",
" a: First number (int or float)\n",
" b: Second number (int or float)\n",
" \n",
" Returns:\n",
" Sum of a and b\n",
" \"\"\"\n",
" ### BEGIN SOLUTION\n",
" # YOUR CODE HERE\n",
" raise NotImplementedError()\n",
" ### END SOLUTION"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "693d10ea",
"metadata": {
"lines_to_next_cell": 1,
"nbgrader": {
"grade": false,
"grade_id": "systeminfo_class",
"locked": false,
"schema_version": 3,
"solution": true,
"task": false
}
},
"outputs": [],
"source": [
"#| export\n",
"class SystemInfo:\n",
" \"\"\"\n",
" A class for collecting and displaying system information.\n",
" \"\"\"\n",
" \n",
" def __init__(self):\n",
" \"\"\"\n",
" Initialize the SystemInfo object.\n",
" Collect Python version, platform, and machine information.\n",
" \"\"\"\n",
" ### BEGIN SOLUTION\n",
" # YOUR CODE HERE\n",
" raise NotImplementedError()\n",
" ### END SOLUTION\n",
" \n",
" def __str__(self):\n",
" \"\"\"\n",
" Return a formatted string representation of system information.\n",
" Format: \"Python X.Y.Z on Platform (Architecture)\"\n",
" \"\"\"\n",
" ### BEGIN SOLUTION\n",
" # YOUR CODE HERE\n",
" raise NotImplementedError()\n",
" ### END SOLUTION\n",
" \n",
" def is_compatible(self):\n",
" \"\"\"\n",
" Check if the Python version is compatible (>= 3.8).\n",
" Returns True if compatible, False otherwise.\n",
" \"\"\"\n",
" ### BEGIN SOLUTION\n",
" # YOUR CODE HERE\n",
" raise NotImplementedError()\n",
" ### END SOLUTION"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "77e585a9",
"metadata": {
"nbgrader": {
"grade": false,
"grade_id": "developer_profile_class",
"locked": false,
"schema_version": 3,
"solution": true,
"task": false
}
},
"outputs": [],
"source": [
"#| export\n",
"class DeveloperProfile:\n",
" \"\"\"\n",
" A class representing a developer profile.\n",
" \"\"\"\n",
" \n",
" def __init__(self, name=\"Student\", email=\"student@example.com\", affiliation=\"TinyTorch Community\", specialization=\"ML Systems\"):\n",
" \"\"\"\n",
" Initialize a developer profile.\n",
" \n",
" Args:\n",
" name: Developer's name\n",
" email: Developer's email\n",
" affiliation: Developer's affiliation or organization\n",
" specialization: Developer's area of specialization\n",
" \"\"\"\n",
" ### BEGIN SOLUTION\n",
" # YOUR CODE HERE\n",
" raise NotImplementedError()\n",
" ### END SOLUTION\n",
" \n",
" def __str__(self):\n",
" \"\"\"\n",
" Return a basic string representation of the developer.\n",
" Format: \"Name (email)\"\n",
" \"\"\"\n",
" ### BEGIN SOLUTION\n",
" # YOUR CODE HERE\n",
" raise NotImplementedError()\n",
" ### END SOLUTION\n",
" \n",
" def get_signature(self):\n",
" \"\"\"\n",
" Return a formatted signature for the developer.\n",
" Should include name, affiliation, and specialization.\n",
" \"\"\"\n",
" ### BEGIN SOLUTION\n",
" # YOUR CODE HERE\n",
" raise NotImplementedError()\n",
" ### END SOLUTION\n",
" \n",
" def get_profile_info(self):\n",
" \"\"\"\n",
" Return comprehensive profile information as a dictionary.\n",
" \"\"\"\n",
" ### BEGIN SOLUTION\n",
" # YOUR CODE HERE\n",
" raise NotImplementedError()\n",
" ### END SOLUTION "
]
}
],
"metadata": {
"jupytext": {
"main_language": "python"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

434
assignments/source/00_setup/setup_dev.ipynb
View File

@@ -2,86 +2,64 @@
"cells": [
{
"cell_type": "markdown",
"id": "8dad0504",
"id": "fc1bae4e",
"metadata": {
"cell_marker": "\"\"\""
},
"source": [
"# Module 0: Setup - Tiny🔥Torch Development Workflow\n",
"# Assignment 0: Setup - TinyTorch Development Environment (INSTRUCTOR VERSION)\n",
"\n",
"Welcome to TinyTorch! This module teaches you the development workflow you'll use throughout the course.\n",
"\n",
"## Learning Goals\n",
"- Understand the nbdev notebook-to-Python workflow\n",
"- Write your first TinyTorch code\n",
"- Run tests and use the CLI tools\n",
"- Get comfortable with the development rhythm\n",
"\n",
"## The TinyTorch Development Cycle\n",
"\n",
"1. **Write code** in this notebook using `#| export` \n",
"2. **Export code** with `python bin/tito.py sync --module setup`\n",
"3. **Run tests** with `python bin/tito.py test --module setup`\n",
"4. **Check progress** with `python bin/tito.py info`\n",
"\n",
"Let's get started!"
"This is the instructor solution version showing how solutions are filled in."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d8986dab",
"id": "876aef2e",
"metadata": {},
"outputs": [],
"source": [
"#| default_exp core.utils\n",
"\n",
"# Setup imports and environment\n",
"#| default_exp core.utils"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e9d346d7",
"metadata": {},
"outputs": [],
"source": [
"#| export\n",
"# Required imports for TinyTorch utilities\n",
"import sys\n",
"import platform\n",
"from datetime import datetime\n",
"import os\n",
"from pathlib import Path\n",
"\n",
"print(\"🔥 TinyTorch Development Environment\")\n",
"print(f\"Python {sys.version}\")\n",
"print(f\"Platform: {platform.system()} {platform.release()}\")\n",
"print(f\"Started: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\")"
"from pathlib import Path"
]
},
{
"cell_type": "markdown",
"id": "9389fc51",
"id": "687b365f",
"metadata": {
"cell_marker": "\"\"\"",
"lines_to_next_cell": 1
},
"source": [
"## Step 1: Understanding the Module → Package Structure\n",
"## Problem 1: Hello Function (10 points)\n",
"\n",
"**🎓 Teaching vs. 🔧 Building**: This course has two sides:\n",
"- **Teaching side**: You work in `modules/setup/setup_dev.ipynb` (learning-focused)\n",
"- **Building side**: Your code exports to `tinytorch/core/utils.py` (production package)\n",
"\n",
"**Key Concept**: The `#| default_exp core.utils` directive at the top tells nbdev to export all `#| export` cells to `tinytorch/core/utils.py`.\n",
"\n",
"This separation allows us to:\n",
"- Organize learning by **concepts** (modules) \n",
"- Organize code by **function** (package structure)\n",
"- Build a real ML framework while learning systematically\n",
"\n",
"Let's write a simple \"Hello World\" function with the `#| export` directive:"
"Write a function that displays a welcome message for TinyTorch."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "17145cf3",
"id": "c8f5870e",
"metadata": {
"lines_to_next_cell": 1,
"nbgrader": {
"grade": false,
"grade_id": "hello-function",
"grade_id": "hello_function",
"locked": false,
"schema_version": 3,
"solution": true,
@@ -93,36 +71,29 @@
"#| export\n",
"def hello_tinytorch():\n",
" \"\"\"\n",
" A simple hello world function for TinyTorch.\n",
" Display a welcome message for TinyTorch.\n",
" \n",
" TODO: Implement this function to display TinyTorch ASCII art and welcome message.\n",
" Load the flame art from tinytorch_flame.txt file with graceful fallback.\n",
" \n",
" HINTS:\n",
" 1. Try to load ASCII art from 'tinytorch_flame.txt' in current directory\n",
" 2. If file exists, read and print the content\n",
" 3. Add \"Tiny🔥Torch\" and \"Build ML Systems from Scratch!\" messages\n",
" 4. If file doesn't exist, just print the emoji version\n",
" 5. Handle any exceptions gracefully\n",
" \n",
" EXAMPLE OUTPUT:\n",
" [ASCII art from file]\n",
" Tiny🔥Torch\n",
" Build ML Systems from Scratch!\n",
" This function should:\n",
" 1. Try to load ASCII art from 'tinytorch_flame.txt' if it exists\n",
" 2. If the file doesn't exist, display a simple text banner\n",
" 3. Print \"TinyTorch\" and \"Build ML Systems from Scratch!\"\n",
" 4. Handle any exceptions gracefully\n",
" \"\"\"\n",
" ### BEGIN SOLUTION\n",
" # YOUR CODE HERE\n",
" raise NotImplementedError()"
" raise NotImplementedError()\n",
" ### END SOLUTION"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6793b66f",
"id": "3a6d8a5a",
"metadata": {
"lines_to_next_cell": 1,
"nbgrader": {
"grade": false,
"grade_id": "add-function",
"grade_id": "add_function",
"locked": false,
"schema_version": 3,
"solution": true,
@@ -136,111 +107,28 @@
" \"\"\"\n",
" Add two numbers together.\n",
" \n",
" TODO: Implement addition of two numbers.\n",
" This is the foundation of all mathematical operations in ML.\n",
" \n",
" Args:\n",
" a: First number (int or float)\n",
" b: Second number (int or float)\n",
" \n",
" Returns:\n",
" Sum of a and b\n",
" \n",
" EXAMPLE:\n",
" add_numbers(2, 3) should return 5\n",
" add_numbers(1.5, 2.5) should return 4.0\n",
" \"\"\"\n",
" ### BEGIN SOLUTION\n",
" # YOUR CODE HERE\n",
" raise NotImplementedError()"
]
},
{
"cell_type": "markdown",
"id": "6840983e",
"metadata": {
"cell_marker": "\"\"\""
},
"source": [
"### 🧪 Test Your Implementation\n",
"\n",
"Once you implement the functions above, run this cell to test them:"
" raise NotImplementedError()\n",
" ### END SOLUTION"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fb7631b3",
"metadata": {
"nbgrader": {
"grade": true,
"grade_id": "test-hello-function",
"locked": true,
"points": 3,
"schema_version": 3,
"solution": false,
"task": false
}
},
"outputs": [],
"source": [
"# Test hello_tinytorch function\n",
"print(\"Testing hello_tinytorch():\")\n",
"try:\n",
" hello_tinytorch()\n",
" print(\"✅ hello_tinytorch() executed successfully!\")\n",
"except NotImplementedError:\n",
" print(\"❌ hello_tinytorch() not implemented yet\")\n",
" raise"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "60c14231",
"metadata": {
"nbgrader": {
"grade": true,
"grade_id": "test-add-function",
"locked": true,
"points": 2,
"schema_version": 3,
"solution": false,
"task": false
}
},
"outputs": [],
"source": [
"# Test add_numbers function\n",
"print(\"Testing add_numbers():\")\n",
"assert add_numbers(2, 3) == 5, \"add_numbers(2, 3) should return 5\"\n",
"assert add_numbers(0, 0) == 0, \"add_numbers(0, 0) should return 0\"\n",
"assert add_numbers(-1, 1) == 0, \"add_numbers(-1, 1) should return 0\"\n",
"assert abs(add_numbers(1.5, 2.5) - 4.0) < 1e-10, \"add_numbers(1.5, 2.5) should return 4.0\"\n",
"print(\"✅ All addition tests passed!\")"
]
},
{
"cell_type": "markdown",
"id": "ec449d60",
"metadata": {
"cell_marker": "\"\"\"",
"lines_to_next_cell": 1
},
"source": [
"## Step 2: A Simple Class\n",
"\n",
"Let's create a simple class that will help us understand system information. This is still basic, but shows how to structure classes in TinyTorch."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "43affa0b",
"id": "693d10ea",
"metadata": {
"lines_to_next_cell": 1,
"nbgrader": {
"grade": false,
"grade_id": "systeminfo-class",
"grade_id": "systeminfo_class",
"locked": false,
"schema_version": 3,
"solution": true,
@@ -252,168 +140,48 @@
"#| export\n",
"class SystemInfo:\n",
" \"\"\"\n",
" Simple system information class.\n",
" \n",
" TODO: Implement this class to collect and display system information.\n",
" \n",
" REQUIREMENTS:\n",
" 1. __init__: Collect Python version, platform, and machine information\n",
" 2. __str__: Return formatted system info string\n",
" 3. is_compatible: Check if Python version >= 3.8\n",
" \n",
" HINTS:\n",
" - Use sys.version_info for Python version\n",
" - Use platform.system() for platform name \n",
" - Use platform.machine() for machine architecture\n",
" - Store these as instance attributes in __init__\n",
" A class for collecting and displaying system information.\n",
" \"\"\"\n",
" \n",
" def __init__(self):\n",
" \"\"\"\n",
" Initialize system information collection.\n",
" \n",
" TODO: Collect Python version, platform, and machine information.\n",
" Store as instance attributes: self.python_version, self.platform, self.machine\n",
" Initialize the SystemInfo object.\n",
" Collect Python version, platform, and machine information.\n",
" \"\"\"\n",
" ### BEGIN SOLUTION\n",
" # YOUR CODE HERE\n",
" raise NotImplementedError()\n",
" ### END SOLUTION\n",
" \n",
" def __str__(self):\n",
" \"\"\"\n",
" Return human-readable system information.\n",
" \n",
" TODO: Format system info as a readable string.\n",
" FORMAT: \"Python X.Y on Platform (Architecture)\"\n",
" EXAMPLE: \"Python 3.9 on Darwin (arm64)\"\n",
" Return a formatted string representation of system information.\n",
" Format: \"Python X.Y.Z on Platform (Architecture)\"\n",
" \"\"\"\n",
" ### BEGIN SOLUTION\n",
" # YOUR CODE HERE\n",
" raise NotImplementedError()\n",
" ### END SOLUTION\n",
" \n",
" def is_compatible(self):\n",
" \"\"\"\n",
" Check if system meets minimum requirements.\n",
" \n",
" TODO: Check if Python version >= 3.8\n",
" Return True if compatible, False otherwise\n",
" Check if the Python version is compatible (>= 3.8).\n",
" Returns True if compatible, False otherwise.\n",
" \"\"\"\n",
" ### BEGIN SOLUTION\n",
" # YOUR CODE HERE\n",
" raise NotImplementedError()"
]
},
{
"cell_type": "markdown",
"id": "81445d5d",
"metadata": {
"cell_marker": "\"\"\""
},
"source": [
"### 🧪 Test Your SystemInfo Class\n",
"\n",
"Once you implement the SystemInfo class above, run this cell to test it:"
" raise NotImplementedError()\n",
" ### END SOLUTION"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "23bbf329",
"id": "77e585a9",
"metadata": {
"nbgrader": {
"grade": true,
"grade_id": "test-systeminfo-creation",
"locked": true,
"points": 2,
"schema_version": 3,
"solution": false,
"task": false
}
},
"outputs": [],
"source": [
"# Test SystemInfo creation\n",
"print(\"Testing SystemInfo creation...\")\n",
"info = SystemInfo()\n",
"assert hasattr(info, 'python_version'), \"SystemInfo should have python_version attribute\"\n",
"assert hasattr(info, 'platform'), \"SystemInfo should have platform attribute\"\n",
"assert hasattr(info, 'machine'), \"SystemInfo should have machine attribute\"\n",
"print(\"✅ SystemInfo creation test passed!\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "621f5acf",
"metadata": {
"nbgrader": {
"grade": true,
"grade_id": "test-systeminfo-str",
"locked": true,
"points": 2,
"schema_version": 3,
"solution": false,
"task": false
}
},
"outputs": [],
"source": [
"# Test SystemInfo string representation\n",
"print(\"Testing SystemInfo string representation...\")\n",
"info = SystemInfo()\n",
"info_str = str(info)\n",
"assert isinstance(info_str, str), \"SystemInfo.__str__() should return a string\"\n",
"assert len(info_str) > 0, \"SystemInfo string should not be empty\"\n",
"assert 'Python' in info_str, \"SystemInfo string should contain 'Python'\"\n",
"print(f\"✅ SystemInfo string: {info_str}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "da5e0d37",
"metadata": {
"nbgrader": {
"grade": true,
"grade_id": "test-systeminfo-compatibility",
"locked": true,
"points": 1,
"schema_version": 3,
"solution": false,
"task": false
}
},
"outputs": [],
"source": [
"# Test SystemInfo compatibility check\n",
"print(\"Testing SystemInfo compatibility...\")\n",
"info = SystemInfo()\n",
"compatible = info.is_compatible()\n",
"assert isinstance(compatible, bool), \"is_compatible() should return a boolean\"\n",
"# Since we're running this test, Python should be >= 3.8\n",
"assert compatible == True, \"Current Python version should be compatible (>= 3.8)\"\n",
"print(\"✅ SystemInfo compatibility test passed!\")"
]
},
{
"cell_type": "markdown",
"id": "55e72365",
"metadata": {
"cell_marker": "\"\"\"",
"lines_to_next_cell": 1
},
"source": [
"## Step 3: Developer Profile (Optional Challenge)\n",
"\n",
"For students who want an extra challenge, implement a DeveloperProfile class:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "75f3279f",
"metadata": {
"lines_to_next_cell": 1,
"nbgrader": {
"grade": false,
"grade_id": "developer-profile",
"grade_id": "developer_profile_class",
"locked": false,
"schema_version": 3,
"solution": true,
@@ -425,72 +193,52 @@
"#| export\n",
"class DeveloperProfile:\n",
" \"\"\"\n",
" Developer profile for personalizing TinyTorch experience.\n",
" \n",
" TODO: OPTIONAL CHALLENGE - Implement this class for extra credit!\n",
" \n",
" REQUIREMENTS:\n",
" 1. Store developer information (name, email, etc.)\n",
" 2. Load ASCII art from file with fallback\n",
" 3. Generate formatted profile display\n",
" 4. Create professional signature\n",
" \n",
" This is an advanced exercise - only attempt after completing the required parts!\n",
" A class representing a developer profile.\n",
" \"\"\"\n",
" \n",
" def __init__(self, name=\"Student\", email=\"student@example.com\"):\n",
" def __init__(self, name=\"Student\", email=\"student@example.com\", affiliation=\"TinyTorch Community\", specialization=\"ML Systems\"):\n",
" \"\"\"\n",
" Initialize developer profile.\n",
" Initialize a developer profile.\n",
" \n",
" TODO: Store developer information with defaults.\n",
" Feel free to customize with your own info!\n",
" Args:\n",
" name: Developer's name\n",
" email: Developer's email\n",
" affiliation: Developer's affiliation or organization\n",
" specialization: Developer's area of specialization\n",
" \"\"\"\n",
" # YOUR CODE HERE (OPTIONAL)\n",
" self.name = name\n",
" self.email = email\n",
" ### BEGIN SOLUTION\n",
" # YOUR CODE HERE\n",
" raise NotImplementedError()\n",
" ### END SOLUTION\n",
" \n",
" def __str__(self):\n",
" \"\"\"\n",
" Return a basic string representation of the developer.\n",
" Format: \"Name (email)\"\n",
" \"\"\"\n",
" ### BEGIN SOLUTION\n",
" # YOUR CODE HERE\n",
" raise NotImplementedError()\n",
" ### END SOLUTION\n",
" \n",
" def get_signature(self):\n",
" \"\"\"\n",
" Get a short signature for code headers.\n",
" \n",
" TODO: Return a signature like \"Built by Name (email)\"\n",
" Return a formatted signature for the developer.\n",
" Should include name, affiliation, and specialization.\n",
" \"\"\"\n",
" # YOUR CODE HERE (OPTIONAL)\n",
" return f\"Built by {self.name} ({self.email})\""
]
},
{
"cell_type": "markdown",
"id": "6efefa44",
"metadata": {
"cell_marker": "\"\"\""
},
"source": [
"## 🎯 Module Summary\n",
"\n",
"Congratulations! You've completed the TinyTorch setup module:\n",
"\n",
"### What You've Accomplished\n",
"✅ **Environment Setup**: Learned the development workflow \n",
"✅ **First Function**: Implemented hello_tinytorch() with file handling \n",
"✅ **Math Operations**: Built add_numbers() for ML foundations \n",
"✅ **Object-Oriented Programming**: Created SystemInfo class with properties \n",
"✅ **Testing**: Verified your implementations with automated tests \n",
"✅ **Package Export**: Used nbdev to build the tinytorch package \n",
"\n",
"### Key Concepts You've Learned\n",
"- **nbdev workflow**: From notebook to production package\n",
"- **File handling**: Reading ASCII art with graceful fallbacks\n",
"- **System information**: Collecting platform and version data\n",
"- **Object-oriented design**: Classes, properties, and methods\n",
"- **Error handling**: Using try/except and fallback strategies\n",
"\n",
"### Next Steps\n",
"1. **Export your code**: Run `python bin/tito.py sync --module setup`\n",
"2. **Run tests**: Use `python bin/tito.py test --module setup`\n",
"3. **Check your work**: Import your functions with `from tinytorch.core.utils import hello_tinytorch`\n",
"\n",
"**Ready for the next challenge?** Let's move on to building tensors!"
" ### BEGIN SOLUTION\n",
" # YOUR CODE HERE\n",
" raise NotImplementedError()\n",
" ### END SOLUTION\n",
" \n",
" def get_profile_info(self):\n",
" \"\"\"\n",
" Return comprehensive profile information as a dictionary.\n",
" \"\"\"\n",
" ### BEGIN SOLUTION\n",
" # YOUR CODE HERE\n",
" raise NotImplementedError()\n",
" ### END SOLUTION "
]
}
],

288
assignments/source/00_setup/setup_dev.py
View File

@@ -10,295 +10,149 @@
# %% [markdown]
"""
# Module 0: Setup - Tiny🔥Torch Development Workflow
# Assignment 0: Setup - TinyTorch Development Environment (INSTRUCTOR VERSION)
Welcome to TinyTorch! This module teaches you the development workflow you'll use throughout the course.
## Learning Goals
- Understand the nbdev notebook-to-Python workflow
- Write your first TinyTorch code
- Run tests and use the CLI tools
- Get comfortable with the development rhythm
## The TinyTorch Development Cycle
1. **Write code** in this notebook using `#| export`
2. **Export code** with `python bin/tito.py sync --module setup`
3. **Run tests** with `python bin/tito.py test --module setup`
4. **Check progress** with `python bin/tito.py info`
Let's get started!
This is the instructor solution version showing how solutions are filled in.
"""
# %%
#| default_exp core.utils
# Setup imports and environment
# %%
#| export
# Required imports for TinyTorch utilities
import sys
import platform
from datetime import datetime
import os
from pathlib import Path
print("🔥 TinyTorch Development Environment")
print(f"Python {sys.version}")
print(f"Platform: {platform.system()} {platform.release()}")
print(f"Started: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}")
# %% [markdown]
"""
## Step 1: Understanding the Module → Package Structure
## Problem 1: Hello Function (10 points)
**🎓 Teaching vs. 🔧 Building**: This course has two sides:
- **Teaching side**: You work in `modules/setup/setup_dev.ipynb` (learning-focused)
- **Building side**: Your code exports to `tinytorch/core/utils.py` (production package)
**Key Concept**: The `#| default_exp core.utils` directive at the top tells nbdev to export all `#| export` cells to `tinytorch/core/utils.py`.
This separation allows us to:
- Organize learning by **concepts** (modules)
- Organize code by **function** (package structure)
- Build a real ML framework while learning systematically
Let's write a simple "Hello World" function with the `#| export` directive:
Write a function that displays a welcome message for TinyTorch.
"""
# %% nbgrader={"grade": false, "grade_id": "hello-function", "locked": false, "schema_version": 3, "solution": true, "task": false}
# %% nbgrader={"grade": false, "grade_id": "hello_function", "locked": false, "schema_version": 3, "solution": true, "task": false}
#| export
def hello_tinytorch():
"""
A simple hello world function for TinyTorch.
Display a welcome message for TinyTorch.
TODO: Implement this function to display TinyTorch ASCII art and welcome message.
Load the flame art from tinytorch_flame.txt file with graceful fallback.
HINTS:
1. Try to load ASCII art from 'tinytorch_flame.txt' in current directory
2. If file exists, read and print the content
3. Add "Tiny🔥Torch" and "Build ML Systems from Scratch!" messages
4. If file doesn't exist, just print the emoji version
5. Handle any exceptions gracefully
EXAMPLE OUTPUT:
[ASCII art from file]
Tiny🔥Torch
Build ML Systems from Scratch!
This function should:
1. Try to load ASCII art from 'tinytorch_flame.txt' if it exists
2. If the file doesn't exist, display a simple text banner
3. Print "TinyTorch" and "Build ML Systems from Scratch!"
4. Handle any exceptions gracefully
"""
### BEGIN SOLUTION
# YOUR CODE HERE
raise NotImplementedError()
### END SOLUTION
# %% nbgrader={"grade": false, "grade_id": "add-function", "locked": false, "schema_version": 3, "solution": true, "task": false}
# %% nbgrader={"grade": false, "grade_id": "add_function", "locked": false, "schema_version": 3, "solution": true, "task": false}
#| export
def add_numbers(a, b):
"""
Add two numbers together.
TODO: Implement addition of two numbers.
This is the foundation of all mathematical operations in ML.
Args:
a: First number (int or float)
b: Second number (int or float)
Returns:
Sum of a and b
EXAMPLE:
add_numbers(2, 3) should return 5
add_numbers(1.5, 2.5) should return 4.0
"""
### BEGIN SOLUTION
# YOUR CODE HERE
raise NotImplementedError()
### END SOLUTION
# %% [markdown]
"""
### 🧪 Test Your Implementation
Once you implement the functions above, run this cell to test them:
"""
# %% nbgrader={"grade": true, "grade_id": "test-hello-function", "locked": true, "points": 3, "schema_version": 3, "solution": false, "task": false}
# Test hello_tinytorch function
print("Testing hello_tinytorch():")
try:
hello_tinytorch()
print("✅ hello_tinytorch() executed successfully!")
except NotImplementedError:
print("❌ hello_tinytorch() not implemented yet")
raise
# %% nbgrader={"grade": true, "grade_id": "test-add-function", "locked": true, "points": 2, "schema_version": 3, "solution": false, "task": false}
# Test add_numbers function
print("Testing add_numbers():")
assert add_numbers(2, 3) == 5, "add_numbers(2, 3) should return 5"
assert add_numbers(0, 0) == 0, "add_numbers(0, 0) should return 0"
assert add_numbers(-1, 1) == 0, "add_numbers(-1, 1) should return 0"
assert abs(add_numbers(1.5, 2.5) - 4.0) < 1e-10, "add_numbers(1.5, 2.5) should return 4.0"
print("✅ All addition tests passed!")
# %% [markdown]
"""
## Step 2: A Simple Class
Let's create a simple class that will help us understand system information. This is still basic, but shows how to structure classes in TinyTorch.
"""
# %% nbgrader={"grade": false, "grade_id": "systeminfo-class", "locked": false, "schema_version": 3, "solution": true, "task": false}
# %% nbgrader={"grade": false, "grade_id": "systeminfo_class", "locked": false, "schema_version": 3, "solution": true, "task": false}
#| export
class SystemInfo:
"""
Simple system information class.
TODO: Implement this class to collect and display system information.
REQUIREMENTS:
1. __init__: Collect Python version, platform, and machine information
2. __str__: Return formatted system info string
3. is_compatible: Check if Python version >= 3.8
HINTS:
- Use sys.version_info for Python version
- Use platform.system() for platform name
- Use platform.machine() for machine architecture
- Store these as instance attributes in __init__
A class for collecting and displaying system information.
"""
def __init__(self):
"""
Initialize system information collection.
TODO: Collect Python version, platform, and machine information.
Store as instance attributes: self.python_version, self.platform, self.machine
Initialize the SystemInfo object.
Collect Python version, platform, and machine information.
"""
### BEGIN SOLUTION
# YOUR CODE HERE
raise NotImplementedError()
### END SOLUTION
def __str__(self):
"""
Return human-readable system information.
TODO: Format system info as a readable string.
FORMAT: "Python X.Y on Platform (Architecture)"
EXAMPLE: "Python 3.9 on Darwin (arm64)"
Return a formatted string representation of system information.
Format: "Python X.Y.Z on Platform (Architecture)"
"""
### BEGIN SOLUTION
# YOUR CODE HERE
raise NotImplementedError()
### END SOLUTION
def is_compatible(self):
"""
Check if system meets minimum requirements.
TODO: Check if Python version >= 3.8
Return True if compatible, False otherwise
Check if the Python version is compatible (>= 3.8).
Returns True if compatible, False otherwise.
"""
### BEGIN SOLUTION
# YOUR CODE HERE
raise NotImplementedError()
### END SOLUTION
# %% [markdown]
"""
### 🧪 Test Your SystemInfo Class
Once you implement the SystemInfo class above, run this cell to test it:
"""
# %% nbgrader={"grade": true, "grade_id": "test-systeminfo-creation", "locked": true, "points": 2, "schema_version": 3, "solution": false, "task": false}
# Test SystemInfo creation
print("Testing SystemInfo creation...")
info = SystemInfo()
assert hasattr(info, 'python_version'), "SystemInfo should have python_version attribute"
assert hasattr(info, 'platform'), "SystemInfo should have platform attribute"
assert hasattr(info, 'machine'), "SystemInfo should have machine attribute"
print("✅ SystemInfo creation test passed!")
# %% nbgrader={"grade": true, "grade_id": "test-systeminfo-str", "locked": true, "points": 2, "schema_version": 3, "solution": false, "task": false}
# Test SystemInfo string representation
print("Testing SystemInfo string representation...")
info = SystemInfo()
info_str = str(info)
assert isinstance(info_str, str), "SystemInfo.__str__() should return a string"
assert len(info_str) > 0, "SystemInfo string should not be empty"
assert 'Python' in info_str, "SystemInfo string should contain 'Python'"
print(f"✅ SystemInfo string: {info_str}")
# %% nbgrader={"grade": true, "grade_id": "test-systeminfo-compatibility", "locked": true, "points": 1, "schema_version": 3, "solution": false, "task": false}
# Test SystemInfo compatibility check
print("Testing SystemInfo compatibility...")
info = SystemInfo()
compatible = info.is_compatible()
assert isinstance(compatible, bool), "is_compatible() should return a boolean"
# Since we're running this test, Python should be >= 3.8
assert compatible == True, "Current Python version should be compatible (>= 3.8)"
print("✅ SystemInfo compatibility test passed!")
# %% [markdown]
"""
## Step 3: Developer Profile (Optional Challenge)
For students who want an extra challenge, implement a DeveloperProfile class:
"""
# %% nbgrader={"grade": false, "grade_id": "developer-profile", "locked": false, "schema_version": 3, "solution": true, "task": false}
# %% nbgrader={"grade": false, "grade_id": "developer_profile_class", "locked": false, "schema_version": 3, "solution": true, "task": false}
#| export
class DeveloperProfile:
"""
Developer profile for personalizing TinyTorch experience.
TODO: OPTIONAL CHALLENGE - Implement this class for extra credit!
REQUIREMENTS:
1. Store developer information (name, email, etc.)
2. Load ASCII art from file with fallback
3. Generate formatted profile display
4. Create professional signature
This is an advanced exercise - only attempt after completing the required parts!
A class representing a developer profile.
"""
def __init__(self, name="Student", email="student@example.com"):
def __init__(self, name="Student", email="student@example.com", affiliation="TinyTorch Community", specialization="ML Systems"):
"""
Initialize developer profile.
Initialize a developer profile.
TODO: Store developer information with defaults.
Feel free to customize with your own info!
Args:
name: Developer's name
email: Developer's email
affiliation: Developer's affiliation or organization
specialization: Developer's area of specialization
"""
# YOUR CODE HERE (OPTIONAL)
self.name = name
self.email = email
### BEGIN SOLUTION
# YOUR CODE HERE
raise NotImplementedError()
### END SOLUTION
def __str__(self):
"""
Return a basic string representation of the developer.
Format: "Name (email)"
"""
### BEGIN SOLUTION
# YOUR CODE HERE
raise NotImplementedError()
### END SOLUTION
def get_signature(self):
"""
Get a short signature for code headers.
TODO: Return a signature like "Built by Name (email)"
Return a formatted signature for the developer.
Should include name, affiliation, and specialization.
"""
# YOUR CODE HERE (OPTIONAL)
return f"Built by {self.name} ({self.email})"
# %% [markdown]
"""
## 🎯 Module Summary
Congratulations! You've completed the TinyTorch setup module:
### What You've Accomplished
✅ **Environment Setup**: Learned the development workflow
✅ **First Function**: Implemented hello_tinytorch() with file handling
✅ **Math Operations**: Built add_numbers() for ML foundations
✅ **Object-Oriented Programming**: Created SystemInfo class with properties
✅ **Testing**: Verified your implementations with automated tests
✅ **Package Export**: Used nbdev to build the tinytorch package
### Key Concepts You've Learned
- **nbdev workflow**: From notebook to production package
- **File handling**: Reading ASCII art with graceful fallbacks
- **System information**: Collecting platform and version data
- **Object-oriented design**: Classes, properties, and methods
- **Error handling**: Using try/except and fallback strategies
### Next Steps
1. **Export your code**: Run `python bin/tito.py sync --module setup`
2. **Run tests**: Use `python bin/tito.py test --module setup`
3. **Check your work**: Import your functions with `from tinytorch.core.utils import hello_tinytorch`
**Ready for the next challenge?** Let's move on to building tensors!
"""
### BEGIN SOLUTION
# YOUR CODE HERE
raise NotImplementedError()
### END SOLUTION
def get_profile_info(self):
"""
Return comprehensive profile information as a dictionary.
"""
### BEGIN SOLUTION
# YOUR CODE HERE
raise NotImplementedError()
### END SOLUTION

1047
assignments/source/01_tensor/01_tensor.ipynb
View File

File diff suppressed because it is too large Load Diff

1047
assignments/source/01_tensor/tensor_dev.ipynb
View File

File diff suppressed because it is too large Load Diff

2
assignments/source/02_activations/README.md
View File

@@ -51,7 +51,7 @@ This module teaches you the mathematical foundations that make deep learning pos
1. **Open the development file**:
```bash
python bin/tito.py jupyter
# Then open modules/activations/activations_dev.py
# Then open assignments/source/02_activations/activations_dev.py
```
2. **Implement the functions**:

1143
assignments/source/02_activations/activations_dev.ipynb
View File

File diff suppressed because it is too large Load Diff

797
assignments/source/03_layers/layers_dev.ipynb
View File

@@ -1,797 +0,0 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "2668bc45",
"metadata": {
"cell_marker": "\"\"\""
},
"source": [
"# Module 2: Layers - Neural Network Building Blocks\n",
"\n",
"Welcome to the Layers module! This is where neural networks begin. You'll implement the fundamental building blocks that transform tensors.\n",
"\n",
"## Learning Goals\n",
"- Understand layers as functions that transform tensors: `y = f(x)`\n",
"- Implement Dense layers with linear transformations: `y = Wx + b`\n",
"- Use activation functions from the activations module for nonlinearity\n",
"- See how neural networks are just function composition\n",
"- Build intuition before diving into training\n",
"\n",
"## Build → Use → Understand\n",
"1. **Build**: Dense layers using activation functions as building blocks\n",
"2. **Use**: Transform tensors and see immediate results\n",
"3. **Understand**: How neural networks transform information\n",
"\n",
"## Module Dependencies\n",
"This module builds on the **activations** module:\n",
"- **activations** → **layers** → **networks**\n",
"- Clean separation of concerns: math functions → layer building blocks → full networks"
]
},
{
"cell_type": "markdown",
"id": "530716e8",
"metadata": {
"cell_marker": "\"\"\""
},
"source": [
"## 📦 Where This Code Lives in the Final Package\n",
"\n",
"**Learning Side:** You work in `assignments/source/03_layers/layers_dev.py` \n",
"**Building Side:** Code exports to `tinytorch.core.layers`\n",
"\n",
"```python\n",
"# Final package structure:\n",
"from tinytorch.core.layers import Dense, Conv2D # All layers together!\n",
"from tinytorch.core.activations import ReLU, Sigmoid, Tanh\n",
"from tinytorch.core.tensor import Tensor\n",
"```\n",
"\n",
"**Why this matters:**\n",
"- **Learning:** Focused modules for deep understanding\n",
"- **Production:** Proper organization like PyTorch's `torch.nn`\n",
"- **Consistency:** All layers (Dense, Conv2D) live together in `core.layers`"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4f63809e",
"metadata": {},
"outputs": [],
"source": [
"#| default_exp core.layers\n",
"\n",
"# Setup and imports\n",
"import numpy as np\n",
"import sys\n",
"from typing import Union, Optional, Callable\n",
"import math"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "00a72b7c",
"metadata": {},
"outputs": [],
"source": [
"#| export\n",
"import numpy as np\n",
"import math\n",
"import sys\n",
"from typing import Union, Optional, Callable\n",
"\n",
"# Import from the main package (rock solid foundation)\n",
"from tinytorch.core.tensor import Tensor\n",
"from tinytorch.core.activations import ReLU, Sigmoid, Tanh\n",
"\n",
"# print(\"🔥 TinyTorch Layers Module\")\n",
"# print(f\"NumPy version: {np.__version__}\")\n",
"# print(f\"Python version: {sys.version_info.major}.{sys.version_info.minor}\")\n",
"# print(\"Ready to build neural network layers!\")"
]
},
{
"cell_type": "markdown",
"id": "a0ad08ea",
"metadata": {
"cell_marker": "\"\"\""
},
"source": [
"## Step 1: What is a Layer?\n",
"\n",
"### Definition\n",
"A **layer** is a function that transforms tensors. Think of it as a mathematical operation that takes input data and produces output data:\n",
"\n",
"```\n",
"Input Tensor → Layer → Output Tensor\n",
"```\n",
"\n",
"### Why Layers Matter in Neural Networks\n",
"Layers are the fundamental building blocks of all neural networks because:\n",
"- **Modularity**: Each layer has a specific job (linear transformation, nonlinearity, etc.)\n",
"- **Composability**: Layers can be combined to create complex functions\n",
"- **Learnability**: Each layer has parameters that can be learned from data\n",
"- **Interpretability**: Different layers learn different features\n",
"\n",
"### The Fundamental Insight\n",
"**Neural networks are just function composition!**\n",
"```\n",
"x → Layer1 → Layer2 → Layer3 → y\n",
"```\n",
"\n",
"Each layer transforms the data, and the final output is the composition of all these transformations.\n",
"\n",
"### Real-World Examples\n",
"- **Dense Layer**: Learns linear relationships between features\n",
"- **Convolutional Layer**: Learns spatial patterns in images\n",
"- **Recurrent Layer**: Learns temporal patterns in sequences\n",
"- **Activation Layer**: Adds nonlinearity to make networks powerful\n",
"\n",
"### Visual Intuition\n",
"```\n",
"Input: [1, 2, 3] (3 features)\n",
"Dense Layer: y = Wx + b\n",
"Weights W: [[0.1, 0.2, 0.3],\n",
" [0.4, 0.5, 0.6]] (2×3 matrix)\n",
"Bias b: [0.1, 0.2] (2 values)\n",
"Output: [0.1*1 + 0.2*2 + 0.3*3 + 0.1,\n",
" 0.4*1 + 0.5*2 + 0.6*3 + 0.2] = [1.4, 3.2]\n",
"```\n",
"\n",
"Let's start with the most important layer: **Dense** (also called Linear or Fully Connected)."
]
},
{
"cell_type": "markdown",
"id": "5d63d076",
"metadata": {
"cell_marker": "\"\"\"",
"lines_to_next_cell": 1
},
"source": [
"## Step 2: Understanding Matrix Multiplication\n",
"\n",
"Before we build layers, let's understand the core operation: **matrix multiplication**. This is what powers all neural network computations.\n",
"\n",
"### Why Matrix Multiplication Matters\n",
"- **Efficiency**: Process multiple inputs at once\n",
"- **Parallelization**: GPU acceleration works great with matrix operations\n",
"- **Batch processing**: Handle multiple samples simultaneously\n",
"- **Mathematical foundation**: Linear algebra is the language of neural networks\n",
"\n",
"### The Math Behind It\n",
"For matrices A (m×n) and B (n×p), the result C (m×p) is:\n",
"```\n",
"C[i,j] = sum(A[i,k] * B[k,j] for k in range(n))\n",
"```\n",
"\n",
"### Visual Example\n",
"```\n",
"A = [[1, 2], B = [[5, 6],\n",
" [3, 4]] [7, 8]]\n",
"\n",
"C = A @ B = [[1*5 + 2*7, 1*6 + 2*8],\n",
" [3*5 + 4*7, 3*6 + 4*8]]\n",
" = [[19, 22],\n",
" [43, 50]]\n",
"```\n",
"\n",
"Let's implement this step by step!"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "82cc8565",
"metadata": {
"lines_to_next_cell": 1
},
"outputs": [],
"source": [
"#| export\n",
"def matmul_naive(A: np.ndarray, B: np.ndarray) -> np.ndarray:\n",
" \"\"\"\n",
" Naive matrix multiplication using explicit for-loops.\n",
" \n",
" This helps you understand what matrix multiplication really does!\n",
" \n",
" Args:\n",
" A: Matrix of shape (m, n)\n",
" B: Matrix of shape (n, p)\n",
" \n",
" Returns:\n",
" Matrix of shape (m, p) where C[i,j] = sum(A[i,k] * B[k,j] for k in range(n))\n",
" \n",
" TODO: Implement matrix multiplication using three nested for-loops.\n",
" \n",
" APPROACH:\n",
" 1. Get the dimensions: m, n from A and n2, p from B\n",
" 2. Check that n == n2 (matrices must be compatible)\n",
" 3. Create output matrix C of shape (m, p) filled with zeros\n",
" 4. Use three nested loops:\n",
" - i loop: rows of A (0 to m-1)\n",
" - j loop: columns of B (0 to p-1) \n",
" - k loop: shared dimension (0 to n-1)\n",
" 5. For each (i,j), compute: C[i,j] += A[i,k] * B[k,j]\n",
" \n",
" EXAMPLE:\n",
" A = [[1, 2], B = [[5, 6],\n",
" [3, 4]] [7, 8]]\n",
" \n",
" C[0,0] = A[0,0]*B[0,0] + A[0,1]*B[1,0] = 1*5 + 2*7 = 19\n",
" C[0,1] = A[0,0]*B[0,1] + A[0,1]*B[1,1] = 1*6 + 2*8 = 22\n",
" C[1,0] = A[1,0]*B[0,0] + A[1,1]*B[1,0] = 3*5 + 4*7 = 43\n",
" C[1,1] = A[1,0]*B[0,1] + A[1,1]*B[1,1] = 3*6 + 4*8 = 50\n",
" \n",
" HINTS:\n",
" - Start with C = np.zeros((m, p))\n",
" - Use three nested for loops: for i in range(m): for j in range(p): for k in range(n):\n",
" - Accumulate the sum: C[i,j] += A[i,k] * B[k,j]\n",
" \"\"\"\n",
" raise NotImplementedError(\"Student implementation required\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ea923f30",
"metadata": {
"lines_to_next_cell": 1
},
"outputs": [],
"source": [
"#| hide\n",
"#| export\n",
"def matmul_naive(A: np.ndarray, B: np.ndarray) -> np.ndarray:\n",
" \"\"\"\n",
" Naive matrix multiplication using explicit for-loops.\n",
" \n",
" This helps you understand what matrix multiplication really does!\n",
" \"\"\"\n",
" m, n = A.shape\n",
" n2, p = B.shape\n",
" assert n == n2, f\"Matrix shapes don't match: A({m},{n}) @ B({n2},{p})\"\n",
" \n",
" C = np.zeros((m, p))\n",
" for i in range(m):\n",
" for j in range(p):\n",
" for k in range(n):\n",
" C[i, j] += A[i, k] * B[k, j]\n",
" return C"
]
},
{
"cell_type": "markdown",
"id": "60fb8544",
"metadata": {
"cell_marker": "\"\"\""
},
"source": [
"### 🧪 Test Your Matrix Multiplication"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "28898e45",
"metadata": {},
"outputs": [],
"source": [
"# Test matrix multiplication\n",
"print(\"Testing matrix multiplication...\")\n",
"\n",
"try:\n",
" # Test case 1: Simple 2x2 matrices\n",
" A = np.array([[1, 2], [3, 4]], dtype=np.float32)\n",
" B = np.array([[5, 6], [7, 8]], dtype=np.float32)\n",
" \n",
" result = matmul_naive(A, B)\n",
" expected = np.array([[19, 22], [43, 50]], dtype=np.float32)\n",
" \n",
" print(f\"✅ Matrix A:\\n{A}\")\n",
" print(f\"✅ Matrix B:\\n{B}\")\n",
" print(f\"✅ Your result:\\n{result}\")\n",
" print(f\"✅ Expected:\\n{expected}\")\n",
" \n",
" assert np.allclose(result, expected), \"❌ Result doesn't match expected!\"\n",
" print(\"🎉 Matrix multiplication works!\")\n",
" \n",
" # Test case 2: Compare with NumPy\n",
" numpy_result = A @ B\n",
" assert np.allclose(result, numpy_result), \"❌ Doesn't match NumPy result!\"\n",
" print(\"✅ Matches NumPy implementation!\")\n",
" \n",
"except Exception as e:\n",
" print(f\"❌ Error: {e}\")\n",
" print(\"Make sure to implement matmul_naive above!\")"
]
},
{
"cell_type": "markdown",
"id": "d8176801",
"metadata": {
"cell_marker": "\"\"\"",
"lines_to_next_cell": 1
},
"source": [
"## Step 3: Building the Dense Layer\n",
"\n",
"Now let's build the **Dense layer**, the most fundamental building block of neural networks. A Dense layer performs a linear transformation: `y = Wx + b`\n",
"\n",
"### What is a Dense Layer?\n",
"- **Linear transformation**: `y = Wx + b`\n",
"- **W**: Weight matrix (learnable parameters)\n",
"- **x**: Input tensor\n",
"- **b**: Bias vector (learnable parameters)\n",
"- **y**: Output tensor\n",
"\n",
"### Why Dense Layers Matter\n",
"- **Universal approximation**: Can approximate any function with enough neurons\n",
"- **Feature learning**: Each neuron learns a different feature\n",
"- **Nonlinearity**: When combined with activation functions, becomes very powerful\n",
"- **Foundation**: All other layers build on this concept\n",
"\n",
"### The Math\n",
"For input x of shape (batch_size, input_size):\n",
"- **W**: Weight matrix of shape (input_size, output_size)\n",
"- **b**: Bias vector of shape (output_size)\n",
"- **y**: Output of shape (batch_size, output_size)\n",
"\n",
"### Visual Example\n",
"```\n",
"Input: x = [1, 2, 3] (3 features)\n",
"Weights: W = [[0.1, 0.2], Bias: b = [0.1, 0.2]\n",
" [0.3, 0.4],\n",
" [0.5, 0.6]]\n",
"\n",
"Step 1: Wx = [0.1*1 + 0.3*2 + 0.5*3, 0.2*1 + 0.4*2 + 0.6*3]\n",
" = [2.2, 3.2]\n",
"\n",
"Step 2: y = Wx + b = [2.2 + 0.1, 3.2 + 0.2] = [2.3, 3.4]\n",
"```\n",
"\n",
"Let's implement this!"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4a916c67",
"metadata": {
"lines_to_next_cell": 1
},
"outputs": [],
"source": [
"#| export\n",
"class Dense:\n",
" \"\"\"\n",
" Dense (Linear) Layer: y = Wx + b\n",
" \n",
" The fundamental building block of neural networks.\n",
" Performs linear transformation: matrix multiplication + bias addition.\n",
" \n",
" Args:\n",
" input_size: Number of input features\n",
" output_size: Number of output features\n",
" use_bias: Whether to include bias term (default: True)\n",
" use_naive_matmul: Whether to use naive matrix multiplication (for learning)\n",
" \n",
" TODO: Implement the Dense layer with weight initialization and forward pass.\n",
" \n",
" APPROACH:\n",
" 1. Store layer parameters (input_size, output_size, use_bias, use_naive_matmul)\n",
" 2. Initialize weights with small random values (Xavier/Glorot initialization)\n",
" 3. Initialize bias to zeros (if use_bias=True)\n",
" 4. Implement forward pass using matrix multiplication and bias addition\n",
" \n",
" EXAMPLE:\n",
" layer = Dense(input_size=3, output_size=2)\n",
" x = Tensor([[1, 2, 3]]) # batch_size=1, input_size=3\n",
" y = layer(x) # shape: (1, 2)\n",
" \n",
" HINTS:\n",
" - Use np.random.randn() for random initialization\n",
" - Scale weights by sqrt(2/(input_size + output_size)) for Xavier init\n",
" - Store weights and bias as numpy arrays\n",
" - Use matmul_naive or @ operator based on use_naive_matmul flag\n",
" \"\"\"\n",
" \n",
" def __init__(self, input_size: int, output_size: int, use_bias: bool = True, \n",
" use_naive_matmul: bool = False):\n",
" \"\"\"\n",
" Initialize Dense layer with random weights.\n",
" \n",
" Args:\n",
" input_size: Number of input features\n",
" output_size: Number of output features\n",
" use_bias: Whether to include bias term\n",
" use_naive_matmul: Use naive matrix multiplication (for learning)\n",
" \n",
" TODO: \n",
" 1. Store layer parameters (input_size, output_size, use_bias, use_naive_matmul)\n",
" 2. Initialize weights with small random values\n",
" 3. Initialize bias to zeros (if use_bias=True)\n",
" \n",
" STEP-BY-STEP:\n",
" 1. Store the parameters as instance variables\n",
" 2. Calculate scale factor for Xavier initialization: sqrt(2/(input_size + output_size))\n",
" 3. Initialize weights: np.random.randn(input_size, output_size) * scale\n",
" 4. If use_bias=True, initialize bias: np.zeros(output_size)\n",
" 5. If use_bias=False, set bias to None\n",
" \n",
" EXAMPLE:\n",
" Dense(3, 2) creates:\n",
" - weights: shape (3, 2) with small random values\n",
" - bias: shape (2,) with zeros\n",
" \"\"\"\n",
" raise NotImplementedError(\"Student implementation required\")\n",
" \n",
" def forward(self, x: Tensor) -> Tensor:\n",
" \"\"\"\n",
" Forward pass: y = Wx + b\n",
" \n",
" Args:\n",
" x: Input tensor of shape (batch_size, input_size)\n",
" \n",
" Returns:\n",
" Output tensor of shape (batch_size, output_size)\n",
" \n",
" TODO: Implement matrix multiplication and bias addition\n",
" - Use self.use_naive_matmul to choose between NumPy and naive implementation\n",
" - If use_naive_matmul=True, use matmul_naive(x.data, self.weights)\n",
" - If use_naive_matmul=False, use x.data @ self.weights\n",
" - Add bias if self.use_bias=True\n",
" \n",
" STEP-BY-STEP:\n",
" 1. Perform matrix multiplication: Wx\n",
" - If use_naive_matmul: result = matmul_naive(x.data, self.weights)\n",
" - Else: result = x.data @ self.weights\n",
" 2. Add bias if use_bias: result += self.bias\n",
" 3. Return Tensor(result)\n",
" \n",
" EXAMPLE:\n",
" Input x: Tensor([[1, 2, 3]]) # shape (1, 3)\n",
" Weights: shape (3, 2)\n",
" Output: Tensor([[val1, val2]]) # shape (1, 2)\n",
" \n",
" HINTS:\n",
" - x.data gives you the numpy array\n",
" - self.weights is your weight matrix\n",
" - Use broadcasting for bias addition: result + self.bias\n",
" - Return Tensor(result) to wrap the result\n",
" \"\"\"\n",
" raise NotImplementedError(\"Student implementation required\")\n",
" \n",
" def __call__(self, x: Tensor) -> Tensor:\n",
" \"\"\"Make layer callable: layer(x) same as layer.forward(x)\"\"\"\n",
" return self.forward(x)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8570d026",
"metadata": {
"lines_to_next_cell": 1
},
"outputs": [],
"source": [
"#| hide\n",
"#| export\n",
"class Dense:\n",
" \"\"\"\n",
" Dense (Linear) Layer: y = Wx + b\n",
" \n",
" The fundamental building block of neural networks.\n",
" Performs linear transformation: matrix multiplication + bias addition.\n",
" \"\"\"\n",
" \n",
" def __init__(self, input_size: int, output_size: int, use_bias: bool = True, \n",
" use_naive_matmul: bool = False):\n",
" \"\"\"\n",
" Initialize Dense layer with random weights.\n",
" \n",
" Args:\n",
" input_size: Number of input features\n",
" output_size: Number of output features\n",
" use_bias: Whether to include bias term\n",
" use_naive_matmul: Use naive matrix multiplication (for learning)\n",
" \"\"\"\n",
" # Store parameters\n",
" self.input_size = input_size\n",
" self.output_size = output_size\n",
" self.use_bias = use_bias\n",
" self.use_naive_matmul = use_naive_matmul\n",
" \n",
" # Xavier/Glorot initialization\n",
" scale = np.sqrt(2.0 / (input_size + output_size))\n",
" self.weights = np.random.randn(input_size, output_size).astype(np.float32) * scale\n",
" \n",
" # Initialize bias\n",
" if use_bias:\n",
" self.bias = np.zeros(output_size, dtype=np.float32)\n",
" else:\n",
" self.bias = None\n",
" \n",
" def forward(self, x: Tensor) -> Tensor:\n",
" \"\"\"\n",
" Forward pass: y = Wx + b\n",
" \n",
" Args:\n",
" x: Input tensor of shape (batch_size, input_size)\n",
" \n",
" Returns:\n",
" Output tensor of shape (batch_size, output_size)\n",
" \"\"\"\n",
" # Matrix multiplication\n",
" if self.use_naive_matmul:\n",
" result = matmul_naive(x.data, self.weights)\n",
" else:\n",
" result = x.data @ self.weights\n",
" \n",
" # Add bias\n",
" if self.use_bias:\n",
" result += self.bias\n",
" \n",
" return Tensor(result)\n",
" \n",
" def __call__(self, x: Tensor) -> Tensor:\n",
" \"\"\"Make layer callable: layer(x) same as layer.forward(x)\"\"\"\n",
" return self.forward(x)"
]
},
{
"cell_type": "markdown",
"id": "90197c65",
"metadata": {
"cell_marker": "\"\"\""
},
"source": [
"### 🧪 Test Your Dense Layer"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9d9e4d64",
"metadata": {},
"outputs": [],
"source": [
"# Test Dense layer\n",
"print(\"Testing Dense layer...\")\n",
"\n",
"try:\n",
" # Test basic Dense layer\n",
" layer = Dense(input_size=3, output_size=2, use_bias=True)\n",
" x = Tensor([[1, 2, 3]]) # batch_size=1, input_size=3\n",
" \n",
" print(f\"✅ Input shape: {x.shape}\")\n",
" print(f\"✅ Layer weights shape: {layer.weights.shape}\")\n",
" print(f\"✅ Layer bias shape: {layer.bias.shape}\")\n",
" \n",
" y = layer(x)\n",
" print(f\"✅ Output shape: {y.shape}\")\n",
" print(f\"✅ Output: {y}\")\n",
" \n",
" # Test without bias\n",
" layer_no_bias = Dense(input_size=2, output_size=1, use_bias=False)\n",
" x2 = Tensor([[1, 2]])\n",
" y2 = layer_no_bias(x2)\n",
" print(f\"✅ No bias output: {y2}\")\n",
" \n",
" # Test naive matrix multiplication\n",
" layer_naive = Dense(input_size=2, output_size=2, use_naive_matmul=True)\n",
" x3 = Tensor([[1, 2]])\n",
" y3 = layer_naive(x3)\n",
" print(f\"✅ Naive matmul output: {y3}\")\n",
" \n",
" print(\"\\n🎉 All Dense layer tests passed!\")\n",
" \n",
"except Exception as e:\n",
" print(f\"❌ Error: {e}\")\n",
" print(\"Make sure to implement the Dense layer above!\")"
]
},
{
"cell_type": "markdown",
"id": "37532e4d",
"metadata": {
"cell_marker": "\"\"\""
},
"source": [
"## Step 4: Composing Layers with Activations\n",
"\n",
"Now let's see how layers work together! A neural network is just layers composed with activation functions.\n",
"\n",
"### Why Layer Composition Matters\n",
"- **Nonlinearity**: Activation functions make networks powerful\n",
"- **Feature learning**: Each layer learns different levels of features\n",
"- **Universal approximation**: Can approximate any function\n",
"- **Modularity**: Easy to experiment with different architectures\n",
"\n",
"### The Pattern\n",
"```\n",
"Input → Dense → Activation → Dense → Activation → Output\n",
"```\n",
"\n",
"### Real-World Example\n",
"```\n",
"Input: [1, 2, 3] (3 features)\n",
"Dense(3→2): [1.4, 2.8] (linear transformation)\n",
"ReLU: [1.4, 2.8] (nonlinearity)\n",
"Dense(2→1): [3.2] (final prediction)\n",
"```\n",
"\n",
"Let's build a simple network!"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d6e1d85c",
"metadata": {},
"outputs": [],
"source": [
"# Test layer composition\n",
"print(\"Testing layer composition...\")\n",
"\n",
"try:\n",
" # Create a simple network: Dense → ReLU → Dense\n",
" dense1 = Dense(input_size=3, output_size=2)\n",
" relu = ReLU()\n",
" dense2 = Dense(input_size=2, output_size=1)\n",
" \n",
" # Test input\n",
" x = Tensor([[1, 2, 3]])\n",
" print(f\"✅ Input: {x}\")\n",
" \n",
" # Forward pass through the network\n",
" h1 = dense1(x)\n",
" print(f\"✅ After Dense1: {h1}\")\n",
" \n",
" h2 = relu(h1)\n",
" print(f\"✅ After ReLU: {h2}\")\n",
" \n",
" y = dense2(h2)\n",
" print(f\"✅ Final output: {y}\")\n",
" \n",
" print(\"\\n🎉 Layer composition works!\")\n",
" print(\"This is how neural networks work: layers + activations!\")\n",
" \n",
"except Exception as e:\n",
" print(f\"❌ Error: {e}\")\n",
" print(\"Make sure all your layers and activations are working!\")"
]
},
{
"cell_type": "markdown",
"id": "5f2f8a48",
"metadata": {
"cell_marker": "\"\"\""
},
"source": [
"## Step 5: Performance Comparison\n",
"\n",
"Let's compare our naive matrix multiplication with NumPy's optimized version to understand why optimization matters in ML.\n",
"\n",
"### Why Performance Matters\n",
"- **Training time**: Neural networks train for hours/days\n",
"- **Inference speed**: Real-time applications need fast predictions\n",
"- **GPU utilization**: Optimized operations use hardware efficiently\n",
"- **Scalability**: Large models need efficient implementations"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b6f490a2",
"metadata": {},
"outputs": [],
"source": [
"# Performance comparison\n",
"print(\"Comparing naive vs NumPy matrix multiplication...\")\n",
"\n",
"try:\n",
" import time\n",
" \n",
" # Create test matrices\n",
" A = np.random.randn(100, 100).astype(np.float32)\n",
" B = np.random.randn(100, 100).astype(np.float32)\n",
" \n",
" # Time naive implementation\n",
" start_time = time.time()\n",
" result_naive = matmul_naive(A, B)\n",
" naive_time = time.time() - start_time\n",
" \n",
" # Time NumPy implementation\n",
" start_time = time.time()\n",
" result_numpy = A @ B\n",
" numpy_time = time.time() - start_time\n",
" \n",
" print(f\"✅ Naive time: {naive_time:.4f} seconds\")\n",
" print(f\"✅ NumPy time: {numpy_time:.4f} seconds\")\n",
" print(f\"✅ Speedup: {naive_time/numpy_time:.1f}x faster\")\n",
" \n",
" # Verify correctness\n",
" assert np.allclose(result_naive, result_numpy), \"Results don't match!\"\n",
" print(\"✅ Results are identical!\")\n",
" \n",
" print(\"\\n💡 This is why we use optimized libraries in production!\")\n",
" \n",
"except Exception as e:\n",
" print(f\"❌ Error: {e}\")"
]
},
{
"cell_type": "markdown",
"id": "35efc1ca",
"metadata": {
"cell_marker": "\"\"\""
},
"source": [
"## 🎯 Module Summary\n",
"\n",
"Congratulations! You've built the foundation of neural network layers:\n",
"\n",
"### What You've Accomplished\n",
"✅ **Matrix Multiplication**: Understanding the core operation \n",
"✅ **Dense Layer**: Linear transformation with weights and bias \n",
"✅ **Layer Composition**: Combining layers with activations \n",
"✅ **Performance Awareness**: Understanding optimization importance \n",
"✅ **Testing**: Immediate feedback on your implementations \n",
"\n",
"### Key Concepts You've Learned\n",
"- **Layers** are functions that transform tensors\n",
"- **Matrix multiplication** powers all neural network computations\n",
"- **Dense layers** perform linear transformations: `y = Wx + b`\n",
"- **Layer composition** creates complex functions from simple building blocks\n",
"- **Performance** matters for real-world ML applications\n",
"\n",
"### What's Next\n",
"In the next modules, you'll build on this foundation:\n",
"- **Networks**: Compose layers into complete models\n",
"- **Training**: Learn parameters with gradients and optimization\n",
"- **Convolutional layers**: Process spatial data like images\n",
"- **Recurrent layers**: Process sequential data like text\n",
"\n",
"### Real-World Connection\n",
"Your Dense layer is now ready to:\n",
"- Learn patterns in data through weight updates\n",
"- Transform features for classification and regression\n",
"- Serve as building blocks for complex architectures\n",
"- Integrate with the rest of the TinyTorch ecosystem\n",
"\n",
"**Ready for the next challenge?** Let's move on to building complete neural networks!"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9c9187ca",
"metadata": {},
"outputs": [],
"source": [
"# Final verification\n",
"print(\"\\n\" + \"=\"*50)\n",
"print(\"🎉 LAYERS MODULE COMPLETE!\")\n",
"print(\"=\"*50)\n",
"print(\"✅ Matrix multiplication understanding\")\n",
"print(\"✅ Dense layer implementation\")\n",
"print(\"✅ Layer composition with activations\")\n",
"print(\"✅ Performance awareness\")\n",
"print(\"✅ Comprehensive testing\")\n",
"print(\"\\n🚀 Ready to build networks in the next module!\") "
]
}
],
"metadata": {
"jupytext": {
"main_language": "python"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

1437
assignments/source/04_networks/networks_dev.ipynb
View File

File diff suppressed because it is too large Load Diff

816
assignments/source/05_cnn/cnn_dev.ipynb
View File

@@ -1,816 +0,0 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "955f6ea8",
"metadata": {
"cell_marker": "\"\"\""
},
"source": [
"# Module X: CNN - Convolutional Neural Networks\n",
"\n",
"Welcome to the CNN module! Here you'll implement the core building block of modern computer vision: the convolutional layer.\n",
"\n",
"## Learning Goals\n",
"- Understand the convolution operation (sliding window, local connectivity, weight sharing)\n",
"- Implement Conv2D with explicit for-loops\n",
"- Visualize how convolution builds feature maps\n",
"- Compose Conv2D with other layers to build a simple ConvNet\n",
"- (Stretch) Explore stride, padding, pooling, and multi-channel input\n",
"\n",
"## Build → Use → Understand\n",
"1. **Build**: Conv2D layer using sliding window convolution\n",
"2. **Use**: Transform images and see feature maps\n",
"3. **Understand**: How CNNs learn spatial patterns"
]
},
{
"cell_type": "markdown",
"id": "60a60cd1",
"metadata": {
"cell_marker": "\"\"\""
},
"source": [
"## 📦 Where This Code Lives in the Final Package\n",
"\n",
"**Learning Side:** You work in `assignments/source/05_cnn/cnn_dev.py` \n",
"**Building Side:** Code exports to `tinytorch.core.layers`\n",
"\n",
"```python\n",
"# Final package structure:\n",
"from tinytorch.core.layers import Dense, Conv2D # Both layers together!\n",
"from tinytorch.core.activations import ReLU\n",
"from tinytorch.core.tensor import Tensor\n",
"```\n",
"\n",
"**Why this matters:**\n",
"- **Learning:** Focused modules for deep understanding\n",
"- **Production:** Proper organization like PyTorch's `torch.nn`\n",
"- **Consistency:** All layers (Dense, Conv2D) live together in `core.layers`"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f0294e6a",
"metadata": {},
"outputs": [],
"source": [
"#| default_exp core.cnn"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7e1c6590",
"metadata": {},
"outputs": [],
"source": [
"#| export\n",
"import numpy as np\n",
"from typing import List, Tuple, Optional\n",
"from tinytorch.core.tensor import Tensor\n",
"\n",
"# Setup and imports (for development)\n",
"import matplotlib.pyplot as plt\n",
"from tinytorch.core.layers import Dense\n",
"from tinytorch.core.activations import ReLU"
]
},
{
"cell_type": "markdown",
"id": "fed284f4",
"metadata": {
"cell_marker": "\"\"\"",
"lines_to_next_cell": 1
},
"source": [
"## Step 1: What is Convolution?\n",
"\n",
"### Definition\n",
"A **convolutional layer** applies a small filter (kernel) across the input, producing a feature map. This operation captures local patterns and is the foundation of modern vision models.\n",
"\n",
"### Why Convolution Matters in Computer Vision\n",
"- **Local connectivity**: Each output value depends only on a small region of the input\n",
"- **Weight sharing**: The same filter is applied everywhere (translation invariance)\n",
"- **Spatial hierarchy**: Multiple layers build increasingly complex features\n",
"- **Parameter efficiency**: Much fewer parameters than fully connected layers\n",
"\n",
"### The Fundamental Insight\n",
"**Convolution is pattern matching!** The kernel learns to detect specific patterns:\n",
"- **Edge detectors**: Find boundaries between objects\n",
"- **Texture detectors**: Recognize surface patterns\n",
"- **Shape detectors**: Identify geometric forms\n",
"- **Feature detectors**: Combine simple patterns into complex features\n",
"\n",
"### Real-World Examples\n",
"- **Image processing**: Detect edges, blur, sharpen\n",
"- **Computer vision**: Recognize objects, faces, text\n",
"- **Medical imaging**: Detect tumors, analyze scans\n",
"- **Autonomous driving**: Identify traffic signs, pedestrians\n",
"\n",
"### Visual Intuition\n",
"```\n",
"Input Image: Kernel: Output Feature Map:\n",
"[1, 2, 3] [1, 0] [1*1+2*0+4*0+5*(-1), 2*1+3*0+5*0+6*(-1)]\n",
"[4, 5, 6] [0, -1] [4*1+5*0+7*0+8*(-1), 5*1+6*0+8*0+9*(-1)]\n",
"[7, 8, 9]\n",
"```\n",
"\n",
"The kernel slides across the input, computing dot products at each position.\n",
"\n",
"### The Math Behind It\n",
"For input I (H×W) and kernel K (kH×kW), the output O (out_H×out_W) is:\n",
"```\n",
"O[i,j] = sum(I[i+di, j+dj] * K[di, dj] for di in range(kH), dj in range(kW))\n",
"```\n",
"\n",
"Let's implement this step by step!"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "64e2c944",
"metadata": {
"lines_to_next_cell": 1
},
"outputs": [],
"source": [
"#| export\n",
"def conv2d_naive(input: np.ndarray, kernel: np.ndarray) -> np.ndarray:\n",
" \"\"\"\n",
" Naive 2D convolution (single channel, no stride, no padding).\n",
" \n",
" Args:\n",
" input: 2D input array (H, W)\n",
" kernel: 2D filter (kH, kW)\n",
" Returns:\n",
" 2D output array (H-kH+1, W-kW+1)\n",
" \n",
" TODO: Implement the sliding window convolution using for-loops.\n",
" \n",
" APPROACH:\n",
" 1. Get input dimensions: H, W = input.shape\n",
" 2. Get kernel dimensions: kH, kW = kernel.shape\n",
" 3. Calculate output dimensions: out_H = H - kH + 1, out_W = W - kW + 1\n",
" 4. Create output array: np.zeros((out_H, out_W))\n",
" 5. Use nested loops to slide the kernel:\n",
" - i loop: output rows (0 to out_H-1)\n",
" - j loop: output columns (0 to out_W-1)\n",
" - di loop: kernel rows (0 to kH-1)\n",
" - dj loop: kernel columns (0 to kW-1)\n",
" 6. For each (i,j), compute: output[i,j] += input[i+di, j+dj] * kernel[di, dj]\n",
" \n",
" EXAMPLE:\n",
" Input: [[1, 2, 3], Kernel: [[1, 0],\n",
" [4, 5, 6], [0, -1]]\n",
" [7, 8, 9]]\n",
" \n",
" Output[0,0] = 1*1 + 2*0 + 4*0 + 5*(-1) = 1 - 5 = -4\n",
" Output[0,1] = 2*1 + 3*0 + 5*0 + 6*(-1) = 2 - 6 = -4\n",
" Output[1,0] = 4*1 + 5*0 + 7*0 + 8*(-1) = 4 - 8 = -4\n",
" Output[1,1] = 5*1 + 6*0 + 8*0 + 9*(-1) = 5 - 9 = -4\n",
" \n",
" HINTS:\n",
" - Start with output = np.zeros((out_H, out_W))\n",
" - Use four nested loops: for i in range(out_H): for j in range(out_W): for di in range(kH): for dj in range(kW):\n",
" - Accumulate the sum: output[i,j] += input[i+di, j+dj] * kernel[di, dj]\n",
" \"\"\"\n",
" raise NotImplementedError(\"Student implementation required\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ff21cee2",
"metadata": {
"lines_to_next_cell": 1
},
"outputs": [],
"source": [
"#| hide\n",
"#| export\n",
"def conv2d_naive(input: np.ndarray, kernel: np.ndarray) -> np.ndarray:\n",
" H, W = input.shape\n",
" kH, kW = kernel.shape\n",
" out_H, out_W = H - kH + 1, W - kW + 1\n",
" output = np.zeros((out_H, out_W), dtype=input.dtype)\n",
" for i in range(out_H):\n",
" for j in range(out_W):\n",
" for di in range(kH):\n",
" for dj in range(kW):\n",
" output[i, j] += input[i + di, j + dj] * kernel[di, dj]\n",
" return output"
]
},
{
"cell_type": "markdown",
"id": "2d3d2a95",
"metadata": {
"cell_marker": "\"\"\""
},
"source": [
"### 🧪 Test Your Conv2D Implementation\n",
"\n",
"Try your function on this simple example:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ac0cac68",
"metadata": {},
"outputs": [],
"source": [
"# Test case for conv2d_naive\n",
"input = np.array([\n",
" [1, 2, 3],\n",
" [4, 5, 6],\n",
" [7, 8, 9]\n",
"], dtype=np.float32)\n",
"kernel = np.array([\n",
" [1, 0],\n",
" [0, -1]\n",
"], dtype=np.float32)\n",
"\n",
"expected = np.array([\n",
" [1*1+2*0+4*0+5*(-1), 2*1+3*0+5*0+6*(-1)],\n",
" [4*1+5*0+7*0+8*(-1), 5*1+6*0+8*0+9*(-1)]\n",
"], dtype=np.float32)\n",
"\n",
"try:\n",
" output = conv2d_naive(input, kernel)\n",
" print(\"✅ Input:\\n\", input)\n",
" print(\"✅ Kernel:\\n\", kernel)\n",
" print(\"✅ Your output:\\n\", output)\n",
" print(\"✅ Expected:\\n\", expected)\n",
" assert np.allclose(output, expected), \"❌ Output does not match expected!\"\n",
" print(\"🎉 conv2d_naive works!\")\n",
"except Exception as e:\n",
" print(f\"❌ Error: {e}\")\n",
" print(\"Make sure to implement conv2d_naive above!\")"
]
},
{
"cell_type": "markdown",
"id": "c0771c94",
"metadata": {
"cell_marker": "\"\"\""
},
"source": [
"## Step 2: Understanding What Convolution Does\n",
"\n",
"Let's visualize how different kernels detect different patterns:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c9e63d70",
"metadata": {},
"outputs": [],
"source": [
"# Visualize different convolution kernels\n",
"print(\"Visualizing different convolution kernels...\")\n",
"\n",
"try:\n",
" # Test different kernels\n",
" test_input = np.array([\n",
" [1, 1, 1, 0, 0],\n",
" [1, 1, 1, 0, 0],\n",
" [1, 1, 1, 0, 0],\n",
" [0, 0, 0, 0, 0],\n",
" [0, 0, 0, 0, 0]\n",
" ], dtype=np.float32)\n",
" \n",
" # Edge detection kernel (horizontal)\n",
" edge_kernel = np.array([\n",
" [1, 1, 1],\n",
" [0, 0, 0],\n",
" [-1, -1, -1]\n",
" ], dtype=np.float32)\n",
" \n",
" # Sharpening kernel\n",
" sharpen_kernel = np.array([\n",
" [0, -1, 0],\n",
" [-1, 5, -1],\n",
" [0, -1, 0]\n",
" ], dtype=np.float32)\n",
" \n",
" # Test edge detection\n",
" edge_output = conv2d_naive(test_input, edge_kernel)\n",
" print(\"✅ Edge detection kernel:\")\n",
" print(\" Detects horizontal edges (boundaries between light and dark)\")\n",
" print(\" Output:\\n\", edge_output)\n",
" \n",
" # Test sharpening\n",
" sharpen_output = conv2d_naive(test_input, sharpen_kernel)\n",
" print(\"✅ Sharpening kernel:\")\n",
" print(\" Enhances edges and details\")\n",
" print(\" Output:\\n\", sharpen_output)\n",
" \n",
" print(\"\\n💡 Different kernels detect different patterns!\")\n",
" print(\" Neural networks learn these kernels automatically!\")\n",
" \n",
"except Exception as e:\n",
" print(f\"❌ Error: {e}\")"
]
},
{
"cell_type": "markdown",
"id": "ef10d9f8",
"metadata": {
"cell_marker": "\"\"\"",
"lines_to_next_cell": 1
},
"source": [
"## Step 3: Conv2D Layer Class\n",
"\n",
"Now let's wrap your convolution function in a layer class for use in networks. This makes it consistent with other layers like Dense.\n",
"\n",
"### Why Layer Classes Matter\n",
"- **Consistent API**: Same interface as Dense layers\n",
"- **Learnable parameters**: Kernels can be learned from data\n",
"- **Composability**: Can be combined with other layers\n",
"- **Integration**: Works seamlessly with the rest of TinyTorch\n",
"\n",
"### The Pattern\n",
"```\n",
"Input Tensor → Conv2D → Output Tensor\n",
"```\n",
"\n",
"Just like Dense layers, but with spatial operations instead of linear transformations."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3ae72cc4",
"metadata": {
"lines_to_next_cell": 1
},
"outputs": [],
"source": [
"#| export\n",
"class Conv2D:\n",
" \"\"\"\n",
" 2D Convolutional Layer (single channel, single filter, no stride/pad).\n",
" \n",
" Args:\n",
" kernel_size: (kH, kW) - size of the convolution kernel\n",
" \n",
" TODO: Initialize a random kernel and implement the forward pass using conv2d_naive.\n",
" \n",
" APPROACH:\n",
" 1. Store kernel_size as instance variable\n",
" 2. Initialize random kernel with small values\n",
" 3. Implement forward pass using conv2d_naive function\n",
" 4. Return Tensor wrapped around the result\n",
" \n",
" EXAMPLE:\n",
" layer = Conv2D(kernel_size=(2, 2))\n",
" x = Tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]]) # shape (3, 3)\n",
" y = layer(x) # shape (2, 2)\n",
" \n",
" HINTS:\n",
" - Store kernel_size as (kH, kW)\n",
" - Initialize kernel with np.random.randn(kH, kW) * 0.1 (small values)\n",
" - Use conv2d_naive(x.data, self.kernel) in forward pass\n",
" - Return Tensor(result) to wrap the result\n",
" \"\"\"\n",
" def __init__(self, kernel_size: Tuple[int, int]):\n",
" \"\"\"\n",
" Initialize Conv2D layer with random kernel.\n",
" \n",
" Args:\n",
" kernel_size: (kH, kW) - size of the convolution kernel\n",
" \n",
" TODO: \n",
" 1. Store kernel_size as instance variable\n",
" 2. Initialize random kernel with small values\n",
" 3. Scale kernel values to prevent large outputs\n",
" \n",
" STEP-BY-STEP:\n",
" 1. Store kernel_size as self.kernel_size\n",
" 2. Unpack kernel_size into kH, kW\n",
" 3. Initialize kernel: np.random.randn(kH, kW) * 0.1\n",
" 4. Convert to float32 for consistency\n",
" \n",
" EXAMPLE:\n",
" Conv2D((2, 2)) creates:\n",
" - kernel: shape (2, 2) with small random values\n",
" \"\"\"\n",
" raise NotImplementedError(\"Student implementation required\")\n",
" \n",
" def forward(self, x: Tensor) -> Tensor:\n",
" \"\"\"\n",
" Forward pass: apply convolution to input.\n",
" \n",
" Args:\n",
" x: Input tensor of shape (H, W)\n",
" \n",
" Returns:\n",
" Output tensor of shape (H-kH+1, W-kW+1)\n",
" \n",
" TODO: Implement convolution using conv2d_naive function.\n",
" \n",
" STEP-BY-STEP:\n",
" 1. Use conv2d_naive(x.data, self.kernel)\n",
" 2. Return Tensor(result)\n",
" \n",
" EXAMPLE:\n",
" Input x: Tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]]) # shape (3, 3)\n",
" Kernel: shape (2, 2)\n",
" Output: Tensor([[val1, val2], [val3, val4]]) # shape (2, 2)\n",
" \n",
" HINTS:\n",
" - x.data gives you the numpy array\n",
" - self.kernel is your learned kernel\n",
" - Use conv2d_naive(x.data, self.kernel)\n",
" - Return Tensor(result) to wrap the result\n",
" \"\"\"\n",
" raise NotImplementedError(\"Student implementation required\")\n",
" \n",
" def __call__(self, x: Tensor) -> Tensor:\n",
" \"\"\"Make layer callable: layer(x) same as layer.forward(x)\"\"\"\n",
" return self.forward(x)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "48e50d1b",
"metadata": {
"lines_to_next_cell": 1
},
"outputs": [],
"source": [
"#| hide\n",
"#| export\n",
"class Conv2D:\n",
" def __init__(self, kernel_size: Tuple[int, int]):\n",
" self.kernel_size = kernel_size\n",
" kH, kW = kernel_size\n",
" # Initialize with small random values\n",
" self.kernel = np.random.randn(kH, kW).astype(np.float32) * 0.1\n",
" \n",
" def forward(self, x: Tensor) -> Tensor:\n",
" return Tensor(conv2d_naive(x.data, self.kernel))\n",
" \n",
" def __call__(self, x: Tensor) -> Tensor:\n",
" return self.forward(x)"
]
},
{
"cell_type": "markdown",
"id": "4c29a5dd",
"metadata": {
"cell_marker": "\"\"\""
},
"source": [
"### 🧪 Test Your Conv2D Layer"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ab358d43",
"metadata": {},
"outputs": [],
"source": [
"# Test Conv2D layer\n",
"print(\"Testing Conv2D layer...\")\n",
"\n",
"try:\n",
" # Test basic Conv2D layer\n",
" conv = Conv2D(kernel_size=(2, 2))\n",
" x = Tensor(np.array([\n",
" [1, 2, 3],\n",
" [4, 5, 6],\n",
" [7, 8, 9]\n",
" ], dtype=np.float32))\n",
" \n",
" print(f\"✅ Input shape: {x.shape}\")\n",
" print(f\"✅ Kernel shape: {conv.kernel.shape}\")\n",
" print(f\"✅ Kernel values:\\n{conv.kernel}\")\n",
" \n",
" y = conv(x)\n",
" print(f\"✅ Output shape: {y.shape}\")\n",
" print(f\"✅ Output: {y}\")\n",
" \n",
" # Test with different kernel size\n",
" conv2 = Conv2D(kernel_size=(3, 3))\n",
" y2 = conv2(x)\n",
" print(f\"✅ 3x3 kernel output shape: {y2.shape}\")\n",
" \n",
" print(\"\\n🎉 Conv2D layer works!\")\n",
" \n",
"except Exception as e:\n",
" print(f\"❌ Error: {e}\")\n",
" print(\"Make sure to implement the Conv2D layer above!\")"
]
},
{
"cell_type": "markdown",
"id": "9700071c",
"metadata": {
"cell_marker": "\"\"\"",
"lines_to_next_cell": 1
},
"source": [
"## Step 4: Building a Simple ConvNet\n",
"\n",
"Now let's compose Conv2D layers with other layers to build a complete convolutional neural network!\n",
"\n",
"### Why ConvNets Matter\n",
"- **Spatial hierarchy**: Each layer learns increasingly complex features\n",
"- **Parameter sharing**: Same kernel applied everywhere (efficiency)\n",
"- **Translation invariance**: Can recognize objects regardless of position\n",
"- **Real-world success**: Power most modern computer vision systems\n",
"\n",
"### The Architecture\n",
"```\n",
"Input Image → Conv2D → ReLU → Flatten → Dense → Output\n",
"```\n",
"\n",
"This simple architecture can learn to recognize patterns in images!"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "abbf0682",
"metadata": {
"lines_to_next_cell": 1
},
"outputs": [],
"source": [
"#| export\n",
"def flatten(x: Tensor) -> Tensor:\n",
" \"\"\"\n",
" Flatten a 2D tensor to 1D (for connecting to Dense).\n",
" \n",
" TODO: Implement flattening operation.\n",
" \n",
" APPROACH:\n",
" 1. Get the numpy array from the tensor\n",
" 2. Use .flatten() to convert to 1D\n",
" 3. Add batch dimension with [None, :]\n",
" 4. Return Tensor wrapped around the result\n",
" \n",
" EXAMPLE:\n",
" Input: Tensor([[1, 2], [3, 4]]) # shape (2, 2)\n",
" Output: Tensor([[1, 2, 3, 4]]) # shape (1, 4)\n",
" \n",
" HINTS:\n",
" - Use x.data.flatten() to get 1D array\n",
" - Add batch dimension: result[None, :]\n",
" - Return Tensor(result)\n",
" \"\"\"\n",
" raise NotImplementedError(\"Student implementation required\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1b107a4c",
"metadata": {
"lines_to_next_cell": 1
},
"outputs": [],
"source": [
"#| hide\n",
"#| export\n",
"def flatten(x: Tensor) -> Tensor:\n",
" \"\"\"Flatten a 2D tensor to 1D (for connecting to Dense).\"\"\"\n",
" return Tensor(x.data.flatten()[None, :])"
]
},
{
"cell_type": "markdown",
"id": "7729a18f",
"metadata": {
"cell_marker": "\"\"\""
},
"source": [
"### 🧪 Test Your Flatten Function"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "075746b0",
"metadata": {},
"outputs": [],
"source": [
"# Test flatten function\n",
"print(\"Testing flatten function...\")\n",
"\n",
"try:\n",
" # Test flattening\n",
" x = Tensor([[1, 2, 3], [4, 5, 6]]) # shape (2, 3)\n",
" flattened = flatten(x)\n",
" \n",
" print(f\"✅ Input shape: {x.shape}\")\n",
" print(f\"✅ Flattened shape: {flattened.shape}\")\n",
" print(f\"✅ Flattened values: {flattened}\")\n",
" \n",
" # Verify the flattening worked correctly\n",
" expected = np.array([[1, 2, 3, 4, 5, 6]])\n",
" assert np.allclose(flattened.data, expected), \"❌ Flattening incorrect!\"\n",
" print(\"✅ Flattening works correctly!\")\n",
" \n",
"except Exception as e:\n",
" print(f\"❌ Error: {e}\")\n",
" print(\"Make sure to implement the flatten function above!\")"
]
},
{
"cell_type": "markdown",
"id": "506b9eb7",
"metadata": {
"cell_marker": "\"\"\""
},
"source": [
"## Step 5: Composing a Complete ConvNet\n",
"\n",
"Now let's build a simple convolutional neural network that can process images!"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "952a65f9",
"metadata": {},
"outputs": [],
"source": [
"# Compose a simple ConvNet\n",
"print(\"Building a simple ConvNet...\")\n",
"\n",
"try:\n",
" # Create network components\n",
" conv = Conv2D((2, 2))\n",
" relu = ReLU()\n",
" dense = Dense(input_size=4, output_size=1) # 4 features from 2x2 output\n",
" \n",
" # Test input (small 3x3 \"image\")\n",
" x = Tensor(np.random.randn(3, 3).astype(np.float32))\n",
" print(f\"✅ Input shape: {x.shape}\")\n",
" print(f\"✅ Input: {x}\")\n",
" \n",
" # Forward pass through the network\n",
" conv_out = conv(x)\n",
" print(f\"✅ After Conv2D: {conv_out}\")\n",
" \n",
" relu_out = relu(conv_out)\n",
" print(f\"✅ After ReLU: {relu_out}\")\n",
" \n",
" flattened = flatten(relu_out)\n",
" print(f\"✅ After flatten: {flattened}\")\n",
" \n",
" final_out = dense(flattened)\n",
" print(f\"✅ Final output: {final_out}\")\n",
" \n",
" print(\"\\n🎉 Simple ConvNet works!\")\n",
" print(\"This network can learn to recognize patterns in images!\")\n",
" \n",
"except Exception as e:\n",
" print(f\"❌ Error: {e}\")\n",
" print(\"Check your Conv2D, flatten, and Dense implementations!\")"
]
},
{
"cell_type": "markdown",
"id": "92177b19",
"metadata": {
"cell_marker": "\"\"\""
},
"source": [
"## Step 6: Understanding the Power of Convolution\n",
"\n",
"Let's see how convolution captures different types of patterns:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "255f7e00",
"metadata": {},
"outputs": [],
"source": [
"# Demonstrate pattern detection\n",
"print(\"Demonstrating pattern detection...\")\n",
"\n",
"try:\n",
" # Create a simple \"image\" with a pattern\n",
" image = np.array([\n",
" [0, 0, 0, 0, 0],\n",
" [0, 1, 1, 1, 0],\n",
" [0, 1, 1, 1, 0],\n",
" [0, 1, 1, 1, 0],\n",
" [0, 0, 0, 0, 0]\n",
" ], dtype=np.float32)\n",
" \n",
" # Different kernels detect different patterns\n",
" edge_kernel = np.array([\n",
" [1, 1, 1],\n",
" [1, -8, 1],\n",
" [1, 1, 1]\n",
" ], dtype=np.float32)\n",
" \n",
" blur_kernel = np.array([\n",
" [1/9, 1/9, 1/9],\n",
" [1/9, 1/9, 1/9],\n",
" [1/9, 1/9, 1/9]\n",
" ], dtype=np.float32)\n",
" \n",
" # Test edge detection\n",
" edge_result = conv2d_naive(image, edge_kernel)\n",
" print(\"✅ Edge detection:\")\n",
" print(\" Detects boundaries around the white square\")\n",
" print(\" Result:\\n\", edge_result)\n",
" \n",
" # Test blurring\n",
" blur_result = conv2d_naive(image, blur_kernel)\n",
" print(\"✅ Blurring:\")\n",
" print(\" Smooths the image\")\n",
" print(\" Result:\\n\", blur_result)\n",
" \n",
" print(\"\\n💡 Different kernels = different feature detectors!\")\n",
" print(\" Neural networks learn these automatically from data!\")\n",
" \n",
"except Exception as e:\n",
" print(f\"❌ Error: {e}\")"
]
},
{
"cell_type": "markdown",
"id": "47b66d7c",
"metadata": {
"cell_marker": "\"\"\""
},
"source": [
"## 🎯 Module Summary\n",
"\n",
"Congratulations! You've built the foundation of convolutional neural networks:\n",
"\n",
"### What You've Accomplished\n",
"✅ **Convolution Operation**: Understanding the sliding window mechanism \n",
"✅ **Conv2D Layer**: Learnable convolutional layer implementation \n",
"✅ **Pattern Detection**: Visualizing how kernels detect different features \n",
"✅ **ConvNet Architecture**: Composing Conv2D with other layers \n",
"✅ **Real-world Applications**: Understanding computer vision applications \n",
"\n",
"### Key Concepts You've Learned\n",
"- **Convolution** is pattern matching with sliding windows\n",
"- **Local connectivity** means each output depends on a small input region\n",
"- **Weight sharing** makes CNNs parameter-efficient\n",
"- **Spatial hierarchy** builds complex features from simple patterns\n",
"- **Translation invariance** allows recognition regardless of position\n",
"\n",
"### What's Next\n",
"In the next modules, you'll build on this foundation:\n",
"- **Advanced CNN features**: Stride, padding, pooling\n",
"- **Multi-channel convolution**: RGB images, multiple filters\n",
"- **Training**: Learning kernels from data\n",
"- **Real applications**: Image classification, object detection\n",
"\n",
"### Real-World Connection\n",
"Your Conv2D layer is now ready to:\n",
"- Learn edge detectors, texture recognizers, and shape detectors\n",
"- Process real images for computer vision tasks\n",
"- Integrate with the rest of the TinyTorch ecosystem\n",
"- Scale to complex architectures like ResNet, VGG, etc.\n",
"\n",
"**Ready for the next challenge?** Let's move on to training these networks!"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4f9cbe7e",
"metadata": {},
"outputs": [],
"source": [
"# Final verification\n",
"print(\"\\n\" + \"=\"*50)\n",
"print(\"🎉 CNN MODULE COMPLETE!\")\n",
"print(\"=\"*50)\n",
"print(\"✅ Convolution operation understanding\")\n",
"print(\"✅ Conv2D layer implementation\")\n",
"print(\"✅ Pattern detection visualization\")\n",
"print(\"✅ ConvNet architecture composition\")\n",
"print(\"✅ Real-world computer vision context\")\n",
"print(\"\\n🚀 Ready to train networks in the next module!\") "
]
}
],
"metadata": {
"jupytext": {
"main_language": "python"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

1699
assignments/source/06_dataloader/dataloader_dev.ipynb
View File

File diff suppressed because it is too large Load Diff