mirror of
https://github.com/MLSysBook/TinyTorch.git
synced 2026-05-27 02:05:52 -05:00
4b1f290b8a
- Add tensor_dev.ipynb converted from tensor_dev.py - Add activations_dev.ipynb converted from activations_dev.py These notebooks provide interactive learning environments for students to explore tensor operations and activation functions.
1557 lines
64 KiB
Plaintext
1557 lines
64 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "2d3eeb54",
|
||
"metadata": {
|
||
"cell_marker": "\"\"\""
|
||
},
|
||
"source": [
|
||
"# Tensor - Core Data Structure\n",
|
||
"\n",
|
||
"Welcome to the Tensor module! This is where TinyTorch really begins. You'll implement the fundamental data structure that powers all ML systems.\n",
|
||
"\n",
|
||
"## Learning Goals\n",
|
||
"- Understand tensors as N-dimensional arrays with ML-specific operations\n",
|
||
"- Implement a complete Tensor class with arithmetic operations\n",
|
||
"- Handle shape management, data types, and memory layout\n",
|
||
"- Build the foundation for neural networks and automatic differentiation\n",
|
||
"- Master the NBGrader workflow with comprehensive testing\n",
|
||
"\n",
|
||
"## Build → Use → Understand\n",
|
||
"1. **Build**: Create the Tensor class with core operations\n",
|
||
"2. **Use**: Perform tensor arithmetic and transformations\n",
|
||
"3. **Understand**: How tensors form the foundation of ML systems"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "5cee10fc",
|
||
"metadata": {
|
||
"nbgrader": {
|
||
"grade": false,
|
||
"grade_id": "tensor-imports",
|
||
"locked": false,
|
||
"schema_version": 3,
|
||
"solution": false,
|
||
"task": false
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"#| default_exp core.tensor\n",
|
||
"\n",
|
||
"#| export\n",
|
||
"import numpy as np\n",
|
||
"import sys\n",
|
||
"from typing import Union, List, Tuple, Optional, Any"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "3b504db9",
|
||
"metadata": {
|
||
"nbgrader": {
|
||
"grade": false,
|
||
"grade_id": "tensor-setup",
|
||
"locked": false,
|
||
"schema_version": 3,
|
||
"solution": false,
|
||
"task": false
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"print(\"🔥 TinyTorch Tensor Module\")\n",
|
||
"print(f\"NumPy version: {np.__version__}\")\n",
|
||
"print(f\"Python version: {sys.version_info.major}.{sys.version_info.minor}\")\n",
|
||
"print(\"Ready to build tensors!\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "bc5a3789",
|
||
"metadata": {
|
||
"cell_marker": "\"\"\""
|
||
},
|
||
"source": [
|
||
"## 📦 Where This Code Lives in the Final Package\n",
|
||
"\n",
|
||
"**Learning Side:** You work in `modules/source/01_tensor/tensor_dev.py` \n",
|
||
"**Building Side:** Code exports to `tinytorch.core.tensor`\n",
|
||
"\n",
|
||
"```python\n",
|
||
"# Final package structure:\n",
|
||
"from tinytorch.core.tensor import Tensor # The foundation of everything!\n",
|
||
"from tinytorch.core.activations import ReLU, Sigmoid, Tanh\n",
|
||
"from tinytorch.core.layers import Dense, Conv2D\n",
|
||
"```\n",
|
||
"\n",
|
||
"**Why this matters:**\n",
|
||
"- **Learning:** Focused modules for deep understanding\n",
|
||
"- **Production:** Proper organization like PyTorch's `torch.Tensor`\n",
|
||
"- **Consistency:** All tensor operations live together in `core.tensor`\n",
|
||
"- **Foundation:** Every other module depends on Tensor"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "edc31a73",
|
||
"metadata": {
|
||
"cell_marker": "\"\"\""
|
||
},
|
||
"source": [
|
||
"## 🔧 DEVELOPMENT"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "86d5f66b",
|
||
"metadata": {
|
||
"cell_marker": "\"\"\""
|
||
},
|
||
"source": [
|
||
"## Step 1: What is a Tensor?\n",
|
||
"\n",
|
||
"### Definition\n",
|
||
"A **tensor** is an N-dimensional array with ML-specific operations. Think of it as a container that can hold data in multiple dimensions:\n",
|
||
"\n",
|
||
"- **Scalar** (0D): A single number - `5.0`\n",
|
||
"- **Vector** (1D): A list of numbers - `[1, 2, 3]` \n",
|
||
"- **Matrix** (2D): A 2D array - `[[1, 2], [3, 4]]`\n",
|
||
"- **Higher dimensions**: 3D, 4D, etc. for images, video, batches\n",
|
||
"\n",
|
||
"### The Mathematical Foundation: From Scalars to Tensors\n",
|
||
"Understanding tensors requires building from mathematical fundamentals:\n",
|
||
"\n",
|
||
"#### **Scalars (Rank 0)**\n",
|
||
"- **Definition**: A single number with no direction\n",
|
||
"- **Examples**: Temperature (25°C), mass (5.2 kg), probability (0.7)\n",
|
||
"- **Operations**: Addition, multiplication, comparison\n",
|
||
"- **ML Context**: Loss values, learning rates, regularization parameters\n",
|
||
"\n",
|
||
"#### **Vectors (Rank 1)**\n",
|
||
"- **Definition**: An ordered list of numbers with direction and magnitude\n",
|
||
"- **Examples**: Position [x, y, z], RGB color [255, 128, 0], word embedding [0.1, -0.5, 0.8]\n",
|
||
"- **Operations**: Dot product, cross product, norm calculation\n",
|
||
"- **ML Context**: Feature vectors, gradients, model parameters\n",
|
||
"\n",
|
||
"#### **Matrices (Rank 2)**\n",
|
||
"- **Definition**: A 2D array organizing data in rows and columns\n",
|
||
"- **Examples**: Image (height × width), weight matrix (input × output), covariance matrix\n",
|
||
"- **Operations**: Matrix multiplication, transpose, inverse, eigendecomposition\n",
|
||
"- **ML Context**: Linear layer weights, attention matrices, batch data\n",
|
||
"\n",
|
||
"#### **Higher-Order Tensors (Rank 3+)**\n",
|
||
"- **Definition**: Multi-dimensional arrays extending matrices\n",
|
||
"- **Examples**: \n",
|
||
" - **3D**: Video frames (time × height × width), RGB images (height × width × channels)\n",
|
||
" - **4D**: Image batches (batch × height × width × channels)\n",
|
||
" - **5D**: Video batches (batch × time × height × width × channels)\n",
|
||
"- **Operations**: Tensor products, contractions, decompositions\n",
|
||
"- **ML Context**: Convolutional features, RNN states, transformer attention\n",
|
||
"\n",
|
||
"### Why Tensors Matter in ML: The Computational Foundation\n",
|
||
"\n",
|
||
"#### **1. Unified Data Representation**\n",
|
||
"Tensors provide a consistent way to represent all ML data:\n",
|
||
"```python\n",
|
||
"# All of these are tensors with different shapes\n",
|
||
"scalar_loss = Tensor(0.5) # Shape: ()\n",
|
||
"feature_vector = Tensor([1, 2, 3]) # Shape: (3,)\n",
|
||
"weight_matrix = Tensor([[1, 2], [3, 4]]) # Shape: (2, 2)\n",
|
||
"image_batch = Tensor(np.random.rand(32, 224, 224, 3)) # Shape: (32, 224, 224, 3)\n",
|
||
"```\n",
|
||
"\n",
|
||
"#### **2. Efficient Batch Processing**\n",
|
||
"ML systems process multiple samples simultaneously:\n",
|
||
"```python\n",
|
||
"# Instead of processing one image at a time:\n",
|
||
"for image in images:\n",
|
||
" result = model(image) # Slow: 1000 separate operations\n",
|
||
"\n",
|
||
"# Process entire batch at once:\n",
|
||
"batch_result = model(image_batch) # Fast: 1 vectorized operation\n",
|
||
"```\n",
|
||
"\n",
|
||
"#### **3. Hardware Acceleration**\n",
|
||
"Modern hardware (GPUs, TPUs) excels at tensor operations:\n",
|
||
"- **Parallel processing**: Multiple operations simultaneously\n",
|
||
"- **Vectorization**: SIMD (Single Instruction, Multiple Data) operations\n",
|
||
"- **Memory optimization**: Contiguous memory layout for cache efficiency\n",
|
||
"\n",
|
||
"#### **4. Automatic Differentiation**\n",
|
||
"Tensors enable gradient computation through computational graphs:\n",
|
||
"```python\n",
|
||
"# Each tensor operation creates a node in the computation graph\n",
|
||
"x = Tensor([1, 2, 3])\n",
|
||
"y = x * 2 # Node: multiplication\n",
|
||
"z = y + 1 # Node: addition\n",
|
||
"loss = z.sum() # Node: summation\n",
|
||
"# Gradients flow backward through this graph\n",
|
||
"```\n",
|
||
"\n",
|
||
"### Real-World Examples: Tensors in Action\n",
|
||
"\n",
|
||
"#### **Computer Vision**\n",
|
||
"- **Grayscale image**: 2D tensor `(height, width)` - `(28, 28)` for MNIST\n",
|
||
"- **Color image**: 3D tensor `(height, width, channels)` - `(224, 224, 3)` for RGB\n",
|
||
"- **Image batch**: 4D tensor `(batch, height, width, channels)` - `(32, 224, 224, 3)`\n",
|
||
"- **Video**: 5D tensor `(batch, time, height, width, channels)`\n",
|
||
"\n",
|
||
"#### **Natural Language Processing**\n",
|
||
"- **Word embedding**: 1D tensor `(embedding_dim,)` - `(300,)` for Word2Vec\n",
|
||
"- **Sentence**: 2D tensor `(sequence_length, embedding_dim)` - `(50, 768)` for BERT\n",
|
||
"- **Batch of sentences**: 3D tensor `(batch, sequence_length, embedding_dim)`\n",
|
||
"\n",
|
||
"#### **Audio Processing**\n",
|
||
"- **Audio signal**: 1D tensor `(time_steps,)` - `(16000,)` for 1 second at 16kHz\n",
|
||
"- **Spectrogram**: 2D tensor `(time_frames, frequency_bins)`\n",
|
||
"- **Batch of audio**: 3D tensor `(batch, time_steps, features)`\n",
|
||
"\n",
|
||
"#### **Time Series**\n",
|
||
"- **Single series**: 2D tensor `(time_steps, features)`\n",
|
||
"- **Multiple series**: 3D tensor `(batch, time_steps, features)`\n",
|
||
"- **Multivariate forecasting**: 4D tensor `(batch, time_steps, features, predictions)`\n",
|
||
"\n",
|
||
"### Why Not Just Use NumPy?\n",
|
||
"\n",
|
||
"While we use NumPy internally, our Tensor class adds ML-specific functionality:\n",
|
||
"\n",
|
||
"#### **1. ML-Specific Operations**\n",
|
||
"- **Gradient tracking**: For automatic differentiation (coming in Module 7)\n",
|
||
"- **GPU support**: For hardware acceleration (future extension)\n",
|
||
"- **Broadcasting semantics**: ML-friendly dimension handling\n",
|
||
"\n",
|
||
"#### **2. Consistent API**\n",
|
||
"- **Type safety**: Predictable behavior across operations\n",
|
||
"- **Error checking**: Clear error messages for debugging\n",
|
||
"- **Integration**: Seamless work with other TinyTorch components\n",
|
||
"\n",
|
||
"#### **3. Educational Value**\n",
|
||
"- **Conceptual clarity**: Understand what tensors really are\n",
|
||
"- **Implementation insight**: See how frameworks work internally\n",
|
||
"- **Debugging skills**: Trace through tensor operations step by step\n",
|
||
"\n",
|
||
"#### **4. Extensibility**\n",
|
||
"- **Future features**: Ready for gradients, GPU, distributed computing\n",
|
||
"- **Customization**: Add domain-specific operations\n",
|
||
"- **Optimization**: Profile and optimize specific use cases\n",
|
||
"\n",
|
||
"### Performance Considerations: Building Efficient Tensors\n",
|
||
"\n",
|
||
"#### **Memory Layout**\n",
|
||
"- **Contiguous arrays**: Better cache locality and performance\n",
|
||
"- **Data types**: `float32` vs `float64` trade-offs\n",
|
||
"- **Memory sharing**: Avoid unnecessary copies\n",
|
||
"\n",
|
||
"#### **Vectorization**\n",
|
||
"- **SIMD operations**: Single Instruction, Multiple Data\n",
|
||
"- **Broadcasting**: Efficient operations on different shapes\n",
|
||
"- **Batch operations**: Process multiple samples simultaneously\n",
|
||
"\n",
|
||
"#### **Numerical Stability**\n",
|
||
"- **Precision**: Balancing speed and accuracy\n",
|
||
"- **Overflow/underflow**: Handling extreme values\n",
|
||
"- **Gradient flow**: Maintaining numerical stability for training\n",
|
||
"\n",
|
||
"Let's start building our tensor foundation!"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "81319d29",
|
||
"metadata": {
|
||
"cell_marker": "\"\"\""
|
||
},
|
||
"source": [
|
||
"## 🧠 The Mathematical Foundation\n",
|
||
"\n",
|
||
"### Linear Algebra Refresher\n",
|
||
"Tensors are generalizations of scalars, vectors, and matrices:\n",
|
||
"\n",
|
||
"```\n",
|
||
"Scalar (0D): 5\n",
|
||
"Vector (1D): [1, 2, 3]\n",
|
||
"Matrix (2D): [[1, 2], [3, 4]]\n",
|
||
"Tensor (3D): [[[1, 2], [3, 4]], [[5, 6], [7, 8]]]\n",
|
||
"```\n",
|
||
"\n",
|
||
"### Why This Matters for Neural Networks\n",
|
||
"- **Forward Pass**: Matrix multiplication between layers\n",
|
||
"- **Batch Processing**: Multiple samples processed simultaneously\n",
|
||
"- **Convolutions**: 3D operations on image data\n",
|
||
"- **Gradients**: Derivatives computed across all dimensions\n",
|
||
"\n",
|
||
"### Connection to Real ML Systems\n",
|
||
"Every major ML framework uses tensors:\n",
|
||
"- **PyTorch**: `torch.Tensor`\n",
|
||
"- **TensorFlow**: `tf.Tensor`\n",
|
||
"- **JAX**: `jax.numpy.ndarray`\n",
|
||
"- **TinyTorch**: `tinytorch.core.tensor.Tensor` (what we're building!)\n",
|
||
"\n",
|
||
"### Performance Considerations\n",
|
||
"- **Memory Layout**: Contiguous arrays for cache efficiency\n",
|
||
"- **Vectorization**: SIMD operations for speed\n",
|
||
"- **Broadcasting**: Efficient operations on different shapes\n",
|
||
"- **Type Consistency**: Avoiding unnecessary conversions"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "11c6ee5d",
|
||
"metadata": {
|
||
"cell_marker": "\"\"\"",
|
||
"lines_to_next_cell": 1
|
||
},
|
||
"source": [
|
||
"## Step 2: The Tensor Class Foundation\n",
|
||
"\n",
|
||
"### 🔍 The Wrapper Pattern: Why Every ML Framework Uses It\n",
|
||
"\n",
|
||
"Before we dive into implementation, let's understand a critical design pattern that ALL ML frameworks use:\n",
|
||
"\n",
|
||
"#### **The Two-Layer Architecture**\n",
|
||
"```\n",
|
||
"┌─────────────────────────────────────┐\n",
|
||
"│ User-Facing API Layer (Python) │ ← What you interact with\n",
|
||
"│ - Tensor class │\n",
|
||
"│ - Automatic differentiation │\n",
|
||
"│ - High-level operations │\n",
|
||
"├─────────────────────────────────────┤\n",
|
||
"│ Efficient Compute Layer (C/C++) │ ← What does the actual work\n",
|
||
"│ - Optimized matrix operations │\n",
|
||
"│ - SIMD/vectorization │\n",
|
||
"│ - GPU kernels │\n",
|
||
"└─────────────────────────────────────┘\n",
|
||
"```\n",
|
||
"\n",
|
||
"#### **Why This Pattern?**\n",
|
||
"1. **Performance**: Low-level languages (C/C++/CUDA) for speed\n",
|
||
"2. **Usability**: High-level language (Python) for ease of use\n",
|
||
"3. **Flexibility**: Add ML-specific features without modifying core compute\n",
|
||
"4. **Portability**: Swap compute backends (CPU/GPU/TPU) transparently\n",
|
||
"\n",
|
||
"### Core Concept: Wrapping NumPy with ML Intelligence\n",
|
||
"Our Tensor class wraps NumPy arrays with ML-specific functionality. This design pattern is used by all major ML frameworks:\n",
|
||
"\n",
|
||
"- **PyTorch**: `torch.Tensor` wraps ATen (C++ tensor library)\n",
|
||
"- **TensorFlow**: `tf.Tensor` wraps Eigen (C++ linear algebra library)\n",
|
||
"- **JAX**: `jax.numpy.ndarray` wraps XLA (Google's linear algebra compiler)\n",
|
||
"- **TinyTorch**: `Tensor` wraps NumPy (Python's numerical computing library)\n",
|
||
"\n",
|
||
"### Design Requirements Analysis\n",
|
||
"\n",
|
||
"#### **1. Input Flexibility**\n",
|
||
"Our tensor must handle diverse input types:\n",
|
||
"```python\n",
|
||
"# Scalars (Python numbers)\n",
|
||
"t1 = Tensor(5) # int → numpy array\n",
|
||
"t2 = Tensor(3.14) # float → numpy array\n",
|
||
"\n",
|
||
"# Lists (Python sequences)\n",
|
||
"t3 = Tensor([1, 2, 3]) # list → numpy array\n",
|
||
"t4 = Tensor([[1, 2], [3, 4]]) # nested list → 2D array\n",
|
||
"\n",
|
||
"# NumPy arrays (existing arrays)\n",
|
||
"t5 = Tensor(np.array([1, 2, 3])) # array → tensor wrapper\n",
|
||
"```\n",
|
||
"\n",
|
||
"#### **2. Type Management**\n",
|
||
"ML systems need consistent, predictable types:\n",
|
||
"- **Default behavior**: Auto-detect appropriate types\n",
|
||
"- **Explicit control**: Allow manual type specification\n",
|
||
"- **Performance optimization**: Prefer `float32` over `float64`\n",
|
||
"- **Memory efficiency**: Use appropriate precision\n",
|
||
"\n",
|
||
"#### **3. Property Access**\n",
|
||
"Essential tensor properties for ML operations:\n",
|
||
"- **Shape**: Dimensions for compatibility checking\n",
|
||
"- **Size**: Total elements for memory estimation\n",
|
||
"- **Data type**: For numerical computation planning\n",
|
||
"- **Data access**: For integration with other libraries\n",
|
||
"\n",
|
||
"#### **4. Arithmetic Operations**\n",
|
||
"Support for mathematical operations:\n",
|
||
"- **Element-wise**: Addition, multiplication, subtraction, division\n",
|
||
"- **Broadcasting**: Operations on different shapes\n",
|
||
"- **Type promotion**: Consistent result types\n",
|
||
"- **Error handling**: Clear messages for incompatible operations\n",
|
||
"\n",
|
||
"### Implementation Strategy\n",
|
||
"\n",
|
||
"#### **Memory Management**\n",
|
||
"- **Copy vs. Reference**: When to copy data vs. share memory\n",
|
||
"- **Type conversion**: Efficient dtype changes\n",
|
||
"- **Contiguous layout**: Ensure optimal memory access patterns\n",
|
||
"\n",
|
||
"#### **Error Handling**\n",
|
||
"- **Input validation**: Check for valid input types\n",
|
||
"- **Shape compatibility**: Verify operations are mathematically valid\n",
|
||
"- **Informative messages**: Help users debug issues quickly\n",
|
||
"\n",
|
||
"#### **Performance Optimization**\n",
|
||
"- **Lazy evaluation**: Defer expensive operations when possible\n",
|
||
"- **Vectorization**: Use NumPy's optimized operations\n",
|
||
"- **Memory reuse**: Minimize unnecessary allocations\n",
|
||
"\n",
|
||
"### Learning Objectives for Implementation\n",
|
||
"\n",
|
||
"By implementing this Tensor class, you'll learn:\n",
|
||
"1. **Wrapper pattern**: How to extend existing libraries\n",
|
||
"2. **Type system design**: Managing data types in numerical computing\n",
|
||
"3. **API design**: Creating intuitive, consistent interfaces\n",
|
||
"4. **Performance considerations**: Balancing flexibility and speed\n",
|
||
"5. **Error handling**: Providing helpful feedback to users\n",
|
||
"\n",
|
||
"### 📊 Understanding NumPy Data Types Before We Start\n",
|
||
"\n",
|
||
"Before implementing the constructor, let's understand NumPy's type system:\n",
|
||
"\n",
|
||
"#### **Common Data Types in ML**\n",
|
||
"```python\n",
|
||
"# Integer types (discrete values)\n",
|
||
"'int8', 'int16', 'int32', 'int64' # Signed integers\n",
|
||
"'uint8', 'uint16', 'uint32', 'uint64' # Unsigned integers\n",
|
||
"\n",
|
||
"# Float types (continuous values)\n",
|
||
"'float16' # Half precision (saves memory, less accurate)\n",
|
||
"'float32' # Single precision (ML standard - good balance)\n",
|
||
"'float64' # Double precision (high accuracy, more memory)\n",
|
||
"```\n",
|
||
"\n",
|
||
"#### **Why We Prefer float32 in ML**\n",
|
||
"- **Memory**: Uses half the memory of float64\n",
|
||
"- **Speed**: Faster computation on GPUs\n",
|
||
"- **Sufficient precision**: Good enough for most ML tasks\n",
|
||
"- **Industry standard**: Used by PyTorch, TensorFlow by default\n",
|
||
"\n",
|
||
"Let's implement our tensor foundation!"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "ecd3a418",
|
||
"metadata": {
|
||
"lines_to_next_cell": 1,
|
||
"nbgrader": {
|
||
"grade": false,
|
||
"grade_id": "tensor-class",
|
||
"locked": false,
|
||
"schema_version": 3,
|
||
"solution": true,
|
||
"task": false
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"#| export\n",
|
||
"class Tensor:\n",
|
||
" \"\"\"\n",
|
||
" TinyTorch Tensor: N-dimensional array with ML operations.\n",
|
||
" \n",
|
||
" The fundamental data structure for all TinyTorch operations.\n",
|
||
" Wraps NumPy arrays with ML-specific functionality.\n",
|
||
" \"\"\"\n",
|
||
" \n",
|
||
" def __init__(self, data: Union[int, float, List, np.ndarray], dtype: Optional[str] = None):\n",
|
||
" \"\"\"\n",
|
||
" Create a new tensor from data.\n",
|
||
" \n",
|
||
" Args:\n",
|
||
" data: Input data (scalar, list, or numpy array)\n",
|
||
" dtype: Data type ('float32', 'int32', etc.). Defaults to auto-detect.\n",
|
||
" \n",
|
||
" TODO: Implement tensor creation with proper type handling.\n",
|
||
" \n",
|
||
" APPROACH:\n",
|
||
" 1. Check the input data type using isinstance()\n",
|
||
" - If scalar (int/float): convert to numpy array\n",
|
||
" - If list: convert to numpy array\n",
|
||
" - If numpy array: use directly (but make a copy)\n",
|
||
" 2. Handle the dtype parameter:\n",
|
||
" - If dtype is None: auto-detect the best type\n",
|
||
" - If dtype is specified: convert to that type\n",
|
||
" 3. For auto-detection, prefer:\n",
|
||
" - int32 for integers (not int64)\n",
|
||
" - float32 for floats (not float64)\n",
|
||
" 4. Store the final array in self._data\n",
|
||
" \n",
|
||
" EXAMPLE:\n",
|
||
" >>> t1 = Tensor(5) # Creates int32 array\n",
|
||
" >>> t2 = Tensor(3.14) # Creates float32 array\n",
|
||
" >>> t3 = Tensor([1, 2, 3]) # Creates int32 array\n",
|
||
" >>> t4 = Tensor([1.0, 2.0], 'float64') # Forces float64\n",
|
||
" >>> t5 = Tensor(np.array([1, 2])) # Preserves numpy array\n",
|
||
" \n",
|
||
" HINTS:\n",
|
||
" - Use isinstance(data, (int, float)) to check for scalars\n",
|
||
" - Use isinstance(data, list) for lists\n",
|
||
" - Use isinstance(data, np.ndarray) for numpy arrays\n",
|
||
" - For dtype conversion: np.array(data, dtype=dtype)\n",
|
||
" - Remember: we prefer float32 over float64 for ML efficiency\n",
|
||
" \"\"\"\n",
|
||
" ### BEGIN SOLUTION\n",
|
||
" # Convert input to numpy array\n",
|
||
" if isinstance(data, (int, float, np.number)):\n",
|
||
" # Handle Python and NumPy scalars\n",
|
||
" if dtype is None:\n",
|
||
" # Auto-detect type: int for integers, float32 for floats\n",
|
||
" if isinstance(data, int) or (isinstance(data, np.number) and np.issubdtype(type(data), np.integer)):\n",
|
||
" dtype = 'int32'\n",
|
||
" else:\n",
|
||
" dtype = 'float32'\n",
|
||
" self._data = np.array(data, dtype=dtype)\n",
|
||
" elif isinstance(data, list):\n",
|
||
" # Let NumPy auto-detect type, then convert if needed\n",
|
||
" temp_array = np.array(data)\n",
|
||
" if dtype is None:\n",
|
||
" # Use NumPy's auto-detected type, but prefer float32 for floats\n",
|
||
" if temp_array.dtype == np.float64:\n",
|
||
" dtype = 'float32'\n",
|
||
" else:\n",
|
||
" dtype = str(temp_array.dtype)\n",
|
||
" self._data = np.array(data, dtype=dtype)\n",
|
||
" elif isinstance(data, np.ndarray):\n",
|
||
" # Already a numpy array\n",
|
||
" if dtype is None:\n",
|
||
" # Keep existing dtype, but prefer float32 for float64\n",
|
||
" if data.dtype == np.float64:\n",
|
||
" dtype = 'float32'\n",
|
||
" else:\n",
|
||
" dtype = str(data.dtype)\n",
|
||
" self._data = data.astype(dtype) if dtype != data.dtype else data.copy()\n",
|
||
" else:\n",
|
||
" # Try to convert unknown types\n",
|
||
" self._data = np.array(data, dtype=dtype)\n",
|
||
" ### END SOLUTION\n",
|
||
"\n",
|
||
" @property\n",
|
||
" def data(self) -> np.ndarray:\n",
|
||
" \"\"\"\n",
|
||
" Access underlying numpy array.\n",
|
||
" \n",
|
||
" TODO: Return the stored numpy array.\n",
|
||
" \n",
|
||
" HINT: Return self._data (the array you stored in __init__)\n",
|
||
" \"\"\"\n",
|
||
" ### BEGIN SOLUTION\n",
|
||
" return self._data\n",
|
||
" ### END SOLUTION\n",
|
||
" \n",
|
||
" @property\n",
|
||
" def shape(self) -> Tuple[int, ...]:\n",
|
||
" \"\"\"\n",
|
||
" Get tensor shape.\n",
|
||
" \n",
|
||
" TODO: Return the shape of the stored numpy array.\n",
|
||
" \n",
|
||
" HINT: Use .shape attribute of the numpy array\n",
|
||
" EXAMPLE: Tensor([1, 2, 3]).shape should return (3,)\n",
|
||
" \"\"\"\n",
|
||
" ### BEGIN SOLUTION\n",
|
||
" return self._data.shape\n",
|
||
" ### END SOLUTION\n",
|
||
" \n",
|
||
" @property\n",
|
||
" def size(self) -> int:\n",
|
||
" \"\"\"\n",
|
||
" Get total number of elements.\n",
|
||
" \n",
|
||
" TODO: Return the total number of elements in the tensor.\n",
|
||
" \n",
|
||
" HINT: Use .size attribute of the numpy array\n",
|
||
" EXAMPLE: Tensor([1, 2, 3]).size should return 3\n",
|
||
" \"\"\"\n",
|
||
" ### BEGIN SOLUTION\n",
|
||
" return self._data.size\n",
|
||
" ### END SOLUTION\n",
|
||
" \n",
|
||
" @property\n",
|
||
" def dtype(self) -> np.dtype:\n",
|
||
" \"\"\"\n",
|
||
" Get data type as numpy dtype.\n",
|
||
" \n",
|
||
" TODO: Return the data type of the stored numpy array.\n",
|
||
" \n",
|
||
" HINT: Use .dtype attribute of the numpy array\n",
|
||
" EXAMPLE: Tensor([1, 2, 3]).dtype should return dtype('int32')\n",
|
||
" \"\"\"\n",
|
||
" ### BEGIN SOLUTION\n",
|
||
" return self._data.dtype\n",
|
||
" ### END SOLUTION\n",
|
||
" \n",
|
||
" def __repr__(self) -> str:\n",
|
||
" \"\"\"\n",
|
||
" String representation.\n",
|
||
" \n",
|
||
" TODO: Create a clear string representation of the tensor.\n",
|
||
" \n",
|
||
" APPROACH:\n",
|
||
" 1. Convert the numpy array to a list for readable output\n",
|
||
" 2. Include the shape and dtype information\n",
|
||
" 3. Format: \"Tensor([data], shape=shape, dtype=dtype)\"\n",
|
||
" \n",
|
||
" EXAMPLE:\n",
|
||
" Tensor([1, 2, 3]) → \"Tensor([1, 2, 3], shape=(3,), dtype=int32)\"\n",
|
||
" \n",
|
||
" HINTS:\n",
|
||
" - Use .tolist() to convert numpy array to list\n",
|
||
" - Include shape and dtype information\n",
|
||
" - Keep format consistent and readable\n",
|
||
" \"\"\"\n",
|
||
" ### BEGIN SOLUTION\n",
|
||
" return f\"Tensor({self._data.tolist()}, shape={self.shape}, dtype={self.dtype})\"\n",
|
||
" ### END SOLUTION\n",
|
||
"\n",
|
||
" def add(self, other: 'Tensor') -> 'Tensor':\n",
|
||
" \"\"\"\n",
|
||
" Add two tensors element-wise.\n",
|
||
" \n",
|
||
" TODO: Implement tensor addition.\n",
|
||
" \n",
|
||
" APPROACH:\n",
|
||
" 1. Add the numpy arrays using +\n",
|
||
" 2. Return a new Tensor with the result\n",
|
||
" 3. Handle broadcasting automatically\n",
|
||
" \n",
|
||
" EXAMPLE:\n",
|
||
" Tensor([1, 2]) + Tensor([3, 4]) → Tensor([4, 6])\n",
|
||
" \n",
|
||
" HINTS:\n",
|
||
" - Use self._data + other._data\n",
|
||
" - Return Tensor(result)\n",
|
||
" - NumPy handles broadcasting automatically\n",
|
||
" \"\"\"\n",
|
||
" ### BEGIN SOLUTION\n",
|
||
" result = self._data + other._data\n",
|
||
" return Tensor(result)\n",
|
||
" ### END SOLUTION\n",
|
||
"\n",
|
||
" def multiply(self, other: 'Tensor') -> 'Tensor':\n",
|
||
" \"\"\"\n",
|
||
" Multiply two tensors element-wise.\n",
|
||
" \n",
|
||
" TODO: Implement tensor multiplication.\n",
|
||
" \n",
|
||
" APPROACH:\n",
|
||
" 1. Multiply the numpy arrays using *\n",
|
||
" 2. Return a new Tensor with the result\n",
|
||
" 3. Handle broadcasting automatically\n",
|
||
" \n",
|
||
" EXAMPLE:\n",
|
||
" Tensor([1, 2]) * Tensor([3, 4]) → Tensor([3, 8])\n",
|
||
" \n",
|
||
" HINTS:\n",
|
||
" - Use self._data * other._data\n",
|
||
" - Return Tensor(result)\n",
|
||
" - This is element-wise, not matrix multiplication\n",
|
||
" \"\"\"\n",
|
||
" ### BEGIN SOLUTION\n",
|
||
" result = self._data * other._data\n",
|
||
" return Tensor(result)\n",
|
||
" ### END SOLUTION\n",
|
||
"\n",
|
||
" def __add__(self, other: Union['Tensor', int, float]) -> 'Tensor':\n",
|
||
" \"\"\"\n",
|
||
" Addition operator overload for natural syntax: tensor + other.\n",
|
||
" \n",
|
||
" TODO: Implement the + operator to enable natural addition syntax for tensors.\n",
|
||
" \n",
|
||
" APPROACH:\n",
|
||
" 1. Check if other is a Tensor or scalar (int/float)\n",
|
||
" 2. If scalar, convert to Tensor for uniform handling\n",
|
||
" 3. Delegate to the add() method for actual computation\n",
|
||
" 4. Return the resulting Tensor\n",
|
||
" \n",
|
||
" EXAMPLE:\n",
|
||
" >>> a = Tensor([1, 2, 3])\n",
|
||
" >>> b = Tensor([4, 5, 6])\n",
|
||
" >>> c = a + b # Element-wise addition\n",
|
||
" >>> print(c.data) # [5, 7, 9]\n",
|
||
" >>> d = a + 10 # Broadcasting with scalar\n",
|
||
" >>> print(d.data) # [11, 12, 13]\n",
|
||
" \n",
|
||
" HINTS:\n",
|
||
" - Use isinstance(other, Tensor) to check type\n",
|
||
" - Convert scalars with Tensor(other)\n",
|
||
" - Delegate to self.add() for consistency\n",
|
||
" - Python automatically calls this for + operator\n",
|
||
" \n",
|
||
" LEARNING CONNECTIONS:\n",
|
||
" - This is how torch.Tensor implements natural syntax\n",
|
||
" - Operator overloading makes math operations intuitive\n",
|
||
" - Broadcasting happens automatically in the add() method\n",
|
||
" \"\"\"\n",
|
||
" ### BEGIN SOLUTION\n",
|
||
" if isinstance(other, Tensor):\n",
|
||
" return self.add(other)\n",
|
||
" else:\n",
|
||
" return self.add(Tensor(other))\n",
|
||
" ### END SOLUTION\n",
|
||
"\n",
|
||
" def __mul__(self, other: Union['Tensor', int, float]) -> 'Tensor':\n",
|
||
" \"\"\"\n",
|
||
" Multiplication operator overload for natural syntax: tensor * other.\n",
|
||
" \n",
|
||
" TODO: Implement the * operator for element-wise multiplication (Hadamard product).\n",
|
||
" \n",
|
||
" APPROACH:\n",
|
||
" 1. Check if other is a Tensor or scalar (int/float)\n",
|
||
" 2. If scalar, convert to Tensor for uniform handling\n",
|
||
" 3. Delegate to the multiply() method for computation\n",
|
||
" 4. Return the resulting Tensor\n",
|
||
" \n",
|
||
" EXAMPLE:\n",
|
||
" >>> a = Tensor([2, 3, 4])\n",
|
||
" >>> b = Tensor([5, 6, 7])\n",
|
||
" >>> c = a * b # Element-wise multiplication\n",
|
||
" >>> print(c.data) # [10, 18, 28]\n",
|
||
" >>> d = a * 3 # Scalar multiplication\n",
|
||
" >>> print(d.data) # [6, 9, 12]\n",
|
||
" \n",
|
||
" HINTS:\n",
|
||
" - This is element-wise, NOT matrix multiplication\n",
|
||
" - For matrix multiplication, use @ operator (matmul)\n",
|
||
" - Scalar multiplication broadcasts automatically\n",
|
||
" - Delegate to self.multiply() for consistency\n",
|
||
" \n",
|
||
" LEARNING CONNECTIONS:\n",
|
||
" - In PyTorch: * is element-wise, @ is matrix multiplication\n",
|
||
" - Neural networks use this for element-wise activations\n",
|
||
" - Scalar multiplication is used for learning rate scaling\n",
|
||
" \"\"\"\n",
|
||
" ### BEGIN SOLUTION\n",
|
||
" if isinstance(other, Tensor):\n",
|
||
" return self.multiply(other)\n",
|
||
" else:\n",
|
||
" return self.multiply(Tensor(other))\n",
|
||
" ### END SOLUTION\n",
|
||
"\n",
|
||
" def __sub__(self, other: Union['Tensor', int, float]) -> 'Tensor':\n",
|
||
" \"\"\"\n",
|
||
" Subtraction operator overload for natural syntax: tensor - other.\n",
|
||
" \n",
|
||
" TODO: Implement the - operator for element-wise subtraction.\n",
|
||
" \n",
|
||
" APPROACH:\n",
|
||
" 1. Check if other is a Tensor or scalar\n",
|
||
" 2. Access the underlying NumPy arrays with ._data\n",
|
||
" 3. Perform subtraction using NumPy's broadcasting\n",
|
||
" 4. Wrap the result in a new Tensor and return\n",
|
||
" \n",
|
||
" EXAMPLE:\n",
|
||
" >>> a = Tensor([10, 20, 30])\n",
|
||
" >>> b = Tensor([3, 5, 7])\n",
|
||
" >>> c = a - b # Element-wise subtraction\n",
|
||
" >>> print(c.data) # [7, 15, 23]\n",
|
||
" >>> d = a - 5 # Broadcasting with scalar\n",
|
||
" >>> print(d.data) # [5, 15, 25]\n",
|
||
" \n",
|
||
" HINTS:\n",
|
||
" - Access data with self._data and other._data\n",
|
||
" - NumPy handles broadcasting automatically\n",
|
||
" - Always return a new Tensor (immutability)\n",
|
||
" - Handle both Tensor and scalar cases\n",
|
||
" \n",
|
||
" LEARNING CONNECTIONS:\n",
|
||
" - Used in gradient computation (loss - target)\n",
|
||
" - Batch normalization uses mean subtraction\n",
|
||
" - Residual connections compute (output - input)\n",
|
||
" \"\"\"\n",
|
||
" ### BEGIN SOLUTION\n",
|
||
" if isinstance(other, Tensor):\n",
|
||
" result = self._data - other._data\n",
|
||
" else:\n",
|
||
" result = self._data - other\n",
|
||
" return Tensor(result)\n",
|
||
" ### END SOLUTION\n",
|
||
"\n",
|
||
" def __truediv__(self, other: Union['Tensor', int, float]) -> 'Tensor':\n",
|
||
" \"\"\"\n",
|
||
" Division operator overload for natural syntax: tensor / other.\n",
|
||
" \n",
|
||
" TODO: Implement the / operator for element-wise division.\n",
|
||
" \n",
|
||
" APPROACH:\n",
|
||
" 1. Check if other is a Tensor or scalar\n",
|
||
" 2. Access the underlying NumPy arrays\n",
|
||
" 3. Perform division (watch for division by zero!)\n",
|
||
" 4. Wrap the result in a new Tensor and return\n",
|
||
" \n",
|
||
" EXAMPLE:\n",
|
||
" >>> a = Tensor([10, 20, 30])\n",
|
||
" >>> b = Tensor([2, 4, 5])\n",
|
||
" >>> c = a / b # Element-wise division\n",
|
||
" >>> print(c.data) # [5.0, 5.0, 6.0]\n",
|
||
" >>> d = a / 10 # Scalar division\n",
|
||
" >>> print(d.data) # [1.0, 2.0, 3.0]\n",
|
||
" \n",
|
||
" HINTS:\n",
|
||
" - NumPy handles division by zero → inf/nan\n",
|
||
" - Results are always float type\n",
|
||
" - Consider using safe_divide() for production\n",
|
||
" - Broadcasting rules apply here too\n",
|
||
" \n",
|
||
" LEARNING CONNECTIONS:\n",
|
||
" - Normalization often uses division (x / std)\n",
|
||
" - Softmax involves division by sum\n",
|
||
" - Learning rate scheduling uses division\n",
|
||
" - Numerical stability is critical in ML\n",
|
||
" \"\"\"\n",
|
||
" ### BEGIN SOLUTION\n",
|
||
" if isinstance(other, Tensor):\n",
|
||
" result = self._data / other._data\n",
|
||
" else:\n",
|
||
" result = self._data / other\n",
|
||
" return Tensor(result)\n",
|
||
" ### END SOLUTION\n",
|
||
"\n",
|
||
" def mean(self) -> 'Tensor':\n",
|
||
" \"\"\"Computes the mean of the tensor's elements.\"\"\"\n",
|
||
" return Tensor(np.mean(self.data))\n",
|
||
"\n",
|
||
" # --- Matmul ---\n",
|
||
" def matmul(self, other: 'Tensor') -> 'Tensor':\n",
|
||
" \"\"\"\n",
|
||
" Perform matrix multiplication between two tensors.\n",
|
||
" \n",
|
||
" TODO: Implement matrix multiplication.\n",
|
||
" \n",
|
||
" APPROACH:\n",
|
||
" 1. Use np.matmul() to perform matrix multiplication\n",
|
||
" 2. Return a new Tensor with the result\n",
|
||
" 3. Handle broadcasting automatically\n",
|
||
" \n",
|
||
" EXAMPLE:\n",
|
||
" Tensor([[1, 2], [3, 4]]) @ Tensor([[5, 6], [7, 8]]) → Tensor([[19, 22], [43, 50]])\n",
|
||
" \n",
|
||
" HINTS:\n",
|
||
" - Use np.matmul(self._data, other._data)\n",
|
||
" - Return Tensor(result)\n",
|
||
" - This is matrix multiplication, not element-wise multiplication\n",
|
||
" \"\"\"\n",
|
||
" ### BEGIN SOLUTION\n",
|
||
" result = np.matmul(self._data, other._data)\n",
|
||
" return Tensor(result)\n",
|
||
" ### END SOLUTION\n",
|
||
" \n",
|
||
" # --- Error Handling ---\n",
|
||
" def validate_operation(self, other: 'Tensor', operation: str) -> None:\n",
|
||
" \"\"\"\n",
|
||
" Validate tensor compatibility for operations with informative error messages.\n",
|
||
" \n",
|
||
" TODO: Implement validation for tensor operations that checks shape compatibility\n",
|
||
" and provides educational error messages when operations would fail.\n",
|
||
" \n",
|
||
" APPROACH:\n",
|
||
" 1. Check if the operation is element-wise or matrix multiplication\n",
|
||
" 2. For element-wise ops, verify broadcasting compatibility\n",
|
||
" 3. For matmul, verify matrix dimension alignment\n",
|
||
" 4. Raise informative ValueError with hints if incompatible\n",
|
||
" \n",
|
||
" EXAMPLE:\n",
|
||
" >>> a = Tensor([[1, 2, 3]]) # Shape: (1, 3)\n",
|
||
" >>> b = Tensor([[1], [2]]) # Shape: (2, 1)\n",
|
||
" >>> a.validate_operation(b, 'add') # Should pass (broadcasting)\n",
|
||
" >>> a.validate_operation(b, 'matmul') # Should raise with helpful message\n",
|
||
" \n",
|
||
" HINTS:\n",
|
||
" - Use np.broadcast_shapes() to check broadcasting compatibility\n",
|
||
" - For matmul, check that self.shape[-1] == other.shape[-2]\n",
|
||
" - Include shapes in error messages to help debugging\n",
|
||
" - Suggest fixes in the error message\n",
|
||
" \n",
|
||
" LEARNING CONNECTIONS:\n",
|
||
" - PyTorch raises similar errors with shape information\n",
|
||
" - Good error messages accelerate learning and debugging\n",
|
||
" - Understanding shape compatibility is crucial for ML\n",
|
||
" \"\"\"\n",
|
||
" ### BEGIN SOLUTION\n",
|
||
" if operation in ['add', 'mul', 'sub', 'div']:\n",
|
||
" # Check broadcasting compatibility for element-wise operations\n",
|
||
" try:\n",
|
||
" np.broadcast_shapes(self.shape, other.shape)\n",
|
||
" except ValueError:\n",
|
||
" raise ValueError(\n",
|
||
" f\"Cannot {operation} tensors with incompatible shapes: \"\n",
|
||
" f\"{self.shape} and {other.shape}. \"\n",
|
||
" f\"Hint: Shapes must be broadcastable. Either dimensions match \"\n",
|
||
" f\"or one of them is 1. Consider reshaping with tensor.reshape().\"\n",
|
||
" )\n",
|
||
" elif operation == 'matmul':\n",
|
||
" # Check matrix multiplication compatibility\n",
|
||
" if len(self.shape) == 0 or len(other.shape) == 0:\n",
|
||
" raise ValueError(\n",
|
||
" f\"Cannot perform matrix multiplication with scalar tensors. \"\n",
|
||
" f\"Shapes: {self.shape} @ {other.shape}. \"\n",
|
||
" f\"Hint: Use regular multiplication (*) for scalar operations.\"\n",
|
||
" )\n",
|
||
" if self.shape[-1] != other.shape[-2 if len(other.shape) > 1 else 0]:\n",
|
||
" raise ValueError(\n",
|
||
" f\"Cannot perform matrix multiplication with incompatible shapes: \"\n",
|
||
" f\"{self.shape} @ {other.shape}. \"\n",
|
||
" f\"Hint: For A @ B, A's last dimension ({self.shape[-1]}) must match \"\n",
|
||
" f\"B's second-to-last dimension ({other.shape[-2 if len(other.shape) > 1 else 0]}). \"\n",
|
||
" f\"Consider transposing with tensor.T or reshaping.\"\n",
|
||
" )\n",
|
||
" ### END SOLUTION\n",
|
||
" \n",
|
||
" def safe_divide(self, other: Union['Tensor', int, float]) -> 'Tensor':\n",
|
||
" \"\"\"\n",
|
||
" Perform division with zero-division protection and informative warnings.\n",
|
||
" \n",
|
||
" TODO: Implement safe division that handles division by zero gracefully\n",
|
||
" and provides educational feedback about numerical stability.\n",
|
||
" \n",
|
||
" APPROACH:\n",
|
||
" 1. Check for zero values in the divisor\n",
|
||
" 2. Use np.where to handle zero divisions appropriately \n",
|
||
" 3. Optionally warn about near-zero values that might cause instability\n",
|
||
" 4. Return result with inf/nan handled according to NumPy conventions\n",
|
||
" \n",
|
||
" EXAMPLE:\n",
|
||
" >>> a = Tensor([1, 2, 3, 4])\n",
|
||
" >>> b = Tensor([2, 0, 1, 0])\n",
|
||
" >>> result = a.safe_divide(b) # Handles zeros gracefully\n",
|
||
" >>> print(result.data) # [0.5, inf, 3.0, inf] with warning\n",
|
||
" \n",
|
||
" HINTS:\n",
|
||
" - Use np.where(divisor == 0, np.inf, normal_division)\n",
|
||
" - Consider np.errstate(divide='ignore') context manager\n",
|
||
" - Check for small values with np.abs(divisor) < epsilon\n",
|
||
" - Return informative warning messages for learning\n",
|
||
" \n",
|
||
" LEARNING CONNECTIONS:\n",
|
||
" - Numerical stability is critical in ML (gradient explosions)\n",
|
||
" - Many ML bugs come from division by zero in normalizations\n",
|
||
" - Production systems need robust error handling\n",
|
||
" \"\"\"\n",
|
||
" ### BEGIN SOLUTION\n",
|
||
" other_data = other._data if isinstance(other, Tensor) else other\n",
|
||
" \n",
|
||
" # Check for exact zeros\n",
|
||
" if np.any(other_data == 0):\n",
|
||
" import warnings\n",
|
||
" warnings.warn(\n",
|
||
" \"Division by zero detected! Results will contain inf values. \"\n",
|
||
" \"In ML, this often happens in normalization or loss calculations. \"\n",
|
||
" \"Consider adding a small epsilon (e.g., 1e-8) to prevent instability.\",\n",
|
||
" category=RuntimeWarning\n",
|
||
" )\n",
|
||
" \n",
|
||
" # Check for near-zero values that might cause instability\n",
|
||
" epsilon = 1e-10\n",
|
||
" if np.any(np.abs(other_data) < epsilon):\n",
|
||
" import warnings\n",
|
||
" warnings.warn(\n",
|
||
" f\"Near-zero divisor detected (values < {epsilon}). \"\n",
|
||
" \"This may cause numerical instability in gradients. \"\n",
|
||
" \"Consider using stabilized versions of your operations.\",\n",
|
||
" category=RuntimeWarning\n",
|
||
" )\n",
|
||
" \n",
|
||
" # Perform division with NumPy's default handling (inf for division by zero)\n",
|
||
" with np.errstate(divide='ignore', invalid='ignore'):\n",
|
||
" result = self._data / other_data\n",
|
||
" \n",
|
||
" return Tensor(result)\n",
|
||
" ### END SOLUTION"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "a354d1b5",
|
||
"metadata": {
|
||
"cell_marker": "\"\"\""
|
||
},
|
||
"source": [
|
||
"### 🎯 Checkpoint: Basic Tensor Creation\n",
|
||
"Before moving on, make sure you can:\n",
|
||
"```python\n",
|
||
"# Create different tensor types\n",
|
||
"scalar = Tensor(5) # 0D tensor\n",
|
||
"vector = Tensor([1, 2, 3]) # 1D tensor \n",
|
||
"matrix = Tensor([[1, 2], [3, 4]]) # 2D tensor\n",
|
||
"\n",
|
||
"# Check their properties\n",
|
||
"print(f\"Scalar shape: {scalar.shape}\") # ()\n",
|
||
"print(f\"Vector shape: {vector.shape}\") # (3,)\n",
|
||
"print(f\"Matrix shape: {matrix.shape}\") # (2, 2)\n",
|
||
"```"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "df030d32",
|
||
"metadata": {
|
||
"cell_marker": "\"\"\""
|
||
},
|
||
"source": [
|
||
"## Step 3: Understanding Broadcasting for Arithmetic Operations\n",
|
||
"\n",
|
||
"### 📊 Broadcasting: The Magic of Flexible Operations\n",
|
||
"\n",
|
||
"Broadcasting allows operations between tensors of different shapes without explicit reshaping. It's what makes code like `matrix + vector` or `tensor * scalar` work seamlessly.\n",
|
||
"\n",
|
||
"#### **Broadcasting Rules (Simple to Complex)**\n",
|
||
"\n",
|
||
"**Rule 1: Scalar Broadcasting**\n",
|
||
"```\n",
|
||
"Scalar + Any Shape = Element-wise Operation\n",
|
||
" 5 + [1,2,3] = [6,7,8]\n",
|
||
"```\n",
|
||
"\n",
|
||
"**Rule 2: Vector Broadcasting**\n",
|
||
"```\n",
|
||
"Matrix + Vector (matching last dimension)\n",
|
||
"[[1,2,3] [10,20,30] [[11,22,33]\n",
|
||
" [4,5,6]] + = [14,25,36]]\n",
|
||
"```\n",
|
||
"\n",
|
||
"**Rule 3: Dimension Alignment**\n",
|
||
"```\n",
|
||
"Shapes are aligned from the right:\n",
|
||
"(3, 4) and (4,) → Compatible (4 matches)\n",
|
||
"(3, 4) and (3,) → Not compatible (3 doesn't match 4)\n",
|
||
"```\n",
|
||
"\n",
|
||
"#### **Visual Examples**\n",
|
||
"```python\n",
|
||
"# Scalar broadcasting\n",
|
||
"[1, 2, 3] + 5 = [1+5, 2+5, 3+5] = [6, 7, 8]\n",
|
||
"\n",
|
||
"# Row broadcasting \n",
|
||
"[[1, 2] [10, 20] [[1+10, 2+20] [[11, 22]\n",
|
||
" [3, 4]] + = [3+10, 4+20]] = [13, 24]]\n",
|
||
"\n",
|
||
"# Column broadcasting\n",
|
||
"[[1, 2] [[10] [[1+10, 2+10] [[11, 12]\n",
|
||
" [3, 4]] + [20]] = [3+20, 4+20]] = [23, 24]]\n",
|
||
"```\n",
|
||
"\n",
|
||
"### Why Broadcasting Matters in ML\n",
|
||
"- **Batch operations**: Add bias to all samples in a batch\n",
|
||
"- **Normalization**: Subtract mean from all features\n",
|
||
"- **Scaling**: Multiply by weights across dimensions\n",
|
||
"- **Efficiency**: No memory copying, just smart iteration\n",
|
||
"\n",
|
||
"With this understanding, the arithmetic operations in our Tensor class will automatically support broadcasting through NumPy!"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "1bd1d01d",
|
||
"metadata": {
|
||
"cell_marker": "\"\"\""
|
||
},
|
||
"source": [
|
||
"### 🧪 Unit Test: Tensor Creation\n",
|
||
"\n",
|
||
"Let's test your tensor creation implementation right away! This gives you immediate feedback on whether your `__init__` method works correctly.\n",
|
||
"\n",
|
||
"**This is a unit test** - it tests one specific function (tensor creation) in isolation."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "b3fbade8",
|
||
"metadata": {
|
||
"nbgrader": {
|
||
"grade": true,
|
||
"grade_id": "test-tensor-creation-immediate",
|
||
"locked": true,
|
||
"points": 5,
|
||
"schema_version": 3,
|
||
"solution": false,
|
||
"task": false
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Test tensor creation immediately after implementation\n",
|
||
"print(\"🔬 Unit Test: Tensor Creation...\")\n",
|
||
"\n",
|
||
"# Test basic tensor creation\n",
|
||
"try:\n",
|
||
" # Test scalar\n",
|
||
" scalar = Tensor(5.0)\n",
|
||
" assert hasattr(scalar, '_data'), \"Tensor should have _data attribute\"\n",
|
||
" assert scalar._data.shape == (), f\"Scalar should have shape (), got {scalar._data.shape}\"\n",
|
||
" print(\"✅ Scalar creation works\")\n",
|
||
" \n",
|
||
" # Test vector\n",
|
||
" vector = Tensor([1, 2, 3])\n",
|
||
" assert vector._data.shape == (3,), f\"Vector should have shape (3,), got {vector._data.shape}\"\n",
|
||
" print(\"✅ Vector creation works\")\n",
|
||
" \n",
|
||
" # Test matrix\n",
|
||
" matrix = Tensor([[1, 2], [3, 4]])\n",
|
||
" assert matrix._data.shape == (2, 2), f\"Matrix should have shape (2, 2), got {matrix._data.shape}\"\n",
|
||
" print(\"✅ Matrix creation works\")\n",
|
||
" \n",
|
||
" print(\"📈 Progress: Tensor Creation ✓\")\n",
|
||
" \n",
|
||
"except Exception as e:\n",
|
||
" print(f\"❌ Tensor creation test failed: {e}\")\n",
|
||
" raise\n",
|
||
"\n",
|
||
"print(\"🎯 Tensor creation behavior:\")\n",
|
||
"print(\" Converts data to NumPy arrays\")\n",
|
||
"print(\" Preserves shape and data type\")\n",
|
||
"print(\" Stores in _data attribute\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "1b74cbc1",
|
||
"metadata": {
|
||
"cell_marker": "\"\"\""
|
||
},
|
||
"source": [
|
||
"### 🧪 Unit Test: Tensor Properties\n",
|
||
"\n",
|
||
"Now let's test that your tensor properties work correctly. This tests the @property methods you implemented.\n",
|
||
"\n",
|
||
"**This is a unit test** - it tests specific properties (shape, size, dtype, data) in isolation."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "c02437ea",
|
||
"metadata": {
|
||
"nbgrader": {
|
||
"grade": true,
|
||
"grade_id": "test-tensor-properties-immediate",
|
||
"locked": true,
|
||
"points": 5,
|
||
"schema_version": 3,
|
||
"solution": false,
|
||
"task": false
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Test tensor properties immediately after implementation\n",
|
||
"print(\"🔬 Unit Test: Tensor Properties...\")\n",
|
||
"\n",
|
||
"# Test properties with simple examples\n",
|
||
"try:\n",
|
||
" # Test with a simple matrix\n",
|
||
" tensor = Tensor([[1, 2, 3], [4, 5, 6]])\n",
|
||
" \n",
|
||
" # Test shape property\n",
|
||
" assert tensor.shape == (2, 3), f\"Shape should be (2, 3), got {tensor.shape}\"\n",
|
||
" print(\"✅ Shape property works\")\n",
|
||
" \n",
|
||
" # Test size property\n",
|
||
" assert tensor.size == 6, f\"Size should be 6, got {tensor.size}\"\n",
|
||
" print(\"✅ Size property works\")\n",
|
||
" \n",
|
||
" # Test data property\n",
|
||
" assert np.array_equal(tensor.data, np.array([[1, 2, 3], [4, 5, 6]])), \"Data property should return numpy array\"\n",
|
||
" print(\"✅ Data property works\")\n",
|
||
" \n",
|
||
" # Test dtype property\n",
|
||
" assert tensor.dtype in [np.int32, np.int64], f\"Dtype should be int32 or int64, got {tensor.dtype}\"\n",
|
||
" print(\"✅ Dtype property works\")\n",
|
||
" \n",
|
||
" print(\"📈 Progress: Tensor Properties ✓\")\n",
|
||
" \n",
|
||
"except Exception as e:\n",
|
||
" print(f\"❌ Tensor properties test failed: {e}\")\n",
|
||
" raise\n",
|
||
"\n",
|
||
"print(\"🎯 Tensor properties behavior:\")\n",
|
||
"print(\" shape: Returns tuple of dimensions\")\n",
|
||
"print(\" size: Returns total number of elements\")\n",
|
||
"print(\" data: Returns underlying NumPy array\")\n",
|
||
"print(\" dtype: Returns NumPy data type\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "2ceb7f26",
|
||
"metadata": {
|
||
"cell_marker": "\"\"\""
|
||
},
|
||
"source": [
|
||
"### 🧪 Unit Test: Tensor Arithmetic\n",
|
||
"\n",
|
||
"Let's test your tensor arithmetic operations. This tests the __add__, __mul__, __sub__, __truediv__ methods.\n",
|
||
"\n",
|
||
"**This is a unit test** - it tests specific arithmetic operations in isolation."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "fcf0369b",
|
||
"metadata": {
|
||
"nbgrader": {
|
||
"grade": true,
|
||
"grade_id": "test-tensor-arithmetic-immediate",
|
||
"locked": true,
|
||
"points": 5,
|
||
"schema_version": 3,
|
||
"solution": false,
|
||
"task": false
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Test tensor arithmetic immediately after implementation\n",
|
||
"print(\"🔬 Unit Test: Tensor Arithmetic...\")\n",
|
||
"\n",
|
||
"# Test basic arithmetic with simple examples\n",
|
||
"try:\n",
|
||
" # Test addition\n",
|
||
" a = Tensor([1, 2, 3])\n",
|
||
" b = Tensor([4, 5, 6])\n",
|
||
" result = a + b\n",
|
||
" expected = np.array([5, 7, 9])\n",
|
||
" assert np.array_equal(result.data, expected), f\"Addition failed: expected {expected}, got {result.data}\"\n",
|
||
" print(\"✅ Addition works\")\n",
|
||
" \n",
|
||
" # Test scalar addition\n",
|
||
" result_scalar = a + 10\n",
|
||
" expected_scalar = np.array([11, 12, 13])\n",
|
||
" assert np.array_equal(result_scalar.data, expected_scalar), f\"Scalar addition failed: expected {expected_scalar}, got {result_scalar.data}\"\n",
|
||
" print(\"✅ Scalar addition works\")\n",
|
||
" \n",
|
||
" # Test multiplication\n",
|
||
" result_mul = a * b\n",
|
||
" expected_mul = np.array([4, 10, 18])\n",
|
||
" assert np.array_equal(result_mul.data, expected_mul), f\"Multiplication failed: expected {expected_mul}, got {result_mul.data}\"\n",
|
||
" print(\"✅ Multiplication works\")\n",
|
||
" \n",
|
||
" # Test scalar multiplication\n",
|
||
" result_scalar_mul = a * 2\n",
|
||
" expected_scalar_mul = np.array([2, 4, 6])\n",
|
||
" assert np.array_equal(result_scalar_mul.data, expected_scalar_mul), f\"Scalar multiplication failed: expected {expected_scalar_mul}, got {result_scalar_mul.data}\"\n",
|
||
" print(\"✅ Scalar multiplication works\")\n",
|
||
" \n",
|
||
" print(\"📈 Progress: Tensor Arithmetic ✓\")\n",
|
||
" \n",
|
||
"except Exception as e:\n",
|
||
" print(f\"❌ Tensor arithmetic test failed: {e}\")\n",
|
||
" raise\n",
|
||
"\n",
|
||
"print(\"🎯 Tensor arithmetic behavior:\")\n",
|
||
"print(\" Element-wise operations on tensors\")\n",
|
||
"print(\" Broadcasting with scalars\")\n",
|
||
"print(\" Returns new Tensor objects\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "2d2645b0",
|
||
"metadata": {
|
||
"cell_marker": "\"\"\""
|
||
},
|
||
"source": [
|
||
"Congratulations! You've successfully implemented the core Tensor class for TinyTorch:\n",
|
||
"\n",
|
||
"### What You've Accomplished\n",
|
||
"✅ **Tensor Creation**: Handle scalars, vectors, matrices, and higher-dimensional arrays \n",
|
||
"✅ **Data Types**: Proper dtype handling with auto-detection and conversion \n",
|
||
"✅ **Properties**: Shape, size, dtype, and data access \n",
|
||
"✅ **Arithmetic**: Addition, multiplication, subtraction, division \n",
|
||
"✅ **Operators**: Natural Python syntax with `+`, `-`, `*`, `/` \n",
|
||
"✅ **Broadcasting**: Automatic shape compatibility like NumPy \n",
|
||
"\n",
|
||
"### Key Concepts You've Learned\n",
|
||
"- **Tensors** are the fundamental data structure for ML systems\n",
|
||
"- **NumPy backend** provides efficient computation with ML-friendly API\n",
|
||
"- **Operator overloading** makes tensor operations feel natural\n",
|
||
"- **Broadcasting** enables flexible operations between different shapes\n",
|
||
"- **Type safety** ensures consistent behavior across operations\n",
|
||
"\n",
|
||
"### Next Steps\n",
|
||
"1. **Export your code**: `tito package nbdev --export 01_tensor`\n",
|
||
"2. **Test your implementation**: `tito module test 01_tensor`\n",
|
||
"3. **Use your tensors**: \n",
|
||
" ```python\n",
|
||
" from tinytorch.core.tensor import Tensor\n",
|
||
" t = Tensor([1, 2, 3])\n",
|
||
" print(t + 5) # Your tensor in action!\n",
|
||
" ```\n",
|
||
"4. **Move to Module 2**: Start building activation functions!\n",
|
||
"\n",
|
||
"**Ready for the next challenge?** Let's add the mathematical functions that make neural networks powerful!"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "eeb499fa",
|
||
"metadata": {
|
||
"cell_marker": "\"\"\"",
|
||
"lines_to_next_cell": 1
|
||
},
|
||
"source": [
|
||
"### 🧪 Unit Test: Tensor Creation\n",
|
||
"\n",
|
||
"This test validates your `Tensor` class constructor, ensuring it correctly handles scalars, vectors, matrices, and higher-dimensional arrays with proper shape detection."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "c4e8cc6b",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"def test_unit_tensor_creation():\n",
|
||
" \"\"\"Comprehensive test of tensor creation with all data types and shapes.\"\"\"\n",
|
||
" print(\"🔬 Testing comprehensive tensor creation...\")\n",
|
||
" \n",
|
||
" # Test scalar creation\n",
|
||
" scalar_int = Tensor(42)\n",
|
||
" assert scalar_int.shape == ()\n",
|
||
" \n",
|
||
" # Test vector creation\n",
|
||
" vector_int = Tensor([1, 2, 3])\n",
|
||
" assert vector_int.shape == (3,)\n",
|
||
"\n",
|
||
" # Test matrix creation\n",
|
||
" matrix_2x2 = Tensor([[1, 2], [3, 4]])\n",
|
||
" assert matrix_2x2.shape == (2, 2)\n",
|
||
" print(\"✅ Tensor creation tests passed!\")\n",
|
||
"\n",
|
||
"# Run the test\n",
|
||
"test_unit_tensor_creation()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "9624c0a7",
|
||
"metadata": {
|
||
"cell_marker": "\"\"\"",
|
||
"lines_to_next_cell": 1
|
||
},
|
||
"source": [
|
||
"### 🧪 Unit Test: Tensor Properties\n",
|
||
"\n",
|
||
"This test validates your tensor property methods (shape, size, dtype, data), ensuring they correctly reflect the tensor's dimensional structure and data characteristics."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "54cef0a2",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"def test_unit_tensor_properties():\n",
|
||
" \"\"\"Comprehensive test of tensor properties (shape, size, dtype, data access).\"\"\"\n",
|
||
" print(\"🔬 Testing comprehensive tensor properties...\")\n",
|
||
"\n",
|
||
" tensor = Tensor([[1, 2, 3], [4, 5, 6]])\n",
|
||
" \n",
|
||
" # Test shape property\n",
|
||
" assert tensor.shape == (2, 3)\n",
|
||
" \n",
|
||
" # Test size property\n",
|
||
" assert tensor.size == 6\n",
|
||
" \n",
|
||
" # Test data property\n",
|
||
" assert np.array_equal(tensor.data, np.array([[1, 2, 3], [4, 5, 6]]))\n",
|
||
" \n",
|
||
" # Test dtype property\n",
|
||
" assert tensor.dtype in [np.int32, np.int64]\n",
|
||
" print(\"✅ Tensor properties tests passed!\")\n",
|
||
"\n",
|
||
"# Run the test\n",
|
||
"test_unit_tensor_properties()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "53e41625",
|
||
"metadata": {
|
||
"cell_marker": "\"\"\"",
|
||
"lines_to_next_cell": 1
|
||
},
|
||
"source": [
|
||
"### 🧪 Unit Test: Tensor Arithmetic Operations\n",
|
||
"\n",
|
||
"This test validates your tensor arithmetic implementation (addition, multiplication, subtraction, division) and operator overloading, ensuring mathematical operations work correctly with proper broadcasting."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "923f3b8c",
|
||
"metadata": {
|
||
"lines_to_next_cell": 1
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"def test_unit_tensor_arithmetic():\n",
|
||
" \"\"\"Comprehensive test of tensor arithmetic operations.\"\"\"\n",
|
||
" print(\"🔬 Testing comprehensive tensor arithmetic...\")\n",
|
||
" \n",
|
||
" a = Tensor([1, 2, 3])\n",
|
||
" b = Tensor([4, 5, 6])\n",
|
||
" \n",
|
||
" # Test addition\n",
|
||
" c = a + b\n",
|
||
" expected = np.array([5, 7, 9])\n",
|
||
" assert np.array_equal(c.data, expected)\n",
|
||
" \n",
|
||
" # Test multiplication\n",
|
||
" d = a * b\n",
|
||
" expected = np.array([4, 10, 18])\n",
|
||
" assert np.array_equal(d.data, expected)\n",
|
||
"\n",
|
||
" # Test subtraction\n",
|
||
" e = b - a\n",
|
||
" expected = np.array([3, 3, 3])\n",
|
||
" assert np.array_equal(e.data, expected)\n",
|
||
"\n",
|
||
" # Test division\n",
|
||
" f = b / a\n",
|
||
" expected = np.array([4.0, 2.5, 2.0])\n",
|
||
" assert np.allclose(f.data, expected)\n",
|
||
" print(\"✅ Tensor arithmetic tests passed!\")\n",
|
||
"\n",
|
||
"# Run the test\n",
|
||
"test_unit_tensor_arithmetic()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "0eb9f796",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"def test_module_tensor_numpy_integration():\n",
|
||
" \"\"\"\n",
|
||
" Integration test for tensor operations with NumPy arrays.\n",
|
||
" \n",
|
||
" Tests that tensors properly integrate with NumPy operations and maintain\n",
|
||
" compatibility with the scientific Python ecosystem.\n",
|
||
" \"\"\"\n",
|
||
" print(\"🔬 Running Integration Test: Tensor-NumPy Integration...\")\n",
|
||
" \n",
|
||
" # Test 1: Tensor from NumPy array\n",
|
||
" numpy_array = np.array([[1, 2, 3], [4, 5, 6]])\n",
|
||
" tensor_from_numpy = Tensor(numpy_array)\n",
|
||
" \n",
|
||
" assert tensor_from_numpy.shape == (2, 3), \"Tensor should preserve NumPy array shape\"\n",
|
||
" assert np.array_equal(tensor_from_numpy.data, numpy_array), \"Tensor should preserve NumPy array data\"\n",
|
||
" \n",
|
||
" # Test 2: Tensor arithmetic with NumPy-compatible operations\n",
|
||
" a = Tensor([1.0, 2.0, 3.0])\n",
|
||
" b = Tensor([4.0, 5.0, 6.0])\n",
|
||
" \n",
|
||
" # Test operations that would be used in neural networks\n",
|
||
" dot_product_result = np.dot(a.data, b.data) # Common in layers\n",
|
||
" assert np.isclose(dot_product_result, 32.0), \"Dot product should work with tensor data\"\n",
|
||
" \n",
|
||
" # Test 3: Broadcasting compatibility\n",
|
||
" matrix = Tensor([[1, 2], [3, 4]])\n",
|
||
" scalar = Tensor(10)\n",
|
||
" \n",
|
||
" result = matrix + scalar\n",
|
||
" expected = np.array([[11, 12], [13, 14]])\n",
|
||
" assert np.array_equal(result.data, expected), \"Broadcasting should work like NumPy\"\n",
|
||
" \n",
|
||
" # Test 4: Integration with scientific computing patterns\n",
|
||
" data = Tensor([1, 4, 9, 16, 25])\n",
|
||
" sqrt_result = Tensor(np.sqrt(data.data)) # Using NumPy functions on tensor data\n",
|
||
" expected_sqrt = np.array([1., 2., 3., 4., 5.])\n",
|
||
" assert np.allclose(sqrt_result.data, expected_sqrt), \"Should integrate with NumPy functions\"\n",
|
||
" \n",
|
||
" print(\"✅ Integration Test Passed: Tensor-NumPy integration works correctly.\")\n",
|
||
"\n",
|
||
"# Run the integration test\n",
|
||
"test_module_tensor_numpy_integration()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "c8c32c33",
|
||
"metadata": {
|
||
"cell_marker": "\"\"\""
|
||
},
|
||
"source": [
|
||
"## 🎯 MODULE SUMMARY: Tensor Foundation\n",
|
||
"\n",
|
||
"Congratulations! You've successfully implemented the fundamental data structure that powers all machine learning:\n",
|
||
"\n",
|
||
"### ✅ What You've Built\n",
|
||
"- **Tensor Class**: N-dimensional array wrapper with professional interfaces\n",
|
||
"- **Core Operations**: Creation, property access, and arithmetic operations\n",
|
||
"- **Shape Management**: Automatic shape tracking and validation\n",
|
||
"- **Data Types**: Proper NumPy integration and type handling\n",
|
||
"- **Foundation**: The building block for all subsequent TinyTorch modules\n",
|
||
"\n",
|
||
"### ✅ Key Learning Outcomes\n",
|
||
"- **Understanding**: How tensors work as the foundation of machine learning\n",
|
||
"- **Implementation**: Built tensor operations from scratch\n",
|
||
"- **Professional patterns**: Clean APIs, proper error handling, comprehensive testing\n",
|
||
"- **Real-world connection**: Understanding PyTorch/TensorFlow tensor foundations\n",
|
||
"- **Systems thinking**: Building reliable, reusable components\n",
|
||
"\n",
|
||
"### ✅ Mathematical Foundations Mastered\n",
|
||
"- **N-dimensional arrays**: Shape, size, and dimensionality concepts\n",
|
||
"- **Element-wise operations**: Addition, subtraction, multiplication, division\n",
|
||
"- **Broadcasting**: Understanding how operations work with different shapes\n",
|
||
"- **Memory management**: Efficient data storage and access patterns\n",
|
||
"\n",
|
||
"### ✅ Professional Skills Developed\n",
|
||
"- **API design**: Clean, intuitive interfaces for tensor operations\n",
|
||
"- **Error handling**: Graceful handling of invalid operations and edge cases\n",
|
||
"- **Testing methodology**: Comprehensive validation of tensor functionality\n",
|
||
"- **Documentation**: Clear, educational documentation with examples\n",
|
||
"\n",
|
||
"### ✅ Ready for Advanced Applications\n",
|
||
"Your tensor implementation now enables:\n",
|
||
"- **Neural Networks**: Foundation for all layer implementations\n",
|
||
"- **Automatic Differentiation**: Gradient computation through computational graphs\n",
|
||
"- **Complex Models**: CNNs, RNNs, Transformers - all built on tensors\n",
|
||
"- **Real Applications**: Training models on real datasets\n",
|
||
"\n",
|
||
"### 🔗 Connection to Real ML Systems\n",
|
||
"Your implementation mirrors production systems:\n",
|
||
"- **PyTorch**: `torch.Tensor` provides identical functionality\n",
|
||
"- **TensorFlow**: `tf.Tensor` implements similar concepts\n",
|
||
"- **NumPy**: `numpy.ndarray` serves as the foundation\n",
|
||
"- **Industry Standard**: Every major ML framework uses these exact principles\n",
|
||
"\n",
|
||
"### 🎯 The Power of Tensors\n",
|
||
"You've built the fundamental data structure of modern AI:\n",
|
||
"- **Universality**: Tensors represent all data: images, text, audio, video\n",
|
||
"- **Efficiency**: Vectorized operations enable fast computation\n",
|
||
"- **Scalability**: Handles everything from single numbers to massive matrices\n",
|
||
"- **Flexibility**: Foundation for any mathematical operation\n",
|
||
"\n",
|
||
"### 🚀 What's Next\n",
|
||
"Your tensor implementation is the foundation for:\n",
|
||
"- **Activations**: Nonlinear functions that enable complex learning\n",
|
||
"- **Layers**: Linear transformations and neural network building blocks\n",
|
||
"- **Networks**: Composing layers into powerful architectures\n",
|
||
"- **Training**: Optimizing networks to solve real problems\n",
|
||
"\n",
|
||
"**Next Module**: Activation functions - adding the nonlinearity that makes neural networks powerful!\n",
|
||
"\n",
|
||
"You've built the foundation of modern AI. Now let's add the mathematical functions that enable machines to learn complex patterns!"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"jupytext": {
|
||
"main_language": "python"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 5
|
||
}
|