TinyTorch/modules/01_tensor/ABOUT.md

title, description, module_number, tier, difficulty, time_estimate, prerequisites, next_module, learning_objectives

title	description	module_number	tier	difficulty	time_estimate	prerequisites	next_module	learning_objectives
Tensor	Core tensor data structure and operations	1	foundation	beginner	4-6 hours	Environment Setup	02. Activations	Understand tensors as N-dimensional arrays and their role in ML systems Implement a complete Tensor class with arithmetic and shape operations Handle memory management, data types, and broadcasting efficiently Recognize how tensor operations form the foundation of PyTorch/TensorFlow Analyze computational complexity and memory usage of tensor operations

01. Tensor

🏗️ FOUNDATION TIER | Difficulty: ⭐ (1/4) | Time: 4-6 hours

Build N-dimensional arrays from scratch - the foundation of all ML computations.

---

What You'll Build

The Tensor class is the fundamental data structure of machine learning. It represents N-dimensional arrays and provides operations for manipulation, computation, and transformation.

By the end of this module, you'll have a working Tensor implementation that handles:

• Creating and initializing N-dimensional arrays
• Arithmetic operations (addition, multiplication, division, powers)
• Shape manipulation (reshape, transpose, broadcasting)
• Reductions (sum, mean, min, max along any axis)
• Memory-efficient data storage and copying

Example Usage

from tinytorch.core.tensor import Tensor

# Create tensors
x = Tensor([[1.0, 2.0], [3.0, 4.0]])
y = Tensor([[0.5, 1.5], [2.5, 3.5]])

# Properties
print(x.shape)    # (2, 2)
print(x.size)     # 4
print(x.dtype)    # float64

# Operations
z = x + y         # Addition
w = x * y         # Element-wise multiplication
p = x ** 2        # Exponentiation

# Shape manipulation
reshaped = x.reshape(4, 1)
transposed = x.T

# Reductions
total = x.sum()             # Scalar sum
means = x.mean(axis=0)      # Mean along axis

---

Learning Pattern: Build → Use → Understand

1. Build

Implement the Tensor class from scratch using NumPy as the underlying array library. You'll create constructors, operator overloading, shape manipulation methods, and reduction operations.

2. Use

Apply your Tensor implementation to real problems: matrix multiplication, data normalization, statistical computations. Test with various shapes and data types.

3. Understand

Grasp the systems-level implications: why tensor operations dominate compute time, how memory layout affects performance, and how broadcasting enables efficient computations without data copying.

---

Learning Objectives

By completing this module, you will:

1. Systems Understanding: Recognize tensors as the universal data structure in ML frameworks, understanding how all neural network operations decompose into tensor primitives

2. Core Implementation: Build a complete Tensor class supporting arithmetic, shape manipulation, and reductions with proper error handling

3. Pattern Recognition: Understand broadcasting rules and how they enable efficient computations across different tensor shapes

4. Framework Connection: See how your implementation mirrors PyTorch's torch.Tensor and TensorFlow's tf.Tensor design

5. Performance Trade-offs: Analyze memory usage vs computation speed, understanding when to copy data vs create views

---

Why This Matters

Production Context

Every modern ML framework is built on tensors:

• PyTorch: torch.Tensor is the core class - all operations work with tensors
• TensorFlow: tf.Tensor represents data flowing through computation graphs
• JAX: jax.numpy.ndarray extends NumPy with automatic differentiation
• NumPy: The foundation - understanding tensors starts here

By building your own Tensor class, you'll understand what happens when you call torch.matmul() or tf.reduce_sum() - not just the API, but the actual computation.

Systems Reality Check

Performance Note: Tensor operations dominate training time. A single matrix multiplication in a linear layer might take 90% of forward pass time. Understanding tensor internals is essential for optimization.

Memory Note: Large models store billions of parameters as tensors. A GPT-3 scale model requires 350GB of memory just for weights (175B parameters × 2 bytes for FP16). Efficient tensor memory management is critical.

---

Implementation Guide

Prerequisites Check

Verify your environment is ready:

tito system doctor

All checks should pass before starting implementation.

Development Workflow

# Navigate to tensor module
cd modules/01_tensor/

# Open development file (choose your preferred method)
jupyter lab tensor_dev.py          # Jupytext (recommended)
# OR
code tensor_dev.py                 # Direct Python editing

Step-by-Step Build

Step 1: Tensor Class Foundation

Create the basic Tensor class with initialization and properties:

class Tensor:
    def __init__(self, data, dtype=None):
        """Initialize tensor from Python list or NumPy array"""
        self.data = np.array(data, dtype=dtype)
    
    @property
    def shape(self):
        """Return tensor shape"""
        return self.data.shape
    
    @property
    def size(self):
        """Return total number of elements"""
        return self.data.size

Why this matters: Properties enable clean API design - users can write x.shape instead of x.get_shape(), matching PyTorch conventions.

Step 2: Arithmetic Operations

Implement operator overloading for element-wise operations:

def __add__(self, other):
    """Element-wise addition"""
    return Tensor(self.data + other.data)

def __mul__(self, other):
    """Element-wise multiplication"""
    return Tensor(self.data * other.data)

Systems insight: These operations vectorize automatically via NumPy, achieving ~100x speedup over Python loops. This is why frameworks use tensors.

Step 3: Shape Manipulation

Implement reshape, transpose, and broadcasting:

def reshape(self, *shape):
    """Return tensor with new shape"""
    return Tensor(self.data.reshape(*shape))

@property
def T(self):
    """Return transposed tensor"""
    return Tensor(self.data.T)

Memory consideration: Reshape and transpose often return views (no data copying) for efficiency. Understanding views vs copies is crucial for memory optimization.

Step 4: Reductions

Implement aggregation operations along axes:

def sum(self, axis=None):
    """Sum tensor elements along axis"""
    return Tensor(self.data.sum(axis=axis))

def mean(self, axis=None):
    """Mean of tensor elements along axis"""
    return Tensor(self.data.mean(axis=axis))

Production pattern: Reductions are fundamental - every loss function uses them. Understanding axis semantics prevents bugs in multi-dimensional operations.

---

Testing Your Implementation

Inline Tests

Test within your development file:

# Create test tensors
x = Tensor([[1, 2], [3, 4]])
y = Tensor([[5, 6], [7, 8]])

# Test operations
assert x.shape == (2, 2)
assert (x + y).data.tolist() == [[6, 8], [10, 12]]
assert x.sum().data == 10
print("✓ Basic operations working")

Module Export & Validation

# Export your implementation to TinyTorch package
tito export 01

# Run comprehensive test suite
tito test 01

Expected output:

✓ All tests passed! [25/25]
✓ Module 01 complete!

---

Where This Code Lives

After export, your Tensor implementation becomes part of the TinyTorch package:

# Other modules and future code can now import YOUR implementation:
from tinytorch.core.tensor import Tensor

# Used throughout TinyTorch:
from tinytorch.core.layers import Linear      # Uses Tensor for weights
from tinytorch.core.activations import ReLU   # Operates on Tensors
from tinytorch.core.autograd import backward  # Computes Tensor gradients

Package structure:

tinytorch/
├── core/
│   ├── tensor.py  ← YOUR implementation exports here
│   ├── activations.py
│   ├── layers.py
│   └── ...

---

Systems Thinking Questions

Reflect on these questions as you build (no right/wrong answers):

1. Complexity Analysis: Why is matrix multiplication O(n³) for n×n matrices? How does this affect training time for large models?

2. Memory Trade-offs: When should reshape create a view vs copy data? What are the performance implications?

3. Production Scaling: A GPT-3 scale model has 175 billion parameters. How much memory is required to store these as FP32 tensors? As FP16?

4. Design Decisions: Why do frameworks like PyTorch store data as NumPy arrays internally? What are alternatives?

5. Framework Comparison: How does your Tensor class differ from torch.Tensor? What features are missing? Why might those features matter?

---

Real-World Connections

Industry Applications

• Deep Learning Training: All neural network layers operate on tensors (Linear, Conv2d, Attention all perform tensor operations)
• Scientific Computing: Tensors represent multidimensional data (climate models, molecular simulations)
• Computer Vision: Images are 3D tensors (height × width × channels)
• NLP: Text embeddings are 2D tensors (sequence_length × embedding_dim)

Research Applications

• Automatic Differentiation: Frameworks like PyTorch track tensor operations to compute gradients
• Distributed Training: Large models split tensors across GPUs using tensor parallelism
• Quantization: Tensors can be stored in reduced precision (INT8 instead of FP32) for efficiency

---

What's Next?

Congratulations! You've built the foundation of TinyTorch. Your Tensor class will power everything that follows - from activation functions to complete neural networks.

Next, you'll add nonlinearity to enable networks to learn complex patterns.

Module 02: Activations - Implement ReLU, Sigmoid, Tanh, and other activation functions that transform tensor values

Continue to Module 02: Activations →

---

Need Help?

• Ask in GitHub Discussions
• View Tensor API Reference
• Report Issues