github-starred/cs249r_book
2
0
Fork 0
mirror of https://github.com/harvard-edge/cs249r_book.git synced 2026-11-09 10:39:41 -06:00
Code Issues 276 Packages Projects Releases 26 Wiki Activity

refactor(tinytorch): module 01's page becomes a trailhead

Browse Source 
First page done under the two-artifact split. The website page carries
what you need to start; the book carries why it works.

Migrated into book/01_tensors.qmd first, so nothing is lost:

- Reductions and axis semantics. Sum, Mean and Max are Part 5 of module
  01 and the chapter did not mention them at all: what axis collapses,
  why keepdims matters when a loss gradient has to broadcast back, and
  why collapsing the last axis of a row-major array walks contiguous
  memory while collapsing the first walks with a stride.
- The three seminal references (Harris 2020, Lawson 1979, Baydin 2018).
  The book cited no literature; the guide's Further Reading was the only
  place they appeared.

Then removed from guide/modules/01_tensor.qmd: Core Concepts,
Architecture, Production Context, Key Takeaways, Further Reading, and
the "Why Tensors Matter at Scale" subsection nested under PyTorch.

953 -> 632 lines, and 408 -> 15 lines of Python. What remains is the
trailhead: Overview, Learning Objectives, What You'll Build, API
Reference, Common Errors, the TinyTorch/PyTorch contrast, a pointer to
the chapter, Check Your Understanding, and Get Started.

Two things for the remaining nineteen:

- The cut cannot be driven from H2 boundaries alone. "Why Tensors Matter
  at Scale" is an H3 nested under ## PyTorch, not under ## Production
  Context where it belongs logically, so a section-level pass leaves it
  behind. Each page needs the orphan check by eye.
- The pointer uses the published URL, not a relative path. The Book PDF
  is injected into assets/downloads/ at deploy time and is not in the
  repo, so a relative link fails the internal-link check, which is how
  this one was caught.
This commit is contained in:
Vijay Janapa Reddi
2026-09-17 21:44:13 +02:00
parent 4bec7f18e9
commit 33352a7cb1
2 changed files with 38 additions and 327 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tinytorch/book/01_tensors.qmd
+32
View File
@@ -133,6 +133,32 @@ For the forward pass, the contract is simple: compatible dimensions either agree
After a transpose or a broadcast, the buffer is unchanged but the view no longer walks it in plain row order. Our transposed matrix has shape $(3, 2)$ and strides $(1, 3)$, whereas row-major strides for that new shape would be $(2, 1)$. The broadcast view repeats locations because its row stride is zero. For arrays whose dimensions all exceed one, matching the formula for the array's own shape establishes C-contiguity. A dimension of length one is an exception because we never step along it; its stride does not affect the traversal. Empty arrays are another special case. NumPy's [array flags documentation](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.flags.html) describes these cases, so `C_CONTIGUOUS` is the authoritative check. NumPy can read these layouts through their strides. Writes need more care: a broadcast view can map several coordinates to one location, and `np.broadcast_to` returns a read-only view. TinyTorch avoids sharing the returned tensor with its input by copying at construction. Its `contiguous()` method additionally requests row-major layout, which is a separate property from owning the buffer.
### Reductions remove an axis
Broadcasting adds an axis by walking it with stride zero. A reduction does the
opposite: it collapses one, combining every entry along that axis into a single
value. `Sum`, `Mean`, and `Max` are the three the module builds, and they share
one parameter that decides the output's shape.
For a matrix of shape $(R, C)$, `axis=0` collapses the rows and leaves one value
per column, shape $(C,)$. `axis=1` collapses the columns and leaves one value per
row, shape $(R,)$. `axis=None`, the default, collapses everything to a scalar.
The axis you name is the axis that disappears.
That disappearance is the source of a shape error that recurs in later chapters.
A loss reduces a batch of predictions to one number, and the gradient flowing
back must broadcast against the original shape. If the reduced axis is gone, the
broadcast has nothing to align against. `keepdims=True` retains the collapsed
axis with length one, so $(R, C)$ reduces to $(R, 1)$ rather than $(R,)$ and the
result still broadcasts against the input. @sec-losses relies on this when it
subtracts a per-row maximum from a matrix of logits.
The arithmetic cost is linear in the entries read, since every element is touched
once whichever axis is named. What changes is the access pattern: collapsing the
last axis of a row-major array walks contiguous memory, while collapsing the
first walks with a stride, reading one element per row. Same arithmetic, a
different number of cache lines.
---
## The Code
@@ -375,6 +401,12 @@ Module 01 gives later code two contracts. A `Tensor` holds float32 data with a d
---
## Further reading
- Harris et al. (2020), [Array programming with NumPy](https://doi.org/10.1038/s41586-020-2649-2), *Nature*. The design behind the strides, views, and broadcasting this chapter rests on.
- Lawson et al. (1979), [Basic Linear Algebra Subprograms for Fortran usage](https://doi.org/10.1145/355841.355847), *ACM TOMS*. The Level 1/2/3 split that explains why matrix multiplication is the operation the whole stack is tuned around.
- Baydin et al. (2018), [Automatic differentiation in machine learning: a survey](https://www.jmlr.org/papers/v18/17-468.html), *JMLR*. Background for @sec-autograd, including why reverse mode makes memory the binding constraint.
## Exercises
1. **Permutation.** Generalize the model's `transpose(shape, strides, dim0, dim1)` to `permute(shape, strides, *dims)`, which reorders all dimensions at once. For a tensor of shape $(B, C, H, W)$ with row-major strides, write out the shape and strides that `permute(0, 2, 3, 1)` produces, and confirm against the strides and `C_CONTIGUOUS` flag NumPy reports for `np.transpose(A.data, (0, 2, 3, 1))` that the result is a non-contiguous view of the same buffer when all four dimensions exceed one. Use a concrete shape such as `(2, 3, 4, 5)` for the check.
tinytorch/guide/modules/01_tensor.qmd
+6 -327
View File
@@ -438,277 +438,6 @@ Reduction operations collapse one or more dimensions by aggregating values. The
: **Reduction methods that collapse tensor dimensions.** {#tbl-01-tensor-reductions}
## Core Concepts
Five ideas drive every line of code in this module: dimensionality, broadcasting, views vs. copies, matrix multiplication, and reduction along an axis. They are not TinyTorch-specific. Internalize them here and you will recognize them in PyTorch, JAX, NumPy, and every framework you touch afterwards.
### Tensor Dimensionality
Tensors generalize the shapes you already know. A scalar is a single number — a temperature reading of 72.5 degrees. Stack scalars into a list and you get a vector — a day's worth of readings. Stack vectors and you get a matrix — a spreadsheet of readings across many days. Keep stacking and you reach 3D and 4D tensors that hold video frames, batches of images, or sequences of token embeddings.
One class handles all of these cases. The same code path that adds two scalars adds two 4D tensors, because broadcasting (next section) makes shape compatibility a runtime concern, not a compile-time one.
@tbl-01-tensor-rank-examples anchors each rank in a concrete example you have seen before.
| Rank | Name | Shape | Concrete Example |
|------|------|-------|------------------|
| 0D | Scalar | `()` | Temperature reading: `72.5` |
| 1D | Vector | `(768,)` | Audio sample: 768 measurements |
| 2D | Matrix | `(128, 768)` | Spreadsheet: 128 rows × 768 columns |
| 3D | 3D Tensor | `(32, 224, 224)` | Video frames: 32 grayscale images |
| 4D | 4D Tensor | `(32, 3, 224, 224)` | Video frames: 32 color (RGB) images |
: **Tensor ranks from scalar to 4D with concrete examples.** {#tbl-01-tensor-rank-examples}
### Broadcasting
When you add a vector to a matrix, the shapes don't match. Should this fail? In most programming contexts, yes. But many computations need to apply the same operation across rows or columns. For example, if you want to adjust all values in a spreadsheet by adding a different offset to each column, you need to add a vector to a matrix. NumPy and PyTorch implement broadcasting to handle this: automatically expanding smaller tensors to match larger ones without actually copying data.
Consider adding a bias vector `[10, 20, 30]` to every row of a matrix. Without broadcasting, you'd need to manually tile the vector into a matrix first, wasting memory. With broadcasting, the operation just works, and the framework handles alignment internally.
Here is what `__add__` looks like once broadcasting does the work for you:
```python
def __add__(self, other):
"""Add two tensors element-wise with broadcasting support."""
if isinstance(other, Tensor):
return Add.apply(self, _as_tensor(other)) # dispatches through Function.apply
else:
return Tensor(self.data + other) # Scalar broadcast
```
There is no shape logic in this method. NumPy aligns the operand shapes from right to left, expands dimensions where one side is 1, and refuses the operation otherwise. You inherit that contract for free.
::: {#fig-01_tensor-diag-2 fig-env="figure" fig-pos="htb" fig-cap="**Broadcasting reuses a vector across rows.** Adding rows [1, 2, 3] and [4, 5, 6] to vector [10, 20, 30] produces [11, 22, 33] and [14, 25, 36]. Trailing axes must match or have size one; the result has new storage." fig-alt="Adding rows [1, 2, 3] and [4, 5, 6] to vector [10, 20, 30] produces [11, 22, 33] and [14, 25, 36]. Trailing axes must match or have size one; the result has new storage."}
![](../assets/images/diagrams/01_tensor-diag-2.svg)
:::
The rules are simpler than they look. Compare shapes from right to left. At each position, dimensions are compatible if they're equal or if one of them is 1. Missing dimensions on the left are treated as 1. If any position fails this check, broadcasting fails.
@tbl-01-tensor-broadcasting-rules walks through the compatibility check on representative shape pairs.
| Shape A | Shape B | Result | Valid? |
|---------|---------|--------|--------|
| `(3, 4)` | `(4,)` | `(3, 4)` | ✓ |
| `(3, 4)` | `(3, 1)` | `(3, 4)` | ✓ |
| `(3, 4)` | `(3,)` | Error | ✗ (3 ≠ 4) |
| `(2, 3, 4)` | `(3, 4)` | `(2, 3, 4)` | ✓ |
: **Broadcasting compatibility for representative shape pairs.** {#tbl-01-tensor-broadcasting-rules}
The memory savings are not academic. Adding a `(768,)` vector to a `(32, 512, 768)` tensor would, without broadcasting, require tiling the vector 32×512 times — **48 MB** of redundant data, about **12.6 million** float32 numbers. With broadcasting, you store the original **3 KB** vector and read it through expanded indices.
### Views vs. Copies
When you reshape or transpose an array in NumPy or PyTorch, does it allocate new memory or create a view? In pure NumPy and PyTorch, reshaping and transposing create *views* that share memory with the source (an $O(1)$ metadata update).
However, in **TinyTorch**, `Tensor.__init__` enforces an explicit systems invariant: **all constructed Tensors own independent storage**. Every operation result returned by `Function.apply` passes through `np.array(..., dtype=np.float32)`, creating an isolated copy:
```python
x = Tensor([1, 2, 3, 4])
y = x.reshape(2, 2) # TinyTorch wraps the NumPy view in fresh, copied storage
y.data[0, 0] = 99 # In TinyTorch, x is NOT modified!
print(x.data[0, 0]) # Still 1.0
```
This deliberate design choice eliminates tricky mutation bugs where modifying an intermediate activation accidentally corrupts an earlier layer or invalidates tape recording in autograd.
@tbl-01-tensor-view-vs-copy contrasts the memory and time cost of native view operations against TinyTorch's constructor copy.
| Operation | NumPy / PyTorch Native | TinyTorch `Tensor` | Time Complexity |
|-----------|------------------------|--------------------|-----------------|
| `reshape()` | View (Shared buffer) | Fresh Allocation (Copied buffer) | $O(n)$ copy on wrap |
| `transpose()` | View (Strided swap) | Fresh Allocation (Copied buffer) | $O(n)$ copy on wrap |
| `+ - * /` | New allocation | New allocation | $O(n)$ |
: **Memory and time cost of native views versus TinyTorch copying operations.** {#tbl-01-tensor-view-vs-copy}
### Matrix Multiplication
Matrix multiplication is the computational workhorse of neural networks. Every linear layer, every attention head, every embedding lookup involves matmul. Understanding its mechanics and cost is essential.
The operation is simple in concept: for each output element, compute a dot product of a row from the first matrix with a column from the second. But this simplicity hides cubic complexity. Multiplying two n×n matrices requires n³ multiplications and n³ additions.
Here's how the educational implementation in your module works:
The code in @lst-01-tensor-matmul makes this concrete.
```{python}
#| label: lst-01-tensor-matmul
#| lst-cap: "**Matrix multiplication with explicit loops for the 2D case and shape validation.**"
#| eval: false
def matmul(self, other):
"""Matrix multiplication of two tensors."""
if not isinstance(other, Tensor):
raise TypeError(f"Expected Tensor for matrix multiplication, got {type(other)}")
# Shape validation: inner dimensions must match
if len(self.shape) >= 2 and len(other.shape) >= 2:
if self.shape[-1] != other.shape[-2]:
raise ValueError(
f"Cannot perform matrix multiplication: {self.shape} @ {other.shape}. "
f"Inner dimensions must match: {self.shape[-1]} ≠ {other.shape[-2]}"
)
a = self.data
b = other.data
# Handle 2D matrices with explicit loops (educational)
if len(a.shape) == 2 and len(b.shape) == 2:
M, K = a.shape
K2, N = b.shape
result_data = np.zeros((M, N), dtype=a.dtype)
# Each output element is a dot product
for i in range(M):
for j in range(N):
result_data[i, j] = np.dot(a[i, :], b[:, j])
else:
# For batched operations, use np.matmul
result_data = np.matmul(a, b)
return Tensor(result_data)
```
The explicit loops in the 2D case are intentionally slower than `np.matmul` because they reveal exactly what matrix multiplication does: each output element requires K operations, and there are M×N outputs, giving O(M×K×N) total operations. For square matrices, this is O(n³).
:::{.callout-note title="Systems Implication: Cache Tiling"}
While element-wise operations like addition are **Memory-Bound** (hardware sits idle waiting for data from RAM), Matrix Multiplication is heavily **Compute-Bound** (O(n³) math for O(n²) bytes). A naive nested `for` loop constantly evicts data from the ultra-fast L1 cache back to slow main memory. Optimized libraries like `np.matmul` (which calls BLAS) use *Cache Tiling*—breaking matrices into tiny blocks that fit perfectly into the L1 cache, keeping the ALUs fed without waiting for RAM.
:::
### Shape Manipulation
Shape manipulation operations change how data is interpreted without changing the values themselves. Understanding when data is copied versus viewed is crucial for both correctness and performance.
The `reshape` method reinterprets the same data with different dimensions:
The code in @lst-01-tensor-reshape makes this concrete.
```{python}
#| label: lst-01-tensor-reshape
#| lst-cap: "**Reshape with dimension inference via `-1` and element-count validation.**"
#| eval: false
def reshape(self, *shape):
"""Reshape tensor to new dimensions."""
if len(shape) == 1 and isinstance(shape[0], (tuple, list)):
new_shape = tuple(shape[0])
else:
new_shape = shape
# Handle -1 for automatic dimension inference
if -1 in new_shape:
known_size = 1
unknown_idx = new_shape.index(-1)
for i, dim in enumerate(new_shape):
if i != unknown_idx:
known_size *= dim
unknown_dim = self.size // known_size
new_shape = list(new_shape)
new_shape[unknown_idx] = unknown_dim
new_shape = tuple(new_shape)
# Validate total elements match
if np.prod(new_shape) != self.size:
target_size = int(np.prod(new_shape))
raise ValueError(
f"Cannot reshape {self.shape} to {new_shape}\n"
f" ❌ Element count mismatch: {self.size} elements vs {target_size} elements\n"
f" 💡 Reshape preserves data, so total elements must stay the same\n"
f" 🔧 Use -1 to infer a dimension: reshape(-1, {new_shape[-1] if len(new_shape) > 0 else 1}) lets NumPy calculate"
)
reshaped_data = np.reshape(self.data, new_shape)
return Tensor(reshaped_data, requires_grad=self.requires_grad)
```
The `-1` syntax is particularly useful: it tells NumPy to infer one dimension automatically. When flattening a batch of images, `x.reshape(batch_size, -1)` lets NumPy calculate the feature dimension.
While these mathematical abstractions are powerful, they mask the physical reality of how data is stored and accessed. Understanding the gap between mathematical shape and physical memory layout is what separates a machine learning practitioner from a systems engineer.
:::{.callout-note title="Systems Implication: Contiguous Memory & Strides"}
To the math, `reshape` or `transpose` just change dimensions. To the hardware, they manipulate *strides*. A tensor's data is stored as a flat, 1D contiguous block of bytes in RAM. In pure NumPy a `reshape` is an O(1) metadata update and the buffer is untouched. TinyTorch's `reshape` dispatches through `Function.apply`, whose `Tensor` constructor copies, so here it costs O(n) and you get an owned buffer.
However, `transpose` introduces a systems trap. By swapping dimensions, you swap strides. The memory is no longer sequentially aligned with the new shape. If you iterate over a transposed tensor, the processor jumps back and forth across RAM, destroying spatial locality and causing massive **cache misses**. This is why systems engineers call `.contiguous()` after a transpose—trading a one-time O(n) memory copy to restore sequential alignment and keep the cache fed.
:::
### Computational Complexity
As we saw with the explicit loops in our matrix multiplication implementation, not all tensor operations are equal. Element-wise operations like addition visit each element exactly once, resulting in O(n) time complexity where n is the total number of elements. Reductions like `sum` also visit each element once. But matrix multiplication is fundamentally different.
Multiplying two $n{\times}n$ matrices requires n³ operations: for each of the n² output elements, you compute a dot product of n values. This cubic scaling is why a $2000{\times}2000$ matmul takes 8$\times$ longer than a $1000{\times}1000$ matmul, not 4$\times$. In neural networks, matrix multiplications consume over 90% of compute time. This computational density is precisely why GPUs exist for ML: a modern GPU has thousands of cores that can compute thousands of dot products simultaneously, turning an 800ms CPU operation into an 8ms GPU operation.
@tbl-01-tensor-operation-complexity summarises the time and memory cost of the core operations.
| Operation | Complexity | Notes |
|-----------|------------|-------|
| Element-wise (`+`, `-`, `*`) | O(n) | Linear in tensor size |
| Reductions (`sum`, `mean`) | O(n) | Must visit every element |
| Matrix multiply (`matmul`) | O(n³) | Dominates training time |
: **Computational complexity of core tensor operations.** {#tbl-01-tensor-operation-complexity}
### Axis Semantics
The `axis` parameter in reductions specifies which dimension to collapse. Think of it as "sum along this axis" or "average out this dimension." The result has one fewer dimension than the input.
For a 2D tensor with shape `(rows, columns)`, summing along axis 0 collapses the rows, giving you column totals. Summing along axis 1 collapses the columns, giving you row totals. Summing with `axis=None` collapses everything to a single scalar.
Your reduction implementations simply pass the axis to NumPy:
```python
def sum(self, axis=None, keepdims=False):
"""Sum tensor along specified axis."""
result = np.sum(self.data, axis=axis, keepdims=keepdims)
return Tensor(result)
```
The `keepdims=True` option preserves the reduced dimension as size 1, which is useful for broadcasting the result back.
```
For shape (rows, columns) = (32, 768):
sum(axis=0) → collapse rows → shape (768,) - column totals
sum(axis=1) → collapse columns → shape (32,) - row totals
sum(axis=None) → collapse all → scalar
Visual:
[[1, 2, 3], sum(axis=0) sum(axis=1)
[4, 5, 6]] → [5, 7, 9] or [6, 15]
(down cols) (across rows)
```
## Architecture
Your Tensor sits at the top of a stack that reaches down to hardware. When you call `x + y`, Python calls your `__add__` method, which delegates to NumPy, which calls optimized BLAS libraries written in C and Fortran, which use CPU SIMD instructions that process multiple numbers in a single clock cycle.
::: {#fig-01_tensor-diag-3 fig-env="figure" fig-pos="htb" fig-cap="**The TinyTorch execution stack.** Python Tensor expressions dispatch through Function.apply to NumPy operations on CPU. New Tensor results copy their data; the linked BLAS library may accelerate matrix multiplication." fig-alt="Python Tensor expressions dispatch through Function.apply to NumPy operations on CPU. New Tensor results copy their data; the linked BLAS library may accelerate matrix multiplication."}
![](../assets/images/diagrams/01_tensor-diag-3.svg)
:::
This is the same architecture used by PyTorch and TensorFlow, just with different backends. PyTorch replaces NumPy with a C++ engine and BLAS with CUDA kernels running on GPUs. But the Python interface and the abstractions are identical. When you understand TinyTorch's Tensor, you understand PyTorch's Tensor.
### Module Integration
Your `Tensor` is the only data structure the rest of TinyTorch carries around. Every subsequent module either operates on tensors, returns tensors, or stores tensors as state.
```
Module 01: Tensor ← you are here
├─→ Module 02: Activations (element-wise functions on tensors)
├─→ Module 03: Layers (parameters stored as tensors)
├─→ Module 06: Autograd (gradients tracked alongside tensors)
└─→ Module 08: Training (every step reads and writes tensors)
```
Bugs you ship from this module surface as confusing failures three modules later. The investment in clean shape handling, consistent dtypes, and correct broadcasting pays off through the entire stack.
## Common Errors & Debugging
Five errors account for almost every confused half-hour you will spend in this module. Read them now so you recognize them when they appear, instead of debugging them blind.
@@ -757,28 +486,6 @@ Your arithmetic methods need to handle both Tensor and scalar operands. When som
**Fix**: Check for scalars: `if isinstance(other, (int, float)): ...`
## Production Context
### Your Implementation vs. PyTorch
Your `Tensor` and PyTorch's `torch.Tensor` share the same conceptual design. The differences are in the engine room: PyTorch swaps NumPy for a C++ runtime, runs on CUDA, Metal, or ROCm, and exposes thousands of specialized kernels. The Python-facing API, the broadcasting rules, and the shape semantics are the same.
@tbl-01-tensor-vs-pytorch places your implementation side by side with the production reference for direct comparison.
| Feature | Your Implementation | PyTorch |
|---------|---------------------|---------|
| **Backend** | NumPy (Python) | C++/CUDA |
| **Speed** | 1x (baseline) | 10-100x faster |
| **GPU** | ✗ CPU only | ✓ CUDA, Metal, ROCm |
| **Operations** | ~15 core ops | 2000+ operations |
: **Feature comparison between TinyTorch Tensor and PyTorch Tensor.** {#tbl-01-tensor-vs-pytorch}
### Code Comparison
The following comparison shows equivalent operations in TinyTorch and PyTorch. Notice how closely the APIs mirror each other. This is intentional: by learning TinyTorch's patterns, you're simultaneously learning PyTorch's patterns.
::: {.panel-tabset}
## Your TinyTorch
```python
from tinytorch.core.tensor import Tensor
@@ -813,15 +520,14 @@ Let's walk through each line to understand the comparison:
Broadcasting rules, shape semantics, and API design patterns. When you debug PyTorch shape errors, you'll understand exactly what's happening because you built the same abstractions.
:::
### Why Tensors Matter at Scale
## Where this is explained
The class you are building looks small. The systems built on it do not. Three numbers set the scale:
This page is the trailhead: what you are building, the API you will fill in, and
how to start. The reasoning behind it, why a transpose is a stride swap, what
broadcasting costs, and what a reduction does to a shape, is **Chapter 1,
Tensors and Strided Memory**, in the companion textbook.
- **Large language models** — A 175B-parameter model stored in fp16 is **326 GB** of tensor data. That is one model, one copy, before optimizer state.
- **Image batches** — A batch of 128 RGB images at 224×224 in float32 is **73.5 MB**. A training run streams thousands of these per minute.
- **Self-driving perception** — Multi-camera pipelines run tensor ops at **36 FPS**, so each frame's millions of operations have to finish in 28 ms.
A single matrix multiply can absorb 90% of training time. Every micro-optimization in a real framework — contiguous strides, fused kernels, view-vs-copy decisions — exists because the `Tensor` you are about to write multiplies its mistakes by billions.
Read it here: [TinyTorch: From Tensors to Transformers](https://mlsysbook.ai/tinytorch/assets/downloads/TinyTorch-Book.pdf) (PDF).
## Check Your Understanding
@@ -906,33 +612,6 @@ This deliberate trade-off trades a small memory allocation overhead to guarantee
- **Trade-off** — Reshape incurs an $O(n)$ copy on wrap rather than an $O(1)$ stride metadata update.
:::
## Key Takeaways
- **Tensor as universal carrier:** one class holds every value the rest of the framework touches — scalars, vectors, 4-D image batches — because broadcasting makes shape compatibility a runtime concern, not a compile-time one.
- **Broadcasting replaces tiling:** right-to-left alignment lets a `(768,)` bias merge with a `(32, 512, 768)` activation tensor for the cost of the vector, not the expanded 48 MB copy.
- **Views vs. copies is a systems decision:** pure NumPy views share memory while TinyTorch constructor copies guarantee isolated storage; mixing them up causes either memory blowup or silent mutation.
- **Matmul is the cubic tax:** O(n^3) compute on O(n^2) bytes makes matmul compute-bound and cache-sensitive, which is why every real framework routes it through tiled BLAS kernels.
- **Contiguity is physical, shape is logical:** `transpose` rearranges strides without moving bytes, so iteration order breaks cache locality — hence the `.contiguous()` call after transpose in production code.
**Coming next:** Module 02 adds nonlinearity — ReLU, Sigmoid, Tanh, GELU, Softmax — as element-wise functions that consume and return the `Tensor` you just built.
## Further Reading
The abstractions we rely on today—broadcasting, memory views, and hardware-accelerated matrix multiplication—were not developed overnight. They are the culmination of decades of research bridging mathematics and computer architecture. For students who want to trace the evolution of these ideas from early linear algebra libraries to modern tensor frameworks, the following literature is essential.
### Seminal Papers
- **NumPy: Array Programming** - Harris et al. (2020). The definitive reference for NumPy, which underlies your Tensor implementation. Explains broadcasting, views, and the design philosophy. **Systems Implication:** Standardized memory layouts (strides) and contiguous blocks allowed C-level operations to bypass the slow Python interpreter, maximizing memory bandwidth. [Nature](https://doi.org/10.1038/s41586-020-2649-2)
- **BLAS (Basic Linear Algebra Subprograms)** - Lawson et al. (1979). The foundation of all high-performance matrix operations. Your `np.matmul` ultimately calls BLAS routines optimized over 40+ years. Understanding BLAS levels (1, 2, 3) explains why matmul is special. **Systems Implication:** Defined the memory hierarchy abstractions (registers, L1/L2 cache, RAM) allowing matrix multiplication to be tiled for optimal cache reuse, fundamentally defining modern dense compute limits. [ACM TOMS](https://doi.org/10.1145/355841.355847)
- **Automatic Differentiation in ML** - Baydin et al. (2018). Survey of automatic differentiation techniques. **Systems Implication:** Reverse-mode AD required materializing the intermediate forward pass tensors in memory, making memory capacity a primary bottleneck for deep learning before checkpointing techniques were developed. [JMLR](https://www.jmlr.org/papers/v18/17-468.html)
### Additional Resources
- **Textbook**: ["Deep Learning"](https://www.deeplearningbook.org/) by Goodfellow, Bengio, and Courville - Chapter 2 covers linear algebra foundations including tensor operations
- **Documentation**: [PyTorch Tensor Tutorial](https://pytorch.org/tutorials/beginner/basics/tensorqs_tutorial.html) - See how production frameworks implement similar concepts
## What's Next?
You now have the universal carrier of every value the rest of the framework will ever compute. A `Tensor` holds the data, knows its shape, and supports the arithmetic that linear algebra and eventually backpropagation depend on.