mirror of
https://github.com/MLSysBook/TinyTorch.git
synced 2026-06-02 19:44:44 -05:00
Standardize Module 08 (DataLoader) to professional template
- Add complete YAML frontmatter with metadata - Add INTELLIGENCE tier badge - Standardize to exactly 5 learning objectives - Implement Build → Use → Analyze pedagogical pattern - Add Why This Matters section with production + historical context - Add Implementation Guide with step-by-step instructions - Add Systems Thinking Questions for deeper reflection - Add Real-World Connections to industry applications - Reduce emoji usage significantly (professional tone) - Add clear What's Next navigation to Module 09
This commit is contained in:
Vijay Janapa Reddi
@@ -1,306 +1,332 @@
|
||||
---
|
||||
title: "DataLoader"
|
||||
description: "Dataset interfaces and data loading pipelines"
|
||||
difficulty: "⭐⭐⭐"
|
||||
title: "DataLoader - Data Pipeline Engineering"
|
||||
description: "Build production-grade data loading infrastructure for training at scale"
|
||||
difficulty: 3
|
||||
time_estimate: "5-6 hours"
|
||||
prerequisites: []
|
||||
next_steps: []
|
||||
learning_objectives: []
|
||||
prerequisites: ["Tensor", "Layers", "Training"]
|
||||
next_steps: ["Spatial (CNNs)"]
|
||||
learning_objectives:
|
||||
- "Design scalable data pipeline architectures for production ML systems"
|
||||
- "Implement efficient dataset abstractions with batching and streaming"
|
||||
- "Build preprocessing pipelines for normalization and data augmentation"
|
||||
- "Understand memory-efficient data loading patterns for large datasets"
|
||||
- "Apply systems thinking to I/O optimization and throughput engineering"
|
||||
---
|
||||
|
||||
# Module: DataLoader
|
||||
# 08. DataLoader
|
||||
|
||||
```{div} badges
|
||||
⭐⭐⭐ | ⏱️ 5-6 hours
|
||||
```
|
||||
**🧠 INTELLIGENCE TIER** | Difficulty: ⭐⭐⭐ (3/4) | Time: 5-6 hours
|
||||
|
||||
## Overview
|
||||
|
||||
## 📊 Module Info
|
||||
- **Difficulty**: ⭐⭐⭐ Advanced
|
||||
- **Time Estimate**: 5-7 hours
|
||||
- **Prerequisites**: Tensor, Layers modules
|
||||
- **Next Steps**: Training, Networks modules
|
||||
Build the data engineering infrastructure that feeds neural networks. This module implements production-grade data loading, preprocessing, and batching systems—the critical backbone that enables training on real-world datasets like CIFAR-10.
|
||||
|
||||
Build the data pipeline foundation of TinyTorch! This module implements efficient data loading, preprocessing, and batching systems—the critical infrastructure that feeds neural networks during training and powers real-world ML systems.
|
||||
## Learning Objectives
|
||||
|
||||
## 🎯 Learning Objectives
|
||||
By completing this module, you will be able to:
|
||||
|
||||
By the end of this module, you will be able to:
|
||||
1. **Design scalable data pipeline architectures** for production ML systems with proper abstractions and interfaces
|
||||
2. **Implement efficient dataset abstractions** with batching, shuffling, and streaming for memory-efficient training
|
||||
3. **Build preprocessing pipelines** for normalization, augmentation, and transformation with fit-transform patterns
|
||||
4. **Understand memory-efficient data loading patterns** for large datasets that don't fit in RAM
|
||||
5. **Apply systems thinking** to I/O optimization, caching strategies, and throughput engineering
|
||||
|
||||
- **Design data pipeline architectures**: Understand data engineering as the foundation of scalable ML systems
|
||||
- **Implement reusable dataset abstractions**: Build flexible interfaces that support multiple data sources and formats
|
||||
- **Create efficient data loaders**: Develop batching, shuffling, and streaming systems for optimal training performance
|
||||
- **Build preprocessing pipelines**: Implement normalization, augmentation, and transformation systems
|
||||
- **Apply systems engineering principles**: Handle memory management, I/O optimization, and error recovery in data pipelines
|
||||
## Why This Matters
|
||||
|
||||
## 🧠 Build → Use → Optimize
|
||||
### Production Context
|
||||
|
||||
This module follows TinyTorch's **Build → Use → Optimize** framework:
|
||||
Every production ML system depends on robust data infrastructure:
|
||||
|
||||
1. **Build**: Implement dataset abstractions, data loaders, and preprocessing pipelines from engineering principles
|
||||
2. **Use**: Apply your data system to real CIFAR-10 dataset with complete train/test workflows
|
||||
3. **Optimize**: Analyze performance characteristics, memory usage, and system bottlenecks for production readiness
|
||||
- **Netflix** uses sophisticated data pipelines to train recommendation models on billions of viewing records
|
||||
- **Tesla** processes terabytes of driving sensor data through efficient loading pipelines for autonomous driving
|
||||
- **OpenAI** built custom data loaders to train GPT models on hundreds of billions of tokens
|
||||
- **Meta** developed PyTorch's DataLoader (which you're reimplementing) to power research and production
|
||||
|
||||
## 📚 What You'll Build
|
||||
### Historical Context
|
||||
|
||||
### Complete Data Pipeline System
|
||||
Data loading evolved from bottleneck to optimized system:
|
||||
|
||||
- **Early ML (pre-2010)**: Small datasets fit entirely in memory; data loading was an afterthought
|
||||
- **ImageNet Era (2012)**: AlexNet required efficient loading of 1.2M images; preprocessing became critical
|
||||
- **Big Data ML (2015+)**: Streaming data pipelines became necessary for datasets too large for memory
|
||||
- **Modern Scale (2020+)**: Data loading is now a first-class systems problem with dedicated infrastructure teams
|
||||
|
||||
The patterns you're building are the same ones used in production at scale.
|
||||
|
||||
## Pedagogical Pattern: Build → Use → Analyze
|
||||
|
||||
### 1. Build
|
||||
|
||||
Implement from first principles:
|
||||
- Dataset abstraction with Python protocols (`__getitem__`, `__len__`)
|
||||
- DataLoader with batching, shuffling, and iteration
|
||||
- CIFAR-10 dataset loader with binary file parsing
|
||||
- Normalizer with fit-transform pattern
|
||||
- Memory-efficient streaming for large datasets
|
||||
|
||||
### 2. Use
|
||||
|
||||
Apply to real problems:
|
||||
- Load and preprocess CIFAR-10 (50,000 training images)
|
||||
- Create train/test data loaders with proper batching
|
||||
- Build preprocessing pipelines for normalization
|
||||
- Integrate with training loops from Module 07
|
||||
- Measure throughput and identify bottlenecks
|
||||
|
||||
### 3. Analyze
|
||||
|
||||
Deep-dive into systems behavior:
|
||||
- Profile memory usage patterns with different batch sizes
|
||||
- Measure I/O throughput and identify disk bottlenecks
|
||||
- Compare streaming vs in-memory loading strategies
|
||||
- Analyze the impact of shuffling on training dynamics
|
||||
- Understand trade-offs between batch size and memory
|
||||
|
||||
## Implementation Guide
|
||||
|
||||
### Core Components
|
||||
|
||||
**Dataset Abstraction**
|
||||
```python
|
||||
# End-to-end data pipeline creation
|
||||
train_loader, test_loader, normalizer = create_data_pipeline(
|
||||
dataset_path="data/cifar10/",
|
||||
batch_size=32,
|
||||
normalize=True,
|
||||
shuffle=True
|
||||
)
|
||||
|
||||
# Ready for neural network training
|
||||
for batch_images, batch_labels in train_loader:
|
||||
# batch_images.shape: (32, 3, 32, 32) - normalized pixel values
|
||||
# batch_labels.shape: (32,) - class indices
|
||||
predictions = model(batch_images)
|
||||
loss = compute_loss(predictions, batch_labels)
|
||||
# Continue training loop...
|
||||
```
|
||||
|
||||
### Dataset Abstraction System
|
||||
```python
|
||||
# Flexible interface supporting multiple datasets
|
||||
class Dataset:
|
||||
"""Abstract base class for all datasets.
|
||||
|
||||
Implements Python protocols for indexing and length.
|
||||
Subclasses must implement __getitem__ and __len__.
|
||||
"""
|
||||
def __getitem__(self, index: int):
|
||||
"""Return (data, label) for given index."""
|
||||
raise NotImplementedError
|
||||
|
||||
def __len__(self) -> int:
|
||||
"""Return total number of samples."""
|
||||
raise NotImplementedError
|
||||
```
|
||||
|
||||
**DataLoader Implementation**
|
||||
```python
|
||||
class DataLoader:
|
||||
"""Efficient batch loading with shuffling support.
|
||||
|
||||
Features:
|
||||
- Automatic batching with configurable batch size
|
||||
- Optional shuffling for training randomization
|
||||
- Drop last batch handling for even batch sizes
|
||||
- Memory-efficient iteration without loading all data
|
||||
"""
|
||||
def __init__(self, dataset, batch_size=32, shuffle=False, drop_last=False):
|
||||
self.dataset = dataset
|
||||
self.batch_size = batch_size
|
||||
self.shuffle = shuffle
|
||||
self.drop_last = drop_last
|
||||
|
||||
def __iter__(self):
|
||||
# Generate indices (shuffled or sequential)
|
||||
indices = list(range(len(self.dataset)))
|
||||
if self.shuffle:
|
||||
np.random.shuffle(indices)
|
||||
|
||||
# Yield batches
|
||||
for i in range(0, len(indices), self.batch_size):
|
||||
batch_indices = indices[i:i + self.batch_size]
|
||||
if len(batch_indices) < self.batch_size and self.drop_last:
|
||||
continue
|
||||
yield self._get_batch(batch_indices)
|
||||
```
|
||||
|
||||
**CIFAR-10 Dataset Loader**
|
||||
```python
|
||||
class CIFAR10Dataset(Dataset):
|
||||
"""Load CIFAR-10 dataset with automatic download.
|
||||
|
||||
CIFAR-10: 60,000 32x32 color images in 10 classes
|
||||
- 50,000 training images
|
||||
- 10,000 test images
|
||||
- Classes: airplane, car, bird, cat, deer, dog, frog, horse, ship, truck
|
||||
"""
|
||||
def __init__(self, root='./data', train=True, download=True):
|
||||
self.train = train
|
||||
if download:
|
||||
self._download(root)
|
||||
self.data, self.labels = self._load_batch_files(root, train)
|
||||
|
||||
def __getitem__(self, index):
|
||||
# Return (data, label) for any dataset type
|
||||
pass
|
||||
return self.data[index], self.labels[index]
|
||||
|
||||
def __len__(self):
|
||||
# Enable len() and iteration
|
||||
pass
|
||||
|
||||
# Concrete implementation with real data
|
||||
dataset = CIFAR10Dataset("data/cifar10/", train=True, download=True)
|
||||
print(f"Loaded {len(dataset)} real samples") # 50,000 training images
|
||||
image, label = dataset[0] # Access individual samples
|
||||
print(f"Sample shape: {image.shape}, Label: {label}")
|
||||
return len(self.data)
|
||||
```
|
||||
|
||||
### Efficient Data Loading System
|
||||
**Preprocessing Pipeline**
|
||||
```python
|
||||
# High-performance batching with memory optimization
|
||||
dataloader = DataLoader(
|
||||
dataset=dataset,
|
||||
batch_size=32, # Configurable batch size
|
||||
shuffle=True, # Training randomization
|
||||
drop_last=False # Handle incomplete batches
|
||||
)
|
||||
|
||||
# Pythonic iteration interface
|
||||
for batch_idx, (batch_data, batch_labels) in enumerate(dataloader):
|
||||
print(f"Batch {batch_idx}: {batch_data.shape}")
|
||||
# Automatic batching handles all the complexity
|
||||
class Normalizer:
|
||||
"""Normalize data using fit-transform pattern.
|
||||
|
||||
Fits statistics on training data, applies to all splits.
|
||||
Ensures consistent preprocessing across train/val/test.
|
||||
"""
|
||||
def fit(self, data):
|
||||
"""Compute mean and std from training data."""
|
||||
self.mean = data.mean(axis=0)
|
||||
self.std = data.std(axis=0)
|
||||
return self
|
||||
|
||||
def transform(self, data):
|
||||
"""Apply normalization using fitted statistics."""
|
||||
return (data - self.mean) / (self.std + 1e-8)
|
||||
|
||||
def fit_transform(self, data):
|
||||
"""Fit and transform in one step."""
|
||||
return self.fit(data).transform(data)
|
||||
```
|
||||
|
||||
### Data Preprocessing Pipeline
|
||||
```python
|
||||
# Production-ready normalization system
|
||||
normalizer = Normalizer()
|
||||
### Step-by-Step Implementation
|
||||
|
||||
# Fit on training data (compute statistics once)
|
||||
normalizer.fit(training_images)
|
||||
print(f"Mean: {normalizer.mean}, Std: {normalizer.std}")
|
||||
1. **Create Dataset Base Class**
|
||||
- Implement `__getitem__` and `__len__` protocols
|
||||
- Define the interface all datasets must follow
|
||||
- Test with simple array-based dataset
|
||||
|
||||
# Apply to any dataset (training, validation, test)
|
||||
normalized_images = normalizer.transform(test_images)
|
||||
# Ensures consistent preprocessing across data splits
|
||||
```
|
||||
2. **Build CIFAR-10 Loader**
|
||||
- Implement download and extraction logic
|
||||
- Parse binary batch files (pickle format)
|
||||
- Reshape data from flat arrays to (3, 32, 32) images
|
||||
- Handle train/test split loading
|
||||
|
||||
## 🎯 NEW: CIFAR-10 Support for North Star Goal
|
||||
3. **Implement DataLoader**
|
||||
- Create batching logic with configurable batch size
|
||||
- Add shuffling with random permutation
|
||||
- Implement iterator protocol for Pythonic loops
|
||||
- Handle edge cases (last incomplete batch, empty dataset)
|
||||
|
||||
### Built-in CIFAR-10 Download and Loading
|
||||
This module now includes complete CIFAR-10 support to achieve our semester goal of 75% accuracy:
|
||||
4. **Add Preprocessing**
|
||||
- Build Normalizer with fit-transform pattern
|
||||
- Compute per-channel statistics for RGB images
|
||||
- Apply transformations efficiently across batches
|
||||
- Test normalization correctness (zero mean, unit variance)
|
||||
|
||||
```python
|
||||
from tinytorch.core.dataloader import CIFAR10Dataset, download_cifar10
|
||||
5. **Integration Testing**
|
||||
- Load CIFAR-10 and create data loaders
|
||||
- Iterate through batches and verify shapes
|
||||
- Test with actual training loop from Module 07
|
||||
- Measure data loading throughput
|
||||
|
||||
# Download CIFAR-10 automatically (one-time, ~170MB)
|
||||
dataset_path = download_cifar10() # Downloads to ./data/cifar-10-batches-py
|
||||
## Testing
|
||||
|
||||
# Load training and test data
|
||||
dataset = CIFAR10Dataset(download=True, flatten=False)
|
||||
print(f"✅ Loaded {len(dataset.train_data)} training samples")
|
||||
print(f"✅ Loaded {len(dataset.test_data)} test samples")
|
||||
|
||||
# Create DataLoaders for training
|
||||
from tinytorch.core.dataloader import DataLoader
|
||||
train_loader = DataLoader(dataset.train_data, dataset.train_labels, batch_size=32, shuffle=True)
|
||||
test_loader = DataLoader(dataset.test_data, dataset.test_labels, batch_size=32, shuffle=False)
|
||||
|
||||
# Ready for CNN training!
|
||||
for batch_images, batch_labels in train_loader:
|
||||
print(f"Batch shape: {batch_images.shape}") # (32, 3, 32, 32) for CNNs
|
||||
break
|
||||
```
|
||||
|
||||
### What's New in This Module
|
||||
- ✅ **`download_cifar10()`**: Automatically downloads and extracts CIFAR-10 dataset
|
||||
- ✅ **`CIFAR10Dataset`**: Complete dataset class with train/test splits
|
||||
- ✅ **Real Data Support**: Work with actual 32x32 RGB images, not toy data
|
||||
- ✅ **Production Features**: Shuffling, batching, normalization for real training
|
||||
|
||||
## 🚀 Getting Started
|
||||
|
||||
### Prerequisites
|
||||
Ensure you have the foundational tensor operations:
|
||||
### Inline Tests (During Development)
|
||||
|
||||
Run inline tests while building:
|
||||
```bash
|
||||
# Activate TinyTorch environment
|
||||
source bin/activate-tinytorch.sh
|
||||
|
||||
# Verify prerequisite modules
|
||||
tito test --module tensor
|
||||
tito test --module layers
|
||||
cd modules/source/08_dataloader
|
||||
python dataloader_dev.py
|
||||
```
|
||||
|
||||
### Development Workflow
|
||||
1. **Open the development file**: `modules/source/07_dataloader/dataloader_dev.py`
|
||||
2. **Implement Dataset abstraction**: Create the base interface for all data sources
|
||||
3. **Build CIFAR-10 dataset**: Implement real dataset loading with binary file parsing
|
||||
4. **Create DataLoader system**: Add batching, shuffling, and iteration functionality
|
||||
5. **Add preprocessing tools**: Implement normalizer and transformation pipeline
|
||||
6. **Export and verify**: `tito export --module dataloader && tito test --module dataloader`
|
||||
|
||||
## 🧪 Testing Your Implementation
|
||||
|
||||
### Comprehensive Test Suite
|
||||
Run the full test suite to verify data engineering functionality:
|
||||
|
||||
```bash
|
||||
# TinyTorch CLI (recommended)
|
||||
tito test --module dataloader
|
||||
|
||||
# Direct pytest execution
|
||||
python -m pytest tests/ -k dataloader -v
|
||||
Expected output:
|
||||
```
|
||||
Unit Test: Dataset abstraction...
|
||||
✅ __getitem__ protocol works correctly
|
||||
✅ __len__ returns correct size
|
||||
✅ Indexing returns (data, label) tuples
|
||||
Progress: Dataset Interface ✓
|
||||
|
||||
### Test Coverage Areas
|
||||
- ✅ **Dataset Interface**: Verify abstract base class and concrete implementations
|
||||
- ✅ **Real Data Loading**: Test with actual CIFAR-10 dataset (downloads ~170MB)
|
||||
- ✅ **Batching System**: Ensure correct batch shapes and memory efficiency
|
||||
- ✅ **Data Preprocessing**: Verify normalization statistics and transformations
|
||||
- ✅ **Pipeline Integration**: Test complete train/test workflow with real data
|
||||
Unit Test: CIFAR-10 loading...
|
||||
✅ Downloaded and extracted 170MB dataset
|
||||
✅ Loaded 50,000 training samples
|
||||
✅ Sample shape: (3, 32, 32), label range: [0, 9]
|
||||
Progress: CIFAR-10 Dataset ✓
|
||||
|
||||
### Inline Testing & Real Data Validation
|
||||
The module includes comprehensive feedback using real CIFAR-10 data:
|
||||
```python
|
||||
# Example inline test output
|
||||
🔬 Unit Test: CIFAR-10 dataset loading...
|
||||
📥 Downloading CIFAR-10 dataset (170MB)...
|
||||
✅ Successfully loaded 50,000 training samples
|
||||
✅ Sample shapes correct: (3, 32, 32)
|
||||
✅ Labels in valid range: [0, 9]
|
||||
📈 Progress: CIFAR-10 Dataset ✓
|
||||
|
||||
# DataLoader testing with real data
|
||||
🔬 Unit Test: DataLoader batching...
|
||||
Unit Test: DataLoader batching...
|
||||
✅ Batch shapes correct: (32, 3, 32, 32)
|
||||
✅ Shuffling produces different orders
|
||||
✅ Shuffling produces different orderings
|
||||
✅ Iteration covers all samples exactly once
|
||||
📈 Progress: DataLoader ✓
|
||||
Progress: DataLoader ✓
|
||||
```
|
||||
|
||||
### Manual Testing Examples
|
||||
```python
|
||||
from tinytorch.core.tensor import Tensor
|
||||
from dataloader_dev import CIFAR10Dataset, DataLoader, Normalizer
|
||||
### Export and Validate
|
||||
|
||||
# Test dataset loading with real data
|
||||
dataset = CIFAR10Dataset("data/cifar10/", train=True, download=True)
|
||||
print(f"Dataset size: {len(dataset)}")
|
||||
print(f"Classes: {dataset.get_num_classes()}")
|
||||
After completing the module:
|
||||
```bash
|
||||
# Export to tinytorch package
|
||||
tito export 08_dataloader
|
||||
|
||||
# Test data loading pipeline
|
||||
dataloader = DataLoader(dataset, batch_size=16, shuffle=True)
|
||||
for batch_images, batch_labels in dataloader:
|
||||
print(f"Batch shape: {batch_images.shape}")
|
||||
print(f"Label range: {batch_labels.min()} to {batch_labels.max()}")
|
||||
break # Just test first batch
|
||||
|
||||
# Test preprocessing pipeline
|
||||
normalizer = Normalizer()
|
||||
sample_batch, _ = next(iter(dataloader))
|
||||
normalizer.fit(sample_batch)
|
||||
normalized = normalizer.transform(sample_batch)
|
||||
print(f"Original range: [{sample_batch.min():.2f}, {sample_batch.max():.2f}]")
|
||||
print(f"Normalized range: [{normalized.min():.2f}, {normalized.max():.2f}]")
|
||||
# Run integration tests
|
||||
tito test 08_dataloader
|
||||
```
|
||||
|
||||
## 🎯 Key Concepts
|
||||
### Comprehensive Test Coverage
|
||||
|
||||
### Real-World Applications
|
||||
- **Production ML Systems**: Companies like Netflix, Spotify use similar data pipelines for recommendation training
|
||||
- **Computer Vision**: ImageNet, COCO dataset loaders power research and production vision systems
|
||||
- **Natural Language Processing**: Text preprocessing pipelines enable language model training
|
||||
- **Autonomous Systems**: Real-time data streams from sensors require efficient pipeline architectures
|
||||
The test suite validates:
|
||||
- Dataset interface correctness
|
||||
- CIFAR-10 loading and parsing
|
||||
- Batch shape consistency
|
||||
- Shuffling randomness
|
||||
- Memory efficiency
|
||||
- Preprocessing accuracy
|
||||
|
||||
### Data Engineering Principles
|
||||
- **Interface Design**: Abstract Dataset class enables switching between data sources seamlessly
|
||||
- **Memory Efficiency**: Streaming data loading prevents memory overflow with large datasets
|
||||
- **I/O Optimization**: Batching reduces system calls and improves throughput
|
||||
- **Preprocessing Consistency**: Fit-transform pattern ensures identical preprocessing across data splits
|
||||
|
||||
### Systems Performance Considerations
|
||||
- **Batch Size Trade-offs**: Larger batches improve GPU utilization but increase memory usage
|
||||
- **Shuffling Strategy**: Random access patterns for training vs sequential for inference
|
||||
- **Caching and Storage**: Balance between memory usage and I/O performance
|
||||
- **Error Handling**: Robust handling of corrupted data, network failures, disk issues
|
||||
|
||||
### Production ML Pipeline Patterns
|
||||
- **ETL Design**: Extract (load files), Transform (preprocess), Load (batch) pattern
|
||||
- **Data Versioning**: Reproducible datasets with consistent preprocessing
|
||||
- **Pipeline Monitoring**: Track data quality, distribution shifts, processing times
|
||||
- **Scalability Planning**: Design for growing datasets and distributed processing
|
||||
|
||||
## 🎉 Ready to Build?
|
||||
|
||||
You're about to build the data engineering foundation that powers every successful ML system! From startup prototypes to billion-dollar recommendation engines, they all depend on robust data pipelines like the one you're building.
|
||||
|
||||
This module teaches you the systems thinking that separates hobby projects from production ML systems. You'll work with real data, handle real performance constraints, and build infrastructure that scales. Take your time, think about edge cases, and enjoy building the backbone of machine learning!
|
||||
|
||||
|
||||
|
||||
|
||||
Choose your preferred way to engage with this module:
|
||||
|
||||
````{grid} 1 2 3 3
|
||||
|
||||
```{grid-item-card} 🚀 Launch Binder
|
||||
:link: https://mybinder.org/v2/gh/mlsysbook/TinyTorch/main?filepath=modules/source/08_dataloader/dataloader_dev.ipynb
|
||||
:class-header: bg-light
|
||||
|
||||
Run this module interactively in your browser. No installation required!
|
||||
```
|
||||
|
||||
```{grid-item-card} ⚡ Open in Colab
|
||||
:link: https://colab.research.google.com/github/mlsysbook/TinyTorch/blob/main/modules/source/08_dataloader/dataloader_dev.ipynb
|
||||
:class-header: bg-light
|
||||
|
||||
Use Google Colab for GPU access and cloud compute power.
|
||||
```
|
||||
|
||||
```{grid-item-card} 📖 View Source
|
||||
:link: https://github.com/mlsysbook/TinyTorch/blob/main/modules/source/08_dataloader/dataloader_dev.py
|
||||
:class-header: bg-light
|
||||
|
||||
Browse the Python source code and understand the implementation.
|
||||
```
|
||||
|
||||
````
|
||||
|
||||
```{admonition} 💾 Save Your Progress
|
||||
:class: tip
|
||||
**Binder sessions are temporary!** Download your completed notebook when done, or switch to local development for persistent work.
|
||||
## Where This Code Lives
|
||||
|
||||
```
|
||||
tinytorch/
|
||||
├── core/
|
||||
│ └── dataloader.py # Your implementation goes here
|
||||
└── __init__.py # Exposes DataLoader, Dataset, etc.
|
||||
|
||||
Usage in other modules:
|
||||
>>> from tinytorch.core.dataloader import DataLoader, CIFAR10Dataset
|
||||
>>> dataset = CIFAR10Dataset(download=True)
|
||||
>>> loader = DataLoader(dataset, batch_size=32, shuffle=True)
|
||||
```
|
||||
|
||||
## Systems Thinking Questions
|
||||
|
||||
1. **Memory vs Throughput Trade-off**: Why does increasing batch size improve GPU utilization but increase memory usage? What's the optimal batch size for a 16GB GPU?
|
||||
|
||||
2. **Shuffling Impact**: How does shuffling affect training dynamics and convergence? Why is it critical for training but not for evaluation?
|
||||
|
||||
3. **I/O Bottlenecks**: Your GPU can process 1000 images/sec but your disk reads at 100 images/sec. Where's the bottleneck? How would you fix it?
|
||||
|
||||
4. **Preprocessing Placement**: Should preprocessing happen in the data loader or in the training loop? What are the trade-offs for CPU vs GPU preprocessing?
|
||||
|
||||
5. **Distributed Loading**: If you're training on 8 GPUs, how should you partition the dataset? What challenges arise with shuffling across multiple workers?
|
||||
|
||||
## Real-World Connections
|
||||
|
||||
### Industry Applications
|
||||
|
||||
**Netflix (Recommendation Systems)**
|
||||
- Processes billions of viewing records through custom data pipelines
|
||||
- Uses streaming loaders for datasets that don't fit in memory
|
||||
- Implements sophisticated batching strategies for negative sampling
|
||||
|
||||
**Autonomous Vehicles (Tesla, Waymo)**
|
||||
- Load terabytes of sensor data (camera, LIDAR, radar) for training
|
||||
- Use multi-worker data loading to keep GPUs fully utilized
|
||||
- Implement real-time preprocessing pipelines for online learning
|
||||
|
||||
**Large Language Models (OpenAI, Anthropic)**
|
||||
- Stream hundreds of billions of tokens from distributed storage
|
||||
- Use custom data loaders optimized for sequence data
|
||||
- Implement efficient tokenization and batching for transformers
|
||||
|
||||
### Research Impact
|
||||
|
||||
This module teaches patterns from:
|
||||
- PyTorch DataLoader (2016): The industry-standard data loading API
|
||||
- TensorFlow Dataset API (2017): Google's approach to data pipelines
|
||||
- NVIDIA DALI (2019): GPU-accelerated preprocessing for peak throughput
|
||||
- WebDataset (2020): Efficient loading from cloud storage
|
||||
|
||||
## What's Next?
|
||||
|
||||
In **Module 09: Spatial (CNNs)**, you'll use these data loaders to train convolutional neural networks on CIFAR-10:
|
||||
|
||||
- Apply convolution operations to the RGB images you're loading
|
||||
- Use your DataLoader to iterate through 50,000 training samples
|
||||
- Achieve >75% accuracy on CIFAR-10 classification
|
||||
- Understand how CNNs process spatial data efficiently
|
||||
|
||||
The data infrastructure you built here becomes critical—training CNNs requires efficient batch loading of image data with proper preprocessing.
|
||||
|
||||
---
|
||||
|
||||
<div class="prev-next-area">
|
||||
<a class="left-prev" href="../chapters/07_attention.html" title="previous page">← Previous Module</a>
|
||||
<a class="right-next" href="../chapters/09_dataloader.html" title="next page">Next Module →</a>
|
||||
</div>
|
||||
**Ready to build production data infrastructure?** Open `modules/source/08_dataloader/dataloader_dev.py` and start implementing.
|
||||
|
||||
Reference in New Issue
Block a user