# Real Data, Real Systems Principles

## Core Philosophy

TinyTorch modules must use **real production data** and reflect **real-world ML engineering practices**. No mock data, no synthetic examples, no toy datasets.

## Real Data Requirements

### ✅ GOOD: Production Datasets
```python
# Use actual datasets
dataset = CIFAR10Dataset('data/cifar10/', train=True, download=True)
image, label = dataset[0]
assert image.shape == (3, 32, 32)  # Real image dimensions
assert 0 <= label <= 9  # Real class labels
```

### ❌ BAD: Mock/Synthetic Data
```python
# DON'T DO THIS
class MockDataset:
    def __init__(self, size):
        self.data = np.random.randn(size, 784)  # Fake data

# DON'T DO THIS
data = np.random.randn(100, 784)  # Synthetic data
```

## Progress Feedback Requirements

All long-running operations must provide visual feedback:

```python
def _download_progress_hook(self, count, block_size, total_size):
    """Progress callback for downloads."""
    if total_size > 0:
        percent = min(100, (count * block_size * 100) // total_size)
        mb_downloaded = (count * block_size) / (1024 * 1024)
        mb_total = total_size / (1024 * 1024)
        
        # Visual progress bar
        bar_length = 50
        filled_length = int(bar_length * percent // 100)
        bar = '█' * filled_length + '░' * (bar_length - filled_length)
        
        print(f'\r📥 [{bar}] {percent}% ({mb_downloaded:.1f}/{mb_total:.1f} MB)', 
              end='', flush=True)
```

## Visual Feedback (Development Only)

Provide immediate visual confirmation in development:

```python
def show_data_samples(dataset, num_samples=8):
    """Show grid of actual data samples (development only)."""
    # NOT exported to package - development feedback only
    if not _should_show_plots():
        return
        
    fig, axes = plt.subplots(2, 4, figsize=(12, 6))
    for i in range(num_samples):
        image, label = dataset[i]
        axes[i//4, i%4].imshow(image.transpose(1, 2, 0))
        axes[i//4, i%4].set_title(f'Class: {label}')
    plt.show()
```

## Systems Thinking Requirements

Every module must consider:
- **Performance**: How does this scale with larger datasets?
- **Caching**: Download once, use repeatedly
- **User Experience**: How do users know operations are working?
- **Memory**: Efficient handling of large datasets
- **Error Handling**: Graceful failure with helpful messages

## Real-World Scale

- Use actual dataset sizes (CIFAR-10: 50,000 training images)
- Test with realistic batch sizes (32, 64, 128)
- Consider actual network/disk constraints
- Handle production-scale data efficiently

## Development vs Export Separation

- **Development files**: Rich with visual feedback, progress bars, debugging info
- **Package exports**: Clean, minimal dependencies, production-ready
- **Conditional display**: Use `_should_show_plots()` to control visual output

## Quality Standards

Every module must:
- ✅ Use real production datasets
- ✅ Include progress feedback for long operations
- ✅ Provide visual confirmation (development only)
- ✅ Handle realistic data scales
- ✅ Consider performance and caching
- ✅ Reflect real-world ML engineering practices

## Common Datasets

- **CIFAR-10**: Image classification (32x32 RGB images, 10 classes)
- **MNIST**: Handwritten digits (28x28 grayscale, 10 classes)
- **ImageNet**: Large-scale image classification
- **Fashion-MNIST**: Fashion items (28x28 grayscale, 10 classes)

Use these datasets consistently across modules for coherent learning experience.
description:
globs:
alwaysApply: false
---
