mirror of
https://github.com/harvard-edge/cs249r_book.git
synced 2026-07-16 06:07:17 -05:00
d2c3a57b25
DatasetProfile makes test_examples/num_classes Optional by design (MSWC sources only the 23.4M corpus total), but the zoo table called .magnitude unconditionally and crashed the MLSysIM docs-site build with AttributeError on the MSWC row. Guard the optional fields the same way the cell already guards image dimensions.
43 lines
1.3 KiB
Plaintext
43 lines
1.3 KiB
Plaintext
---
|
||
title: "The Datasets Zoo"
|
||
subtitle: "Reference Dataset Profiles for Data-Scale Reasoning"
|
||
---
|
||
|
||
The Datasets Zoo provides **canonical dataset sizes** — training/test example counts, image
|
||
dimensions, and class counts — for back-of-envelope data-pipeline and storage analysis.
|
||
|
||
```{python}
|
||
#| echo: false
|
||
#| output: asis
|
||
import mlsysim
|
||
|
||
print("| Dataset | Train Examples | Test Examples | Classes | Resolution |")
|
||
print("|:---|:---:|:---:|:---:|:---:|")
|
||
def _count(q):
|
||
# Split counts are Optional in DatasetProfile (e.g. MSWC sources only the
|
||
# corpus total) — render missing fields as "---" instead of crashing.
|
||
return f"{int(q.magnitude):,}" if q is not None else "---"
|
||
|
||
|
||
for d in mlsysim.Datasets.list():
|
||
res = f"{d.image_width}×{d.image_height}×{d.image_channels}" if d.image_width else "---"
|
||
classes = d.num_classes if d.num_classes is not None else "---"
|
||
print(
|
||
f"| **{d.name}** | {_count(d.training_examples)} | "
|
||
f"{_count(d.test_examples)} | {classes} | {res} |"
|
||
)
|
||
```
|
||
|
||
## Python Access
|
||
|
||
```python
|
||
import mlsysim
|
||
|
||
imagenet = mlsysim.Datasets.ImageNet
|
||
cifar = mlsysim.Datasets.CIFAR10
|
||
mnist = mlsysim.Datasets.MNIST
|
||
```
|
||
|
||
Pair dataset profiles with [`DataModel`](../api/solvers.DataModel.qmd) and the
|
||
*Data Engineering* textbook chapters when reasoning about ingestion bandwidth and epoch time.
|