mirror of
https://github.com/harvard-edge/cs249r_book.git
synced 2026-03-11 17:49:25 -05:00
704f7555fe
Restoration pass (selective, based on Three-Job Rule audit): - introduction: restore fn-eliza-brittleness, fn-dartmouth-systems, fn-bobrow-student - data_engineering: restore fn-soc-always-on (always-on island architecture) - benchmarking: restore fn-glue-saturation (Goodhart's Law arc, 1-year saturation) Group A surgical edits: - nn_computation: remove fn-overfitting (Context=1, Tether=1 — only confirmed failure) - training: strip dead etymology from fn-convergence-training, fn-hyperparameter-training - model_serving: enrich fn-onnx-runtime-serving with 5–15% TensorRT throughput figure Group B new footnotes: - nn_computation: add fn-alexnet-gpu-split (GTX 580 3 GB ceiling → model parallelism lineage) - responsible_engr: add fn-zillow-dam (D·A·M decomposition of $304M failure)
3733 lines
365 KiB
Plaintext
3733 lines
365 KiB
Plaintext
---
|
||
quiz: data_engineering_quizzes.json
|
||
concepts: data_engineering_concepts.yml
|
||
glossary: data_engineering_glossary.json
|
||
engine: jupyter
|
||
---
|
||
|
||
# Data Engineering {#sec-data-engineering}
|
||
|
||
\index{Data Engineering}
|
||
|
||
::: {layout-narrow}
|
||
::: {.column-margin}
|
||
\chapterminitoc
|
||
:::
|
||
|
||
\noindent
|
||
{fig-alt="Illustration showing data engineering concept with raw data sources flowing through processing pipelines to refined datasets and storage systems."}
|
||
|
||
:::
|
||
|
||
## Purpose {.unnumbered}
|
||
|
||
\begin{marginfigure}
|
||
\mlsysstack{15}{10}{10}{25}{10}{15}{10}{90}
|
||
\end{marginfigure}
|
||
|
||
_Why does data represent the actual source code of machine learning systems while traditional code merely describes how to compile it?_
|
||
|
||
In conventional software, programmers write logic that computers execute. In machine learning, programmers write optimization procedures that extract operational logic *from data*.\index{Data Engineering!data as source code} This inversion makes data the true source code: changing the data changes what the system does, regardless of whether a single line of traditional code has been modified. A dataset with subtle labeling inconsistencies produces a model with subtle behavioral inconsistencies. A dataset missing edge cases produces a model that fails on edge cases. A dataset reflecting historical biases produces a model that perpetuates those biases. No architecture, hyperparameter, or training trick can recover information that was never present or correct errors that were baked in from the start. And unlike traditional source code, which sits inert until a programmer modifies it, data is *alive*: the distribution it captures drifts as the world changes, silently invalidating the model's learned behavior even when nothing in the codebase has been touched. This is why data engineering consumes the majority of effort in most ML projects—not because the work is tedious, but because it is *consequential*. Every decision in the data pipeline—what to collect, how to label, when to filter, how to split—propagates forward to constrain model architecture, training dynamics, and deployment viability. Data engineering is therefore not preprocessing but programming in a different language,\index{Data Engineering!as programming} one where quality control, versioning, and monitoring determine whether the compiled system works today and continues working tomorrow.
|
||
|
||
::: {.content-visible when-format="pdf"}
|
||
\newpage
|
||
:::
|
||
|
||
::: {.callout-tip title="Learning Objectives"}
|
||
|
||
- Explain how **Data Cascades** propagate errors through ML pipelines and apply the **Four Pillars framework** (quality, reliability, scalability, governance) to diagnose data system failures
|
||
- Evaluate data acquisition strategies (curated datasets, crowdsourcing, synthetic generation) using cost-quality trade-offs
|
||
- Compare batch vs. streaming ingestion and **ETL** vs. **ELT** pipeline patterns with validation and graceful degradation
|
||
- Implement **training-serving consistency** through idempotent transformations, and operationalize the **Degradation Equation** via drift detection
|
||
- Build data labeling systems balancing accuracy, throughput, and cost with quality control and AI-assisted scaling
|
||
- Evaluate storage architectures and file formats for ML workloads, including **feature store** concepts for training-serving consistency
|
||
- Identify **Data Debt** categories and apply systematic debugging to production pipeline failures
|
||
|
||
:::
|
||
|
||
```{python}
|
||
#| echo: false
|
||
#| label: chapter-start
|
||
# ┌─────────────────────────────────────────────────────────────────────────────
|
||
# │ CHAPTER START
|
||
# ├─────────────────────────────────────────────────────────────────────────────
|
||
# │ Context: Chapter initialization — must be the first cell
|
||
# │
|
||
# │ Goal: Initialize the chapter and register with the mlsys registry.
|
||
# │ Show: Correct registration for cross-chapter constant resolution.
|
||
# │ How: Invoke start_chapter from the mlsys registry module.
|
||
# │
|
||
# │ Imports: mlsys.registry (start_chapter)
|
||
# │ Exports: (none — side-effect only)
|
||
# └─────────────────────────────────────────────────────────────────────────────
|
||
from mlsys.registry import start_chapter
|
||
|
||
start_chapter("vol1:data_engineering")
|
||
```
|
||
|
||
```{python}
|
||
#| label: data-eng-setup
|
||
#| echo: false
|
||
# ┌─────────────────────────────────────────────────────────────────────────────
|
||
# │ DATA ENGINEERING SETUP
|
||
# ├─────────────────────────────────────────────────────────────────────────────
|
||
# │ Context: Multiple tables and callouts throughout this chapter
|
||
# │
|
||
# │ Goal: Centralize hardware and model parameters for the entire chapter.
|
||
# │ Show: A single source of truth for energy, scale, and cost constants.
|
||
# │ How: Retrieve constants from mlsys.constants and Digital Twins.
|
||
# │
|
||
# │ Imports: mlsys.constants (*), mlsys.formatting (fmt, sci, md_math),
|
||
# │ mlsys.formulas (model_memory)
|
||
# │ Exports: energy_*, a100_*, gpt3_*, alexnet_*, kws_*, labeling_cost_*,
|
||
# │ storage_cost_*, retrieval_cost_*, data_prep_*, model_growth_*,
|
||
# │ dram_*, network_*, transfer_days_10g_str, transfer_time_10g_md
|
||
# └─────────────────────────────────────────────────────────────────────────────
|
||
from mlsys.constants import *
|
||
from mlsys import Hardware, Models
|
||
from mlsys.formatting import fmt, sci, md_math
|
||
from mlsys.formulas import model_memory
|
||
|
||
# ┌── P.I.C.O. ISOLATED SCENARIO ───────────────────────────────────────────────
|
||
class DataloaderStats:
|
||
"""
|
||
Namespace for Dataloader Choke Point statistics.
|
||
"""
|
||
resnet_worker_count = 8
|
||
resnet_worker_count_str = f"{resnet_worker_count}"
|
||
|
||
# ┌── EXPORTS (Bridge to Text) ─────────────────────────────────────────────────
|
||
resnet_worker_count_str = DataloaderStats.resnet_worker_count_str
|
||
|
||
# ┌── P.I.C.O. ISOLATED SCENARIO ───────────────────────────────────────────────
|
||
class DataEngineeringSetup:
|
||
"""
|
||
Namespace for Data Engineering chapter setup.
|
||
Scenario: Constants for tables, KWS case study, and physical invariants.
|
||
"""
|
||
|
||
# ┌── 1. PARAMETERS (Inputs) ───────────────────────────────────────────────
|
||
# Digital Twins
|
||
h_a100 = Hardware.Cloud.A100
|
||
m_gpt3 = Models.GPT3
|
||
m_alexnet = Models.Vision.ALEXNET
|
||
|
||
# Energy
|
||
energy_add_int32_pj = ENERGY_ADD_INT32_PJ.m_as(ureg.picojoule)
|
||
energy_add_fp32_pj = ENERGY_ADD_FP32_PJ.m_as(ureg.picojoule)
|
||
energy_network_1kb_pj = NETWORK_ENERGY_1KB_PJ.m_as(ureg.picojoule)
|
||
alexnet_params_m = m_alexnet.parameters.m_as(Mparam)
|
||
|
||
# Data Prep Effort
|
||
data_prep_effort_low = 60
|
||
data_prep_effort_high = 80
|
||
data_cleaning_effort = 60
|
||
|
||
# KWS Case Study
|
||
kws_accuracy_target = 98
|
||
kws_latency_limit_ms = 200
|
||
kws_memory_limit_kb_tiny = 800
|
||
kws_memory_limit_design_kb = 64
|
||
kws_memory_limit_edge_kb = 16
|
||
kws_dataset_size_m = 23.4
|
||
kws_languages = 50
|
||
kws_dataset_size_gb = 736
|
||
kws_sample_duration_ms_low = 500
|
||
kws_sample_duration_ms_high = 800
|
||
kws_snr_threshold_db = 20
|
||
kws_false_wake_month = 1
|
||
kws_budget = 150000
|
||
kws_timeline_months = 6
|
||
kws_label_seconds_per_example = 10
|
||
|
||
# Labeling Costs
|
||
labeling_cost_crowd_low = LABELING_COST_CROWD_LOW.m_as(USD)
|
||
labeling_cost_crowd_high = LABELING_COST_CROWD_HIGH.m_as(USD)
|
||
labeling_cost_expert_low = LABELING_COST_EXPERT_LOW.m_as(USD)
|
||
labeling_cost_expert_high = LABELING_COST_EXPERT_HIGH.m_as(USD)
|
||
labeling_cost_box_low = LABELING_COST_BOX_LOW.m_as(USD)
|
||
labeling_cost_box_high = LABELING_COST_BOX_HIGH.m_as(USD)
|
||
labeling_cost_seg_low = LABELING_COST_SEG_LOW.m_as(USD)
|
||
labeling_cost_seg_high = LABELING_COST_SEG_HIGH.m_as(USD)
|
||
labeling_cost_medical_low = LABELING_COST_MEDICAL_LOW.m_as(USD)
|
||
labeling_cost_medical_high = LABELING_COST_MEDICAL_HIGH.m_as(USD)
|
||
|
||
# Storage Costs
|
||
storage_cost_s3 = STORAGE_COST_S3_STD.m_as(USD / TB / ureg.month)
|
||
storage_cost_glacier = STORAGE_COST_GLACIER.m_as(USD / TB / ureg.month)
|
||
storage_cost_nvme_low = STORAGE_COST_NVME_LOW.m_as(USD / TB / ureg.month)
|
||
storage_cost_nvme_high = STORAGE_COST_NVME_HIGH.m_as(USD / TB / ureg.month)
|
||
retrieval_cost_glacier = RETRIEVAL_COST_GLACIER.m_as(USD / GB)
|
||
|
||
# Data Gravity Preview
|
||
_dataset_pb = 1
|
||
_network_10g_gbs = 10 / 8 # 10 Gbps in GB/s
|
||
|
||
# ┌── 2. CALCULATION (The Physics) ─────────────────────────────────────────
|
||
# Hardware/Energy
|
||
a100_tflops_fp16 = h_a100.peak_flops.m_as(TFLOPs/second)
|
||
a100_tflops_fp16_sparse = (h_a100.peak_flops * 2).m_as(TFLOPs/second)
|
||
|
||
energy_dram_pj = ENERGY_DRAM_ACCESS_PJ.m_as(ureg.picojoule)
|
||
energy_flop_fp32_pj = ENERGY_FLOP_FP32_PJ.m_as(ureg.picojoule / ureg.count)
|
||
energy_flop_int8_pj = ENERGY_FLOP_INT8_PJ.m_as(ureg.picojoule / ureg.count)
|
||
energy_flop_fp32_pj_per_bit = energy_flop_fp32_pj / 32
|
||
energy_local_ssd_pj = 10_000
|
||
energy_network_pj = 50_000
|
||
|
||
dram_vs_int = int(energy_dram_pj / energy_add_int32_pj)
|
||
network_vs_int = int(energy_network_1kb_pj / energy_add_int32_pj)
|
||
|
||
# Models
|
||
gpt3_params_b = m_gpt3.parameters.m_as(Bparam)
|
||
gpt3_fp32_gb = m_gpt3.size_in_bytes(BYTES_FP32).m_as(GB)
|
||
gpt3_fp16_gb = m_gpt3.size_in_bytes(BYTES_FP16).m_as(GB)
|
||
model_growth_factor = int(gpt3_params_b * BILLION / (alexnet_params_m * MILLION))
|
||
|
||
# KWS Labeling
|
||
kws_label_hours = kws_dataset_size_m * MILLION * kws_label_seconds_per_example / SEC_PER_HOUR
|
||
kws_label_years = kws_label_hours / 2000
|
||
|
||
# KWS Storage
|
||
kws_dataset_size_tb = (kws_dataset_size_gb * GB).m_as(TB)
|
||
kws_storage_s3_cost = kws_dataset_size_tb * storage_cost_s3
|
||
kws_storage_nvme_low_cost = kws_dataset_size_tb * storage_cost_nvme_low
|
||
kws_storage_nvme_high_cost = kws_dataset_size_tb * storage_cost_nvme_high
|
||
|
||
# Data Gravity
|
||
_transfer_days_10g = (_dataset_pb * MILLION) / _network_10g_gbs / (
|
||
SECONDS_PER_MINUTE * MINUTES_PER_HOUR * HOURS_PER_DAY
|
||
)
|
||
|
||
# ┌── 4. OUTPUTS (Formatting) ──────────────────────────────────────────────
|
||
# Energy/Hardware
|
||
a100_tflops_fp16_str = fmt(a100_tflops_fp16, precision=0, commas=False)
|
||
a100_tflops_fp16_sparse_str = fmt(a100_tflops_fp16_sparse, precision=0, commas=False)
|
||
energy_dram_str = fmt(energy_dram_pj, precision=0, commas=False)
|
||
energy_flop_fp32_str = fmt(energy_flop_fp32_pj, precision=1, commas=False)
|
||
energy_flop_int8_str = fmt(energy_flop_int8_pj, precision=1, commas=False)
|
||
energy_add_int32_str = f"{energy_add_int32_pj}"
|
||
energy_add_fp32_str = f"{energy_add_fp32_pj}"
|
||
dram_vs_int_str = f"{dram_vs_int:,}"
|
||
network_vs_int_str = f"{network_vs_int:,}"
|
||
dram_multiplier_str = f"{int(ENERGY_DRAM_ACCESS_PJ.m_as(ureg.picojoule) / ENERGY_FLOP_FP32_PJ.m_as(ureg.picojoule / ureg.count)):,}"
|
||
local_ssd_multiplier_str = fmt(energy_local_ssd_pj / energy_flop_fp32_pj_per_bit, precision=0, commas=True)
|
||
network_multiplier_str = fmt(energy_network_pj / energy_flop_fp32_pj_per_bit, precision=0, commas=True)
|
||
|
||
# Models
|
||
gpt3_params_str = fmt(gpt3_params_b, precision=0, commas=False)
|
||
gpt3_fp32_str = fmt(gpt3_fp32_gb, precision=0, commas=False)
|
||
gpt3_fp16_str = fmt(gpt3_fp16_gb, precision=0, commas=False)
|
||
alexnet_params_str = f"{alexnet_params_m}"
|
||
model_growth_str = f"{model_growth_factor:,}"
|
||
|
||
# Data Prep
|
||
data_prep_effort_low_str = f"{data_prep_effort_low}"
|
||
data_prep_effort_high_str = f"{data_prep_effort_high}"
|
||
data_cleaning_effort_str = f"{data_cleaning_effort}"
|
||
|
||
# KWS
|
||
kws_accuracy_target_str = f"{kws_accuracy_target}"
|
||
kws_latency_limit_ms_str = f"{kws_latency_limit_ms}"
|
||
kws_memory_limit_kb_tiny_str = f"{kws_memory_limit_kb_tiny}"
|
||
kws_memory_limit_design_kb_str = f"{kws_memory_limit_design_kb}"
|
||
kws_memory_limit_edge_kb_str = f"{kws_memory_limit_edge_kb}"
|
||
kws_dataset_size_m_str = f"{kws_dataset_size_m}"
|
||
kws_dataset_size_m_round_str = fmt(kws_dataset_size_m, precision=0)
|
||
kws_label_hours_str = fmt(kws_label_hours, precision=0, commas=True)
|
||
kws_label_years_str = fmt(kws_label_years, precision=1, commas=False)
|
||
kws_languages_str = f"{kws_languages}"
|
||
kws_dataset_size_gb_str = f"{kws_dataset_size_gb}"
|
||
kws_sample_duration_ms_low_str = f"{kws_sample_duration_ms_low}"
|
||
kws_sample_duration_ms_high_str = f"{kws_sample_duration_ms_high}"
|
||
kws_snr_threshold_db_str = f"{kws_snr_threshold_db}"
|
||
kws_false_wake_month_str = f"{kws_false_wake_month}"
|
||
kws_budget_str = fmt(kws_budget/1000, precision=0, commas=False) + "K"
|
||
kws_timeline_months_str = f"{kws_timeline_months}"
|
||
|
||
# Labeling
|
||
labeling_cost_crowd_low_str = fmt(labeling_cost_crowd_low, precision=2, commas=False)
|
||
labeling_cost_crowd_high_str = fmt(labeling_cost_crowd_high, precision=2, commas=False)
|
||
labeling_cost_expert_low_str = fmt(labeling_cost_expert_low, precision=2, commas=False)
|
||
labeling_cost_expert_high_str = fmt(labeling_cost_expert_high, precision=2, commas=False)
|
||
labeling_cost_box_low_str = fmt(labeling_cost_box_low, precision=2, commas=False)
|
||
labeling_cost_box_high_str = fmt(labeling_cost_box_high, precision=2, commas=False)
|
||
labeling_cost_seg_low_str = fmt(labeling_cost_seg_low, precision=0, commas=False)
|
||
labeling_cost_seg_high_str = fmt(labeling_cost_seg_high, precision=0, commas=False)
|
||
labeling_cost_medical_low_str = fmt(labeling_cost_medical_low, precision=0, commas=False)
|
||
labeling_cost_medical_high_str = fmt(labeling_cost_medical_high, precision=0, commas=False)
|
||
|
||
# Storage
|
||
storage_cost_s3_str = fmt(storage_cost_s3, precision=0, commas=False)
|
||
storage_cost_glacier_str = fmt(storage_cost_glacier, precision=0, commas=False)
|
||
storage_cost_nvme_low_str = fmt(storage_cost_nvme_low, precision=0, commas=False)
|
||
storage_cost_nvme_high_str = fmt(storage_cost_nvme_high, precision=0, commas=False)
|
||
retrieval_cost_glacier_str = fmt(retrieval_cost_glacier, precision=2, commas=False)
|
||
|
||
kws_storage_s3_cost_str = fmt(kws_storage_s3_cost, precision=0, commas=False)
|
||
kws_storage_nvme_low_cost_str = fmt(kws_storage_nvme_low_cost, precision=0, commas=False)
|
||
kws_storage_nvme_high_cost_str = fmt(kws_storage_nvme_high_cost, precision=0, commas=False)
|
||
|
||
# Data Gravity
|
||
transfer_days_10g_str = fmt(_transfer_days_10g, precision=0, commas=False)
|
||
transfer_time_10g_md = md_math(f"T = D_{{vol}}/BW \\approx {transfer_days_10g_str} \\text{{ days}}")
|
||
|
||
# ┌── EXPORTS (Bridge to Text) ─────────────────────────────────────────────────
|
||
a100_tflops_fp16_str = DataEngineeringSetup.a100_tflops_fp16_str
|
||
a100_tflops_fp16_sparse_str = DataEngineeringSetup.a100_tflops_fp16_sparse_str
|
||
energy_dram_str = DataEngineeringSetup.energy_dram_str
|
||
energy_flop_fp32_str = DataEngineeringSetup.energy_flop_fp32_str
|
||
energy_flop_int8_str = DataEngineeringSetup.energy_flop_int8_str
|
||
energy_add_int32_str = DataEngineeringSetup.energy_add_int32_str
|
||
energy_add_fp32_str = DataEngineeringSetup.energy_add_fp32_str
|
||
dram_vs_int_str = DataEngineeringSetup.dram_vs_int_str
|
||
network_vs_int_str = DataEngineeringSetup.network_vs_int_str
|
||
dram_multiplier_str = DataEngineeringSetup.dram_multiplier_str
|
||
local_ssd_multiplier_str = DataEngineeringSetup.local_ssd_multiplier_str
|
||
network_multiplier_str = DataEngineeringSetup.network_multiplier_str
|
||
gpt3_params_str = DataEngineeringSetup.gpt3_params_str
|
||
gpt3_fp32_str = DataEngineeringSetup.gpt3_fp32_str
|
||
gpt3_fp16_str = DataEngineeringSetup.gpt3_fp16_str
|
||
alexnet_params_str = DataEngineeringSetup.alexnet_params_str
|
||
model_growth_str = DataEngineeringSetup.model_growth_str
|
||
data_prep_effort_low_str = DataEngineeringSetup.data_prep_effort_low_str
|
||
data_prep_effort_high_str = DataEngineeringSetup.data_prep_effort_high_str
|
||
data_cleaning_effort_str = DataEngineeringSetup.data_cleaning_effort_str
|
||
kws_accuracy_target_str = DataEngineeringSetup.kws_accuracy_target_str
|
||
kws_latency_limit_ms_str = DataEngineeringSetup.kws_latency_limit_ms_str
|
||
kws_memory_limit_kb_tiny_str = DataEngineeringSetup.kws_memory_limit_kb_tiny_str
|
||
kws_memory_limit_design_kb_str = DataEngineeringSetup.kws_memory_limit_design_kb_str
|
||
kws_memory_limit_edge_kb_str = DataEngineeringSetup.kws_memory_limit_edge_kb_str
|
||
kws_dataset_size_m_str = DataEngineeringSetup.kws_dataset_size_m_str
|
||
kws_dataset_size_m_round_str = DataEngineeringSetup.kws_dataset_size_m_round_str
|
||
kws_label_hours_str = DataEngineeringSetup.kws_label_hours_str
|
||
kws_label_years_str = DataEngineeringSetup.kws_label_years_str
|
||
kws_languages_str = DataEngineeringSetup.kws_languages_str
|
||
kws_dataset_size_gb_str = DataEngineeringSetup.kws_dataset_size_gb_str
|
||
kws_sample_duration_ms_low_str = DataEngineeringSetup.kws_sample_duration_ms_low_str
|
||
kws_sample_duration_ms_high_str = DataEngineeringSetup.kws_sample_duration_ms_high_str
|
||
kws_snr_threshold_db_str = DataEngineeringSetup.kws_snr_threshold_db_str
|
||
kws_false_wake_month_str = DataEngineeringSetup.kws_false_wake_month_str
|
||
kws_budget_str = DataEngineeringSetup.kws_budget_str
|
||
kws_timeline_months_str = DataEngineeringSetup.kws_timeline_months_str
|
||
labeling_cost_crowd_low_str = DataEngineeringSetup.labeling_cost_crowd_low_str
|
||
labeling_cost_crowd_high_str = DataEngineeringSetup.labeling_cost_crowd_high_str
|
||
labeling_cost_expert_low_str = DataEngineeringSetup.labeling_cost_expert_low_str
|
||
labeling_cost_expert_high_str = DataEngineeringSetup.labeling_cost_expert_high_str
|
||
labeling_cost_box_low_str = DataEngineeringSetup.labeling_cost_box_low_str
|
||
labeling_cost_box_high_str = DataEngineeringSetup.labeling_cost_box_high_str
|
||
labeling_cost_seg_low_str = DataEngineeringSetup.labeling_cost_seg_low_str
|
||
labeling_cost_seg_high_str = DataEngineeringSetup.labeling_cost_seg_high_str
|
||
labeling_cost_medical_low_str = DataEngineeringSetup.labeling_cost_medical_low_str
|
||
labeling_cost_medical_high_str = DataEngineeringSetup.labeling_cost_medical_high_str
|
||
storage_cost_s3_str = DataEngineeringSetup.storage_cost_s3_str
|
||
storage_cost_glacier_str = DataEngineeringSetup.storage_cost_glacier_str
|
||
storage_cost_nvme_low_str = DataEngineeringSetup.storage_cost_nvme_low_str
|
||
storage_cost_nvme_high_str = DataEngineeringSetup.storage_cost_nvme_high_str
|
||
retrieval_cost_glacier_str = DataEngineeringSetup.retrieval_cost_glacier_str
|
||
kws_storage_s3_cost_str = DataEngineeringSetup.kws_storage_s3_cost_str
|
||
kws_storage_nvme_low_cost_str = DataEngineeringSetup.kws_storage_nvme_low_cost_str
|
||
kws_storage_nvme_high_cost_str = DataEngineeringSetup.kws_storage_nvme_high_cost_str
|
||
transfer_days_10g_str = DataEngineeringSetup.transfer_days_10g_str
|
||
transfer_time_10g_md = DataEngineeringSetup.transfer_time_10g_md
|
||
```
|
||
|
||
## Dataset Compilation {#sec-data-engineering-data-engineering-dataset-compilation-0496}
|
||
|
||
The workflow stages from @sec-ml-workflow establish the *when* and *why* of data preparation, showing that data work consumes `{python} data_prep_effort_low_str` to `{python} data_prep_effort_high_str` percent of ML project effort (as @fig-ds-time quantified) and represents the foundational "D" in the AI Triad (Data, Algorithm, Machine) introduced in @sec-introduction. Executing those stages at scale, however, requires dedicated infrastructure. If the workflow is the plan, data engineering is the factory floor. This chapter uses **Keyword Spotting (KWS)** to demonstrate data engineering under extreme resource constraints—where every byte and operation matters.
|
||
|
||
::: {.callout-definition title="Data Engineering"}
|
||
|
||
***Data Engineering***\index{Data Engineering!definition} is the infrastructure layer that manages the lifecycle of data from source to model, encompassing acquisition, transformation, storage, and governance.
|
||
|
||
1. **Significance (Quantitative):** Its critical function is ensuring **Training-Serving Consistency**, preventing **Silent Degradation** by decoupling the model from the volatility of raw data. Within the **Iron Law**, it governs the **Data Volume ($D_{vol}$)** and ensures that it remains representative of the target distribution.
|
||
2. **Distinction (Durable):** Unlike **Data Science**, which focuses on **Inference and Insight**, Data Engineering addresses the **Scalability and Reliability** of the data pipeline.
|
||
3. **Common Pitfall:** A frequent misconception is that Data Engineering is "data cleaning." In reality, it is **Dataset Compilation**: transforming raw, noisy observations into an optimized binary that the model consumes.
|
||
|
||
:::
|
||
|
||
We reframe data engineering not as "data cleaning," but as *Dataset Compilation*. Just as a compiler transforms human-readable source code into an optimized binary executable, a data pipeline transforms raw, noisy observations into a clean, optimized training set that the model consumes. The analogy to compiler design is instructive. Just as a compiler transforms source code through a series of increasingly refined representations—tokens, abstract syntax trees, intermediate representations, machine code—a data pipeline transforms raw observations into training-ready tensors through analogous stages. Filtering corrupted records, outliers, and irrelevant features corresponds to *dead code elimination*: stripping material that contributes nothing to the learned representation. Augmentation—synthetically expanding limited examples by rotating images, pitch-shifting audio, or injecting noise—mirrors *loop unrolling*, exposing the model to more variations of the underlying pattern without collecting new data. Deduplication\index{Deduplication!duplicate removal} plays the role of *common subexpression elimination*, identifying and merging duplicate records that would otherwise bias gradient estimates and waste compute. And schema validation—enforcing strict types and ranges on every record—is the data pipeline's *type checker*, rejecting malformed inputs before they crash the "runtime" of model training.
|
||
|
||
The engineering implication is direct: datasets must be *versioned* (like git), *unit-tested* (data quality checks), and *debugged*. *Deleting a row of training data is the engineering equivalent of deleting a line of code, and retraining a model is simply recompiling the binary.*
|
||
|
||
This compilation metaphor establishes the engineering mindset that runs through the chapter. Just as a compiler has distinct phases (lexing, parsing, optimization, code generation), our dataset compiler has phases too: acquisition, ingestion, processing, labeling, storage, and ongoing maintenance. A **Four Pillars Framework**—Quality, Reliability, Scalability, and Governance—organizes design decisions across all phases. Each stage is illustrated through a Keyword Spotting (KWS) case study that demonstrates data engineering under extreme resource constraints, where every byte and operation matters.
|
||
|
||
Before any of these pipeline stages can be designed well, however, we need to understand the physical properties that constrain them. Just as a civil engineer must understand soil mechanics before designing foundations, a data engineer must understand the physics of data movement and information density before making pipeline decisions. These physics impose hard constraints that no amount of clever software can circumvent.
|
||
|
||
## Physics of Data {#sec-data-engineering-physics-data-cdcb}
|
||
|
||
\index{Data!physics of} The "data as code" metaphor captures *what* data does (determines system behavior) but not *why* moving it is so expensive. To engineer data systems effectively, we must also treat data as a physical substance with measurable properties. Just as diverse materials have density and viscosity, datasets have **Information Entropy** and **Data Gravity**.
|
||
|
||
### Data Gravity {#sec-data-engineering-data-gravity-adcb}
|
||
|
||
\index{Data Gravity!physics of} Data gravity is the cost of movement. It is a function of volume ($D_{vol}$) and network bandwidth ($BW$). The time to move a petabyte dataset across a 10 Gbps link is fixed by physics (`{python} transfer_time_10g_md`); the "Physics of Data Gravity" callout below quantifies these transfer times for a 100 Gbps link. This gravity dictates architecture: because moving 1PB to the compute is slow and expensive, we must move the compute to the data. This explains the rise of "Data Lakehouse" architectures[^fn-lakehouse-gravity]\index{Data Lakehouse!architecture} [@armbrust2021lakehouse] where processing engines (Spark, Presto) run directly on storage nodes. In contrast, **Data Mesh**\index{Data Mesh!decentralized ownership} [@dehghani2022data] proposes decentralizing ownership to manage this scale organizationally, treating data as a product owned by domain teams.
|
||
|
||
[^fn-lakehouse-gravity]: **Data Lakehouse**: Combines data lake storage (cheap, schema-less) with warehouse query semantics (ACID transactions, schema enforcement) using open table formats like Delta Lake and Apache Iceberg. For ML workloads, the lakehouse eliminates the ETL copy between lake and warehouse, enabling direct feature computation on the storage layer where data already resides -- a direct response to data gravity, since moving petabytes to a separate warehouse doubles the $D_{vol}/BW$ cost. \index{Data Lakehouse!data gravity}
|
||
|
||
### Information Entropy {#sec-data-engineering-information-entropy-5622}
|
||
|
||
\index{Information Entropy!data density} Information entropy is the density of signal. A dataset of 1 million identical images has high gravity (TB of storage) but zero entropy (one image worth of information). A dataset of 10,000 diverse edge cases has low gravity but high entropy. Let **Information Entropy** measure signal density (bits of information per byte) and **Data Gravity** capture movement cost (data volume / bandwidth, i.e., transfer time). This relationship appears in @eq-data-selection-gain:
|
||
|
||
$$ \text{Data Selection Gain} \propto \frac{\text{Information Entropy}}{\text{Data Gravity}} $$ {#eq-data-selection-gain}
|
||
\index{Data Selection!gain formula}
|
||
|
||
```{python}
|
||
#| label: feeding-tax-calc
|
||
#| echo: false
|
||
# ┌─────────────────────────────────────────────────────────────────────────────
|
||
# │ THE FEEDING TAX CALCULATION
|
||
# ├─────────────────────────────────────────────────────────────────────────────
|
||
# │ Context: "The Feeding Problem" section
|
||
# │
|
||
# │ Goal: Demonstrate the mismatch between accelerator compute and disk bandwidth.
|
||
# │ Show: That a standard cloud disk causes 80%+ idle time for an A100.
|
||
# │ How: Calculate required bandwidth for ResNet-50 inference at peak TFLOPS.
|
||
# │
|
||
# │ Imports: mlsys.constants, mlsys.formatting
|
||
# │ Exports: req_bw_gbs_str, disk_bw_mbs_str, feeding_tax_pct_str, img_per_sec_str
|
||
# └─────────────────────────────────────────────────────────────────────────────
|
||
from mlsys.constants import (
|
||
A100_FLOPS_FP16_TENSOR, RESNET50_FLOPs,
|
||
IMAGE_DIM_RESNET, IMAGE_CHANNELS_RGB, BYTES_FP32,
|
||
TFLOPs, second, GB, MB, MILLION
|
||
)
|
||
from mlsys.formatting import fmt, check
|
||
|
||
# ┌── P.I.C.O. ISOLATED SCENARIO ───────────────────────────────────────────────
|
||
class FeedingProblem:
|
||
"""
|
||
Namespace for The Feeding Problem calculation.
|
||
Scenario: Saturating an A100 with ResNet-50 images from a standard disk.
|
||
"""
|
||
|
||
# ┌── 1. PARAMETERS (Inputs) ───────────────────────────────────────────────
|
||
gpu_flops = A100_FLOPS_FP16_TENSOR
|
||
model_flops = RESNET50_FLOPs
|
||
|
||
# Image: 224x224x3 @ FP32
|
||
img_size_bytes = IMAGE_DIM_RESNET * IMAGE_DIM_RESNET * IMAGE_CHANNELS_RGB * BYTES_FP32.m_as('B')
|
||
|
||
# Standard Cloud Disk (e.g. AWS gp3 baseline)
|
||
disk_bw_mbs = 250.0
|
||
|
||
# ┌── 2. CALCULATION (The Physics) ─────────────────────────────────────────
|
||
# Throughput (Images/sec) = GPU_Peak / Model_FLOPs
|
||
img_per_sec = (gpu_flops / model_flops).to_base_units().m_as('count/second')
|
||
|
||
# Required Bandwidth (Bytes/sec) = img_per_sec * img_size_bytes
|
||
req_bw_bytes_sec = img_per_sec * img_size_bytes
|
||
req_bw_gbs = req_bw_bytes_sec / (1 * GB).m_as('B')
|
||
|
||
# Efficiency (eta) = Disk_BW / (Required_BW in MB/s)
|
||
eta = min(disk_bw_mbs / (req_bw_bytes_sec / (1 * MB).m_as('B')), 1.0)
|
||
feeding_tax_pct = (1.0 - eta) * 100
|
||
|
||
# ┌── 3. INVARIANTS (Guardrails) ───────────────────────────────────────────
|
||
check(req_bw_gbs > 1.0, f"Required bandwidth ({req_bw_gbs:.1f} GB/s) is lower than expected.")
|
||
check(feeding_tax_pct > 50, f"Feeding tax ({feeding_tax_pct:.1f}%) should be significant for standard disks.")
|
||
|
||
# ┌── 4. OUTPUTS (Formatting) ──────────────────────────────────────────────
|
||
req_bw_gbs_str = fmt(req_bw_gbs, precision=1, commas=False)
|
||
disk_bw_mbs_str = fmt(disk_bw_mbs, precision=0, commas=False)
|
||
feeding_tax_pct_str = fmt(feeding_tax_pct, precision=0, commas=False)
|
||
img_per_sec_str = fmt(img_per_sec, precision=0, commas=True)
|
||
|
||
# ┌── EXPORTS (Bridge to Text) ─────────────────────────────────────────────────
|
||
req_bw_gbs_str = FeedingProblem.req_bw_gbs_str
|
||
disk_bw_mbs_str = FeedingProblem.disk_bw_mbs_str
|
||
feeding_tax_pct_str = FeedingProblem.feeding_tax_pct_str
|
||
img_per_sec_str = FeedingProblem.img_per_sec_str
|
||
```
|
||
|
||
### The Feeding Problem: Flow Rate and the "Feeding Tax" {#sec-data-engineering-feeding-problem}
|
||
|
||
Data Gravity establishes the cost of moving the entire mass; **The Feeding Problem**\index{Feeding Problem!data pipeline throughput} establishes the cost of delivering it. In the Hennessy & Patterson tradition, we analyze this as a **Flow Rate** problem: the struggle to saturate a high-throughput "Machine" from a low-bandwidth "Data" source.
|
||
|
||
According to the **Iron Law**, the system is only as fast as its slowest term. If a modern accelerator can process `{python} img_per_sec_str` samples per second, but the storage pipeline delivers only `{python} disk_bw_mbs_str` MB/s, the expensive silicon sits idle. We quantify this as **The Feeding Tax**\index{Feeding Tax!I/O wait cost}: the wall-clock time lost to I/O wait, which directly reduces the **System Efficiency ($\eta$)** term. For a standard cloud volume, the feeding tax can exceed **`{python} feeding_tax_pct_str`%**, meaning the GPU spends the majority of its time waiting for bits. This tax transforms the Data Pipeline from a simple storage concern into the primary regulator of the system's Duty Cycle. To saturate a 300 TFLOPS processor, the pipeline must often sustain **`{python} req_bw_gbs_str` GB/s** transfer rates—a requirement that forces the shift from traditional file systems to the specialized storage architectures we examine in @sec-data-engineering-storage-architectures-ml-workloads-96ab.
|
||
|
||
Underlying these physical properties is a fundamental constraint we call the *energy-movement invariant*: moving data always dominates the energy budget.
|
||
|
||
::: {.callout-perspective title="The Energy-Movement Invariant"}
|
||
A corollary of the **Iron Law** (@sec-introduction-iron-law-ml-systems-c32a) for data engineering is that **moving a bit costs 100--1,000$\times$ more energy than computing on it.** While @sec-model-compression examines the energy cost inside the processor, we must also consider the cost of the information flow from the external world. The following table quantifies just how dramatic these differences are:
|
||
|
||
| **Operation** | **Energy (pJ)** | **Relative Cost** |
|
||
|:-----------------------------------|-----------------------------------:|-------------------------------------:|
|
||
| **32-bit Floating Point MAC** | `{python} energy_flop_fp32_str` pJ | 1× |
|
||
| **DRAM Memory Access (32-bit)** | **`{python} energy_dram_str` pJ** | **`{python} dram_multiplier_str`×** |
|
||
| **Local SSD Access (per bit)** | ~10,000 pJ | `{python} local_ssd_multiplier_str`× |
|
||
| **Network Transfer (Data Center)** | ~50,000 pJ | `{python} network_multiplier_str`× |
|
||
|
||
**Systems Implication**: Data has physical mass. If you prune 50% of your training data through deduplication, you are not just saving disk space; you are eliminating the most energy-intensive stages of the training lifecycle. This is *why* data selection is the highest-leverage tool in the systems engineer's toolkit: it addresses the problem at the most expensive source.
|
||
:::
|
||
|
||
Effective data engineering maximizes the Data Selection Gain defined in @eq-data-selection-gain. "Data Cleaning"\index{Data Cleaning!signal-to-noise engineering} is not just hygiene; it is **Signal-to-Noise Engineering**.\index{Signal-to-Noise Engineering} Deduplication removes mass without reducing entropy, directly increasing the ratio. Active learning adds high-entropy examples (edge cases) while ignoring low-entropy ones (common cases), again maximizing information per byte. We optimize this ratio to ensure our storage and compute budgets are spent on signal, not noise. To see how data gravity constrains real engineering decisions, consider the following scenario that quantifies *the physics of data gravity* by calculating the cost of moving a petabyte across regions.
|
||
|
||
```{python}
|
||
#| label: data-gravity-calc
|
||
#| echo: false
|
||
# ┌─────────────────────────────────────────────────────────────────────────────
|
||
# │ DATA GRAVITY CALCULATION
|
||
# ├─────────────────────────────────────────────────────────────────────────────
|
||
# │ Context: Callout "The Physics of Data Gravity"
|
||
# │
|
||
# │ Goal: Quantify the physical and economic cost of moving 1 PB across regions.
|
||
# │ Show: That egress fees can exceed $90,000, illustrating why "move compute to data" is mandatory.
|
||
# │ How: Calculate transfer time and cost based on network bandwidth and cloud egress rates.
|
||
# │
|
||
# │ Imports: mlsys.constants (NETWORK_100G_BW, CLOUD_EGRESS_PER_GB,
|
||
# │ TPU_V4_PER_HOUR, SECONDS_PER_MINUTE, MINUTES_PER_HOUR,
|
||
# │ HOURS_PER_DAY, Gbps, GB, USD, hour, second),
|
||
# │ mlsys.formatting (fmt, md_math)
|
||
# │ Exports: transfer_seconds_str, transfer_hours_str, transfer_days_10g_str,
|
||
# │ transfer_time_10g_md, transfer_cost_str, tpu_hours_str,
|
||
# │ network_gbs_str, dataset_gb_str, network_gbps_str,
|
||
# │ egress_cost_per_gb_str, tpu_cost_per_hour_str
|
||
# └─────────────────────────────────────────────────────────────────────────────
|
||
from mlsys import Hardware
|
||
from mlsys.constants import (
|
||
CLOUD_EGRESS_PER_GB, TPU_V4_PER_HOUR,
|
||
SECONDS_PER_MINUTE, MINUTES_PER_HOUR, HOURS_PER_DAY,
|
||
Gbps, GB, USD, hour, second, MILLION,
|
||
SEC_PER_HOUR, SEC_PER_DAY, BITS_PER_BYTE
|
||
)
|
||
from mlsys.formatting import fmt, check, md_math
|
||
|
||
# ┌── P.I.C.O. ISOLATED SCENARIO ───────────────────────────────────────────────
|
||
class DataGravity:
|
||
"""
|
||
Namespace for Data Gravity calculation.
|
||
Scenario: Moving 1PB of data vs Moving the Compute.
|
||
"""
|
||
|
||
# ┌── 1. PARAMETERS (Inputs) ───────────────────────────────────────────────
|
||
dataset_pb = 1
|
||
network = Hardware.Networks.Ethernet_100G
|
||
egress_cost_gb = CLOUD_EGRESS_PER_GB.m_as(USD / GB)
|
||
tpu_hourly_cost = TPU_V4_PER_HOUR.m_as(USD / hour)
|
||
|
||
# ┌── 2. CALCULATION (The Physics) ─────────────────────────────────────────
|
||
dataset_gb = dataset_pb * MILLION
|
||
network_gbs = network.bandwidth.m_as(GB/second)
|
||
network_gbps = network.bandwidth.m_as(Gbps)
|
||
|
||
# Time
|
||
transfer_seconds = dataset_gb / network_gbs
|
||
transfer_hours = transfer_seconds / SEC_PER_HOUR
|
||
# 10Gbps = 1.25 GB/s
|
||
transfer_days_10g = (dataset_gb / (10 / BITS_PER_BYTE)) / SEC_PER_DAY
|
||
|
||
# Cost
|
||
transfer_cost = dataset_gb * egress_cost_gb
|
||
equiv_tpu_hours = transfer_cost / tpu_hourly_cost
|
||
|
||
# ┌── 3. INVARIANTS (Guardrails) ───────────────────────────────────────────
|
||
check(transfer_hours >= 20, f"Transfer time ({transfer_hours:.1f}h) is too fast. Data gravity argument fails.")
|
||
check(transfer_cost >= 10000, f"Transfer cost (${transfer_cost}) is too cheap.")
|
||
|
||
# ┌── 4. OUTPUTS (Formatting) ──────────────────────────────────────────────
|
||
transfer_seconds_str = fmt(transfer_seconds, precision=0, commas=True)
|
||
transfer_hours_str = fmt(transfer_hours, precision=0, commas=False)
|
||
transfer_days_10g_str = fmt(transfer_days_10g, precision=0, commas=False)
|
||
transfer_time_10g_md = md_math(f"T = D_{{vol}}/BW \\approx {transfer_days_10g_str} \\text{{ days}}")
|
||
transfer_cost_str = fmt(transfer_cost, precision=0, commas=True)
|
||
tpu_hours_str = fmt(equiv_tpu_hours, precision=0, commas=True)
|
||
network_gbs_str = fmt(network_gbs, precision=1, commas=False)
|
||
dataset_gb_str = f"{dataset_gb:,.0f}"
|
||
network_gbps_str = f"{network_gbps:.0f}"
|
||
egress_cost_per_gb_str = fmt(egress_cost_gb, precision=2, commas=False)
|
||
tpu_cost_per_hour_str = fmt(tpu_hourly_cost, precision=1, commas=False)
|
||
```
|
||
|
||
::: {.callout-notebook title="The Physics of Data Gravity"}
|
||
**Problem**: You have a **1 PB** training dataset in a US East data center. You want to train a model using a TPU pod in US West. Is it faster to move the data or the compute?
|
||
|
||
**The Physics**:
|
||
|
||
1. **Network Bandwidth**: A dedicated `{python} DataGravity.network_gbps_str` Gbps line = `{python} DataGravity.network_gbs_str` GB/s.
|
||
2. **Transfer Time**: `{python} DataGravity.dataset_gb_str` GB / `{python} DataGravity.network_gbs_str` GB/s = `{python} DataGravity.transfer_seconds_str` seconds ≈ **`{python} DataGravity.transfer_hours_str` hours**.
|
||
3. **Cost**: At USD `{python} DataGravity.egress_cost_per_gb_str`/GB egress, moving 1 PB costs **USD `{python} DataGravity.transfer_cost_str`**. (Baseline: AWS data transfer out pricing, 2024.)
|
||
|
||
**The Engineering Conclusion**:
|
||
If training takes < `{python} DataGravity.transfer_hours_str` hours, you spend more time moving data than training.
|
||
If training costs < USD `{python} DataGravity.transfer_cost_str` (approx. `{python} DataGravity.tpu_hours_str` TPUv4-hours), you spend more on bandwidth than compute.
|
||
|
||
**Rule of Thumb**: For petabyte-scale data, **Code moves to Data**.\index{Data Locality!code-to-data} For gigabyte-scale data, **Data moves to Code**.\index{Data Locality!data-to-code}
|
||
:::
|
||
|
||
These physical constraints govern every decision in production data pipelines. Before proceeding, verify your intuition about these fundamentals.
|
||
|
||
::: {.callout-checkpoint title="The Physics of Data" collapse="false"}
|
||
Data engineering is governed by physical costs. Check your intuition:
|
||
|
||
- [ ] Do you understand **Data Gravity**: why petabyte-scale datasets force compute to move to the data?
|
||
- [ ] Can you explain the **Energy-Movement Invariant**: why moving a byte of data costs orders of magnitude more energy than processing it?
|
||
- [ ] Can you define **Information Entropy** in this context: why a smaller, diverse dataset can be more valuable than a massive, redundant one?
|
||
:::
|
||
|
||
These physical properties impose hard constraints on every pipeline decision: where to store data, how to transform it, and when to move computation rather than bytes. But physics alone does not prevent failures—it merely defines the boundaries within which engineering decisions must be made. A team that understands data gravity perfectly can still build a brittle pipeline if quality checks are ad hoc, error handling is absent, or governance is an afterthought. Translating physical constraints into reliable practice requires a systematic framework that organizes design decisions across every pipeline stage.
|
||
|
||
## Four Pillars Framework {#sec-data-engineering-four-pillars-framework-4ef1}
|
||
|
||
A recommendation system rejects every applicant from a region because an upstream team changed a zip code field from integer to string. A medical imaging model degrades silently for months because camera hardware changed at a partner hospital. A fraud detection system misses a new attack vector because its training data was six months stale. Each failure traces to a different root cause—schema drift, distribution shift, data staleness—yet all share a common pattern: ad hoc data engineering decisions that interacted in ways no one anticipated until deployment. The Four Pillars Framework organizes these concerns into four interdependent dimensions: **Quality**, **Reliability**, **Scalability**, and **Governance**. We begin with the cascading failure patterns that motivate this framework, then define each pillar.
|
||
|
||
### Data Cascades {#sec-data-engineering-data-cascades-systematic-foundations-matter-2efe}
|
||
|
||
Machine learning systems face a unique failure pattern called "Data Cascades,"\index{Data Cascades!definition}\index{Pipeline!cascading failures}[^fn-data-cascades-silent] where poor data quality in early stages amplifies throughout the entire pipeline, causing downstream model failures, project termination, and potential user harm [@sambasivan2021everyone]. Traditional software produces immediate errors when encountering bad inputs. ML systems degrade silently until quality issues become severe enough to necessitate complete system rebuilds.
|
||
|
||
[^fn-data-cascades-silent]: **Data Cascades**: The failure is "silent" because ML data pipelines are designed for numerical tolerance, processing corrupted inputs without the immediate, hard-coded exceptions of traditional software. This allows the initial error to amplify through every downstream stage—from feature engineering to model training—until it finally surfaces as a drop in production accuracy. By then, the root cause is so embedded that the only viable solution is a complete system rebuild, a decision often triggered only after a >5% performance degradation. \index{Data Cascades!silent failure}
|
||
|
||
Trace the arrows in @fig-cascades to follow how a single data collection error propagates through every subsequent pipeline stage. Pay particular attention to how lapses at the earliest stage only surface during model evaluation and deployment—by which point the team may need to abandon the entire model and restart. This pattern motivates a systematic framework guiding technical choices from acquisition through deployment.
|
||
|
||
::: {#fig-cascades fig-env="figure" fig-pos="htb" fig-cap="**Data Quality Cascades**: Errors introduced early in the machine learning workflow amplify across subsequent stages, increasing costs and potentially leading to flawed predictions or harmful outcomes. Source: [@sambasivan2021everyone]." fig-alt="Timeline with 7 stages from problem statement to deployment. Colored arcs show errors from data collection propagating to evaluation and deployment stages."}
|
||
```{.tikz}
|
||
\begin{tikzpicture}[line join=round,font=\small\usefont{T1}{phv}{m}{n}]
|
||
\definecolor{Green}{HTML}{008F45}
|
||
\definecolor{Red}{HTML}{CB202D}
|
||
\definecolor{Orange}{HTML}{CC5500}
|
||
\definecolor{Blue}{HTML}{006395}
|
||
\definecolor{Violet}{HTML}{7030A0}
|
||
|
||
\tikzset{%
|
||
Line/.style={line width=1.0pt,black!50,shorten <=6pt,shorten >=8pt},
|
||
LineD/.style={line width=2.0pt,black!50,shorten <=6pt,shorten >=8pt},
|
||
Text/.style={rotate=60,align=right,anchor=north east,font=\footnotesize\usefont{T1}{phv}{m}{n}},
|
||
Text2/.style={align=left,anchor=north west,font=\footnotesize\usefont{T1}{phv}{m}{n},text depth=0.7}
|
||
}
|
||
|
||
\draw[line width=1.5pt,black!30](0,0)coordinate(P)--(10,0)coordinate(K);
|
||
|
||
\foreach \i in {0,...,6} {
|
||
\path let \n1 = {(\i/6)*10} in coordinate (P\i) at (\n1,0);
|
||
\fill[black] (P\i) circle (2pt);
|
||
}
|
||
|
||
\draw[LineD,Red](P0)to[out=60,in=120](P6);
|
||
\draw[LineD,Red](P0)to[out=60,in=125](P5);
|
||
\draw[LineD,Blue](P1)to[out=60,in=120](P6);
|
||
\draw[LineD,Red](P1)to[out=50,in=125](P6);
|
||
\draw[LineD,Blue](P4)to[out=60,in=125](P6);
|
||
\draw[LineD,Blue](P3)to[out=60,in=120](P6);
|
||
%
|
||
\draw[Line,Orange](P1)to[out=44,in=132](P6);
|
||
\draw[Line,Green](P1)to[out=38,in=135](P6);
|
||
\draw[Line,Orange](P1)to[out=30,in=135](P5);
|
||
\draw[Line,Green](P1)to[out=36,in=130](P5);
|
||
%
|
||
\draw[Line,Orange](P2)to[out=40,in=135](P6);
|
||
\draw[Line,Orange](P2)to[out=40,in=135](P5);
|
||
%
|
||
\draw[draw=none,fill=VioletLine!50]($(P5)+(-0.1,0.15)$)to[bend left=10]($(P5)+(-0.1,0.61)$)--
|
||
($(P5)+(-0.25,0.50)$)--($(P5)+(-0.85,1.20)$)to[bend left=20]($(P5)+(-1.38,0.76)$)--
|
||
($(P5)+(-0.51,0.33)$)to[bend left=10]($(P5)+(-0.64,0.22)$)to[bend left=10]cycle;
|
||
\draw[draw=none,fill=VioletLine!50]($(P6)+(-0.1,0.15)$)to[bend left=10]($(P6)+(-0.1,0.61)$)--
|
||
($(P6)+(-0.25,0.50)$)--($(P6)+(-0.7,1.30)$)to[bend left=20]($(P6)+(-1.38,0.70)$)--
|
||
($(P6)+(-0.51,0.33)$)to[bend left=10]($(P6)+(-0.64,0.22)$)to[bend left=10]cycle;
|
||
%
|
||
\draw[dashed,red,thick,-latex](P1)--++(90:2)to[out=90,in=0](0.8,2.7);
|
||
\draw[dashed,red,thick,-latex](P6)--++(90:2)to[out=90,in=0](9.1,2.7);
|
||
\node[below=0.1of P0,Text]{Problem\\ Statement};
|
||
\node[below=0.1of P1,Text]{Data collection \\and labeling};
|
||
\node[below=0.1of P2,Text]{Data analysis\\ and cleaning};
|
||
\node[below=0.1of P3,Text]{Model \\selection};
|
||
\node[below=0.1of P4,Text]{Model\\ training};
|
||
\node[below=0.1of P5,Text]{Model\\ evaluation};
|
||
\node[below=0.1of P6,Text]{Model\\ deployment};
|
||
%Legend
|
||
\node[circle,minimum size=4pt,fill=Blue](L1)at(11.5,2.6){};
|
||
\node[above right=0.1 and 0.1of L1,Text2]{Interacting with physical\\ world brittleness};
|
||
|
||
\node[circle,minimum size=4pt,fill=Red,below =0.5 of L1](L2){};
|
||
\node[above right=0.1 and 0.1of L2,Text2]{Inadequate \\application-domain expertise};
|
||
|
||
\node[circle,minimum size=4pt,fill=Green,below =0.5 of L2](L3){};
|
||
\node[above right=0.1 and 0.1of L3,Text2]{Conflicting reward\\ systems};
|
||
|
||
\node[circle,minimum size=4pt,fill=Orange,below =0.5 of L3](L4){};
|
||
\node[above right=0.1 and 0.1of L4,Text2]{Poor cross-organizational\\ documentation};
|
||
|
||
\draw[-{Triangle[width=8pt,length=8pt]}, line width=3pt,Violet](11.4,-0.85)--++(0:0.8)coordinate(L5);
|
||
\node[above right=0.23 and 0of L5,Text2]{Impacts of cascades};
|
||
|
||
\draw[-{Triangle[width=4pt,length=8pt]}, line width=2pt,Red,dashed](11.4,-1.35)--++(0:0.8)coordinate(L6);
|
||
\node[above right=0.23 and 0of L6,Text2]{Abandon / re-start process};
|
||
\end{tikzpicture}
|
||
```
|
||
:::
|
||
|
||
Data cascades occur when teams skip establishing clear quality criteria, reliability requirements, and governance principles before beginning data collection and processing. Preventing them requires a systematic framework that guides technical choices from data acquisition through production deployment. The cascade pattern from @fig-cascades becomes concrete when examining how a single upstream change propagates through a *pipeline jungle* without validation.
|
||
|
||
::: {.callout-example title="The Pipeline Jungle"}
|
||
|
||
\index{Pipeline Jungle}\index{Data Contracts!schema validation}**The Failure**: A credit scoring model suddenly started rejecting all applicants from a specific region.
|
||
|
||
**The Root Cause**: An upstream team changed the schema of the `zip_code` field from `integer` to `string` to handle international codes.
|
||
* The data pipeline silently cast "02139" (string) to 2139 (integer).
|
||
* The leading zero was lost.
|
||
* The model, treating `zip_code` as a categorical feature, saw "2139" as a completely new, unknown category and defaulted to "high risk" behavior.
|
||
|
||
**The Systems Lesson**: This is a **Pipeline Jungle** failure. Without explicit **Data Contracts** and schema validation at the ingestion interface, changes in one system ("we need string zip codes") cause catastrophic, silent failures in downstream systems. Data engineering is the defense against this entropy.
|
||
:::
|
||
|
||
With the cascade pattern understood, we now define the four pillars that prevent these failures.
|
||
|
||
### Four Foundational Pillars {#sec-data-engineering-four-foundational-pillars-c119}
|
||
|
||
Every data engineering decision, from choosing storage formats to designing ingestion pipelines, should be evaluated against four principles. Study @fig-four-pillars to see how these pillars surround and support the central ML data system—note especially the dashed trade-off lines connecting them, which highlight that strengthening one pillar can create tension with another. Balancing these interdependencies is the core challenge of systematic data engineering.
|
||
|
||
::: {#fig-four-pillars fig-env="figure" fig-pos="htb" fig-cap="**The Four Pillars of Data Engineering**: Quality, Reliability, Scalability, and Governance form the foundational framework for ML data systems. Each pillar contributes essential capabilities (solid arrows), while trade-offs between pillars (dashed lines) require careful balancing: validation overhead affects throughput, consistency constraints limit distributed scale, privacy requirements impact performance, and bias mitigation may reduce available training data." fig-alt="Four boxes labeled Quality, Reliability, Scalability, and Governance surround a central ML Data System circle. Solid arrows connect each box to center showing contributions; dashed lines between boxes indicate trade-offs."}
|
||
```{.tikz}
|
||
\begin{tikzpicture}[line join=round,font=\usefont{T1}{phv}{m}{n}]
|
||
\tikzset{
|
||
Box/.style={align=center, inner xsep=2pt,draw=GreenLine, line width=1pt,fill=none, minimum width=45mm, minimum height=25mm},
|
||
Circle1/.style={circle, minimum size=33mm, draw=none, fill=BrownLine!20},
|
||
LineD/.style={dashed,BrownLine!70, line width=1.1pt,latex-latex,text=black},
|
||
LineA/.style={violet!50,line width=4.0pt,{{Triangle[width=1.5*6pt,length=2.0*5pt]}-{Triangle[width=1.5*6pt,length=2.0*5pt]}},shorten <=1pt,shorten >=1pt},
|
||
ALine/.style={black!50, line width=1.1pt,{{Triangle[width=0.9*6pt,length=1.2*6pt]}-}},
|
||
Larrow/.style={fill=violet!50, single arrow, inner sep=2pt, single arrow head extend=3pt,
|
||
single arrow head indent=0pt,minimum height=10mm, minimum width=3pt}
|
||
}
|
||
%%%
|
||
%vaga
|
||
\tikzset{
|
||
pics/vaga/.style = {
|
||
code = {
|
||
\pgfkeys{/channel/.cd, #1}
|
||
\begin{scope}[shift={($(0,0)+(0,0)$)},scale=\scalefac,every node/.append style={transform shape}]
|
||
\node[rectangle,minimum width=2mm,minimum height=22mm,
|
||
draw=none, fill=\filllcolor,line width=\Linewidth](1R) at (0,-0.95){};
|
||
\fill[fill=\filllcolor!60!black](230:2.8)arc(230:310:2.8)--cycle;%circle(2.9);
|
||
%LT
|
||
\node [semicircle, shape border rotate=180, anchor=chord center,
|
||
minimum size=11mm, draw=none, fill=\filllcirclecolor](LT) at (-2,-0.5) {};
|
||
\node [circle, minimum size=4mm, draw=none, fill=\filllcirclecolor](T1) at (-2,1.25) {};
|
||
\draw[draw=\drawcolor,,line width=1.2*\Linewidth,shorten <=3pt,shorten >=3pt](T1)--(LT);
|
||
\draw[draw=\drawcolor,,line width=1.2*\Linewidth,shorten <=3pt,shorten >=3pt](T1)--(LT.30);
|
||
\draw[draw=\drawcolor,,line width=1.2*\Linewidth,shorten <=3pt,shorten >=3pt](T1)--(LT.150);
|
||
%DT
|
||
\node [semicircle, shape border rotate=180, anchor=chord center,
|
||
minimum size=11mm, draw=none, fill=\filllcirclecolor!70!black](DT) at (2,-0.5) {};
|
||
\node [circle, minimum size=4mm, draw=none, fill=\filllcirclecolor!70!black](T2) at (2,1.25) {};
|
||
\draw[draw=\drawcolor,line width=1.2*\Linewidth,shorten <=3pt,shorten >=3pt](T2)--(DT);
|
||
\draw[draw=\drawcolor,,line width=1.2*\Linewidth,shorten <=3pt,shorten >=3pt](T2)--(DT.30);
|
||
\draw[draw=\drawcolor,,line width=1.2*\Linewidth,shorten <=3pt,shorten >=3pt](T2)--(DT.150);
|
||
%
|
||
\node[draw=none,rectangle,minimum width=32mm,minimum height=1.5mm,inner sep=0pt,
|
||
fill=\filllcolor!60!black]at(0,1.25){};
|
||
\node[draw=white,fill=\filllcolor,line width=2*\Linewidth,ellipse,minimum width=9mm, minimum height=15mm](EL)at(0,0.85){};
|
||
\node[draw=white,fill=\filllcolor!60!black,line width=2*\Linewidth,,circle,minimum size=10mm](2C)at(0,2.05){};
|
||
\end{scope}
|
||
}
|
||
}
|
||
}
|
||
%stit
|
||
\def\inset{3.2pt} %
|
||
\def\myshape{%
|
||
(0,1.34) to[out=220,in=0] (-1.20,1.03) --
|
||
(-1.20,-0.23) to[out=280,in=160] (0,-1.53) to[out=20,in=260] (1.20,-0.23) --
|
||
(1.20,1.03) to[out=180,in=320] cycle
|
||
}
|
||
\tikzset{
|
||
pics/stit/.style = {
|
||
code = {
|
||
\pgfkeys{/channel/.cd, #1}
|
||
\begin{scope}[shift={($(0,0)+(0,0)$)},scale=\scalefac,every node/.append style={transform shape}]
|
||
\fill[fill=\filllcolor!60] \myshape;
|
||
%
|
||
\begin{scope}
|
||
\clip \myshape;
|
||
\draw[draw=\filllcolor!60, line width=2*\inset,fill=white] \myshape; % stroke color and width
|
||
\end{scope}
|
||
\fill[fill=\filllcolor!60](0,0)circle(0.4)coordinate(ST\picname);
|
||
\end{scope}
|
||
}
|
||
}
|
||
}
|
||
%AI style
|
||
\tikzset{
|
||
pics/llm/.style = {
|
||
code = {
|
||
\pgfkeys{/channel/.cd, #1}
|
||
\begin{scope}[shift={($(0,0)+(0,0)$)},scale=\scalefac,every node/.append style={transform shape}]
|
||
\node[circle,minimum size=12mm,draw=\drawcolor, fill=\filllcolor!70,line width=1.25*\Linewidth](C\picname) at (0,0){};
|
||
\def\startangle{90}
|
||
\def\radius{1.15}
|
||
\def\radiusI{1.1}
|
||
\foreach \i [evaluate=\i as \j using \i+1] [count =\k] in {0,2,4,6,8} {
|
||
\pgfmathsetmacro{\angle}{\startangle - \i * (360/8)}
|
||
\draw[draw=black,-{Circle[black ,fill=\filllcirclecolor,length=5.5pt,line width=0.5*\Linewidth]},line width=1.5*\Linewidth](C\picname)--++(\startangle - \i*45:\radius) ;
|
||
\node[circle,draw=black,fill=\filllcirclecolor!80!red!50,inner sep=3pt,line width=0.5*\Linewidth](2C\k)at(\startangle - \j*45:\radiusI) {};
|
||
}
|
||
\draw[line width=1.5*\Linewidth](2C1)--++(-0.5,0)|-(2C2);
|
||
\draw[line width=1.5*\Linewidth](2C3)--++(0.5,0)|-(2C4);
|
||
\node[circle,,minimum size=12mm,draw=\drawcolor, fill=\filllcolor!70,line width=0.5*\Linewidth]at (0,0){};
|
||
\end{scope}
|
||
}
|
||
}
|
||
}
|
||
%brain
|
||
\tikzset{pics/brain/.style = {
|
||
code = {
|
||
\pgfkeys{/channel/.cd, #1}
|
||
\begin{scope}[local bounding box=BRAIN,scale=\scalefac, every node/.append style={transform shape}]
|
||
\draw[fill=\filllcolor,line width=\Linewidth](-0.3,-0.10)to(0.08,0.60)
|
||
to[out=60,in=50,distance=3](-0.1,0.69)to[out=160,in=80](-0.26,0.59)to[out=170,in=90](-0.46,0.42)
|
||
to[out=170,in=110](-0.54,0.25)to[out=210,in=150](-0.54,0.04)
|
||
to[out=240,in=130](-0.52,-0.1)to[out=300,in=240]cycle;
|
||
\draw[fill=\filllcolor,line width=\Linewidth]
|
||
(-0.04,0.64)to[out=120,in=0](-0.1,0.69)(-0.19,0.52)to[out=120,in=330](-0.26,0.59)
|
||
(-0.4,0.33)to[out=150,in=280](-0.46,0.42)
|
||
%
|
||
(-0.44,-0.03)to[bend left=30](-0.34,-0.04)
|
||
(-0.33,0.08)to[bend left=40](-0.37,0.2) (-0.37,0.12)to[bend left=40](-0.45,0.14)
|
||
(-0.26,0.2)to[bend left=30](-0.24,0.13)
|
||
(-0.16,0.32)to[bend right=30](-0.27,0.3)to[bend right=30](-0.29,0.38)
|
||
(-0.13,0.49)to[bend left=30](-0.04,0.51);
|
||
\draw[thick,rounded corners=0.8pt,\drawcircle,-{Circle[fill=\filllcolor,length=2.5pt]}](-0.23,0.03)--(-0.15,-0.03)--(-0.19,-0.18)--(-0.04,-0.28);
|
||
\draw[thick,rounded corners=0.8pt,\drawcircle,-{Circle[fill=\filllcolor,length=2.5pt]}](-0.17,0.13)--(-0.04,0.05)--(-0.06,-0.06)--(0.14,-0.11);
|
||
\draw[thick,rounded corners=0.8pt,\drawcircle,-{Circle[fill=\filllcolor,length=2.5pt]}](-0.12,0.23)--(0.31,0.0);
|
||
\draw[thick,rounded corners=0.8pt,\drawcircle,-{Circle[fill=\filllcolor,length=2.5pt]}](-0.07,0.32)--(0.06,0.26)--(0.16,0.33)--(0.34,0.2);
|
||
\draw[thick,rounded corners=0.8pt,\drawcircle,-{Circle[fill=\filllcolor,length=2.5pt]}](-0.01,0.43)--(0.06,0.39)--(0.18,0.51)--(0.31,0.4);
|
||
\end{scope}
|
||
}
|
||
}
|
||
}
|
||
%graph
|
||
\tikzset{pics/graph/.style = {
|
||
code = {
|
||
\pgfkeys{/channel/.cd, #1}
|
||
\begin{scope}[local bounding box=GRAPH,scale=\scalefac, every node/.append style={transform shape}]
|
||
\draw[line width=2*\Linewidth,draw = \drawcolor](-0.20,0)--(2.2,0);
|
||
\draw[line width=2*\Linewidth,draw = \drawcolor](-0.20,0)--(-0.20,2.0);
|
||
\foreach \i/\vi in {0/4,0.5/8,1/12,1.5/16}{
|
||
\node[draw, minimum width =4mm, minimum height = \vi mm, inner sep = 0pt,
|
||
draw = \drawcolor, fill=\filllcolor!50, line width=\Linewidth,anchor=south west](COM)at(\i,0.2){};
|
||
}
|
||
%lupa
|
||
\coordinate(PO)at(1.2,0.9);
|
||
\node[circle,draw=white,line width=0.75pt,fill=\filllcirclecolor,minimum size=9mm,inner sep=0pt](LV)at(PO){};
|
||
\node[draw=none,rotate=40,rounded corners=2pt,rectangle,minimum width=2.2mm,inner sep=1pt,
|
||
fill=\filllcirclecolor,minimum height=11mm,anchor=north]at(PO){};
|
||
\node[circle,draw=none,fill=white,minimum size=5.0mm,inner sep=0pt](LM)at(PO){};
|
||
\node[font=\small\bfseries]at(LM){...};
|
||
\end{scope}
|
||
}
|
||
}
|
||
}
|
||
%target
|
||
\tikzset{
|
||
pics/target/.style = {
|
||
code = {
|
||
\pgfkeys{/channel/.cd, #1}
|
||
\begin{scope}[shift={($(0,0)+(0,0)$)},scale=\scalefac,every node/.append style={transform shape}]
|
||
\definecolor{col1}{RGB}{62,100,125}
|
||
\definecolor{col2}{RGB}{219,253,166}
|
||
\colorlet{col1}{\filllcolor}
|
||
\colorlet{col2}{\filllcirclecolor}
|
||
\foreach\i/\col [count=\k]in {22mm/col1,17mm/col2,12mm/col1,7mm/col2,2.5mm/col1}{
|
||
\node[circle,inner sep=0pt,draw=\drawcolor,fill=\col,minimum size=\i,line width=\Linewidth](C\k){};
|
||
}
|
||
\draw[thick,fill=brown,xscale=-1](0,0)--++(111:0.13)--++(135:1)--++(225:0.1)--++(315:1)--cycle;
|
||
\path[green,xscale=-1](0,0)--(135:0.85)coordinate(XS1);
|
||
\draw[thick,fill=yellow,xscale=-1](XS1)--++(80:0.2)--++(135:0.37)--++(260:0.2)--++(190:0.2)--++(315:0.37)--cycle;
|
||
\end{scope}
|
||
}
|
||
}
|
||
}
|
||
%server
|
||
\tikzset {
|
||
pics/server/.style = {
|
||
code = {
|
||
% \colorlet{red}{black}
|
||
\pgfkeys{/channel/.cd, #1}
|
||
\begin{scope}[anchor=center, transform shape,scale=\scalefac, every node/.append style={transform shape}]
|
||
\draw[draw=\drawcolor,line width=\Linewidth,fill=\filllcolor](-0.55,-0.5) rectangle (0.55,0.5);
|
||
\foreach \i in {-0.25,0,0.25} {
|
||
\draw[line width=\Linewidth]( -0.55,\i) -- (0.55, \i);
|
||
}
|
||
\foreach \i in {-0.375, -0.125, 0.125, 0.375} {
|
||
\draw[line width=\Linewidth](-0.45,\i)--(0,\i);
|
||
\fill[](0.35,\i) circle (1.5pt);
|
||
}
|
||
|
||
\draw[draw=\drawcolor,line width=1.5*\Linewidth](0,-0.53) |- (-0.55,-0.7);
|
||
\draw[draw=\drawcolor,line width=1.5*\Linewidth](0,-0.53) |- (0.55,-0.7);
|
||
\end{scope}
|
||
}
|
||
}
|
||
}
|
||
\pgfkeys{
|
||
/channel/.cd,
|
||
Depth/.store in=\Depth,
|
||
Height/.store in=\Height,
|
||
Width/.store in=\Width,
|
||
filllcirclecolor/.store in=\filllcirclecolor,
|
||
filllcolor/.store in=\filllcolor,
|
||
drawcolor/.store in=\drawcolor,
|
||
drawcircle/.store in=\drawcircle,
|
||
scalefac/.store in=\scalefac,
|
||
Linewidth/.store in=\Linewidth,
|
||
picname/.store in=\picname,
|
||
filllcolor=BrownLine,
|
||
filllcirclecolor=violet!20,
|
||
drawcolor=black,
|
||
drawcircle=violet,
|
||
scalefac=1,
|
||
Linewidth=0.5pt,
|
||
Depth=1.3,
|
||
Height=0.8,
|
||
Width=1.1,
|
||
picname=C
|
||
}
|
||
|
||
\node[Circle1](CI1){};
|
||
%AI
|
||
\pic[shift={(0,0)}] at (CI1){llm={scalefac=1.2,picname=1,drawcolor=GreenD,filllcolor=GreenD!20!, Linewidth=1pt,filllcirclecolor=red}};
|
||
%brain
|
||
\pic[shift={(0.12,-0.23)}] at (C1){brain={scalefac=1.1,picname=2,filllcolor=orange!30!, filllcirclecolor=cyan!55!black!60, Linewidth=0.75pt}};
|
||
%Quality
|
||
\node[Box,above left=1 and 1.5 of CI1](B1){};
|
||
\node[below=1pt of CI1,font=\usefont{T1}{phv}{b}{n}\small,align=center]{ML Data System};
|
||
\fill[green!07](B1.north west) rectangle ($(B1.north east)!0.6!(B1.south east)$)coordinate(B1DE);
|
||
\fill[green!20](B1.south east) rectangle ($(B1.north west)!0.6!(B1.south west)$)coordinate(B1LE);
|
||
\node[Box,above left=1 and 1.5 of CI1](){};
|
||
\tikzset{Text2/.style={font=\usefont{T1}{phv}{b}{n}\small,align=center}}
|
||
\node[Text2]at($(B1.south west)!0.5!(B1DE)$){Quality\\ {\footnotesize Accuracy \& Fitness}};
|
||
\coordinate(Q1)at($(B1.north west)!0.5!(B1DE)$);
|
||
%Quality - target
|
||
\pic[shift={(0,0)}] at (Q1){target={scalefac=0.55,picname=1,drawcolor=BlueD,filllcolor=cyan!90!,Linewidth=0.7pt, filllcirclecolor=cyan!20}};
|
||
%Reliability - stit
|
||
\node[Box,above right=1 and 1.5 of CI1](B2){};
|
||
\fill[cyan!07](B2.north west) rectangle ($(B2.north east)!0.6!(B2.south east)$)coordinate(B2DE);
|
||
\fill[cyan!20](B2.south east) rectangle ($(B2.north west)!0.6!(B2.south west)$)coordinate(B2LE);
|
||
\node[Text2]at($(B2.south west)!0.5!(B2DE)$){Reliability\\ {\footnotesize Consistency \& Fault Tolerance}};
|
||
\coordinate(R1)at($(B2.north west)!0.5!(B2DE)$);
|
||
\node[Box,above right=1 and 1.5 of CI1,draw=BlueD](B2){};
|
||
%Reliability - stit
|
||
\pic[shift={(0,0.03)}] at (R1){stit={scalefac=0.48,picname=1,drawcolor=orange,filllcolor=red!80!}};
|
||
\pic[shift={(0,0.03)}] at (ST1){server={scalefac=0.52,picname=1,drawcolor= black,filllcolor=orange!30!,Linewidth=0.75pt}};
|
||
%governance
|
||
\node[Box,below left=1 and 1.5 of CI1](B3){};
|
||
\fill[violet!07](B3.north west) rectangle ($(B3.north east)!0.6!(B3.south east)$)coordinate(B3DE);
|
||
\fill[violet!20](B3.south east) rectangle ($(B3.north west)!0.6!(B3.south west)$)coordinate(B3LE);
|
||
\node[Text2]at($(B3.south west)!0.5!(B3DE)$){Governance\\ {\footnotesize Ethics \& Compliance}};
|
||
\coordinate(G1)at($(B3.north west)!0.5!(B3DE)$);
|
||
\node[Box,below left=1 and 1.5 of CI1,draw=violet](){};
|
||
%Governance - icon
|
||
\pic[shift={(-0.70,-0.6)}] at (G1){graph={scalefac=0.6,picname=1,filllcirclecolor=RedLine,filllcolor=green!70!black, Linewidth=0.65pt}};
|
||
%Scalability - graph
|
||
\node[Box,below right=1 and 1.5 of CI1](B4){};
|
||
\fill[orange!07](B4.north west) rectangle ($(B4.north east)!0.6!(B4.south east)$)coordinate(B4DE);
|
||
\fill[orange!20](B4.south east) rectangle ($(B4.north west)!0.6!(B4.south west)$)coordinate(B4LE);
|
||
\node[Text2]at($(B4.south west)!0.5!(B4DE)$){Scalability\\ {\footnotesize Growth \& Performance}};
|
||
\coordinate(S1)at($(B4.north west)!0.5!(B4DE)$);
|
||
\node[Box,below right=1 and 1.5 of CI1,draw=OrangeLine](){};
|
||
%governance
|
||
\pic[shift={(0,0.05)}] at (S1){vaga={scalefac=0.25,picname=1,filllcolor=BlueLine, Linewidth=0.75pt,filllcirclecolor=orange}};
|
||
%arrows
|
||
\tikzset{Text/.style={,font=\usefont{T1}{phv}{m}{n}\small,align=center}}
|
||
\draw[LineD](B1)--node[above,Text]{Validation overhead vs.\\ throughput}(B2);
|
||
\draw[LineD](B1)--node[left,Text]{Bias mitigation vs.\\ data availability}(B3);
|
||
\draw[LineD](B2)--node[right,Text]{Consistency vs.\\ distributed scale}(B4);
|
||
\draw[LineD](B3)--node[below,Text]{Performance vs.\\ privacy constraints}(B4);
|
||
%
|
||
\tikzset{Text1/.style={font=\usefont{T1}{phv}{m}{n}\footnotesize,align=center,text=black}}
|
||
\draw[LineA,draw=green!70!black](B1.south east)--node[below left,Text1,green!60!black]{High-quality\\ training data}(CI1);
|
||
\draw[LineA,draw=cyan!70!black](B2.south west)--node[below right,Text1,cyan!70!black]{Consistent\\ processing}(CI1);
|
||
\draw[LineA,draw=violet!80!black!40](B3.north east)--node[above left,Text1,violet]{Compliance \& \\accountability}(CI1);
|
||
\draw[LineA,draw=orange!80](B4.north west)--node[above right,Text1,orange]{Handle growing\\ data volumes}(CI1);
|
||
\end{tikzpicture}
|
||
```
|
||
:::
|
||
|
||
Data quality provides the foundation, as the cascade pattern in @sec-data-engineering-data-cascades-systematic-foundations-matter-2efe demonstrates: when upstream data is wrong, everything downstream fails. Quality encompasses not just correctness (are values accurate?) but also coverage (does the data represent the deployment environment?) and freshness (does the distribution still match production?). The quality pillar motivates validation, monitoring, and drift detection infrastructure examined throughout this chapter.
|
||
|
||
Quality alone is insufficient, however, if the systems delivering that data are fragile. Reliability means building pipelines that continue operating despite component failures, data anomalies, or unexpected load spikes. A pipeline that produces perfect features 99% of the time but crashes during traffic surges delivers no value during the most critical moments. Error handling, circuit breakers, and graceful degradation strategies transform brittle pipelines into resilient ones.
|
||
|
||
Even reliable, high-quality pipelines face a third constraint: they must keep pace with growth. Scalability addresses this challenge as ML systems evolve from prototypes handling gigabytes to production services processing petabytes. Systems must handle growing data volumes, user bases, and computational demands without complete redesigns. Scalability must also be cost-effective, since raw capacity means little if infrastructure costs grow faster than business value. Our recommendation lighthouse illustrates this challenge at its most extreme.
|
||
|
||
::: {.callout-lighthouse title="DLRM (Recommendation Lighthouse)"}
|
||
|
||
\index{DLRM!scalability challenge}**Why it matters:** Recommendation systems like DLRM exemplify the **scalability** challenge of modern data engineering. They rely on high-cardinality categorical features (like User IDs or Product IDs) that must be mapped to dense vectors via embedding tables.
|
||
|
||
| **Property** | **Value** | **System Implication** |
|
||
|:---------------|:---------------------|:----------------------------------------------------------|
|
||
| **Data Scale** | Billion+ users/items | Embedding tables grow to TB/PB scale. |
|
||
| **Constraint** | Memory Capacity | Tables exceed single-GPU memory; requires sharding. |
|
||
| **Bottleneck** | Sparse Access | Random lookups stress memory bandwidth more than compute. |
|
||
|
||
Unlike ResNet (compute-bound) or GPT-2 (bandwidth-bound), DLRM is limited by **memory capacity** and the sheer logistics of storing and accessing massive lookup tables efficiently.
|
||
:::
|
||
|
||
Governance provides the framework within which quality, reliability, and scalability operate. A perfectly scalable, reliable, high-quality pipeline that violates GDPR or perpetuates demographic biases creates liability rather than value. Data governance ensures systems operate within legal, ethical, and business constraints—privacy protection, bias mitigation, regulatory compliance, and clear data ownership—while maintaining the transparency and accountability necessary to demonstrate compliance when audited.
|
||
|
||
When ML systems exhibit failures, the four pillars provide a diagnostic lens for identifying root causes. Model accuracy that degrades gradually—not through sudden breakage but through a slow downward trend—almost always traces to the quality pillar: data drift has shifted the serving distribution away from training, or label quality has degraded as annotator pools change. Pipeline failures that appear intermittent—succeeding most runs but crashing unpredictably—point to the reliability pillar, suggesting inadequate error handling, missing retry logic, or resource exhaustion under peak load. Training that takes too long despite adequate hardware implicates scalability: somewhere in the pipeline a bottleneck—a single-threaded transformation, an unpartitioned shuffle, a storage tier too slow for the data volume—prevents parallel resources from being used. Compliance gaps discovered during audits indicate governance debt: lineage tracking is incomplete, access controls are stale, or data retention policies have not kept pace with regulatory changes.
|
||
|
||
The most insidious failures span multiple pillars. Features that differ between training and serving implicate both quality (the values are wrong) and reliability (the computation is inconsistent). Diagnosing such cross-pillar failures requires checking consistency contracts, comparing feature distributions across environments, and tracing transformation lineage—techniques examined in detail throughout this chapter. The key diagnostic insight is that most practitioners instinctively investigate the model first, but production experience consistently shows that data infrastructure failures outnumber model failures by a wide margin.
|
||
|
||
### KWS Case Study {#sec-data-engineering-framework-application-keyword-spotting-case-study-c8e9}
|
||
|
||
Keyword Spotting (KWS) systems provide an ideal case study for applying our four-pillar framework to real-world data engineering challenges. These systems power voice-activated devices like smartphones and smart speakers, detecting specific wake words such as "OK, Google" or "Alexa" within continuous audio streams while operating under strict resource constraints.[^fn-kws-enrollment-pipeline]
|
||
|
||
[^fn-kws-enrollment-pipeline]: **Voice Match Enrollment**: Setting up "OK Google" requires repeating the wake phrase several times, a micro-scale data collection pipeline running on-device. Even this single-user workflow exercises all four data engineering pillars: quality (re-record if ambient noise is too high), reliability (must succeed on first attempt), scalability (model must fit in the SoC's always-on memory), and governance (voice prints stored locally, never uploaded). The enrollment constraint illustrates why data engineering is not just a cloud-scale problem -- it applies wherever data determines system behavior. \index{Voice Match!enrollment pipeline}
|
||
|
||
Look at @fig-keywords to see how a KWS system operates as a lightweight, always-on front-end that triggers more complex voice processing systems. Even this seemingly simple architecture surfaces interconnected challenges across all four pillars: Quality (accuracy across diverse environments), Reliability (consistent battery-powered operation), Scalability (severe memory constraints), and Governance (privacy protection). These constraints limit KWS systems to a few dozen languages: collecting high-quality, representative voice data for smaller linguistic populations proves prohibitively difficult. All four pillars must work together to achieve successful deployment.
|
||
|
||
{#fig-keywords width=55% fig-alt="Diagram showing voice-activated device with microphone, always-on wake word detector, and connection to main voice assistant that activates upon keyword detection."}
|
||
|
||
With this understanding established, we apply the problem definition approach to the KWS example, demonstrating how the four pillars guide practical engineering decisions.
|
||
|
||
The core problem is deceptively simple: detect specific keywords amidst ambient sounds and other spoken words, with high accuracy, low latency, and minimal false activations, on devices with severely limited computational resources. A well-specified problem definition identifies the desired keywords, the envisioned application, and the deployment scenario. The objectives that follow must balance competing requirements: performance targets of `{python} kws_accuracy_target_str`% accuracy in keyword detection with latency under `{python} kws_latency_limit_ms_str` milliseconds, alongside resource constraints demanding minimal power consumption and model sizes optimized for available device memory.
|
||
|
||
Success metrics for KWS extend beyond simple accuracy to include true positive rate (correctly identified keywords relative to all spoken keywords), false positive rate (non-keywords incorrectly identified as keywords), and detection/error tradeoff curves that compare false accepts per hour against false rejection rate on streaming audio representative of real-world deployment, as demonstrated by @nayak2022improving.
|
||
|
||
Of these metrics, the false positive rate deserves particular attention for always-on systems. Because KWS listens continuously—every second of every day—even a seemingly negligible false positive rate compounds across millions of evaluation windows. The following calculation reveals just how stringent the *false positive targets* must be.
|
||
|
||
```{python}
|
||
#| label: false-positive-calc
|
||
#| echo: false
|
||
# ┌─────────────────────────────────────────────────────────────────────────────
|
||
# │ FALSE POSITIVE CALCULATION
|
||
# ├─────────────────────────────────────────────────────────────────────────────
|
||
# │ Context: Callout "False Positive Targets"
|
||
# │
|
||
# │ Goal: Derive the false-positive-rate ceiling for an always-on KWS system.
|
||
# │ Show: How a "1 false wake-up per month" SLA translates to a >99.999% rejection requirement.
|
||
# │ How: Calculate total monthly detection windows and required accuracy per window.
|
||
# │
|
||
# │ Imports: mlsys.constants (SECONDS_PER_MINUTE, MINUTES_PER_HOUR,
|
||
# │ HOURS_PER_DAY, DAYS_PER_MONTH),
|
||
# │ mlsys.formatting (fmt)
|
||
# │ Exports: sec_str, min_str, hr_str, day_str, windows_per_month_str,
|
||
# │ fpr_str, rejection_pct_str
|
||
# └─────────────────────────────────────────────────────────────────────────────
|
||
from mlsys.constants import (
|
||
SECONDS_PER_MINUTE, MINUTES_PER_HOUR, HOURS_PER_DAY, DAYS_PER_MONTH,
|
||
SEC_PER_HOUR
|
||
)
|
||
from mlsys.formatting import fmt, check
|
||
|
||
# ┌── P.I.C.O. ISOLATED SCENARIO ───────────────────────────────────────────────
|
||
class FalsePositiveTarget:
|
||
"""
|
||
Namespace for KWS False Positive Target calculation.
|
||
Scenario: Always-on device (24h) with 1 false wake-up tolerance per month.
|
||
"""
|
||
|
||
# ┌── 1. PARAMETERS (Inputs) ───────────────────────────────────────────────
|
||
duty_cycle_hours = 24
|
||
window_sec = 1
|
||
tolerance_per_month = 1
|
||
|
||
days_month = DAYS_PER_MONTH
|
||
|
||
# ┌── 2. CALCULATION (The Physics) ─────────────────────────────────────────
|
||
windows_per_month = (days_month * duty_cycle_hours * SEC_PER_HOUR) / window_sec
|
||
target_fpr = tolerance_per_month / windows_per_month
|
||
rejection_pct = (1 - target_fpr) * 100
|
||
|
||
# ┌── 3. INVARIANTS (Guardrails) ───────────────────────────────────────────
|
||
check(rejection_pct >= 99.999, f"Rejection target ({rejection_pct:.4f}%) is too lenient. Should be > 99.999%.")
|
||
|
||
# ┌── 4. OUTPUTS (Formatting) ──────────────────────────────────────────────
|
||
sec_str = "60"
|
||
min_str = "60"
|
||
hr_str = "24"
|
||
day_str = fmt(days_month, precision=0, commas=False)
|
||
windows_per_month_str = f"{windows_per_month:,.0f}"
|
||
fpr_str = f"{target_fpr:.1e}"
|
||
rejection_pct_str = fmt(rejection_pct, precision=5, commas=False)
|
||
|
||
# ┌── EXPORTS (Bridge to Text) ─────────────────────────────────────────────────
|
||
sec_str = FalsePositiveTarget.sec_str
|
||
min_str = FalsePositiveTarget.min_str
|
||
hr_str = FalsePositiveTarget.hr_str
|
||
day_str = FalsePositiveTarget.day_str
|
||
windows_per_month_str = FalsePositiveTarget.windows_per_month_str
|
||
fpr_str = FalsePositiveTarget.fpr_str
|
||
rejection_pct_str = FalsePositiveTarget.rejection_pct_str
|
||
```
|
||
|
||
::: {.callout-notebook title="False Positive Targets"}
|
||
**Constraint**: User tolerance is max 1 false wake-up per month.
|
||
|
||
**Operational Parameters**
|
||
|
||
- **Duty Cycle**: Always-on (24 hours/day).
|
||
- **Window Size**: 1 second classification windows.
|
||
- **Windows per Month**: `{python} sec_str`$\times$ `{python} min_str`$\times$ `{python} hr_str`$\times$ `{python} day_str` = `{python} windows_per_month_str` windows.
|
||
|
||
**Required Accuracy**
|
||
|
||
- **False Positive Rate (FPR)**: 1 / windows ≈ `{python} fpr_str`
|
||
- **Precision Requirement**: `{python} rejection_pct_str`% rejection of non-keywords.
|
||
|
||
**Implication**: Standard accuracy metrics (e.g., "99% accuracy") are meaningless here. We must evaluate specifically on **False Accepts per Hour (FA/Hr)**.
|
||
:::
|
||
|
||
Operational metrics further track response time (keyword utterance to system response) and power consumption (average power used during keyword detection).
|
||
|
||
Stakeholder priorities create additional tension. Device manufacturers prioritize low power consumption, software developers emphasize ease of integration, and end users demand accuracy and responsiveness. Balancing these competing requirements shapes system architecture decisions throughout development.
|
||
|
||
Embedded device constraints impose hard boundaries on these architectural choices. Memory limitations require extremely lightweight models, often in the tens-of-kilobytes range, to fit in the always-on island of the SoC[^fn-soc-always-on]; this constraint covers only model weights, and preprocessing code must also fit within tight memory bounds. Limited computational capabilities (often a few hundred MHz of clock speed) demand aggressive model optimization. Most embedded devices run on batteries, so KWS systems target sub-milliwatt power consumption during continuous listening. Devices must also function across diverse deployment scenarios ranging from quiet bedrooms to noisy industrial settings.
|
||
|
||
[^fn-soc-always-on]: **SoC Always-On Island**: Modern System-on-Chip designs partition power domains so a low-power "always-on" island (typically achieving sub-milliwatt draw) monitors for wake triggers while the main processor sleeps. The critical constraint is that this island must hold both the model weights *and* the audio preprocessing code within its dedicated SRAM — a split budget that forces KWS architectures to optimize for total footprint, not just parameter count. \index{SoC!always-on island}
|
||
|
||
Data quality and diversity ultimately determine whether these constraints can be met. The dataset must capture demographic diversity (speakers with various accents, ages, and genders) to ensure broad recognition. Keyword variations require attention since people pronounce wake words differently, and background noise diversity proves essential for training models that perform across real-world scenarios from quiet environments to noisy conditions. Once a prototype system is developed, iterative feedback and refinement keep the system aligned with objectives as deployment scenarios evolve, requiring testing in real-world conditions and systematic refinement based on observed failure patterns.
|
||
|
||
### KWS Design Space {#sec-data-engineering-kws-design-space-eadd}
|
||
|
||
These requirements create a multi-dimensional design space where data engineering choices cascade through system performance. @tbl-kws-design-space quantifies key trade-offs, enabling principled decisions rather than ad-hoc selection.
|
||
|
||
| **Design Choice** | **Quality Impact** | **Latency Impact** | **Cost Impact** | **Memory Impact** |
|
||
|:-----------------------------------|---------------------:|:-------------------|---------------------:|:-------------------|
|
||
| **16kHz vs 8kHz sampling** | +2–4% accuracy | 2× storage | 2× processing | 2× feature size |
|
||
| **13 vs 40 MFCC coefficients** | +3–5% accuracy | 3× feature compute | Minimal | 3× feature memory |
|
||
| **1M vs 10M training examples** | +5–8% accuracy | 10× training time | 10× labeling cost | 10× storage |
|
||
| **Clean vs noisy training data** | +10–15% real-world | Minimal | 3× collection cost | Minimal |
|
||
| **Local vs cloud inference** | −2% accuracy (quant) | 10 ms vs 100 ms | \$0 vs \$0.001/query | 16 KB vs unlimited |
|
||
| **Synthetic vs real augmentation** | +3–5% robustness | Minimal | 10× cheaper | Minimal |
|
||
|
||
: **KWS Data Engineering Design Space**: Each design choice creates quantifiable trade-offs across the four pillars. Higher sampling rates improve quality but double storage and processing (scalability impact). More training data improves accuracy but multiplies labeling costs (governance/cost impact). Local inference eliminates latency but requires aggressive quantization (quality/reliability trade-off). This design space analysis guides systematic optimization rather than intuition-based decisions. {#tbl-kws-design-space}
|
||
|
||
The following worked example demonstrates how to apply this design space analysis to a concrete engineering scenario.
|
||
|
||
```{python}
|
||
#| label: budget-allocation-calc
|
||
#| echo: false
|
||
# ┌─────────────────────────────────────────────────────────────────────────────
|
||
# │ BUDGET ALLOCATION CALCULATION
|
||
# ├─────────────────────────────────────────────────────────────────────────────
|
||
# │ Context: Callout "Optimizing the KWS Design Space"
|
||
# │
|
||
# │ Goal: Demonstrate data budget planning as an engineering optimization problem.
|
||
# │ Show: How a $150K budget is allocated across labeling, storage, and governance.
|
||
# │ How: Calculate total labeled examples given labor costs and review overhead.
|
||
# │
|
||
# │ Imports: mlsys.formatting (fmt)
|
||
# │ Exports: cost_per_label_str, review_overhead_pct_str, labeling_k,
|
||
# │ storage_k, governance_k, effective_cost_str, labeled_k
|
||
# └─────────────────────────────────────────────────────────────────────────────
|
||
from mlsys.formatting import fmt
|
||
|
||
# ┌── P.I.C.O. ISOLATED SCENARIO ───────────────────────────────────────────────
|
||
class BudgetAllocation:
|
||
"""Budget planning as an engineering optimization problem."""
|
||
|
||
# ┌── 1. PARAMETERS (Inputs) ──────────────────────────────────────────────
|
||
total_budget = 150_000
|
||
labeling_pct = 0.60
|
||
storage_pct = 0.25
|
||
governance_pct = 0.15
|
||
cost_per_label = 0.10
|
||
review_overhead = 0.20
|
||
|
||
# ┌── 2. CALCULATION (The Physics) ────────────────────────────────────────
|
||
labeling_budget = total_budget * labeling_pct
|
||
storage_budget = total_budget * storage_pct
|
||
governance_budget = total_budget * governance_pct
|
||
effective_cost = cost_per_label * (1 + review_overhead)
|
||
labeled_examples = labeling_budget / effective_cost
|
||
review_overhead_pct = int(review_overhead * 100)
|
||
|
||
# ┌── 4. OUTPUTS (Formatting) ─────────────────────────────────────────────
|
||
cost_per_label_str = fmt(cost_per_label, precision=2, commas=False)
|
||
review_overhead_pct_str = fmt(review_overhead_pct, precision=0, commas=False)
|
||
labeling_k = f"{labeling_budget/1000:.0f}"
|
||
storage_k = f"{storage_budget/1000:.1f}"
|
||
governance_k = f"{governance_budget/1000:.1f}"
|
||
effective_cost_str = fmt(effective_cost, precision=2, commas=False)
|
||
labeled_k = f"{labeled_examples/1000:.0f}"
|
||
|
||
# ┌── EXPORTS (Bridge to Text) ─────────────────────────────────────────────────
|
||
cost_per_label_str = BudgetAllocation.cost_per_label_str
|
||
review_overhead_pct_str = BudgetAllocation.review_overhead_pct_str
|
||
labeling_k = BudgetAllocation.labeling_k
|
||
storage_k = BudgetAllocation.storage_k
|
||
governance_k = BudgetAllocation.governance_k
|
||
effective_cost_str = BudgetAllocation.effective_cost_str
|
||
labeled_k = BudgetAllocation.labeled_k
|
||
```
|
||
|
||
::: {.callout-example title="Optimizing the KWS Design Space"}
|
||
**Scenario**: You're building a KWS system for a smart speaker with these constraints:
|
||
|
||
- **Target**: `{python} kws_accuracy_target_str`% accuracy, <`{python} kws_false_wake_month_str` false wake per month
|
||
- **Budget**: \$`{python} kws_budget_str` total data engineering budget
|
||
- **Memory**: `{python} kws_memory_limit_design_kb_str` KB model size limit (always-on island)
|
||
- **Timeline**: `{python} kws_timeline_months_str` months to production
|
||
|
||
#### Step 1: Apply Constraints to Eliminate Options {.unnumbered}
|
||
|
||
From @tbl-kws-design-space, the 64 KB memory limit eliminates:
|
||
|
||
- 40 MFCC coefficients (3$\times$ memory) → Must use 13 MFCCs
|
||
- Cloud inference (requires network stack) → Must use local inference
|
||
|
||
#### Step 2: Calculate Budget Allocation {.unnumbered}
|
||
|
||
Allocating our \$150K budget across three cost categories (refer to the Key Data Engineering Numbers above):
|
||
|
||
- **Labeling** (~60%): `{python} labeling_k` K available
|
||
- **Storage/Processing** (~25%): `{python} storage_k` K
|
||
- **Governance/Other** (~15%): `{python} governance_k` K
|
||
|
||
At USD `{python} cost_per_label_str`/label with `{python} review_overhead_pct_str`% review overhead: `{python} labeling_k` K ÷ `{python} effective_cost_str` = **`{python} labeled_k` K labeled examples**
|
||
|
||
This falls between 1M and 10M in our design space, closer to 1M, suggesting +5–6% accuracy contribution from data volume.
|
||
|
||
#### Step 3: Maximize Remaining Accuracy {.unnumbered}
|
||
|
||
Current accuracy budget:
|
||
|
||
- Base model: ~90% (minimal data)
|
||
- +5–6% from 750K examples
|
||
- Need: +2–3% more to reach 98%
|
||
|
||
Options from design space:
|
||
|
||
- 16kHz sampling: +2–4% accuracy, 2$\times$ storage cost ✓ (fits budget)
|
||
- Noisy training data: +10–15% real-world accuracy, 3$\times$ collection cost
|
||
- Synthetic augmentation: +3–5% robustness, 10$\times$ cheaper than real data ✓
|
||
|
||
#### Step 4: Final Configuration {.unnumbered}
|
||
|
||
| **Choice** | **Selection** | **Rationale** |
|
||
|:----------------------|:-------------------------|:--------------------------------------------|
|
||
| **Sampling rate** | 16kHz | +3% accuracy worth 2× storage within budget |
|
||
| **MFCC coefficients** | 13 | Memory-constrained, non-negotiable |
|
||
| **Training examples** | 750K real + 2M synthetic | Budget-optimal mix |
|
||
| **Data diversity** | Noisy + clean mix | Critical for real-world deployment |
|
||
| **Inference** | Local, 8-bit quantized | Memory-constrained |
|
||
| **Augmentation** | Heavy synthetic | 10× cost efficiency |
|
||
|
||
**Projected Outcome**: 97–99% accuracy (meeting target), \$145K spend (under budget), 48 KB model (under limit).
|
||
|
||
**The Engineering Lesson**: Systematic design space analysis transformed intuition ("we need more data") into quantified decisions ("750K real + 2M synthetic maximizes accuracy per dollar given memory constraints").
|
||
:::
|
||
|
||
With optimal parameters selected from our design space, implementation requires combining multiple data collection approaches. Our KWS system demonstrates how these approaches work together across the project lifecycle. Pre-existing datasets like Google's Speech Commands [@warden2018speech] provide a foundation for initial development, offering carefully curated voice samples for common wake words. For multilingual coverage, the Multilingual Spoken Words Corpus (MSWC) [@mazumder2021multilingual] extends this foundation to 50 languages with over 23 million examples. However, even these large-scale datasets often lack diversity in accents, environments, and recording conditions, necessitating additional strategies.
|
||
|
||
To address coverage gaps, web scraping supplements baseline datasets by gathering diverse voice samples from video platforms and speech databases, capturing natural speech patterns and wake word variations. Crowdsourcing platforms like Amazon Mechanical Turk[^fn-mturk-labeling-scale] enable targeted collection of wake word samples across different demographics and environments. This approach is particularly valuable for underrepresented languages or specific acoustic conditions.
|
||
|
||
[^fn-mturk-labeling-scale]: **Amazon Mechanical Turk (MTurk)**: A crowdsourcing platform that routes tasks to workers matching specific demographic or environmental profiles, enabling targeted acquisition of wake word recordings from specific accents, age groups, or noisy environments that synthetic data cannot authentically reproduce. The trade-off is cost versus validation overhead: raw MTurk collection runs 10--50$\times$ cheaper per sample than professional field recording, but each submission requires acoustic quality verification (SNR, duration, speaker proximity) to prevent low-quality recordings from degrading the KWS model's accuracy on precisely the underrepresented conditions the collection was designed to cover. \index{Mechanical Turk!labeling scale}
|
||
|
||
Finally, synthetic data generation fills remaining gaps through speech synthesis [@werchniak2021exploring] and audio augmentation, creating unlimited wake word variations across acoustic environments, speaker characteristics, and background conditions. The combination enables KWS systems that perform well across diverse real-world conditions.
|
||
|
||
These complementary strategies, each with distinct trade-offs across quality, cost, and scale, require systematic evaluation rather than ad-hoc selection. The cost and time constants below provide essential context for acquisition decisions.
|
||
|
||
::: {.callout-perspective title="Key Data Engineering Numbers"}
|
||
Just as systems engineers memorize latency numbers, ML engineers should internalize these data engineering constants:
|
||
|
||
**Costs (2024 estimates)**
|
||
|
||
| **Operation** | **Cost** | **Notes** |
|
||
|:-----------------------------|-------------------------------------------------------------------------------------:|:-----------------------|
|
||
| **Crowdsourced image label** | \$`{python} labeling_cost_crowd_low_str`–`{python} labeling_cost_crowd_high_str` | Simple classification |
|
||
| **Bounding box annotation** | \$`{python} labeling_cost_box_low_str`–`{python} labeling_cost_box_high_str` | Per box, simple scenes |
|
||
| **Expert medical label** | \$`{python} labeling_cost_medical_low_str`–`{python} labeling_cost_medical_high_str` | Per study, radiologist |
|
||
| **S3 storage (Standard)** | \$`{python} storage_cost_s3_str`/TB/month | Hot storage |
|
||
| **S3 retrieval (Glacier)** | \$`{python} retrieval_cost_glacier_str`/GB | Standard: 3-5 hours |
|
||
| **GPU training hour (A100)** | \$2–4 | Cloud spot pricing |
|
||
| **Human review hour** | \$15–50 | Depending on expertise |
|
||
|
||
**Time Constants**
|
||
|
||
| **Operation** | **Duration** | **Bottleneck** |
|
||
|:-----------------------------------|-------------:|:------------------------------------|
|
||
| **Label 1M images (crowdsourced)** | 2–4 weeks | Annotation throughput |
|
||
| **Train ResNet-50 on ImageNet** | 4–6 hours | Compute (8$\times$ A100, optimized) |
|
||
| **Feature store lookup** | 1–10 ms | Network + cache |
|
||
|
||
The contrast matters: **weeks** for human labeling, **hours** for GPU training, **milliseconds** for serving. Labeling is the bottleneck.
|
||
|
||
**The 1000$\times$ Rule**: Labeling typically costs 1,000--3,000$\times$ more than the compute to train on that data. A \$100K labeling budget buys data that trains on \$30–100 of GPU time.
|
||
|
||
**The 80/20 Split**: 80% of data engineering effort goes to 20% of features: the "long tail" of edge cases, rare categories, and quality exceptions.
|
||
:::
|
||
|
||
All cost figures reflect approximate 2024 cloud provider rates and are intended to convey relative magnitudes rather than exact pricing.[^fn-pricing-cost-ratios]
|
||
|
||
[^fn-pricing-cost-ratios]: **Pricing Ratios**: The absolute dollar amounts in this chapter will shift with provider pricing, but the ratios between tiers are remarkably stable because they reflect physical constraints, not business decisions. S3 Glacier retrieval illustrates the pattern: standard (\$0.01/GB, 3--5 hours), expedited (\$0.03/GB, 1--5 minutes), and bulk (free, 5--12 hours) span a 30$\times$ cost range that maps directly to the $D_{vol}/BW$ trade-off in the Iron Law. Engineers who memorize ratios rather than prices make storage decisions that survive the next pricing revision. \index{Cloud Pricing!cost-latency trade-off}
|
||
|
||
::: {.callout-checkpoint title="Four Pillars Framework" collapse="false"}
|
||
The Four Pillars provide a systems lens for every pipeline choice.
|
||
|
||
**The Pillars**
|
||
|
||
- [ ] **Quality**: Can you distinguish correctness from *coverage* (edge cases, subgroup coverage) and explain why both matter?
|
||
- [ ] **Reliability**: Can you name concrete failure modes (schema drift, missing values, skew) that cause silent degradation rather than explicit crashes?
|
||
- [ ] **Scalability**: Can you explain which parts of a pipeline become bottlenecks as data volume grows (storage bandwidth, compute, coordination), and why?
|
||
- [ ] **Governance**: Can you articulate what must be tracked to make a dataset auditable (provenance, consent constraints, access controls, documentation)?
|
||
|
||
**Trade-offs**
|
||
|
||
- [ ] Given one pipeline change (e.g., stricter validation, stronger privacy constraints, or more data sources), can you predict which pillar improves and which pillar(s) you stress?
|
||
:::
|
||
|
||
The Four Pillars provide the evaluative lens; now we apply it to the first concrete engineering decision: where does the data come from? Acquisition strategy determines the raw material that every subsequent stage refines.
|
||
|
||
## Data Acquisition {#sec-data-engineering-strategic-data-acquisition-418f}
|
||
|
||
\index{Data Acquisition!strategies}ImageNet's 14.2 million labeled images took 49,000 crowdsourced workers to assemble. GPT-3's training corpus required petabytes of web text filtered through elaborate quality pipelines. Our KWS system needs 23 million audio samples spanning 50 languages—a volume that no single collection method can economically provide. These examples illustrate a recurring reality: data acquisition is a strategic decision that determines a system's capabilities and limitations. Sourcing strategies—existing datasets, web scraping, crowdsourcing, synthetic generation—trade off quality, cost, scale, and ethics differently. No single approach satisfies all requirements; successful ML systems combine multiple strategies, balancing complementary strengths against competing constraints.
|
||
|
||
Our KWS system illustrates these interconnected requirements. Achieving 98% accuracy across diverse acoustic environments requires representative data spanning accents, ages, and recording conditions (quality). Maintaining consistent detection despite device variations demands data from varied hardware (reliability). Supporting millions of concurrent users requires data volumes that manual collection cannot economically provide (scalability). Protecting user privacy in always-listening systems constrains collection methods and requires careful anonymization (governance). These interconnected requirements demonstrate why acquisition strategy must be evaluated systematically rather than through ad-hoc source selection.
|
||
|
||
### Data Source Evaluation and Selection {#sec-data-engineering-data-source-evaluation-selection-cd87}
|
||
|
||
\index{Data Sources!evaluation}The choice between curated datasets, expert crowdsourcing, and controlled web scraping depends on accuracy targets, domain expertise needed, and benchmark requirements. Achieving quality requires understanding not just that data appears correct but that it accurately represents the deployment environment and provides sufficient coverage of edge cases that might cause failures.
|
||
|
||
\index{Fei-Fei Li!ImageNet creation}
|
||
Pre-existing datasets from repositories such as Kaggle [@kaggle_website], UCI [@uci_repo], and ImageNet\index{ImageNet!benchmark dataset}[^fn-imagenet-data-scale] [@imagenet_website] offer cost-efficient starting points with established benchmarks for consistent performance comparison.
|
||
|
||
[^fn-imagenet-data-scale]: **ImageNet**: The canonical "cost-efficient starting point" -- 14.2 million images labeled by 49,000 Mechanical Turk workers across 21,841 categories (2009). Its value as a benchmark is inseparable from its data engineering: Fei-Fei Li's team spent two years building the labeling infrastructure, yet every subsequent team reuses that investment for free. The catch is benchmark overfitting: models tuned to ImageNet's distribution systematically underperform on production data with different lighting, occlusion, or demographic composition, making it a starting point that must be augmented, never a finishing line. \index{ImageNet!data scale}
|
||
|
||
Documentation quality directly affects reproducibility, an ongoing crisis in machine learning research [@pineau2021improving; @henderson2018deep]. Good documentation captures collection methodology, variable definitions, and baseline performance, enabling validation and replication. At scale, volume and variety compound quality challenges [@gudivada2017data], requiring systematic validation pipelines rather than ad-hoc inspection.
|
||
|
||
Context matters as much as content. Popular benchmarks like ImageNet invite overfitting that inflates performance metrics [@beyer2020we], and curated datasets frequently fail to reflect real-world deployment distributions [@venturebeat_datasets]. This disconnect creates systemic risk when organizations rely exclusively on standard datasets. Follow the arrows in @fig-misalignment to see what happens: when multiple ML systems all train on the same data, they propagate shared biases and limitations throughout an entire ecosystem of deployed models.
|
||
|
||
::: {#fig-misalignment fig-env="figure" fig-pos="htb" fig-cap="**Shared Dataset Bias Propagation**: Five models (A through E) all train on a single central dataset repository. Arrows show how shared limitations, biases, and blind spots propagate from the common dataset to every downstream model, leading to correlated failures across the ecosystem." fig-alt="Five model boxes labeled A through E at center all connect upward to one central training dataset repository. Arrows downward show shared limitations, biases, and blind spots propagating to all models."}
|
||
```{.tikz}
|
||
\begin{tikzpicture}[font=\small\usefont{T1}{phv}{m}{n}]
|
||
%
|
||
\tikzset{%
|
||
Line/.style={line width=1.0pt,black!50,text=black},
|
||
Box/.style={align=flush center,
|
||
inner xsep=2pt,
|
||
node distance=1.4,
|
||
draw=GreenLine,
|
||
line width=0.75pt,
|
||
fill=GreenL,
|
||
text width=17mm,
|
||
minimum width=17mm, minimum height=9mm
|
||
},
|
||
Text/.style={%
|
||
inner sep=2pt,
|
||
draw=none,
|
||
line width=0.75pt,
|
||
fill=TextColor!80,
|
||
text=black,
|
||
font=\footnotesize\usefont{T1}{phv}{m}{n},
|
||
align=flush center,
|
||
minimum width=7mm, minimum height=5mm
|
||
},
|
||
}
|
||
%
|
||
\node[Box](B1){Model A};
|
||
\node[Box,right=of B1](B2){Model B};
|
||
\node[Box,right=of B2](B3){Model C};
|
||
\node[Box,right=of B3](B4){Model D};
|
||
\node[Box,right=of B4](B5){Model E};
|
||
\node[Box, fill=OrangeL,draw=OrangeLine,above=1.5 of B3,text width=53mm](G){Central Training Dataset Repository};
|
||
\node[Box, fill=RedL,draw=RedLine,below=1.3 of B3,text width=53mm](D){Limited Real-World Alignment};
|
||
%
|
||
\scoped[on background layer]
|
||
\node[draw=BackLine,inner xsep=6mm,inner ysep=3mm,yshift=0mm,
|
||
fill=BackColor,minimum width=113mm,fit=(B1)(B5)(D),line width=0.75pt](BB1){};
|
||
\node[above=11pt of BB1.south east,anchor=east]{Potential Issues};
|
||
\draw[latex-,Line](B2)--node[Text,pos=0.9]{Same Data}++(90:1.5)--(G);
|
||
\draw[latex-,Line](B3)--node[Text,pos=0.9]{Same Data}++(90:1.5)--(G);
|
||
\draw[latex-,Line](B4)--node[Text,pos=0.9]{Same Data}++(90:1.5)--(G);
|
||
\draw[latex-,Line](B1)|-node[Text,pos=0.22]{Training Data}(G);
|
||
\draw[latex-,Line](B5)|-node[Text,pos=0.22]{Same Data}(G);
|
||
%
|
||
\draw[-latex,Line](B2)--node[Text,pos=0.6]{Shared Limitations}++(270:1.5)--(D);
|
||
\draw[-latex,Line](B3)--node[Text,pos=0.6]{Dataset Blind Spots}++(270:1.5)--(D);
|
||
\draw[-latex,Line](B4)--node[Text,pos=0.6]{Common Weaknesses}++(270:1.5)--(D);
|
||
\draw[-latex,Line](B1)|-node[Text,pos=0.17]{Propagated Biases}(D);
|
||
\draw[-latex,Line](B5)|-node[Text,pos=0.17]{Systemic Issues}(D);
|
||
\end{tikzpicture}
|
||
```
|
||
:::
|
||
|
||
For our KWS lighthouse, the datasets introduced in @sec-data-engineering-framework-application-keyword-spotting-case-study-c8e9—Speech Commands, MSWC, and supplementary web-scraped and synthetic sources—provide essential starting points for rapid prototyping and baseline performance. For the visual component of the broader smart doorbell application, the Wake Vision dataset [@banbury2024wakevisiontailoreddataset] serves as the standard benchmark for person detection on microcontrollers. However, evaluating these sources against quality requirements immediately reveals coverage gaps: limited accent diversity in audio, lack of low-light scenarios in vision, and predominantly clean recording environments for both. Quality-driven acquisition strategy recognizes these limitations and plans complementary approaches, demonstrating how framework-based thinking guides source selection beyond simply choosing available datasets.
|
||
|
||
### Scalability and Cost Optimization {#sec-data-engineering-scalability-cost-optimization-b9b3}
|
||
|
||
\index{Data Acquisition!scalability}\index{Web Scraping!ML training data}Quality-focused approaches face inherent scaling limitations.\index{Data Acquisition!scale vs quality} When scale requirements dominate—needing millions or billions of examples that manual curation cannot economically provide—web scraping and synthetic generation offer paths to massive datasets. Scalability requires understanding the economic models underlying different acquisition strategies: cost per labeled example, throughput limitations, and how these scale with data volume. Cost-effectiveness inverts with scale: what works at thousands of examples becomes prohibitive at millions, while high-setup-cost approaches amortize favorably at large volumes.
|
||
|
||
Web scraping enables dataset construction at scales that manual curation cannot match. Major vision datasets like ImageNet [@imagenet_website] and OpenImages [@openimages_website] were built through systematic scraping, and large language models depend on web-scale text corpora [@groeneveld2024olmo]. Targeted scraping of domain-specific sources, such as code repositories [@chen2021evaluating], further demonstrates the approach's versatility. However, production systems that rely on continuous scraping face pipeline reliability challenges: website structure changes break extractors, rate limiting throttles collection throughput, and dynamic content introduces inconsistencies that degrade model performance. Scraped data can also contain unexpected noise, such as historical images appearing in contemporary searches (@fig-traffic-light), requiring systematic validation and cleaning stages.
|
||
|
||
Consider what happens when you scrape the web for "traffic light" images: search engines return not only modern LED signals but also historical photographs like the one below. A model trained on such data might learn that traffic lights are sometimes operated by uniformed officers standing in the street, a spurious correlation that would cause failures in any real-world deployment.
|
||
|
||
{#fig-traffic-light fig-alt="Historical black-and-white photograph from 1914 showing early traffic control with manual semaphore signals, illustrating how outdated images can appear in modern web scraping results."}
|
||
|
||
This example reveals why the quality pillar cannot be satisfied by scale alone: no amount of additional scraped data removes the need for validation that detects and filters anachronistic or contextually inappropriate content. Beyond technical quality challenges, legal and ethical constraints further bound what scraping can achieve. Not all websites permit scraping, and ongoing litigation around training data usage illustrates the consequences of non-compliance [@harvard_law_chatgpt]. Teams must document data provenance, ensure compliance with terms of service and copyright law, and apply anonymization procedures when scraping user-generated content.
|
||
|
||
Crowdsourcing\index{Crowdsourcing!data labeling}\index{Crowdsourcing!annotation} distributes annotation tasks across a global workforce, enabling rapid labeling at scales that in-house teams cannot match. Platforms like Amazon Mechanical Turk [@mturk_website] demonstrated this at landmark scale with ImageNet, where distributed contributors categorized millions of images into thousands of classes [@alexnet2012]. Crowdsourcing offers two systems advantages: scalability through parallel microtask distribution, and diversity through the range of perspectives, cultural contexts, and linguistic variations that a global contributor pool introduces. This diversity directly improves model generalization across populations. Tasks can also be adjusted dynamically based on initial results, enabling iterative refinement of collection strategies as quality gaps emerge.
|
||
|
||
Moving beyond human-generated data entirely, synthetic data generation\index{Synthetic Data!generation}\index{Data Augmentation!synthetic data}\index{Synthetic Data!scalability} represents the ultimate scalability solution, creating unlimited training examples through algorithmic generation rather than manual collection. This approach changes the economics of data acquisition by removing human labor from the equation. Follow the pipeline in @fig-synthetic-data to see how synthetic data merges with historical datasets, producing training sets of a size and diversity that would be impractical to collect manually.
|
||
|
||
::: {#fig-synthetic-data fig-env="figure" fig-pos="htb" fig-cap="**Synthetic Data Augmentation**: A four-node pipeline where historical data and simulation outputs feed into a synthetic data generation process, producing an expanded combined training dataset with greater size and diversity than either source alone. Source: AnyLogic [@anylogic_synthetic]." fig-alt="Diagram showing historical data icon and simulation cloud icon both feeding into synthetic data generation process, producing an expanded combined training dataset."}
|
||
```{.tikz}
|
||
\begin{tikzpicture}[line join=round,font=\usefont{T1}{phv}{m}{n}\small]
|
||
\tikzset{
|
||
Line/.style={line width=0.35pt,black!50,text=black},
|
||
LineDO/.style={single arrow, draw=VioletLine, fill=VioletLine!50,
|
||
minimum width = 10pt, single arrow head extend=3pt,
|
||
minimum height=10mm},
|
||
ALineA/.style={violet!80!black!50,line width=3pt,shorten <=2pt,shorten >=2pt,
|
||
{Triangle[width=1.1*6pt,length=0.8*6pt]}-{Triangle[width=1.1*6pt,length=0.8*6pt]}},
|
||
LineD/.style={line width=0.75pt,black!50,text=black,dashed,dash pattern=on 5pt off 3pt},
|
||
Circle/.style={inner xsep=2pt,
|
||
% node distance=1.15,
|
||
circle,
|
||
draw=BrownLine,
|
||
line width=0.75pt,
|
||
fill=BrownL!40,
|
||
minimum size=18mm
|
||
},
|
||
circles/.pic={
|
||
\pgfkeys{/channel/.cd, #1}
|
||
\node[circle,draw=\channelcolor,line width=\Linewidth,fill=\channelcolor!10,
|
||
minimum size=2.5mm](\picname){};
|
||
}
|
||
}
|
||
\tikzset {
|
||
pics/cloud/.style = {
|
||
code = {
|
||
\colorlet{red}{RedLine}
|
||
\begin{scope}[local bounding box=CLO,scale=0.5, every node/.append style={transform shape},,shift={($(SIM)+(0,0)$)},]
|
||
\draw[red,fill=white,line width=0.9pt](0.67,1.21)to[out=55,in=90,distance=13](1.5,0.96)
|
||
to[out=360,in=30,distance=9](1.68,0.42);
|
||
\draw[red,fill=white,line width=0.9pt](0,0)to[out=170,in=180,distance=11](0.1,0.61)
|
||
to[out=90,in=105,distance=17](1.07,0.71)
|
||
to[out=20,in=75,distance=7](1.48,0.36)
|
||
to[out=350,in=0,distance=7](1.48,0)--(0,0);
|
||
\draw[red,fill=white,line width=0.9pt](0.27,0.71)to[bend left=25](0.49,0.96);
|
||
|
||
\end{scope}
|
||
}
|
||
}
|
||
}
|
||
%streaming
|
||
\tikzset{%
|
||
LineST/.style={-{Circle[\channelcolor,fill=RedLine,length=4pt]},draw=\channelcolor,line width=\Linewidth,rounded corners},
|
||
ellipseST/.style={fill=\channelcolor,ellipse,minimum width = 2.5mm, inner sep=2pt, minimum height =1.5mm},
|
||
BoxST/.style={line width=\Linewidth,fill=white,draw=\channelcolor,rectangle,minimum width=56,
|
||
minimum height=16,rounded corners=1.2pt},
|
||
pics/streaming/.style = {
|
||
code = {
|
||
\pgfkeys{/channel/.cd, #1}
|
||
\begin{scope}[local bounding box=STREAMING,scale=\scalefac, every node/.append style={transform shape}]
|
||
\node[BoxST,minimum width=44,minimum height=48](\picname-RE1){};
|
||
\foreach \i/\j in{1/north,2/center,3/south}{
|
||
\node[BoxST](\picname-GR\i)at(\picname-RE1.\j){};
|
||
\node[ellipseST]at($(\picname-GR\i.west)!0.2!(\picname-GR\i.east)$){};
|
||
\node[ellipseST]at($(\picname-GR\i.west)!0.4!(\picname-GR\i.east)$){};
|
||
}
|
||
\draw[LineST](\picname-GR3)--++(2,0)coordinate(\picname-C4);
|
||
\draw[LineST](\picname-GR3.320)--++(0,-0.7)--++(0.8,0)coordinate(\picname-C5);
|
||
\draw[LineST](\picname-GR3.220)--++(0,-0.7)--++(-0.8,0)coordinate(\picname-C6);
|
||
\draw[LineST](\picname-GR3)--++(-2,0)coordinate(\picname-C7);
|
||
\end{scope}
|
||
}
|
||
}
|
||
}
|
||
%data
|
||
\tikzset{mycylinder/.style={cylinder, shape border rotate=90, aspect=1.3, draw, fill=white,
|
||
minimum width=25mm,minimum height=11mm,line width=\Linewidth,node distance=-0.15},
|
||
pics/data/.style = {
|
||
code = {
|
||
\pgfkeys{/channel/.cd, #1}
|
||
\begin{scope}[local bounding box=STREAMING,scale=\scalefac, every node/.append style={transform shape}]
|
||
\node[mycylinder,fill=\channelcolor!50] (A) {};
|
||
\node[mycylinder, above=of A,fill=\channelcolor!30] (B) {};
|
||
\node[mycylinder, above=of B,fill=\channelcolor!10] (C) {};
|
||
\end{scope}
|
||
}
|
||
}
|
||
}
|
||
|
||
\tikzset{pics/brain/.style = {
|
||
code = {
|
||
\pgfkeys{/channel/.cd, #1}
|
||
\begin{scope}[local bounding box=BRAIN,scale=\scalefac, every node/.append style={transform shape}]
|
||
\fill[fill=\filllcolor!50](0.1,-0.5)to[out=0,in=180](0.33,-0.5)
|
||
to[out=0,in=270](0.45,-0.38)to(0.45,-0.18)
|
||
to[out=40,in=240](0.57,-0.13)to[out=110,in=310](0.52,-0.05)
|
||
to[out=130,in=290](0.44,0.15)to[out=90,in=340,distance=8](0.08,0.69)
|
||
to[out=160,in=80](-0.42,-0.15)to (-0.48,-0.7)to(0.07,-0.7)to(0.1,-0.5)
|
||
(-0.10,-0.42)to[out=310,in=180](0.1,-0.5);
|
||
\draw[draw=\drawchannelcolor,line width=\Linewidth](0.1,-0.5)to[out=0,in=180](0.33,-0.5)
|
||
to[out=0,in=270](0.45,-0.38)to(0.45,-0.18)
|
||
to[out=40,in=240](0.57,-0.13)to[out=110,in=310](0.52,-0.05)
|
||
to[out=130,in=290](0.44,0.15)to[out=90,in=340,distance=8](0.08,0.69)
|
||
(-0.42,-0.15)to (-0.48,-0.7)
|
||
(0.07,-0.7)to(0.1,-0.5)
|
||
(-0.10,-0.42)to[out=310,in=180](0.1,-0.5);
|
||
%brain
|
||
\draw[fill=\filllcolor,line width=\Linewidth](-0.3,-0.10)to(0.08,0.60)
|
||
to[out=60,in=50,distance=3](-0.1,0.69)to[out=160,in=80](-0.26,0.59)to[out=170,in=90](-0.46,0.42)
|
||
to[out=170,in=110](-0.54,0.25)to[out=210,in=150](-0.54,0.04)
|
||
to[out=240,in=130](-0.52,-0.1)to[out=300,in=240](-0.3,-0.10);
|
||
\draw[fill=\filllcolor,line width=\Linewidth]
|
||
(-0.04,0.64)to[out=120,in=0](-0.1,0.69)(-0.19,0.52)to[out=120,in=330](-0.26,0.59)
|
||
(-0.4,0.33)to[out=150,in=280](-0.46,0.42)
|
||
%
|
||
(-0.44,-0.03)to[bend left=30](-0.34,-0.04)
|
||
(-0.33,0.08)to[bend left=40](-0.37,0.2) (-0.37,0.12)to[bend left=40](-0.45,0.14)
|
||
(-0.26,0.2)to[bend left=30](-0.24,0.13)
|
||
(-0.16,0.32)to[bend right=30](-0.27,0.3)to[bend right=30](-0.29,0.38)
|
||
(-0.13,0.49)to[bend left=30](-0.04,0.51);
|
||
|
||
\draw[rounded corners=0.8pt,\drawcircle,-{Circle[fill=\filllcirclecolor,length=2.5pt]}](-0.23,0.03)--(-0.15,-0.03)--(-0.19,-0.18)--(-0.04,-0.28);
|
||
\draw[rounded corners=0.8pt,\drawcircle,-{Circle[fill=\filllcirclecolor,length=2.5pt]}](-0.17,0.13)--(-0.04,0.05)--(-0.06,-0.06)--(0.14,-0.11);
|
||
\draw[rounded corners=0.8pt,\drawcircle,-{Circle[fill=\filllcirclecolor,length=2.5pt]}](-0.12,0.23)--(0.31,0.0);
|
||
\draw[rounded corners=0.8pt,\drawcircle,-{Circle[fill=\filllcirclecolor,length=2.5pt]}](-0.07,0.32)--(0.06,0.26)--(0.16,0.33)--(0.34,0.2);
|
||
\draw[rounded corners=0.8pt,\drawcircle,-{Circle[fill=\filllcirclecolor,length=2.5pt]}](-0.01,0.43)--(0.06,0.39)--(0.18,0.51)--(0.31,0.4);
|
||
\end{scope}
|
||
}
|
||
}
|
||
}
|
||
|
||
\tikzset{pics/tube/.style = {
|
||
code = {
|
||
\pgfkeys{/channel/.cd, #1}
|
||
\begin{scope}[local bounding box=BRAIN,scale=\scalefac, every node/.append style={transform shape}]
|
||
\draw[draw=\drawchannelcolor,line width=\Linewidth,fill=white](-0.1,0.26)to(-0.1,0.1)to[out=240,in=60](-0.23,-0.14)
|
||
to[out=240,in=180,distance=3](-0.13,-0.27)to(0.09,-0.27)
|
||
to[out=0,in=300,distance=3](0.19,-0.14)
|
||
to[out=120,in=290]((0.06,0.1)to(0.06,0.26)
|
||
to cycle;
|
||
\fill[fill=\filllcolor!50](-0.23,-0.14)
|
||
to[out=240,in=180,distance=3](-0.13,-0.27)to(0.09,-0.27)
|
||
to[out=0,in=300,distance=3](0.19,-0.14)to[out=200,in=20]cycle;
|
||
\draw[draw=\drawchannelcolor,line width=\Linewidth,fill=none](-0.1,0.26)to(-0.1,0.1)to[out=240,in=60](-0.23,-0.14)
|
||
to[out=240,in=180,distance=3](-0.13,-0.27)to(0.09,-0.27)
|
||
to[out=0,in=300,distance=3](0.19,-0.14)
|
||
to[out=120,in=290]((0.06,0.1)to(0.06,0.26)
|
||
to cycle;
|
||
\end{scope}
|
||
}
|
||
}
|
||
}
|
||
|
||
\tikzset{pics/factory/.style = {
|
||
code = {
|
||
\pgfkeys{/channel/.cd, #1}
|
||
\begin{scope}[local bounding box=FACTORY,scale=\scalefac, every node/.append style={transform shape}]
|
||
\node[rectangle,draw=\drawchannelcolor,fill=\filllcolor!50,minimum height=15,minimum width=23,,line width=\Linewidth](R1){};
|
||
\draw[fill=\filllcolor!50,line width=1.0pt]($(R1.40)+(0,-0.01)$)--++(110:0.2)--++(180:0.12)|-($(R1.40)+(0,-0.01)$);
|
||
\draw[line width=\Linewidth,fill=green](-0.68,-0.27)--++(88:0.85)--++(0:0.15)--(-0.48,-0.27)--cycle;
|
||
\draw[line width=2.5pt](-0.8,-0.27)--(0.55,-0.27);
|
||
|
||
\foreach \x in{0.25,0.45,0.65}{
|
||
\node[rectangle,fill=black,minimum height=2,minimum width=5,thick,inner sep=0pt]
|
||
at ($(R1.north)!\x!(R1.south)$){};
|
||
}
|
||
\foreach \x in{0.25,0.45,0.65}{
|
||
\node[rectangle,fill=black,minimum height=2,minimum width=5,thick,inner sep=0pt]
|
||
at ($(R1.130)!\x!(R1.230)$){};
|
||
}
|
||
\foreach \x in{0.25,0.45,0.65}{
|
||
\node[rectangle,fill=black,minimum height=2,minimum width=5,thick,inner sep=0pt]
|
||
at ($(R1.50)!\x!(R1.310)$){};
|
||
}
|
||
\end{scope}
|
||
}
|
||
}
|
||
}
|
||
|
||
\tikzset {
|
||
pics/cloud/.style = {
|
||
code = {
|
||
\colorlet{red}{BrownLine}
|
||
\pgfkeys{/channel/.cd, #1}
|
||
\begin{scope}[local bounding box=CLO,scale=\scalefac, every node/.append style={transform shape}]
|
||
\draw[red,line width=\Linewidth,fill=red!10](0,0)to[out=170,in=180,distance=11](0.1,0.61)
|
||
to[out=90,in=105,distance=17](1.07,0.71)
|
||
to[out=20,in=75,distance=7](1.48,0.36)
|
||
to[out=350,in=0,distance=7](1.48,0)--(0,0);
|
||
\draw[red,line width=\Linewidth](0.27,0.71)to[bend left=25](0.49,0.96);
|
||
%\draw[red,line width=\Linewidth](0.67,1.21)to[out=55,in=90,distance=13](1.5,0.96)
|
||
%to[out=360,in=30,distance=9](1.68,0.42);
|
||
\end{scope}
|
||
}
|
||
}
|
||
}
|
||
\tikzset{
|
||
pics/square/.style = {
|
||
code = {
|
||
\pgfkeys{/channel/.cd, #1}
|
||
\begin{scope}[local bounding box=SQUARE,scale=\scalefac,every node/.append style={transform shape}]
|
||
% Right Face
|
||
\draw[fill=\channelcolor!70,line width=\Linewidth]
|
||
(\Depth,0,0)coordinate(\picname-ZDD)--(\Depth,\Width,0)--(\Depth,\Width,\Height)--(\Depth,0,\Height)--cycle;
|
||
% Front Face
|
||
\draw[fill=\channelcolor!40,line width=\Linewidth]
|
||
(0,0,\Height)coordinate(\picname-DL)--(0,\Width,\Height)coordinate(\picname-GL)--
|
||
(\Depth,\Width,\Height)coordinate(\picname-GD)--(\Depth,0,\Height)coordinate(\picname-DD)--(0,0,\Height);
|
||
% Top Face
|
||
\draw[fill=\channelcolor!20,line width=\Linewidth]
|
||
(0,\Width,0)coordinate(\picname-ZGL)--(0,\Width,\Height)coordinate(\picname-ZGL)--
|
||
(\Depth,\Width,\Height)--(\Depth,\Width,0)coordinate(\picname-ZGD)--cycle;
|
||
\end{scope}
|
||
}
|
||
}
|
||
}
|
||
|
||
\tikzset{
|
||
pics/plus/.style = {
|
||
code = {
|
||
\pgfkeys{/channel/.cd, #1}
|
||
\begin{scope}[local bounding box=PLUS,scale=\scalefac,every node/.append style={transform shape}]
|
||
% Right Face
|
||
\fill[fill=\channelcolor!70] (-0.7,-0.15)rectangle(0.7,0.15);
|
||
\fill[fill=\channelcolor!70] (-0.15,-0.7)rectangle(0.15,0.7);
|
||
\end{scope}
|
||
}
|
||
}
|
||
}
|
||
|
||
\pgfkeys{
|
||
/channel/.cd,
|
||
Depth/.store in=\Depth,
|
||
Height/.store in=\Height,
|
||
Width/.store in=\Width,
|
||
channelcolor/.store in=\channelcolor,
|
||
filllcirclecolor/.store in=\filllcirclecolor,
|
||
filllcolor/.store in=\filllcolor,
|
||
drawchannelcolor/.store in=\drawchannelcolor,
|
||
drawcircle/.store in=\drawcircle,
|
||
scalefac/.store in=\scalefac,
|
||
Linewidth/.store in=\Linewidth,
|
||
picname/.store in=\picname,
|
||
filllcolor=BrownLine,
|
||
filllcirclecolor=violet!20,
|
||
drawchannelcolor=black,
|
||
drawcircle=violet,
|
||
channelcolor=BrownLine,
|
||
scalefac=1,
|
||
Linewidth=1.6pt,
|
||
Depth=1.3,
|
||
Height=0.8,
|
||
Width=1.1,
|
||
picname=C
|
||
}
|
||
|
||
\node[Circle](SIM){};
|
||
\node[Circle,right=2 of SIM,draw=GreenLine,fill=GreenL!40,](SYN){};
|
||
\node[Circle,below=1.75 of SIM,draw=OrangeLine,fill=OrangeL!40,](REA){};
|
||
\node[Circle,right=2 of REA,draw=RedLine,fill=RedL!40,](HIS){};
|
||
%
|
||
\node[Circle, right=3.5 of $(SYN)!0.5!(HIS)$,draw=BlueLine,fill=BlueL!40,](MLA){};
|
||
\node[Circle,right=2 of MLA,draw=VioletLine,fill=VioletL2!40,](TRA){};
|
||
\node[LineDO]at($(SIM)!0.5!(SYN)$){};
|
||
\node[LineDO]at($(REA)!0.5!(HIS)$){};
|
||
\node[LineDO]at($(MLA)!0.5!(TRA)$){};
|
||
\coordinate(LG)at($(SYN.east)+(6mm,0)$);
|
||
\coordinate(LD)at($(HIS.east)+(6mm,0)$);
|
||
\draw[line width=4pt,violet!40](LG)--++(5mm,0)|-coordinate[pos=0.25](S)(LD);
|
||
\node[LineDO]at($(S)!0.1!(MLA)$){};
|
||
%%
|
||
\begin{scope}[local bounding box=CIRCLE1,shift={($(TRA)+(0.04,-0.24)$)},
|
||
scale=0.6, every node/.append style={transform shape}]
|
||
%1 column
|
||
\foreach \j in {1,2,3} {
|
||
\pgfmathsetmacro{\y}{(1.5-\j)*0.53 + 0.7}
|
||
\pic at (-0.8,\y) {circles={channelcolor=green!70!black,picname=1CD\j}};
|
||
}
|
||
%2 column
|
||
\foreach \i in {1,...,4} {
|
||
\pgfmathsetmacro{\y}{(2-\i)*0.53+0.7}
|
||
\pic at (0,\y) {circles={channelcolor=green!70!black, picname=2CD\i}};
|
||
}
|
||
%3 column
|
||
\foreach \j in {1,2} {
|
||
\pgfmathsetmacro{\y}{(1-\j)*0.53 + 0.7}
|
||
\pic at (0.8,\y) {circles={channelcolor=green!70!black,picname=3CD\j}};
|
||
}
|
||
\foreach \i in {1,2,3}{
|
||
\foreach \j in {1,2,3,4}{
|
||
\draw[Line](1CD\i)--(2CD\j);
|
||
}}
|
||
\foreach \i in {1,2,3,4}{
|
||
\foreach \j in {1,2}{
|
||
\draw[Line](2CD\i)--(3CD\j);
|
||
}}
|
||
\end{scope}
|
||
|
||
\tikzset{
|
||
comp/.style = {draw,
|
||
minimum width =18mm,
|
||
minimum height = 15mm,
|
||
inner sep= 0pt,
|
||
rounded corners=1pt,
|
||
draw = BlueLine,
|
||
fill=cyan!10,
|
||
line width=1.2pt
|
||
}
|
||
}
|
||
\begin{scope}[local bounding box=COMPUTER,scale=0.6, every node/.append style={transform shape}]
|
||
\node[comp](COM){};
|
||
\draw[draw = BlueLine,line width=1.0pt]
|
||
($(COM.north west)!0.85!(COM.south west)$)-- ($(COM.north east)!0.85!(COM.south east)$);
|
||
\draw[draw = BlueLine,line width=1.0pt]($(COM.south west)!0.4!(COM.south east)$)--++(270:0.2)coordinate(DL);
|
||
\draw[draw = BlueLine,line width=1.0pt]($(COM.south west)!0.6!(COM.south east)$)--++(270:0.2)coordinate(DD);
|
||
\draw[draw = BlueLine,line width=3.0pt,shorten <=-3mm,shorten >=-3mm](DL)--(DD);
|
||
\end{scope}
|
||
%
|
||
\pic[shift={(0,-0.4)}] at (HIS){data={scalefac=0.35,picname=1,channelcolor=green!70!black, Linewidth=0.4pt}};
|
||
\pic[shift={(0,0)}] at (MLA){brain={scalefac=0.9,picname=1,filllcolor=orange!30!, Linewidth=0.7pt}};
|
||
\pic[shift={(0,-0.4)}] at (SYN){data={scalefac=0.35,picname=1,channelcolor=cyan!70!black, Linewidth=0.4pt}};
|
||
\pic[shift={(0.25,-0.35)}] at (SYN){tube={scalefac=1.2,picname=1,filllcolor=blue!90!, Linewidth=0.5pt}};
|
||
\pic[shift={(0.13,-0.00)}] at (REA){factory={scalefac=0.9,picname=1,filllcolor=brown!, Linewidth=0.5pt}};
|
||
\pic[shift={(-0.32,-0.65)}] at (REA) {cloud={scalefac=0.5, Linewidth=1.0pt}};
|
||
\pic[shift={(-0.16,-0.1)}] at (SIM){square={scalefac=0.35,picname=1,channelcolor=red, Linewidth=0.5pt}};
|
||
%
|
||
\pic[shift={(0,0)}] at ($(SYN)!0.55!(HIS)$){plus={scalefac=0.4,channelcolor=violet}};
|
||
%
|
||
\node[below=1mm of SIM]{Simulation model};
|
||
\node[below=1mm of SYN]{Synthetic data};
|
||
\node[below=1mm of REA]{Real system};
|
||
\node[below=1mm of HIS]{Historical data};
|
||
\node[below=1mm of MLA]{ML algorithm};
|
||
\node[below=1mm of TRA]{Trained ML model};
|
||
\end{tikzpicture}
|
||
|
||
```
|
||
:::
|
||
|
||
Synthetic data\index{Synthetic Data!rare event coverage}\index{Synthetic Data!simulation} is particularly valuable for rare event coverage and data augmentation. Simulation environments enable controlled generation of edge cases that are impractical to collect naturally [@nvidia_simulation], while augmentation techniques like SpecAugment\index{SpecAugment!audio augmentation} [@park2019specaugment] introduce noise, pitch shifts, and temporal variations that improve model generalization across deployment conditions [@shorten2019survey]. The following notebook explores *synthetic data generation* in practice.
|
||
|
||
::: {.callout-notebook title="Synthetic Data Generation"}
|
||
Building synthetic datasets is one of the most cost-effective data engineering techniques. This exercise generates synthetic audio samples using pitch shifting, noise injection, and room impulse simulation, then measures how synthetic-to-real ratios affect KWS model accuracy. @sec-data-selection examines complementary strategies that further optimize what fraction of data actually contributes to learning.
|
||
:::
|
||
|
||
For our KWS system, `{python} kws_dataset_size_m_round_str` million training examples across `{python} kws_languages_str` languages demand a volume that manual collection cannot economically provide. The multi-source strategy introduced in @sec-data-engineering-framework-application-keyword-spotting-case-study-c8e9—curated datasets supplemented by web scraping, crowdsourcing, and synthetic generation—addresses this scale requirement while maintaining coverage across acoustic environments and speaker demographics.
|
||
|
||
### Coverage and Diversity Requirements {#sec-data-engineering-coverage-diversity-requirements-ce87}
|
||
|
||
\index{Data Coverage!requirements}\index{Geographic Bias!training data}Scale alone does not guarantee reliable models. Coverage gaps in even large datasets—geographic bias, demographic underrepresentation, temporal drift, missing edge cases—cause systematic failures that aggregate metrics obscure [@wang2019balanced; @oakden2020hidden]. As @fig-misalignment makes clear, multiple systems training on identical datasets inherit identical blind spots;\index{Dataset Convergence!shared blind spots} diverse sourcing strategies are the defense against correlated failure modes.
|
||
|
||
Governance constraints further shape acquisition: privacy regulations (GDPR, CCPA, HIPAA)\index{GDPR!data collection requirements}\index{CCPA!privacy regulations} limit what data can be collected and how, while ethical sourcing requires fair compensation and transparent use of human contributions. @sec-responsible-engineering-data-governance-compliance-bd1a examines the full governance infrastructure for production ML systems.
|
||
|
||
With acquisition strategies established, the diversity of sources—crowdsourced audio, synthetic waveforms, web-scraped content—creates specific challenges at the boundary where external data enters our controlled pipeline. We now cross that boundary into the infrastructure that receives, validates, and routes this heterogeneous data.
|
||
|
||
## Data Pipeline Architecture {#sec-data-engineering-data-pipeline-architecture-b527}
|
||
|
||
\index{Data Pipeline!architecture}\index{Pipeline!ML data}In our compilation metaphor, pipeline architecture is the *compiler frontend*: it parses heterogeneous raw inputs into a uniform intermediate representation that downstream stages can process reliably. Audio files from crowdsourcing platforms, synthetic waveforms from generation systems, and real-world captures from deployed devices all enter the pipeline in different formats—the pipeline must normalize, validate, and route them into a consistent internal representation. For KWS, this means handling continuous audio streams, maintaining low-latency processing for real-time keyword detection, and scaling from development environments to production deployments handling millions of concurrent streams.
|
||
|
||
Orient yourself with @fig-pipeline-flow, which maps the end-to-end pipeline across its distinct layers: data sources, ingestion, processing, labeling, storage, and ML training.
|
||
|
||
::: {#fig-pipeline-flow fig-env="figure" fig-pos="htb" fig-cap="**Three-Stage Pipeline Flow**: Raw data sources and APIs feed into batch and stream ingestion at the middle layer, then flow to data warehouse and storage destinations at the bottom. Each stage scales independently, enabling modular quality control across the pipeline." fig-alt="Three-tier flow diagram with raw data sources and APIs at top, batch and stream ingestion in middle layer, and data warehouse and storage destinations at bottom connected by arrows."}
|
||
```{.tikz}
|
||
\resizebox{.7\textwidth}{!}{%
|
||
\begin{tikzpicture}[font=\small\usefont{T1}{phv}{m}{n}]
|
||
%
|
||
\tikzset{%
|
||
Line/.style={line width=1.0pt,black!50,text=black},
|
||
Box/.style={align=flush center,
|
||
inner xsep=2pt,
|
||
node distance=0.8,
|
||
draw=GreenLine,
|
||
line width=0.75pt,
|
||
fill=GreenL,
|
||
text width=27mm,
|
||
minimum width=26mm, minimum height=9mm
|
||
},
|
||
}
|
||
%
|
||
\begin{scope}[local bounding box = scope1]
|
||
\node[Box](B1){Raw Data Sources};
|
||
\node[Box,right=of B1](B2){External APIs};
|
||
\node[Box,right=of B2](B3){Streaming Sources};
|
||
\end{scope}
|
||
%
|
||
\begin{scope}[shift={($(scope1.south)+(-2.84,-2.2)$)},anchor=center]
|
||
\node[Box, fill=BlueL,draw=BlueLine](2B1){Batch Ingestion};
|
||
\node[Box, fill=BlueL,draw=BlueLine, node distance=2.8,right=of 2B1](2B2){Stream Processing};
|
||
\end{scope}
|
||
%
|
||
\node[Box, node distance=1.2,below=of $(2B1)!0.5!(2B2)$](3B1){Storage Layer};
|
||
%
|
||
\node[Box, fill=OrangeL,draw=OrangeLine,below left=1 and 0.2 of 3B1](4B1){Training Data};
|
||
\node[Box, fill=RedL,draw=RedLine,node distance=1.3,below right=1 and 0.2of 3B1](4B2){Data Validation \& Quality Checks};
|
||
\node[Box, fill=OrangeL,draw=OrangeLine, node distance=0.6,below =of 4B1](5B1){Model Training};
|
||
\node[Box,fill=RedL,draw=RedLine,node distance=0.6,below =of 4B2](5B2){Transformation};
|
||
\node[Box, fill=RedL,draw=RedLine, node distance=0.6,below =of 5B2](6B1){Feature Creation / Engineering};
|
||
\node[Box, fill=RedL,draw=RedLine, node distance=0.6,below =of 6B1](7B1){Data Labeling};
|
||
%
|
||
\scoped[on background layer]
|
||
\node[draw=BackLine,inner xsep=5mm,inner ysep=5mm,yshift=1mm,
|
||
fill=BackColor,minimum width=113mm,fit=(B1)(B2)(B3),line width=0.75pt](BB1){};
|
||
\node[below=8pt of BB1.north east,anchor=east]{Sources};
|
||
\scoped[on background layer]
|
||
\node[draw=BackLine,inner xsep=5mm,inner ysep=5mm,yshift=1mm,
|
||
fill=BackColor,minimum width=113mm,fit=(2B1)(2B2),line width=0.75pt](BB2){};
|
||
\node[below=8pt of BB2.north east,anchor=east]{Data Ingestion};
|
||
\scoped[on background layer]
|
||
\node[draw=BackLine,inner xsep=9mm,inner ysep=5mm,yshift=-2mm,
|
||
fill=BackColor,fit=(4B1)(5B1),line width=0.75pt](BB3){};
|
||
\node[above=7pt of BB3.south east,anchor=east]{ML Training};
|
||
%
|
||
\scoped[on background layer]
|
||
\node[draw=BackLine,inner xsep=9mm,inner ysep=5mm,yshift=-2mm,
|
||
fill=BackColor,fit=(4B2)(7B1),line width=0.75pt](BB4){};
|
||
\node[above=7pt of BB4.south east,anchor=east]{Processing Layer};
|
||
%
|
||
\scoped[on background layer]
|
||
\node[draw=OrangeLine,inner xsep=3mm,inner ysep=6mm,yshift=3mm,
|
||
fill=none,fit=(BB1)(BB4),line width=0.75pt](BB4){};
|
||
\node[below=4pt of BB4.north,anchor=north]{Data Governance};
|
||
%
|
||
\draw[Line,-latex](B1)--++(270:1.2)-|(2B1);
|
||
\draw[Line,-latex](B2)--++(270:1.2)-|(2B1);
|
||
\draw[Line,-latex](B3)--++(270:1.2)-|(2B2);
|
||
%
|
||
\draw[Line,-latex](2B1)|-(3B1);
|
||
\draw[Line,-latex](2B2)|-(3B1);
|
||
%
|
||
\draw[Line,-latex](3B1)--++(270:0.9)-|(4B1);
|
||
\draw[Line,-latex](3B1)--++(270:0.9)-|(4B2);
|
||
%
|
||
\draw[Line,-latex](4B1)--(5B1);
|
||
\draw[Line,-latex](4B2)--(5B2);
|
||
\draw[Line,-latex](5B2)--(6B1);
|
||
\draw[Line,-latex](6B1)--(7B1);
|
||
\draw[Line,-latex](7B1.east)--++(0:0.6)|-(3B1);
|
||
\end{tikzpicture}}
|
||
```
|
||
:::
|
||
|
||
Each layer plays a specific role in the data preparation workflow. Selecting appropriate technologies requires understanding how our four framework pillars manifest at each stage. Quality requirements at one stage affect scalability constraints at another, reliability needs shape governance implementations, and the pillars interact to determine overall system effectiveness.
|
||
|
||
Data pipeline design is constrained by storage hierarchies and I/O bandwidth limitations rather than CPU capacity. Understanding these constraints enables building efficient systems for modern ML workloads. Storage hierarchy trade-offs, ranging from high-latency object storage (ideal for archival) to low-latency in-memory stores (essential for real-time serving), and bandwidth limitations (spinning disks at 100-200 MB/s versus RAM at 50-200 GB/s) shape every pipeline decision. @sec-data-engineering-strategic-storage-architecture-1a6b covers detailed storage architecture considerations.
|
||
|
||
Choosing between these design patterns requires matching workload characteristics to infrastructure capabilities. Streaming workloads demand attention to message durability (the ability to replay failed processing), ordering guarantees (what sequence is preserved, under what conditions), and geographic distribution. Batch workloads hinge on data volume relative to available memory, processing complexity, and whether computation must be distributed across machines. Single-machine tools suffice for gigabyte-scale data, but terabyte-scale processing often benefits from distributed frameworks that partition work across clusters. These layer interactions, viewed through the four-pillar lens, determine overall system effectiveness.
|
||
|
||
### Quality Through Validation and Monitoring {#sec-data-engineering-quality-validation-monitoring-498f}
|
||
|
||
\index{Data Validation!monitoring}\index{Schema Validation!ML pipelines}A self-driving car company discovered that 15% of their LiDAR point-cloud labels were misaligned by 10–20 centimeters—enough to place pedestrian bounding boxes on empty sidewalk. The mislabeling had persisted for three months, silently degrading the perception model's recall on pedestrians at crosswalks. No schema check flagged the error because every record was structurally valid; only statistical monitoring of label-to-sensor alignment distributions caught the drift.
|
||
|
||
Quality represents the foundation of reliable ML systems, and this example illustrates why. Pipelines implement quality through systematic validation and monitoring at every stage. Data pipeline issues represent a major source of ML failures.\index{Pipeline!failure modes} Schema changes breaking downstream processing, distribution drift degrading model accuracy, and data corruption silently introducing errors account for a substantial fraction of production incidents [@sculley2015hidden]. These failures are insidious because they rarely cause obvious system crashes; instead, they slowly degrade model performance in ways that become apparent only after affecting users. Achieving quality therefore demands proactive monitoring and validation that catches issues before they cascade into model failures.
|
||
|
||
::: {.callout-war-story title="Microsoft Tay (2016)"}
|
||
**The Context**: Microsoft launched "Tay," an AI chatbot designed to learn from user interactions on Twitter in real-time.
|
||
|
||
**The Failure**: The data pipeline ingested user tweets directly into the model's retraining loop without sufficient filtering or "safety" checks. Users quickly realized this and coordinated a "data poisoning" attack, tweeting offensive and racist content at the bot.
|
||
|
||
**The Consequence**: Within 24 hours, the model had learned from this poisoned data and began generating hate speech autonomously. Microsoft had to shut down the service immediately, suffering significant reputational damage.
|
||
|
||
**The Systems Lesson**: Data pipelines are not just plumbing; they are the immune system of your model. Ingesting uncurated, user-generated content without robust filtering is a security vulnerability. "Garbage in, garbage out" happens at the speed of software [@wolf2017we].
|
||
:::
|
||
|
||
Production teams implement monitoring at scale through severity-based alerting systems where different failure types trigger different response protocols. The most critical alerts indicate complete system failure: the pipeline has stopped processing entirely, showing zero throughput for more than 5 minutes, or a primary data source has become unavailable. These situations demand immediate attention because they halt all downstream model training or serving. More subtle degradation patterns require different detection strategies. When throughput drops to 80% of baseline levels, error rates climb above 5%, or quality metrics drift more than 2 standard deviations from training data characteristics, the system signals degradation requiring urgent but not immediate attention. These gradual failures often prove more dangerous than complete outages because they persist undetected for hours or days, silently corrupting model inputs and degrading prediction quality.
|
||
|
||
Consider how these principles apply to a recommendation system processing user interaction events. With a baseline throughput of 50,000 records per second, the monitoring system tracks several interdependent signals. Instantaneous throughput alerts fire if processing drops below 40,000 records per second for more than 10 minutes, accounting for normal traffic variation while catching genuine capacity or processing problems. Each feature in the data stream has its own quality profile: if a feature like user_age shows null values in more than 5% of records when the training data contained less than 1% nulls, something has likely broken in the upstream data source. Duplicate detection runs on sampled data, watching for the same event appearing multiple times—a pattern that might indicate retry logic gone wrong or a database query accidentally returning the same records repeatedly.
|
||
|
||
These monitoring dimensions become particularly important when considering end-to-end latency. The system must track not just whether data arrives, but how long it takes to flow through the entire pipeline from the moment an event occurs to when the resulting features become available for model inference. When 95th percentile latency exceeds 30 seconds in a system with a 10-second service level agreement, the monitoring system needs to pinpoint which pipeline stage introduced the delay: ingestion, transformation, validation, or storage.
|
||
|
||
Quality monitoring extends beyond simple schema validation to statistical properties\index{Data Quality!statistical monitoring} that capture whether serving data resembles training data. Rather than just checking that values fall within valid ranges, production systems track rolling statistics over 24-hour windows. For numerical features like transaction_amount or session_duration, the system computes means and standard deviations continuously, then applies statistical tests like the Kolmogorov-Smirnov test[^fn-ks-test-drift] to compare serving distributions against training distributions.
|
||
|
||
```{python}
|
||
#| label: ks-test-calc
|
||
#| echo: false
|
||
# ┌─────────────────────────────────────────────────────────────────────────────
|
||
# │ K-S TEST CALCULATION
|
||
# ├─────────────────────────────────────────────────────────────────────────────
|
||
# │ Context: Callout "Detecting Drift with K-S Test"
|
||
# │
|
||
# │ Goal: Provide a concrete numeric threshold for detecting distribution drift.
|
||
# │ Show: That a difference of 0.043 marks the threshold for 1000 samples.
|
||
# │ How: Calculate the K-S critical value $D_{\text{crit}}$ at $\alpha = 0.05$.
|
||
# │
|
||
# │ Imports: math (sqrt), mlsys.constants (KS_TEST_COEFFICIENT),
|
||
# │ mlsys.formatting (fmt)
|
||
# │ Exports: ks_dcrit_str
|
||
# └─────────────────────────────────────────────────────────────────────────────
|
||
import math
|
||
from mlsys.constants import KS_TEST_COEFFICIENT
|
||
from mlsys.formatting import fmt, check
|
||
|
||
# ┌── P.I.C.O. ISOLATED SCENARIO ───────────────────────────────────────────────
|
||
class KSTest:
|
||
"""
|
||
Namespace for K-S Test Critical Value calculation.
|
||
Scenario: Detecting drift with n=1000 samples at alpha=0.05.
|
||
"""
|
||
|
||
# ┌── 1. PARAMETERS (Inputs) ───────────────────────────────────────────────
|
||
n = 1000
|
||
coeff = KS_TEST_COEFFICIENT # 1.36 for alpha=0.05
|
||
|
||
# ┌── 2. CALCULATION (The Physics) ─────────────────────────────────────────
|
||
d_crit = coeff / math.sqrt(n)
|
||
|
||
# ┌── 3. INVARIANTS (Guardrails) ───────────────────────────────────────────
|
||
check(d_crit > 0, "Critical value must be positive.")
|
||
check(d_crit <= 0.1, f"Critical value ({d_crit:.3f}) is too loose for n=1000. Check formula.")
|
||
|
||
# ┌── 4. OUTPUTS (Formatting) ──────────────────────────────────────────────
|
||
ks_dcrit_str = fmt(d_crit, precision=3, commas=False)
|
||
|
||
# ┌── EXPORTS (Bridge to Text) ─────────────────────────────────────────────────
|
||
ks_dcrit_str = KSTest.ks_dcrit_str
|
||
```
|
||
|
||
::: {.callout-example title="Detecting Drift with K-S Test"}
|
||
**Scenario**: Monitoring `session_duration` distribution stability between training ($P$) and serving ($Q$).
|
||
|
||
**Methodology**
|
||
|
||
1. **Compute CDFs**: Calculate Cumulative Distribution Functions for both datasets.
|
||
2. **Calculate Statistic ($D_{KS}$)**: Find the maximum absolute difference between the CDFs. Let $F_P(x)$ and $F_Q(x)$ denote the cumulative distribution functions of the training ($P$) and serving ($Q$) datasets.
|
||
$$D_{KS} = \max_x |F_P(x) - F_Q(x)|$$
|
||
|
||
3. **Determine Significance**: Compare $D_{KS}$ to critical value $D_{crit}$ based on sample size $n$ and confidence level ($\alpha = 0.05$).
|
||
$$D_{crit} \approx \frac{1.36}{\sqrt{n}}$$
|
||
|
||
**Example Calculation**
|
||
|
||
- Sample size $n = 1000$.
|
||
- Critical value $D_{crit} \approx 1.36 / \sqrt{1000}$ ≈ `{python} ks_dcrit_str`.
|
||
- If observed max difference $D_{KS} = 0.08$, then 0.08 > `{python} ks_dcrit_str` → **Reject Null Hypothesis**. Significant drift detected. Trigger retraining or investigation.
|
||
:::
|
||
|
||
[^fn-ks-test-drift]: **Kolmogorov-Smirnov (K-S) Test**: A non-parametric test measuring the maximum distance between two cumulative distribution functions, requiring no assumptions about underlying distributions. In ML pipelines, the K-S test serves as the primary continuous-feature drift detector: comparing serving distributions against training baselines, with p-values below 0.05 triggering investigation. Its distribution-free nature makes it robust across feature types, but it applies only to univariate continuous features -- categorical drift requires PSI or chi-squared tests instead. \index{K-S Test!drift detection}
|
||
|
||
The K-S test is one tool for detecting drift in continuous features; @sec-data-engineering-detecting-responding-data-drift-509a provides the complete taxonomy of distribution shifts (covariate, label, concept, and label-quality drift) along with Population Stability Index (PSI) and KL divergence metrics for operationalizing the Degradation Equation.
|
||
|
||
Categorical features require different statistical approaches. Instead of comparing means and variances, monitoring systems track category frequency distributions. When new categories appear that never existed in training data, or when existing categories shift substantially in relative frequency—say, the proportion of "mobile" versus "desktop" traffic changes by more than 20%, the system flags potential data quality issues or genuine distribution shifts. This statistical vigilance catches subtle problems that simple schema validation misses entirely: imagine if age values remain in the valid range of 18-95, but the distribution shifts from primarily 25-45 year olds to primarily 65+ year olds, indicating the data source has changed in ways that will affect model performance.
|
||
|
||
Validation at the pipeline level encompasses multiple strategies working together. Schema validation executes synchronously as data enters the pipeline, rejecting malformed records immediately before they can propagate downstream. Modern tools like TensorFlow Data Validation (TFDV) [@breck2019data] automatically infer schemas from training data, capturing expected data types, value ranges, and presence requirements.
|
||
|
||
This synchronous validation remains simple and fast, checking properties that can be evaluated on individual records in microseconds. More sophisticated validation that requires comparing serving data against training data distributions or aggregating statistics across many records must run asynchronously to avoid blocking the ingestion pipeline. Statistical validation\index{Statistical Validation!sampling strategies} systems typically sample 1-10% of serving traffic, enough to detect meaningful shifts while avoiding the computational cost of analyzing every record. These samples accumulate in rolling windows, commonly 1 hour, 24 hours, and 7 days, with different windows revealing different patterns. Hourly windows detect sudden shifts like a data source failing over to a backup with different characteristics, while weekly windows reveal gradual drift in user populations or behavior.
|
||
|
||
The most insidious validation challenge arises from training-serving skew\index{Training-Serving Skew!causes}: the failure mode where features are computed differently in training versus serving. This typically happens when training pipelines process data in batch using one set of libraries or logic, while serving systems compute features in real-time using different implementations. Even seemingly minor discrepancies (a materialized view[^fn-materialized-view-skew] refreshed weekly versus a complete join recomputed daily) can cause accuracy drops of 10-15% that take weeks to diagnose because the system produces no obvious errors. We formalize this as the **Consistency Imperative** in @sec-data-engineering-ensuring-trainingserving-consistency-c683 and quantify its impact with a concrete example. Detecting training-serving skew requires infrastructure that can recompute training features on serving data for comparison, sampling raw serving data and processing it through both pipelines to measure discrepancies. @sec-ml-operations examines operational monitoring infrastructure for this challenge at scale.
|
||
|
||
[^fn-materialized-view-skew]: **Materialized View**: A database optimization that pre-computes and caches query results as a physical table. For ML systems, the risk is structural: when a materialized view refreshes on a different schedule in training versus serving environments, the feature values the model trained on diverge from what it receives at inference -- a primary mechanism of training-serving skew that produces 10--15% accuracy drops with no error messages. \index{Materialized View!training-serving skew}
|
||
|
||
### Data Quality as Code {#sec-data-engineering-data-quality-code-1cca}
|
||
|
||
Just as unit tests protect software systems, data expectation tests\index{Data Expectation Tests} protect ML pipelines. Using libraries like Great Expectations\index{Great Expectations!data validation} or Pandera,\index{Pandera!schema validation} teams codify quality expectations as executable assertions (@lst-data-expectations) that run on every pipeline execution.
|
||
|
||
::: {.callout-perspective title="Mechanical vs. Semantic Quality"}
|
||
**Why Data Validation feels different from Unit Testing:**
|
||
|
||
In traditional software, quality is **Mechanical**. A null pointer is always a bug. An integer overflow is always a crash. These are binary, deterministic failures.
|
||
|
||
In ML systems, data quality has a second, softer dimension: **Semantic Quality**\index{Data Quality!mechanical vs. semantic}.
|
||
* **Mechanical Check**: "Is `age` an integer?" (Yes/No).
|
||
* **Semantic Check**: "Is the `age` distribution shifting?" (Probabilistic).
|
||
|
||
A dataset can be mechanically perfect (no nulls, correct types) but semantically broken (e.g., all users are suddenly 25 years old due to a default value change). Robust ML systems must validate both the **Container** (Mechanical) and the **Content** (Semantic).
|
||
:::
|
||
|
||
The following listing demonstrates how these mechanical expectations translate to executable assertions using the Great Expectations library.
|
||
|
||
::: {#lst-data-expectations lst-cap="**Data Quality Assertions**: Executable data contracts catch schema violations, missing values, and invalid entries before training begins. Production systems using this pattern detect approximately 60% of data issues at pipeline execution time, preventing cascading failures that would otherwise propagate to model training."}
|
||
```{.python}
|
||
import great_expectations as gx
|
||
|
||
# Create a data context and load training data
|
||
context = gx.get_context()
|
||
batch = context.get_batch("training_users")
|
||
|
||
# Define an expectation suite as executable quality contract
|
||
suite = context.create_expectation_suite("user_data_quality")
|
||
|
||
# Range validation: prevents physiologically impossible values
|
||
batch.expect_column_values_to_be_between(
|
||
column="age", min_value=0, max_value=120
|
||
)
|
||
|
||
# Null detection: ensures primary key integrity for joins
|
||
batch.expect_column_values_to_not_be_null(column="user_id")
|
||
|
||
# Uniqueness: prevents duplicate training examples
|
||
batch.expect_column_values_to_be_unique(column="user_id")
|
||
|
||
# Categorical validation: detects unexpected values
|
||
# from upstream changes
|
||
batch.expect_column_distinct_values_to_be_in_set(
|
||
column="country_code", value_set=["US", "CA", "UK", "DE", "FR"]
|
||
)
|
||
|
||
# Run validation and fail pipeline if expectations not met
|
||
results = context.run_validation_operator(
|
||
"action_list_operator", assets_to_validate=[batch]
|
||
)
|
||
if not results["success"]:
|
||
raise ValueError(f"Data quality check failed: {results}")
|
||
```
|
||
:::
|
||
|
||
CI/CD integration runs expectations in the deployment pipeline. Expectation violations fail deployments before bad data reaches training. A pipeline structured as data ingestion followed by data validation followed by training blocks deployment when validation detects anomalies like age values of 150, triggering alerts for investigation.
|
||
|
||
Expectation suites as artifacts version alongside training code. When training code changes, expectation updates help keep data contracts evolving together. This coupling reduces the risk of silent divergence where code assumes data properties that the upstream pipeline no longer provides.
|
||
|
||
This pattern catches approximately 60% of production data issues before they reach training, based on industry experience with tools like Great Expectations, Pandera, and Pydantic. The remaining issues require runtime monitoring, as some quality problems only emerge in the full production data stream.
|
||
|
||
### Data Drift Detection and Response {#sec-data-engineering-detecting-responding-data-drift-509a}
|
||
|
||
\index{Data Drift!detection}\index{Distribution Shift!monitoring}ML models rest on the assumption that production data resembles training data.\index{Distribution Assumption!ML models} When this assumption breaks—not through immediate schema violations but through gradual statistical shifts—model performance degrades silently without obvious errors or system failures. Unlike the validation and monitoring techniques we have examined that catch immediate data quality issues, drift detection identifies gradual statistical changes in data distributions that compound over time to undermine model effectiveness. Production experience reveals that drift detection and response consume 30–40% of ongoing ML operations effort, making this a core data engineering responsibility rather than an optional advanced topic. @sec-ml-operations builds on this foundation to address operational response orchestration and automated retraining pipelines at scale.
|
||
|
||
The **Degradation Equation** introduced in @sec-introduction-ml-vs-traditional-software-e19a (@eq-degradation) now becomes actionable: the divergence term $D(P_t \| P_0)$ is exactly what PSI and KL divergence measure. Two key metrics operationalize this measurement: the **Population Stability Index (PSI)**\index{Population Stability Index (PSI)!definition}, which quantifies how much a categorical or binned feature distribution has shifted (values above 0.2 indicate significant drift), and **Kullback-Leibler (KL) Divergence**\index{KL Divergence!definition}, which measures the information-theoretic distance between continuous distributions. When PSI exceeds 0.2 or KL divergence crosses threshold, the system signals that $D$ has grown large enough to materially impact $\text{Accuracy}(t)$.
|
||
|
||
Understanding the three core types of drift enables targeted detection and response strategies. Each type manifests differently in production systems and requires distinct monitoring approaches.
|
||
|
||
#### Covariate Shift {#sec-data-engineering-covariate-shift-b3d9}
|
||
|
||
Covariate shift\index{Covariate Shift!definition} occurs when input feature distributions change while the relationship between features and labels remains constant: $P(X)$ changes but $P(Y|X)$ stays the same. A medical imaging system trained on one camera model might see production data from a different camera manufacturer. The disease-image relationship remains unchanged (same pathologies produce same visual indicators), but pixel value distributions shift due to different sensor characteristics, color calibration, or image processing pipelines. Detection focuses on monitoring feature distributions using statistical metrics like PSI or KL divergence applied to input features.
|
||
|
||
#### Label Shift {#sec-data-engineering-label-shift-dd6c}
|
||
|
||
Label shift\index{Label Shift!definition}\index{Distribution Shift!label shift} occurs when the output label distribution changes while the relationship between labels and features remains constant: $P(Y)$ changes but $P(X|Y)$ stays the same. Disease prevalence might change seasonally (flu cases spike in winter) while symptoms remain consistent predictors of each disease. A recommendation system might see label shift when new product categories launch, changing the distribution of user preferences without altering what makes products appealing within each category. Detection monitors prediction distributions for shifts in relative frequencies of predicted classes, which can be done without ground truth labels by tracking model output distributions.
|
||
|
||
#### Concept Drift {#sec-data-engineering-concept-drift-2b83}
|
||
|
||
Concept drift\index{Concept Drift!definition} represents the most challenging case: the relationship between features and labels changes, meaning $P(Y|X)$ evolves over time [@gama2014survey]. Medical treatment protocols change, altering disease outcomes for given symptoms. User preferences shift as social trends evolve, changing what product features drive purchases. Fraud patterns evolve as attackers adapt to detection systems. Concept drift requires ground truth labels for detection since we must monitor whether the feature-to-label relationship has changed, making it inherently more difficult and delayed than detecting covariate or label shift.
|
||
|
||
#### Label Quality Drift {#sec-data-engineering-label-quality-drift-1ce7}
|
||
|
||
Label quality drift\index{Label Quality Drift!detection}[^fn-label-drift-invisible] represents a meta-level shift distinct from the three distribution shifts above: the reliability of ground truth labels degrades over time even when the underlying data distributions remain stable. This drift type proves particularly insidious because standard feature distribution monitoring fails to detect it. Crowdsourced labels may degrade as annotator pools change, training materials become outdated, or labeling guidelines evolve without corresponding model updates. Automated labeling systems accumulate errors as the models powering them drift from their original operating conditions. A recommendation system using click feedback as implicit labels may see label quality degrade as user behavior becomes more exploratory, as bot traffic patterns change, or as interface modifications alter how users interact with content.
|
||
|
||
Detection requires monitoring annotation consistency rather than feature distributions. Inter-annotator agreement metrics like Cohen's kappa[^fn-cohens-kappa-label-quality] ($\kappa$) provide quantitative assessment. Let $p_o$ represent observed agreement between annotators and $p_e$ represent agreement expected by chance. @eq-cohens-kappa defines the statistic:
|
||
|
||
[^fn-cohens-kappa-label-quality]: **Cohen's Kappa ($\kappa$)**: Introduced by psychologist Jacob Cohen in 1960 to measure inter-rater agreement while correcting for chance agreement, which raw percentage agreement ignores. The correction matters: two annotators labeling 90% of images as "not spam" will agree 81% of the time by pure chance, making raw agreement a dangerously misleading quality metric. For ML label quality monitoring, $\kappa$ below 0.4 signals unreliable training data whose noise the model will inherit. \index{Cohen's Kappa!label quality}
|
||
|
||
$$
|
||
\kappa = \frac{p_o - p_e}{1 - p_e}
|
||
$$ {#eq-cohens-kappa}
|
||
|
||
Monitoring $\kappa$ over time windows reveals degradation trends. A medical imaging annotation project might establish a baseline $\kappa = 0.85$ (substantial agreement) during initial data collection, then observe decline to $\kappa = 0.72$ (moderate agreement) after six months as new annotators join without receiving equivalent domain training.
|
||
|
||
For systems with calibrated model probabilities, label confidence entropy provides an alternative detection signal. Let $p_i$ represent the model's probability assigned to label category $i$. @eq-label-entropy defines this measure:
|
||
|
||
$$
|
||
H_{\text{label}} = -\sum_i p_i \log p_i
|
||
$$ {#eq-label-entropy}
|
||
|
||
Rising entropy in model confidence distributions suggests increasing ambiguity or mislabeling in training data, as the model learns from inconsistent supervision.
|
||
|
||
Mitigation strategies depend on root cause analysis. Annotator retraining addresses systematic errors from unclear guidelines at low cost with high effectiveness. Multi-annotator voting with majority or consensus rules provides very high accuracy for high-stakes domains but significantly increases annotation costs. Model-assisted labeling reduces annotator fatigue but risks introducing bias if the assisting model has its own systematic errors. Expert review sampling, where domain specialists audit a random sample of annotations, enables root cause analysis when quality decline is detected but provides medium coverage of the overall annotation stream.
|
||
|
||
[^fn-label-drift-invisible]: **Label Quality Drift**: Degradation in annotation reliability over time, distinct from distribution shifts in the data itself. This drift type is invisible to standard feature monitoring because the features remain stable while the labels degrade -- annotator fatigue, pool turnover, or guideline evolution silently corrupt the ground truth the model learns from. Detection requires monitoring inter-annotator agreement ($\kappa$) over rolling time windows and comparing automated labels against periodic expert audits. \index{Label Quality Drift!silent corruption}
|
||
|
||
Operationalizing the PSI and KL divergence metrics introduced above\index{Population Stability Index (PSI)!drift detection}\index{KL Divergence!distribution comparison} requires connecting them to automated alerts and retraining workflows. Data engineering is responsible for *defining* the drift thresholds (PSI > 0.2, KL divergence above a domain-specific threshold), selecting the monitoring window sizes (hourly for sudden shifts, weekly for gradual trends), and instrumenting pipelines to compute these metrics continuously. The *operational infrastructure* for alerting, escalation, and automated retraining is a core concern of MLOps; @sec-ml-operations provides comprehensive coverage of tiered alerting strategies, cold start monitoring, and automated response orchestration.
|
||
|
||
Drift detection is one dimension of the quality pillar, focused on identifying statistical changes in data distributions over time. Detecting issues, however, is only half the challenge; the other half is ensuring systems continue operating effectively even when problems surface. This leads us from quality monitoring to the reliability pillar, which addresses how pipelines maintain service continuity under adverse conditions.
|
||
|
||
### Reliability Through Graceful Degradation {#sec-data-engineering-reliability-graceful-degradation-f83d}
|
||
|
||
\index{Graceful Degradation!data pipelines}\index{Reliability!error handling}Reliability ensures systems continue operating when problems occur. Pipelines face constant challenges: data sources become temporarily unavailable, network partitions separate components, upstream schema changes break parsing logic, or unexpected load spikes exhaust resources. Robust systems handle these failures gracefully through systematic failure analysis, intelligent error handling, and automated recovery strategies that maintain service continuity even under adverse conditions.
|
||
|
||
Systematic failure mode analysis for ML data pipelines reveals predictable patterns that require specific engineering countermeasures. Data corruption failures occur when upstream systems introduce subtle format changes, encoding issues, or field value modifications that pass basic validation but corrupt model inputs. A date field switching from "YYYY-MM-DD" to "MM/DD/YYYY" format might not trigger schema validation but will break any date-based feature computation. Schema evolution\index{Schema Evolution!failures}[^fn-schema-evolution-skew] failures happen when source systems add fields, rename columns, or change data types without coordination, breaking downstream processing assumptions that expected specific field names or types. Resource exhaustion manifests as gradually degrading performance when data volume growth outpaces capacity planning, eventually causing pipeline failures during peak load periods.
|
||
|
||
[^fn-schema-evolution-skew]: **Schema Evolution**: This failure mode arises from a lack of contract testing between upstream data producers and downstream ML consumers. While "loud" failures like a renamed column break explicit assumptions and cause immediate pipeline crashes, "silent" failures are more dangerous. A field changing type from integer to string can pass validation but corrupt feature logic, often going undetected for weeks and degrading model accuracy by over 5% before discovery. \index{Schema Evolution!silent failure}
|
||
|
||
Building on this failure analysis, effective error handling strategies ensure problems are contained and recovered from systematically. Implementing intelligent retry logic\index{Retry Logic!exponential backoff} for transient errors (network interruptions or temporary service outages) requires exponential backoff strategies to avoid overwhelming recovering services. A simple linear retry that attempts reconnection every second would flood a struggling service with connection attempts, potentially preventing its recovery. Exponential backoff—retrying after 1 second, then 2 seconds, then 4 seconds, doubling with each attempt—gives services breathing room to recover while still maintaining persistence. Many ML systems employ the concept of dead letter queues\index{Reliability!dead letter queues}, using separate storage for data that fails processing after multiple retry attempts. This allows for later analysis and potential reprocessing of problematic data without blocking the main pipeline [@kleppmann2017designing]. A pipeline processing financial transactions that encounters malformed data can route it to a dead letter queue rather than losing critical records or halting all processing.
|
||
|
||
In ML systems, dead letter queues serve dual purposes beyond failure analysis. Production teams implement systematic review of DLQ contents to identify: (1) schema violations indicating upstream changes, (2) edge case patterns the model should handle, and (3) data quality issues requiring source system fixes. For example, a fraud detection system's DLQ revealed transactions from a new payment type the model had never seen, prompting targeted data collection and retraining rather than simply logging the failures. This transforms DLQs from passive error storage into active sources for identifying model blind spots and driving improvement.
|
||
|
||
Moving beyond ad-hoc error handling, cascade failure prevention requires circuit breaker\index{Reliability!circuit breakers}[^fn-circuit-breaker-ml] patterns and bulkhead isolation to prevent single component failures from propagating throughout the system. When a feature computation service fails, the circuit breaker pattern stops calling that service after detecting repeated failures, preventing the caller from waiting on timeouts that would cascade into its own failure.
|
||
|
||
[^fn-circuit-breaker-ml]: **Circuit Breaker**: Named for its three-state behavior -- closed (normal flow), open (faults blocked), half-open (recovery probe) -- after the electrical safety device that interrupts current on overload. In ML data pipelines, the circuit breaker prevents a failing feature computation service from cascading timeouts through the entire serving path: once failure count exceeds a threshold, the breaker opens and the pipeline falls back to cached or default features rather than waiting on a dead service. \index{Circuit Breaker!cascade prevention}
|
||
|
||
Automated recovery engineering extends beyond simple retry logic. Progressive timeout increases prevent overwhelming struggling services while maintaining rapid recovery for transient issues: initial requests timeout after 1 second, but after detecting service degradation, timeouts extend to 5 seconds, then 30 seconds, giving the service time to stabilize. Multi-tier fallback systems provide degraded service when primary data sources fail: serving slightly stale cached features when real-time computation fails, or using approximate features when exact computation times out. A recommendation system unable to compute user preferences from the past 30 days might fall back to preferences from the past 90 days, providing somewhat less accurate but still useful recommendations rather than failing entirely. Comprehensive alerting and escalation procedures ensure human intervention occurs when automated recovery fails, with sufficient diagnostic information captured during the failure to enable rapid debugging.
|
||
|
||
These patterns—retry logic, dead letter queues, circuit breakers—are the runtime error handlers of our dataset compiler: they catch malformed inputs without halting the entire compilation. With these defensive patterns established, we can now examine the specific ingestion mechanisms that feed data into the pipeline. The choice of ingestion pattern—batch versus streaming, ETL versus ELT—determines how quickly new data reaches the model, how much infrastructure the system requires, and how the reliability patterns above are concretely deployed.
|
||
|
||
### Data Ingestion {#sec-data-engineering-data-ingestion-8efc}
|
||
|
||
\index{Data Ingestion!pipeline boundary}\index{IO Bottleneck!data loading}Continuing the compilation analogy, ingestion is the *lexer*: it reads raw source (data streams) and tokenizes them into well-formed records that the rest of the pipeline can process.
|
||
|
||
A critical and often overlooked constraint in ingestion design is the **Input/Output (IO) Bottleneck**. We invest heavily in expensive GPUs, but their utilization depends entirely on whether data arrives fast enough to keep them busy. Training speed is governed by a simple inequality: $T_{step} = \max(T_{compute}, T_{io})$.
|
||
If your data pipeline cannot decode images fast enough to keep the GPU busy, your expensive accelerator sits idle. This phenomenon creates a "Choke Point" where adding more GPUs yields zero speedup—a counterintuitive result that frustrates teams who expect linear scaling from hardware investments. This bottleneck frequently occurs in computer vision, where decoding high-resolution JPEG images on the CPU consumes more time than the GPU requires to perform the forward and backward passes of a lightweight model like ResNet-18. Typically, training ResNet-50 on an H100 requires at least `{python} resnet_worker_count_str` CPU workers just to keep the GPU from starving.
|
||
|
||
Examine @fig-dataloader-choke-point to see this **Dataloader Choke Point** in action. Notice the "Starvation Region" on the left where the CPU limits performance: no matter how powerful your GPU, training throughput is capped by data loading speed until enough workers are allocated to saturate the accelerator. Throughput levels are representative and vary by model and hardware.
|
||
|
||
::: {#fig-dataloader-choke-point fig-env="figure" fig-pos="htb" fig-cap="**The Dataloader Choke Point.** Training Throughput (img/s) vs. Number of DataLoader Workers. The blue curve shows CPU throughput scaling linearly with workers until hitting disk limits. The red dashed line is the GPU's consumption capacity (e.g., ResNet-50 at ~3,000 img/s; illustrative). The system is bottlenecked by whichever is lower. In the 'Starvation Region' (left), the GPU is idle waiting for data. In the 'Saturated Region' (right), the GPU is fully utilized, and adding more workers wastes CPU memory." fig-alt="Line chart of Throughput vs Workers. Blue line (CPU) rises linearly. Red line (GPU) is flat. Where CPU < GPU, system is starved. Where CPU > GPU, system is saturated."}
|
||
```{python}
|
||
#| echo: false
|
||
# ┌─────────────────────────────────────────────────────────────────────────────
|
||
# │ FIGURE: DATALOADER CHOKE POINT
|
||
# ├─────────────────────────────────────────────────────────────────────────────
|
||
# │ Context: @fig-dataloader-choke-point in the data ingestion discussion
|
||
# │
|
||
# │ Goal: Visualize the min(CPU supply, GPU demand) bottleneck.
|
||
# │ Show: The starvation vs. saturation regions in training throughput.
|
||
# │ How: Plot throughput curves showing the "choke point" between CPU and GPU capacity.
|
||
# │
|
||
# │ Imports: numpy (np), mlsys.viz (viz)
|
||
# │ Exports: (figure — no string exports)
|
||
# └─────────────────────────────────────────────────────────────────────────────
|
||
import numpy as np
|
||
from mlsys import viz
|
||
|
||
fig, ax, COLORS, plt = viz.setup_plot()
|
||
|
||
# --- Plot: The Dataloader Choke Point ---
|
||
workers = np.arange(1, 17)
|
||
cpu_throughput = np.minimum(250 * workers, 3500)
|
||
gpu_limit = 3000
|
||
|
||
ax.plot(workers, cpu_throughput, 'o-', color=COLORS['BlueLine'], label='DataLoader (CPU)')
|
||
ax.axhline(gpu_limit, linestyle='--', color=COLORS['RedLine'], label='GPU Capacity')
|
||
ax.fill_between(workers, 0, np.minimum(cpu_throughput, gpu_limit), color=COLORS['BlueL'], alpha=0.2)
|
||
|
||
ax.text(4, 1500, "Starvation", color=COLORS['BlueLine'], fontweight='bold', ha='center', bbox=dict(facecolor='white', alpha=0.8, edgecolor='none', pad=0.5))
|
||
ax.text(12, 3200, "Saturated", color=COLORS['RedLine'], fontweight='bold', ha='center', bbox=dict(facecolor='white', alpha=0.8, edgecolor='none', pad=0.5))
|
||
ax.set_xlabel('DataLoader Workers')
|
||
ax.set_ylabel('Throughput (img/s)')
|
||
ax.legend(loc='lower right', fontsize=8)
|
||
plt.show()
|
||
```
|
||
:::
|
||
|
||
The two primary ingestion patterns, batch and streaming, together with the ETL and ELT processing paradigms that govern how transformations are applied during ingestion, shape the cost, latency, and reliability profile of every downstream pipeline stage.
|
||
|
||
#### Batch vs. Streaming Ingestion Patterns {#sec-data-engineering-batch-vs-streaming-ingestion-patterns-e1b2}
|
||
|
||
\index{Batch Processing!ingestion pattern}\index{Stream Processing!real-time ingestion}ML systems follow two primary ingestion patterns that shape how systems balance latency, throughput, cost, and complexity.
|
||
|
||
Batch ingestion\index{Batch Ingestion!definition} involves collecting data in groups or batches over a specified period before processing. This method proves appropriate when real-time data processing is not critical and data can be processed at scheduled intervals. The batch approach enables efficient use of computational resources by amortizing startup costs across large data volumes and processing when resources are available or least expensive. For example, a retail company might use batch ingestion to process daily sales data overnight, updating their ML models for inventory prediction each morning [@akidau2015dataflow]. The batch job might process gigabytes of transaction data using dozens of machines for 30 minutes, then release those resources for other workloads. This scheduled processing proves far more cost-effective than maintaining always-on infrastructure, particularly when slight staleness in predictions does not affect business outcomes.
|
||
|
||
Batch processing also simplifies error handling and recovery. When a batch job fails midway, the system can retry the entire batch or resume from checkpoints without complex state management. Data scientists can easily inspect failed batches, understand what went wrong, and reprocess after fixes. The deterministic nature of batch processing (processing the same input data always produces the same output) simplifies debugging and validation. These characteristics make batch ingestion attractive for ML workflows even when real-time processing is technically feasible but not required.
|
||
|
||
In contrast to this scheduled approach, stream ingestion\index{Stream Ingestion!definition} processes data in real-time as it arrives, consuming events continuously rather than waiting to accumulate batches. This pattern is essential for applications requiring immediate data processing, scenarios where data loses value quickly, and systems that need to respond to events as they occur. A financial institution might use stream ingestion for real-time fraud detection, processing each transaction as it occurs to flag suspicious activity immediately before completing the transaction. The value of fraud detection drops dramatically if detection occurs hours after the fraudulent transaction completes—by then money has been transferred and accounts compromised.
|
||
|
||
However, stream processing introduces complexity that batch processing avoids. The system must handle backpressure\index{Backpressure!streaming systems} when downstream systems cannot keep pace with incoming data rates. During traffic spikes, when a sudden surge produces data faster than processing capacity, the system must either buffer data (requiring memory and introducing latency), sample (losing some data), or push back to producers (potentially causing their failures). Data freshness Service Level Agreements (SLAs)\index{SLA!data freshness}\index{Data Freshness!SLA} formalize these requirements, specifying maximum acceptable delays between data generation and availability for processing. Meeting a 100-millisecond freshness SLA requires different infrastructure than meeting a 1-hour SLA, affecting everything from networking to storage to processing architectures.
|
||
|
||
Recognizing the limitations of either approach alone, many production ML systems employ hybrid approaches that combine batch and stream ingestion. A recommendation system might use streaming ingestion for real-time user interactions to update session-based recommendations immediately, while using batch ingestion for overnight processing of user profiles and item features.
|
||
|
||
Production systems must balance cost versus latency trade-offs when selecting patterns: real-time processing is often materially more expensive than batch processing, commonly by an order of magnitude or more in total cost per byte processed. This cost differential arises from several factors: streaming systems require always-on infrastructure rather than schedulable resources; they maintain redundant processing for fault tolerance to ensure no events are lost; they need low-latency networking and storage to meet millisecond-scale SLAs; and they cannot benefit from the economies of scale that batch processing achieves by amortizing startup costs across large data volumes. A batch job processing one terabyte might use 100 machines for 10 minutes, while a streaming system processing the same data over 24 hours needs dedicated resources continuously available. This difference drives many architectural decisions about which data truly requires real-time processing. The following analysis quantifies *the cost of real-time*.
|
||
|
||
```{python}
|
||
#| label: realtime-cost-calc
|
||
#| echo: false
|
||
# ┌─────────────────────────────────────────────────────────────────────────────
|
||
# │ REAL-TIME COST CALCULATION
|
||
# ├─────────────────────────────────────────────────────────────────────────────
|
||
# │ Context: Callout "The Cost of Real-Time"
|
||
# │
|
||
# │ Goal: Contrast the dollar costs of streaming vs. batch ingestion.
|
||
# │ Show: The order-of-magnitude premium paid for real-time data processing.
|
||
# │ How: Calculate daily processing costs for a 1M event/sec data stream.
|
||
# │
|
||
# │ Imports: mlsys.constants (KB, GB, TB), mlsys.formatting (fmt)
|
||
# │ Exports: throughput_str, hourly_tb_str, stream_cost_str, batch_cost_str,
|
||
# │ cost_ratio_str, events_per_sec_str, event_size_kb_str,
|
||
# │ stream_cores_str, stream_hours_str, stream_cost_per_hr_str,
|
||
# │ batch_core_hours_str
|
||
# └─────────────────────────────────────────────────────────────────────────────
|
||
from mlsys.constants import KB, GB, TB
|
||
from mlsys.formatting import fmt
|
||
|
||
# ┌── P.I.C.O. ISOLATED SCENARIO ───────────────────────────────────────────────
|
||
class RealtimeCost:
|
||
"""Streaming vs. batch ingestion daily cost comparison."""
|
||
|
||
# ┌── 1. PARAMETERS (Inputs) ──────────────────────────────────────────────
|
||
events_per_sec = 1_000_000
|
||
event_size_kb = 1
|
||
stream_cores = 100
|
||
stream_hours = 24
|
||
stream_cost_per_hr = 0.05
|
||
|
||
# ┌── 2. CALCULATION (The Physics) ────────────────────────────────────────
|
||
throughput_gbs = (events_per_sec * event_size_kb * KB).m_as(GB) # GB/s
|
||
stream_cost_day = stream_cores * stream_hours * stream_cost_per_hr
|
||
hourly_data_tb = (throughput_gbs * SEC_PER_HOUR * GB).m_as(TB) # TB per hour
|
||
batch_core_hours = 200 * (10 / 60) * 24 # 200 cores × 10min/hr × 24 hours
|
||
batch_cost_day = batch_core_hours * stream_cost_per_hr
|
||
cost_ratio = stream_cost_day / batch_cost_day
|
||
|
||
# ┌── 4. OUTPUTS (Formatting) ─────────────────────────────────────────────
|
||
throughput_str = fmt(throughput_gbs, precision=0, commas=False)
|
||
hourly_tb_str = fmt(hourly_data_tb, precision=1, commas=False)
|
||
stream_cost_str = fmt(stream_cost_day, precision=0, commas=False)
|
||
batch_cost_str = fmt(batch_cost_day, precision=0, commas=False)
|
||
cost_ratio_str = fmt(cost_ratio, precision=0, commas=False)
|
||
events_per_sec_str = f"{events_per_sec / MILLION:.0f}M"
|
||
event_size_kb_str = fmt(event_size_kb, precision=0, commas=False)
|
||
stream_cores_str = fmt(stream_cores, precision=0, commas=False)
|
||
stream_hours_str = fmt(stream_hours, precision=0, commas=False)
|
||
stream_cost_per_hr_str = f"{stream_cost_per_hr}"
|
||
batch_core_hours_str = fmt(batch_core_hours, precision=0, commas=False)
|
||
|
||
# ┌── EXPORTS (Bridge to Text) ─────────────────────────────────────────────────
|
||
throughput_str = RealtimeCost.throughput_str
|
||
hourly_tb_str = RealtimeCost.hourly_tb_str
|
||
stream_cost_str = RealtimeCost.stream_cost_str
|
||
batch_cost_str = RealtimeCost.batch_cost_str
|
||
cost_ratio_str = RealtimeCost.cost_ratio_str
|
||
events_per_sec_str = RealtimeCost.events_per_sec_str
|
||
event_size_kb_str = RealtimeCost.event_size_kb_str
|
||
stream_cores_str = RealtimeCost.stream_cores_str
|
||
stream_hours_str = RealtimeCost.stream_hours_str
|
||
stream_cost_per_hr_str = RealtimeCost.stream_cost_per_hr_str
|
||
batch_core_hours_str = RealtimeCost.batch_core_hours_str
|
||
```
|
||
|
||
::: {.callout-notebook title="The Cost of Real-Time"}
|
||
**Problem**: Ingest `{python} events_per_sec_str` events/second. Compare Batch (hourly) vs. Stream (sub-second) costs.
|
||
|
||
**The Physics**:
|
||
|
||
1. **Throughput**: `{python} events_per_sec_str` events/sec$\times$ `{python} event_size_kb_str` KB/event = **`{python} throughput_str` GB/s**.
|
||
2. **Stream Requirements**: To sustain `{python} throughput_str` GB/s with <100 ms latency, you need ~50 dedicated cores + redundant backups (always on). Cost: `{python} stream_cores_str` cores$\times$ `{python} stream_hours_str` hrs$\times$ USD `{python} stream_cost_per_hr_str`/hr = **USD `{python} stream_cost_str`/day**.
|
||
3. **Batch Requirements**: Process `{python} hourly_tb_str` TB (1 hour data) in 10 mins. High throughput (sequential I/O) is efficient. You need 200 cores for 10 mins/hour$\times$ 24 hours = `{python} batch_core_hours_str` core-hours/day. Cost: `{python} batch_core_hours_str`$\times$ USD `{python} stream_cost_per_hr_str` = **USD `{python} batch_cost_str`/day**.
|
||
|
||
**The Engineering Conclusion**: Real-time is **~`{python} cost_ratio_str`$\times$ more expensive** for the same data volume. Only pay this tax if the value of <1s latency justifies it.
|
||
:::
|
||
|
||
#### ETL and ELT Comparison {#sec-data-engineering-etl-elt-comparison-2e2b}
|
||
|
||
\index{ETL (Extract, Transform, Load)!vs ELT}\index{ELT (Extract, Load, Transform)!vs ETL}Beyond choosing ingestion patterns based on timing requirements, designing effective data ingestion pipelines requires understanding the differences between Extract, Transform, Load (ETL)[^fn-etl-quality-first] and Extract, Load, Transform (ELT)[^fn-elt-flexibility-first] approaches. The core distinction is ordering: ETL cleanses and structures data *before* it enters storage, ensuring only validated data reaches the warehouse; ELT loads raw data first and applies transformations *within* the target system, preserving flexibility at the cost of storing uncleaned data. Compare the two flows side by side in @fig-etl-vs-elt—note where the "Transform" step falls relative to "Load" in each paradigm, as this ordering significantly impacts pipeline flexibility and efficiency. The choice between ETL and ELT affects where computational resources are consumed, how quickly data becomes available for analysis, and how easily transformation logic can evolve as requirements change.
|
||
|
||
[^fn-etl-quality-first]: **Extract, Transform, Load (ETL)**: Transforms data before it enters storage, ensuring only validated records reach the warehouse. The trade-off is rigidity: changing feature computation logic requires reprocessing the entire dataset from raw sources, a cost that grows linearly with data volume. For stable ML pipelines with well-defined features, ETL minimizes storage costs; for experimental pipelines where feature definitions change weekly, the reprocessing tax dominates. \index{ETL!reprocessing cost}
|
||
|
||
[^fn-elt-flexibility-first]: **Extract, Load, Transform (ELT)**: This approach enables the flexibility mentioned because transformation logic is just a query run inside the data warehouse, completely decoupled from the ingestion process. Changing a feature definition does not require a slow, expensive data pipeline rerun; it only requires modifying the query. This simple change in ordering reduces the iteration cycle for developing a new feature from hours (or days) of data reprocessing down to the minutes it takes to rewrite a SQL statement. \index{ELT!iteration speed}
|
||
|
||
::: {#fig-etl-vs-elt fig-env="figure" fig-pos="htb" fig-cap="**ETL vs. ELT Comparison**: Side-by-side view of two pipeline paradigms. ETL transforms data before loading into a data warehouse, while ELT loads raw data first and transforms within the warehouse. The choice depends on data volume, transformation complexity, and target storage capabilities." fig-alt="Side-by-side comparison showing ETL pipeline with extract, transform, then load sequence versus ELT pipeline with extract, load, then transform sequence within the data warehouse."}
|
||
```{.tikz}
|
||
\begin{tikzpicture}[line join=round,font=\small\usefont{T1}{phv}{m}{n}]
|
||
\tikzset{
|
||
Line/.style={line width=0.75pt,black!50,text=black},
|
||
LineD/.style={line width=0.5pt,black!50,text=black,dashed},
|
||
}
|
||
|
||
\tikzset{
|
||
channel/.pic={
|
||
\pgfkeys{/channel/.cd, #1}
|
||
\begin{scope}[yscale=\scalefac,xscale=\scalefac,every node/.append style={scale=\scalefac}]
|
||
\node[rectangle,draw=\drawchannelcolor,line width=0.5pt,fill=\channelcolor!50,
|
||
minimum width=50,minimum height=28.5](\picname){};
|
||
\end{scope}
|
||
},
|
||
cyl/.pic={
|
||
\pgfkeys{/channel/.cd, #1}
|
||
\begin{scope}[yscale=\scalefac,xscale=\scalefac,every node/.append style={scale=\scalefac}]
|
||
\node[cylinder, draw=\drawchannelcolor,shape border rotate=90, aspect=1.99,inner ysep=0pt,
|
||
minimum height=20mm,minimum width=21mm, cylinder uses custom fill,
|
||
cylinder body fill=\channelcolor!10,cylinder end fill=\channelcolor!35](\picname){};
|
||
\end{scope}
|
||
},
|
||
tableicon/.pic={
|
||
\pgfkeys{/channel/.cd, #1}
|
||
\begin{scope}[yscale=\scalefac,xscale=\scalefac,every node/.append style={scale=\scalefac}]
|
||
\draw[line width=0.5pt,fill=\channelcolor!20] (0,0)coordinate(DO\picname)
|
||
rectangle (2,1.5)coordinate(GO\picname);
|
||
% Horizontal line
|
||
\foreach \y in {0.5,1} {
|
||
\draw (0,\y) -- (2,\y);
|
||
}
|
||
% Vertical line
|
||
\foreach \x in {0.5,1,1.5} {
|
||
\draw (\x,0) -- (\x,1.5);
|
||
}
|
||
\end{scope}
|
||
}
|
||
}
|
||
|
||
\pgfkeys{
|
||
/channel/.cd,
|
||
channelcolor/.store in=\channelcolor,
|
||
drawchannelcolor/.store in=\drawchannelcolor,
|
||
scalefac/.store in=\scalefac,
|
||
picname/.store in=\picname,
|
||
channelcolor=BrownLine,
|
||
drawchannelcolor=BrownLine,
|
||
scalefac=1,
|
||
picname=C
|
||
}
|
||
% #1 number of teeth
|
||
% #2 radius intern
|
||
% #3 radius extern
|
||
% #4 angle from start to end of the first arc
|
||
% #5 angle to decale the second arc from the first
|
||
% #6 inner radius to cut off
|
||
\newcommand{\gear}[6]{%
|
||
(0:#2)
|
||
\foreach \i [evaluate=\i as \n using {\i-1)*360/#1}] in {1,...,#1}{%
|
||
arc (\n:\n+#4:#2) {[rounded corners=1.5pt] -- (\n+#4+#5:#3)
|
||
arc (\n+#4+#5:\n+360/#1-#5:#3)} -- (\n+360/#1:#2)
|
||
}%
|
||
(0,0) circle[radius=#6];
|
||
}
|
||
\begin{scope}[local bounding box=RIGHT,shift={(0,0)},
|
||
scale=1,every node/.append style={scale=1}]
|
||
\begin{scope}[local bounding box=TARGET,shift={(0,0)},
|
||
scale=1,every node/.append style={scale=1}]
|
||
\scoped[on background layer]
|
||
\pic at(0,0) {cyl={scalefac=1.95,picname=1-CYL}};
|
||
\node[align=center]at($(1-CYL.before top)!0.5!(1-CYL.after top)$){Target\\ (MPP database)};
|
||
%%
|
||
\begin{scope}[local bounding box=GEAR,shift={(-0.80,0.3)},
|
||
scale=1,every node/.append style={scale=1}]
|
||
\colorlet{black}{brown!70!black}
|
||
\fill[draw=none,fill=black,even odd rule,xshift=-2mm]coordinate(GE1)\gear{10}{0.23}{0.28}{10}{2}{0.1};
|
||
\fill[draw=none,fill=black,even odd rule,xshift=2.9mm,yshift=-0.6mm]coordinate(GE2)\gear{10}{0.18}{0.22}{10}{2}{0.08};
|
||
\fill[draw=none,fill=black,even odd rule,xshift=-5.7mm,yshift=-2.8mm]coordinate(GE3)\gear{10}{0.15}{0.19}{10}{2}{0.08};
|
||
\node[draw=none,inner xsep=8,inner ysep=8,yshift=0mm,
|
||
fill=none,fit=(GE1)(GE2)(GE3),line width=1.0pt](BB1){};
|
||
\node[below=-2pt of BB1,align=center]{Staging\\ tables};
|
||
\end{scope}
|
||
|
||
\begin{scope}[local bounding box=TAB,shift={(0.1,-0.25)},
|
||
scale=1,every node/.append style={scale=1}]
|
||
\pic at (0,0) {tableicon={scalefac=0.35,channelcolor=red,picname=T1}}coordinate(GE1);
|
||
\pic at (0.85,0){tableicon={scalefac=0.254,channelcolor=green,picname=T2}}coordinate(GE2);
|
||
\pic at (0.85,0.5){tableicon={scalefac=0.23,channelcolor=cyan,picname=T3}}coordinate(GE3);
|
||
\pic at (0.15,0.65){tableicon={scalefac=0.18,channelcolor=orange,picname=T4}}coordinate(GE4);
|
||
\scoped[on background layer]
|
||
\node[draw=black!60,inner xsep=4,inner ysep=5,yshift=0.5mm,
|
||
fill=yellow!20,fit=(DOT1)(GOT2)(GOT3),line width=1.0pt](BB2){};
|
||
\node[below=1pt of BB2,align=center]{Final\\ tables};
|
||
\end{scope}
|
||
\end{scope}
|
||
|
||
\begin{scope}[local bounding box=SOURCE,shift={(-4.5,0)},
|
||
scale=1,every node/.append style={scale=1}]
|
||
\begin{scope}[local bounding box=SOURCE1,shift={(0,1.8)},
|
||
scale=1,every node/.append style={scale=1}]
|
||
\pic at(0,0) {cyl={scalefac=0.75,channelcolor=violet!80!,picname=2-CYL}};
|
||
\node at(2-CYL){Source 1};
|
||
\end{scope}
|
||
|
||
\begin{scope}[local bounding box=SOURCE2,shift={(0,0.1)},
|
||
scale=1,every node/.append style={scale=1}]
|
||
\scoped[on background layer]
|
||
\pic at(0,0) {cyl={scalefac=0.75,channelcolor=orange!80!,picname=3-CYL}};
|
||
\node at(3-CYL){Source 2};
|
||
\end{scope}
|
||
|
||
\begin{scope}[local bounding box=SOURCE3,shift={(-0.15,-1.2)},
|
||
scale=1,every node/.append style={scale=1}]
|
||
\foreach \j in {1,2,3} {
|
||
\pic at ({\j*0.15}, {-0.15*\j}) {channel={scalefac=1.15,channelcolor=green!40!,picname=\j-CH1}};
|
||
}
|
||
\node at(3-CH1){Source 3};
|
||
\end{scope}
|
||
\foreach \j in {1,2,3} {
|
||
\draw[Line,-latex,shorten >=10pt,shorten <=10pt](SOURCE\j.east)--(TARGET.west);
|
||
}
|
||
\end{scope}
|
||
\path[red](3-CH1.south)--++(0,-0.6)-|coordinate[pos=0.35](SR1)(1-CYL.south);
|
||
\node[single arrow, draw=black,thick, fill=VioletL,
|
||
minimum width = 17pt, single arrow head extend=3pt,
|
||
minimum height=9mm](AR1)at(SR1) {};
|
||
\node[left=6pt of AR1,anchor=east]{Extract \& Load};
|
||
\node[right=6pt of AR1,anchor=west]{Transform};
|
||
\node[below=8pt of AR1]{\normalsize E \textcolor{red}{$\to$ L $\to$ T}};
|
||
\end{scope}
|
||
%%%%%%%%%%%%%
|
||
%LEFT
|
||
\begin{scope}[local bounding box=LEFT,shift={(-8,0)},
|
||
scale=1,every node/.append style={scale=1}]
|
||
\begin{scope}[local bounding box=TARGET,shift={(0,0)},
|
||
scale=1,every node/.append style={scale=1}]
|
||
\pic at(0,0) {cyl={scalefac=1.25,picname=1-CYL}};
|
||
\node at(1-CYL){Target};
|
||
\end{scope}
|
||
%%
|
||
\begin{scope}[local bounding box=GEAR,shift={(-3.2,0.3)},
|
||
scale=1.5,every node/.append style={scale=1}]
|
||
\colorlet{black}{brown!70!black}
|
||
\fill[draw=none,fill=black,even odd rule,xshift=-2mm]coordinate(GE1)\gear{10}{0.23}{0.28}{10}{2}{0.1};
|
||
\fill[draw=none,fill=black,even odd rule,xshift=2.9mm,yshift=-0.6mm]coordinate(GE2)\gear{10}{0.18}{0.22}{10}{2}{0.08};
|
||
\fill[draw=none,fill=black,even odd rule,xshift=-5.7mm,yshift=-2.8mm]coordinate(GE3)\gear{10}{0.15}{0.19}{10}{2}{0.08};
|
||
\node[draw=none,inner xsep=8,inner ysep=8,yshift=0mm,
|
||
fill=none,fit=(GE1)(GE2)(GE3),line width=1.0pt](BB1){};
|
||
\end{scope}
|
||
|
||
\begin{scope}[local bounding box=SOURCE,shift={(-6.9,-0.14)},
|
||
scale=1,every node/.append style={scale=1}]
|
||
\begin{scope}[local bounding box=SOURCE1,shift={(0,1.8)},
|
||
scale=1,every node/.append style={scale=1}]
|
||
\pic at(0,0) {cyl={scalefac=0.75,channelcolor=violet!80!,picname=2-CYL}};
|
||
\node at(2-CYL){Source 1};
|
||
\end{scope}
|
||
|
||
\begin{scope}[local bounding box=SOURCE2,shift={(0,0.1)},
|
||
scale=1,every node/.append style={scale=1}]
|
||
\scoped[on background layer]
|
||
\pic at(0,0) {cyl={scalefac=0.75,channelcolor=orange!80!,picname=3-CYL}};
|
||
\node at(3-CYL){Source 2};
|
||
\end{scope}
|
||
|
||
\begin{scope}[local bounding box=SOURCE3,shift={(-0.15,-1.2)},
|
||
scale=1,every node/.append style={scale=1}]
|
||
\foreach \j in {1,2,3} {
|
||
\pic at ({\j*0.15}, {-0.15*\j}) {channel={scalefac=1.15,channelcolor=green!40!,picname=\j-CH1}};
|
||
}
|
||
\node at(3-CH1){Source 3};
|
||
\end{scope}
|
||
\foreach \j in {1,2,3} {
|
||
\draw[Line,-latex,shorten >=10pt,shorten <=10pt](SOURCE\j.east)--(BB1.west);
|
||
}\draw[Line,-latex,shorten >=5pt,shorten <=5pt](BB1.08)--(TARGET.west);
|
||
\end{scope}
|
||
\path[red](3-CH1.south)--++(0,-0.5)-|coordinate[pos=0.35](SR1)(1-CYL.south);
|
||
\node[single arrow, draw=black,thick, fill=VioletL,
|
||
minimum width = 17pt, single arrow head extend=3pt,
|
||
minimum height=9mm](AR1)at(SR1) {};
|
||
\node[left=6pt of AR1,anchor=east](TRA){Transform};
|
||
\node[right=6pt of AR1,anchor=west]{Load};
|
||
\node[left=4pt of TRA,single arrow, draw=black,thick, fill=VioletL,
|
||
minimum width = 17pt, single arrow head extend=3pt,
|
||
minimum height=9mm](AR2) {};
|
||
\node[left=6pt of AR2,anchor=east]{Extract};
|
||
\node[below=8pt of TRA]{\normalsize E \textcolor{red}{$\to$ T $\to$ L}};
|
||
\end{scope}
|
||
\draw[line width=2pt,red!40]($(LEFT.north east)!0.5!(RIGHT.north west)$)--
|
||
($(LEFT.south east)!0.5!(RIGHT.south west)$);
|
||
\end{tikzpicture}
|
||
```
|
||
:::
|
||
|
||
The ETL pattern\index{ETL (Extract, Transform, Load)!transformation first} transforms data before loading it into the target system. For ML pipelines, this means only validated, schema-conformant data enters the warehouse, enforcing quality and privacy compliance at ingestion time. For instance, an ML system predicting customer churn might use ETL to standardize customer interaction data from multiple sources, converting timestamp formats to UTC, normalizing text encodings, and computing aggregate features like "total purchases last 30 days" before loading [@inmon2005building]. The disadvantage is inflexibility: when feature definitions change, all source data must be reprocessed through the pipeline, a process that can take hours or days for large datasets and slows iteration velocity during development.
|
||
|
||
The ELT pattern\index{ELT (Extract, Load, Transform)!load first} reverses this order, loading raw data first and applying transformations within the target system. For ML development, this enables flexible feature experimentation on the same raw data. Multiple teams can compute different aggregation windows, and when transformation logic bugs are discovered, teams reprocess by rerunning queries rather than re-ingesting from sources. This flexibility accelerates ML experimentation where feature engineering requirements evolve rapidly. The cost is higher storage requirements (raw data is larger than transformed data), repeated computation when multiple models transform the same source data, and greater complexity in enforcing privacy compliance when raw sensitive data persists in storage.
|
||
|
||
Production ML systems rarely use one pattern exclusively. Structured data with stable schemas often flows through ETL for efficiency and compliance, while unstructured data or rapidly evolving feature pipelines benefit from ELT's flexibility. Choosing between these patterns requires understanding the *cost of transformation placement*.
|
||
|
||
Deciding where to run these transformations involves calculating *the cost of transformation placement*, as shown in @lst-etl-elt-cost-comparison.
|
||
|
||
::: {#lst-etl-elt-cost-comparison lst-cap="**ETL vs ELT Cost Comparison**: Calculating storage and compute costs for different transformation placement strategies. ETL processes data before loading (reducing storage but requiring reprocessing on schema changes), while ELT loads raw data first (higher storage costs but flexible reprocessing via SQL)."}
|
||
```{.python}
|
||
# ETL vs ELT cost comparison
|
||
daily_raw_tb = 10
|
||
s3_per_tb_mo = 23
|
||
spark_per_tb = 5
|
||
n_models = 3
|
||
retention_days = 30
|
||
query_cost_per = 5
|
||
|
||
# ETL
|
||
etl_spark_daily = daily_raw_tb * spark_per_tb
|
||
etl_datasets = 3
|
||
etl_tb_each = 2
|
||
etl_storage_tb = etl_datasets * etl_tb_each
|
||
etl_storage_mo = etl_storage_tb * s3_per_tb_mo
|
||
|
||
# ELT
|
||
elt_storage_tb = daily_raw_tb * retention_days
|
||
elt_storage_mo = elt_storage_tb * s3_per_tb_mo
|
||
elt_query_mo = n_models * query_cost_per * retention_days
|
||
|
||
# Savings
|
||
storage_savings_mo = elt_storage_mo - etl_storage_mo
|
||
```
|
||
:::
|
||
|
||
```{python}
|
||
#| label: etl-elt-calc
|
||
#| echo: false
|
||
# ┌─────────────────────────────────────────────────────────────────────────────
|
||
# │ ETL VS ELT COST CALCULATION
|
||
# ├─────────────────────────────────────────────────────────────────────────────
|
||
# │ Context: Callout "The Cost of Transformation Placement"
|
||
# │
|
||
# │ Goal: Contrast the monthly costs of ETL and ELT architectures.
|
||
# │ Show: The storage premium of raw ELT vs. the compute costs of transformed ETL.
|
||
# │ How: Model storage and Spark compute costs for a 10 TB/day data warehouse.
|
||
# │
|
||
# │ Imports: mlsys.formatting (fmt)
|
||
# │ Exports: etl_spark_str, etl_storage_str, elt_storage_tb_str,
|
||
# │ elt_storage_str, elt_query_str, savings_str, daily_raw_tb_str,
|
||
# │ spark_per_tb_str, etl_datasets_str, etl_tb_each_str,
|
||
# │ etl_storage_tb_str2, s3_per_tb_mo_str, retention_days_str,
|
||
# │ query_cost_per_str, n_models_str
|
||
# └─────────────────────────────────────────────────────────────────────────────
|
||
from mlsys.formatting import fmt
|
||
|
||
# ┌── P.I.C.O. ISOLATED SCENARIO ───────────────────────────────────────────────
|
||
class EtlEltCost:
|
||
"""ETL vs. ELT monthly cost comparison for a 10 TB/day data warehouse."""
|
||
|
||
# ┌── 1. PARAMETERS (Inputs) ──────────────────────────────────────────────
|
||
daily_raw_tb = 10
|
||
s3_per_tb_mo = 23
|
||
spark_per_tb = 5
|
||
n_models = 3
|
||
retention_days = 30
|
||
query_cost_per = 5
|
||
etl_datasets = 3
|
||
etl_tb_each = 2
|
||
|
||
# ┌── 2. CALCULATION (The Physics) ────────────────────────────────────────
|
||
etl_spark_daily = daily_raw_tb * spark_per_tb
|
||
etl_storage_tb = etl_datasets * etl_tb_each
|
||
etl_storage_mo = etl_storage_tb * s3_per_tb_mo
|
||
|
||
elt_storage_tb = daily_raw_tb * retention_days
|
||
elt_storage_mo = elt_storage_tb * s3_per_tb_mo
|
||
elt_query_mo = n_models * query_cost_per * retention_days
|
||
|
||
storage_savings_mo = elt_storage_mo - etl_storage_mo
|
||
|
||
# ┌── 4. OUTPUTS (Formatting) ─────────────────────────────────────────────
|
||
etl_spark_str = f"{etl_spark_daily}"
|
||
etl_storage_str = f"{etl_storage_mo}"
|
||
elt_storage_tb_str = f"{elt_storage_tb}"
|
||
elt_storage_str = f"{elt_storage_mo:,}"
|
||
elt_query_str = f"{elt_query_mo}"
|
||
savings_str = f"{storage_savings_mo:,}"
|
||
daily_raw_tb_str = fmt(daily_raw_tb, precision=0, commas=False)
|
||
spark_per_tb_str = fmt(spark_per_tb, precision=0, commas=False)
|
||
etl_datasets_str = fmt(etl_datasets, precision=0, commas=False)
|
||
etl_tb_each_str = fmt(etl_tb_each, precision=0, commas=False)
|
||
etl_storage_tb_str2 = str(etl_storage_tb)
|
||
s3_per_tb_mo_str = fmt(s3_per_tb_mo, precision=0, commas=False)
|
||
retention_days_str = fmt(retention_days, precision=0, commas=False)
|
||
query_cost_per_str = fmt(query_cost_per, precision=0, commas=False)
|
||
n_models_str = fmt(n_models, precision=0, commas=False)
|
||
|
||
# ┌── EXPORTS (Bridge to Text) ─────────────────────────────────────────────────
|
||
etl_spark_str = EtlEltCost.etl_spark_str
|
||
etl_storage_str = EtlEltCost.etl_storage_str
|
||
elt_storage_tb_str = EtlEltCost.elt_storage_tb_str
|
||
elt_storage_str = EtlEltCost.elt_storage_str
|
||
elt_query_str = EtlEltCost.elt_query_str
|
||
savings_str = EtlEltCost.savings_str
|
||
daily_raw_tb_str = EtlEltCost.daily_raw_tb_str
|
||
spark_per_tb_str = EtlEltCost.spark_per_tb_str
|
||
etl_datasets_str = EtlEltCost.etl_datasets_str
|
||
etl_tb_each_str = EtlEltCost.etl_tb_each_str
|
||
etl_storage_tb_str2 = EtlEltCost.etl_storage_tb_str2
|
||
s3_per_tb_mo_str = EtlEltCost.s3_per_tb_mo_str
|
||
retention_days_str = EtlEltCost.retention_days_str
|
||
query_cost_per_str = EtlEltCost.query_cost_per_str
|
||
n_models_str = EtlEltCost.n_models_str
|
||
```
|
||
|
||
::: {.callout-notebook title="The Cost of Transformation Placement"}
|
||
**Problem**: Your team processes `{python} daily_raw_tb_str` TB of raw clickstream data daily. You need to compute user session features for `{python} n_models_str` ML models, each requiring different aggregation windows (1-hour, 24-hour, 7-day). Compare ETL vs. ELT costs.
|
||
|
||
**The Math**:
|
||
|
||
1. **ETL Approach**: Transform before loading. Compute all three aggregation windows in a Spark cluster before loading into the warehouse.
|
||
* Spark compute: `{python} daily_raw_tb_str` TB at USD `{python} spark_per_tb_str`/TB = USD `{python} etl_spark_str`/day
|
||
* Storage: `{python} etl_datasets_str` transformed datasets, ~`{python} etl_tb_each_str` TB each = `{python} etl_storage_tb_str2` TB at USD `{python} s3_per_tb_mo_str`/TB per month = USD `{python} etl_storage_str`/month
|
||
* Schema change cost: Re-run full pipeline (~4 hours) per change
|
||
|
||
2. **ELT Approach**: Load raw data first, transform in warehouse.
|
||
* Storage: `{python} daily_raw_tb_str` TB raw/day, `{python} retention_days_str`-day retention = `{python} elt_storage_tb_str` TB at USD `{python} s3_per_tb_mo_str`/TB per month = USD `{python} elt_storage_str`/month
|
||
* Query compute: `{python} n_models_str` models$\times$ USD `{python} query_cost_per_str`/query$\times$ `{python} retention_days_str` days = USD `{python} elt_query_str`/month
|
||
* Schema change cost: Rewrite SQL query (~30 minutes) per change
|
||
|
||
**The Engineering Conclusion**: ETL saves USD `{python} savings_str`/month in storage but costs 8$\times$ more engineering time per schema change. If your feature definitions change weekly, ELT's flexibility pays for itself. If schemas are stable, ETL's lower storage cost dominates. The break-even point: if you change schemas fewer than once per month, ETL wins on total cost.
|
||
:::
|
||
|
||
When implementing streaming components within ETL/ELT architectures, distributed systems principles become critical. The CAP theorem\index{CAP Theorem!streaming systems}[^fn-cap-theorem-ml-storage]—which states that distributed systems cannot simultaneously guarantee Consistency (all nodes see the same data), Availability (the system remains operational), and Partition tolerance (the system continues despite network failures)—constrains streaming system design choices. Apache Kafka\index{Apache Kafka!event streaming}[^fn-kafka-ordering-guarantee] emphasizes consistency and partition tolerance, making it well-suited for reliable event ordering but potentially experiencing availability issues during network partitions. Apache Pulsar emphasizes availability and partition tolerance, providing better fault tolerance but with relaxed consistency guarantees. Amazon Kinesis balances all three properties through careful configuration but requires understanding these trade-offs for proper deployment.
|
||
|
||
[^fn-kafka-ordering-guarantee]: **Apache Kafka**: Kafka achieves its ordering guarantee by using a partitioned, leader-based log; all writes for a partition must go through a single leader replica. This design prioritizes Consistency and Partition tolerance, as the system will halt writes to a partition if the leader becomes unreachable rather than risk an inconsistent state (sacrificing Availability). This trade-off manifests as a tangible availability gap, where a partition can become unwritable for several seconds during a leader re-election event. \index{Kafka!ordering guarantee}
|
||
|
||
[^fn-cap-theorem-ml-storage]: **CAP Theorem**: Conjectured by Eric Brewer in 2000 [@brewer2000towards] and formally proved by Gilbert and Lynch in 2002 [@gilbert2002brewer]. Distributed systems cannot simultaneously guarantee Consistency, Availability, and Partition tolerance — one must be sacrificed. For ML storage architectures, this forces a concrete choice: a feature store prioritizing consistency (CP) guarantees training and serving see identical feature values but may become unavailable during network partitions, while one prioritizing availability (AP) stays operational but risks serving stale features that diverge from training data. \index{CAP Theorem!feature store trade-off}
|
||
|
||
#### Feature Computation Placement {#sec-data-engineering-feature-computation-placement-a998}
|
||
|
||
\index{Feature Engineering!computation placement}For ML pipelines, an additional decision extends beyond ETL versus ELT: where to compute features. This choice significantly impacts training speed, storage costs, and reproducibility.
|
||
|
||
One approach is to precompute features during ETL and store the results.\index{Feature Computation!pipeline vs loader} Pipeline-computed features offer fast training iteration (features are ready on disk), reproducibility (the same features are used consistently), and reduced training compute. The drawbacks are storage cost (features stored separately from raw data), staleness risk (precomputed features may diverge when logic changes), and inflexibility (any change requires full recomputation).
|
||
|
||
The alternative is computing features on the fly during training. Loader-computed features guarantee always-fresh computation (logic changes are immediately reflected), flexible experimentation (easy to modify features), and reduced storage (only raw data is stored). The cost is slower training (computation repeats each epoch), higher compute expenditure (GPUs often idle waiting for features), and potential non-determinism if not carefully implemented.
|
||
|
||
In practice, hybrid patterns\index{Feature Computation!hybrid patterns} predominate. Expensive, stable features—user embeddings requiring matrix factorization, historical aggregations spanning months of data—are precomputed and materialized. Cheap, time-sensitive features—recency signals, session context, time-based transformations—are computed in the data loader.
|
||
|
||
For example, a recommendation system precomputes user embedding features (expensive, stable over days) while computing time-since-last-interaction features (cheap, time-sensitive) in the data loader. This balances storage costs, computation time, and feature freshness based on each feature's specific characteristics.
|
||
|
||
#### Integration Strategies and KWS Case Study {#sec-data-engineering-multisource-integration-strategies-0e8a}
|
||
|
||
Regardless of whether ETL or ELT approaches are used, integrating diverse data sources remains a core ingestion challenge. Data may originate from databases, APIs, file systems, and IoT devices, each with its own format (relational rows, JSON[^fn-json-ml-overhead] documents, binary streams), access protocol, and update frequency. The systems principle is to standardize at the ingestion boundary: normalize formats, validate schemas, and present a consistent interface to downstream processing regardless of source. This boundary standardization separates the complexity of source diversity from the complexity of feature engineering, allowing each to evolve independently.
|
||
|
||
[^fn-json-ml-overhead]: **JSON (JavaScript Object Notation)**: The schema flexibility that makes JSON a common format for APIs creates a validation bottleneck at the ingestion boundary. Unlike binary formats with predefined schemas, every JSON document requires parsing and schema validation before it can be standardized for downstream use. This per-record overhead can make ingestion over 10x slower than with formats like Protobuf, directly impacting the system's ability to handle high-frequency data streams. \index{JSON!storage overhead}
|
||
|
||
KWS production systems use streaming and batch ingestion in concert. The streaming path handles real-time audio from active devices, using publish-subscribe mechanisms like Apache Kafka to buffer incoming data and distribute it across inference servers within the 200 millisecond latency requirement. The batch path handles training data: new recordings from crowdsourcing efforts discussed in @sec-data-engineering-strategic-data-acquisition-418f, synthetic data addressing coverage gaps, and validated user interactions. Batch processing typically follows an ETL pattern where audio undergoes normalization, noise filtering, and segmentation into consistent durations before storage in training-optimized formats.
|
||
|
||
Error handling in voice interaction systems requires special attention. Dead letter queues store failed recognition attempts for subsequent analysis, revealing edge cases that need coverage in future model iterations. Each incoming audio sample must pass quality validation (signal-to-noise ratio, sample rate, duration bounds, speaker proximity) before entering the processing pipeline. Invalid samples route to analysis queues rather than being discarded, since these failures often indicate acoustic conditions underrepresented in training data. Valid samples flow through to real-time detection while simultaneously being logged for potential inclusion in future training data.
|
||
|
||
This ingestion architecture completes the boundary layer where external data enters our controlled pipeline. But ingested data, however reliably delivered, is still raw: audio at inconsistent sample rates, text with varying encodings, numeric features on incompatible scales. Transforming these heterogeneous records into a uniform, model-ready representation while guaranteeing that the exact same transformations apply during both training and serving is the next challenge.
|
||
|
||
## Systematic Data Processing {#sec-data-engineering-systematic-data-processing-aebc}
|
||
|
||
The ingestion stage lexed raw streams into well-formed records; we now enter the *optimization pass* of our dataset compiler: systematic data processing. Just as compiler optimizations must preserve program semantics while improving performance, data transformations must preserve signal while improving model readiness. This section addresses three interconnected challenges: ensuring transformations remain identical between training and serving, building idempotent transformations that produce consistent results regardless of execution count, and scaling processing while maintaining data lineage.
|
||
|
||
Industry experience suggests that training-serving inconsistency contributes to the majority of silent model degradation issues [@sculley2015hidden]. Consider normalizing transaction amounts during training by removing currency symbols and converting to floats, but forgetting to apply identical preprocessing during serving. This seemingly minor inconsistency can degrade model accuracy by 20-40%. For our KWS system, the tension is concrete: transformations must standardize across diverse recording conditions (varying microphones, noise levels, sample rates) while preserving the acoustic characteristics that distinguish wake words from background speech—and this standardization must be identical in both training and serving paths.
|
||
|
||
### Training-Serving Consistency {#sec-data-engineering-ensuring-trainingserving-consistency-c683}
|
||
|
||
\index{Training-Serving Consistency!ensuring}The consistency challenge extends beyond applying the same code—it requires that parameters computed on training data (normalization constants, encoding dictionaries, vocabulary mappings) are stored and reused during serving. We formalize this requirement as the *consistency imperative*.
|
||
|
||
::: {.callout-definition title="The Consistency Imperative"}
|
||
|
||
***The Consistency Imperative***\index{Consistency Imperative!definition} is the axiom that **Transformation Logic** must be immutable across training and serving environments.
|
||
|
||
1. **Significance (Quantitative):** It predicts that performance degradation is proportional to the **KL Divergence** ($D_{KL}(T \parallel T')$) between the training transformation ($T$) and the serving transformation ($T'$).
|
||
2. **Distinction (Durable):** Unlike **Data Quality**, which focuses on the **Cleanliness** of a single record, the Consistency Imperative focuses on the **Alignment** of the entire transformation pipeline.
|
||
3. **Common Pitfall:** A frequent misconception is that consistency is "fixed" by sharing code. In reality, it is a **State Synchronization** problem: parameters computed on training data (e.g., means, standard deviations) must be stored and reused during serving.
|
||
|
||
:::
|
||
|
||
The stakes are high: violating the Consistency Imperative silently degrades model accuracy in production. Verify your understanding of this critical requirement before we examine cleaning and transformation techniques.
|
||
|
||
::: {.callout-checkpoint title="Defensive Processing" collapse="false"}
|
||
The primary cause of ML system failure is not bad algorithms but **Training-Serving Skew**.
|
||
|
||
- [ ] **The Definition**: Skew happens when the code processing data during training differs from the code processing live requests.
|
||
- [ ] **The Mechanism**: If training normalizes data using `mean=0.5` but serving uses `mean=0.0`, the model sees "alien" data and fails silently.
|
||
- [ ] **The Principle**: Do you understand why *architectural guarantees* (shared code, not copied code) are the only reliable solution to skew?
|
||
:::
|
||
|
||
Data cleaning\index{Data Cleaning!error correction}\index{Data Cleaning!inconsistency removal} involves identifying and correcting errors, inconsistencies, and inaccuracies in datasets. Raw data frequently contains missing values, duplicates, or outliers that degrade model performance. The key insight is that cleaning operations must be deterministic and reproducible: given the same input, they must produce the same output regardless of environment. This requirement shapes which cleaning techniques are safe to use in production.
|
||
|
||
Data cleaning might involve removing duplicate records based on deterministic keys, handling missing values through imputation or deletion using rules that can be applied consistently, and correcting formatting inconsistencies systematically. For instance, in a customer database, names might be inconsistently capitalized or formatted. A data cleaning process would standardize these entries, ensuring that "John Doe," "john doe," and "DOE, John" are all treated as the same entity. The cleaning rules—convert to title case, reorder to "First Last" format—must be captured in code that executes identically in training and serving. As emphasized throughout this chapter, every cleaning operation must be applied identically in both contexts to maintain system reliability.
|
||
|
||
Outlier detection\index{Outlier Detection!data cleaning} and treatment is another important aspect of data cleaning, but one that introduces consistency challenges. Outliers can sometimes represent valuable information about rare events, but they can also result from measurement errors or data corruption. ML practitioners must carefully consider the nature of their data and the requirements of their models when deciding how to handle outliers. Simple threshold-based outlier removal (removing values more than 3 standard deviations from the mean) maintains training-serving consistency if the mean and standard deviation are computed on training data and reused during serving. However, more sophisticated outlier detection methods that consider relationships between features or temporal patterns require careful engineering to ensure consistent application.
|
||
|
||
Quality assessment complements data cleaning by systematically evaluating the reliability and usefulness of data across multiple dimensions: accuracy, completeness, consistency, and timeliness. In production systems, data quality degrades in subtle ways that basic metrics miss: fields that never contain nulls suddenly show sparse patterns, numeric distributions drift from their training ranges, or categorical values appear that were not present during model development.
|
||
|
||
To address these subtle degradation patterns, production quality monitoring requires specific metrics beyond simple missing value counts as discussed in @sec-data-engineering-quality-validation-monitoring-498f. Critical indicators include null value patterns by feature (sudden increases suggest upstream failures), count anomalies (10$\times$ increases often indicate data duplication or pipeline errors), value range violations (prices becoming negative, ages exceeding realistic bounds), and join failure rates between data sources. *Statistical drift* detection[^fn-data-drift-degradation] becomes essential by monitoring means, variances, and quantiles of features over time to catch gradual degradation before it impacts model performance. For example, in an e-commerce recommendation system, the average user session length might gradually increase from 8 minutes to 12 minutes over six months due to improved site design, but a sudden drop to 3 minutes suggests a data collection bug.
|
||
|
||
[^fn-data-drift-degradation]: **Statistical Drift Detection**: The means, variances, and quantiles tracked in quality monitoring are the early-warning signals for the Degradation Equation's divergence term $D(P_t \| P_0)$. A mean session length shifting from 8 to 12 minutes over six months is drift (retraining on recent data restores accuracy); a sudden drop to 3 minutes is a data collection bug (requires a source-system fix, not retraining). The monitoring window determines which signal surfaces first: hourly windows catch bugs, weekly windows catch drift, and the quality engineer must distinguish between them before choosing the intervention. \index{Data Drift!Degradation Equation}
|
||
|
||
Quality assessment tools range from simple statistical measures to complex machine learning-based approaches. Data profiling tools provide summary statistics and visualizations that help identify potential quality issues, while advanced techniques employ unsupervised learning algorithms to detect anomalies or inconsistencies in large datasets. The key is maintaining identical quality standards and validation logic across training and serving to prevent quality issues from creating training-serving skew.
|
||
|
||
Transformation techniques\index{Data Transformation!techniques} convert data from its raw form into a format more suitable for analysis and modeling. This process can include a wide range of operations, from simple conversions to complex mathematical transformations. Central to effective transformation, common transformation tasks include normalization\index{Normalization!feature scaling} and standardization,\index{Standardization!feature scaling} which scale numerical features to a common range or distribution. For example, in a housing price prediction model, features like square footage and number of rooms might be on vastly different scales. Normalizing these features ensures that they contribute more equally to the model's predictions [@bishop2006pattern]. Maintaining training-serving consistency requires that normalization parameters\index{Normalization!parameter persistence} (mean, standard deviation) computed on training data be stored and applied identically during serving. This means persisting these parameters alongside the model itself—often in the model artifact or a separate parameter file—and loading them during serving initialization.
|
||
|
||
Beyond numerical scaling, other transformations might involve encoding categorical variables\index{Categorical Encoding!one-hot encoding}, handling date and time data, or creating derived features. For instance, one-hot encoding is often used to convert categorical variables into a format that can be readily understood by many machine learning algorithms. Categorical encodings must handle both the categories present during training and unknown categories encountered during serving. A reliable approach computes the category vocabulary during training (the set of all observed categories), persists it with the model, and during serving either maps unknown categories to a special "unknown" token or uses default values. Without this discipline, serving encounters categories the model never saw during training, potentially causing errors or degraded performance.
|
||
|
||
A health prediction model receives raw GPS coordinates for each patient visit, but latitude and longitude alone tell the model nothing about healthcare access. An engineer who understands the domain creates a new feature: *distance to nearest hospital*. Suddenly the model discovers that patients more than 50 km from an emergency room have measurably worse outcomes for time-sensitive conditions—a pattern invisible in the raw coordinates.
|
||
|
||
Feature engineering\index{Feature Engineering!definition}\index{Feature Engineering!domain knowledge} is this act of using domain knowledge to create new features that make machine learning algorithms work more effectively. The step is often considered more art than science, requiring creativity and deep understanding of both the data and the problem at hand. Feature engineering might involve combining existing features, extracting information from complex data types, or creating entirely new features based on domain insights. In a retail recommendation system, for example, engineers might create features that capture the recency, frequency, and monetary value of customer purchases, known as RFM analysis\index{RFM Analysis!feature engineering} [@kuhn2013applied].
|
||
|
||
Feature engineering is often the single highest-leverage activity in the ML pipeline. Well-engineered features can often lead to significant improvements in model performance, sometimes outweighing the impact of algorithm selection or hyperparameter tuning. The creativity required for feature engineering must be balanced against the consistency requirements of production systems. Every engineered feature must be computed identically during training and serving. This means feature engineering logic should be implemented in shared libraries or modules, not reimplemented separately for each environment. Many organizations build feature stores[^fn-feature-store-consistency]\index{Feature Store!consistency guarantee}, discussed in @sec-data-engineering-feature-stores-bridging-training-serving-55c8, specifically to ensure feature computation consistency across environments.
|
||
|
||
[^fn-feature-store-consistency]: **Feature Store**: Shared libraries enforce code-level consistency, but the feature store solves the harder problem: *state* consistency. Normalization parameters, encoding vocabularies, and aggregation windows must be computed once on training data and served identically at inference. Uber's Michelangelo pioneered the dual-interface pattern -- batch access for training, low-latency online access for serving, both reading from the same pre-computed values -- specifically because shared code alone still allowed 5--10% accuracy drops from divergent normalization constants across environments. \index{Feature Store!consistency}
|
||
|
||
Applying these processing concepts to our KWS system: the audio recordings flowing through our ingestion pipeline, whether from crowdsourcing, synthetic generation, or real-world captures, require careful cleaning to ensure reliable wake word detection. Raw audio data often contains imperfections that our problem definition anticipated: background noise from various environments (quiet bedrooms to noisy industrial settings), clipped signals from recording level issues, varying volumes across different microphones and speakers, and inconsistent sampling rates from diverse capture devices. The cleaning pipeline must standardize these variations while preserving the acoustic characteristics that distinguish wake words from background speech, a quality-preservation requirement that directly impacts our 98% accuracy target.
|
||
|
||
Quality assessment for KWS extends the general principles with audio-specific metrics. Beyond checking for null values or schema conformance, our system tracks background noise levels (signal-to-noise ratio above `{python} kws_snr_threshold_db_str` decibels), audio clarity scores (frequency spectrum analysis), and speaking rate consistency (wake word duration within `{python} kws_sample_duration_ms_low_str`-`{python} kws_sample_duration_ms_high_str` milliseconds). The quality assessment pipeline automatically flags recordings where background noise would prevent accurate detection, where wake words are spoken too quickly or unclearly for the model to distinguish them, or where clipping or distortion has corrupted the audio signal. This automated filtering ensures only high-quality samples reach model development. Recall how @fig-cascades demonstrated the compounding effects of early data quality failures; this filtering prevents precisely those cascade failures by catching issues at the source.
|
||
|
||
Transforming audio data for KWS involves converting raw waveforms into formats suitable for ML models while maintaining training-serving consistency. Raw audio waveforms (sequences of amplitude values sampled thousands of times per second) are high-dimensional and difficult for neural networks to process directly. Instead, we transform them into compact representations that emphasize the frequencies and temporal patterns most relevant to speech. To understand what this transformation produces, study @fig-spectrogram-example: notice how the raw waveform (left) becomes a 2D image-like representation (right) where the horizontal axis is time, the vertical axis is frequency, and color intensity shows energy. These standardized feature representations, typically Mel-frequency cepstral coefficients (MFCCs)\index{MFCC (Mel-Frequency Cepstral Coefficients)}[^fn-mfcc-dimensionality] or spectrograms,\index{Spectrogram!audio features}[^fn-spectrogram-cnn] emphasize speech-relevant characteristics while reducing noise and variability across different recording conditions.
|
||
|
||
[^fn-mfcc-dimensionality]: **MFCC (Mel-Frequency Cepstral Coefficients)**: This transformation achieves its compactness and noise resistance by applying mel-scale filtering, which selectively emphasizes the frequencies humans use to distinguish speech. This process reduces thousands of raw audio samples from a small time window (e.g., 25ms) into just 13-39 coefficients, the aggressive dimensionality reduction required for always-on, kilobyte-scale hardware. Any mismatch in the parameters governing this transformation between training and serving will create feature skew, degrading the model's accuracy. \index{MFCC!dimensionality reduction}
|
||
|
||
[^fn-spectrogram-cnn]: **Spectrogram**: The Short-Time Fourier Transform (STFT) computes this 2D representation, architecturally repurposing standard image-based CNNs for audio processing by converting the 1D waveform into a visual format. This repurposing creates a rigid dependency: any mismatch in STFT parameters (e.g., a 25ms vs. 30ms window) between training and serving invalidates the learned patterns, causing performance to collapse. \index{Spectrogram!CNN bridge}
|
||
|
||
{#fig-spectrogram-example fig-pos="t!" fig-alt="Two-panel visualization showing raw audio waveform on left transforming into spectrogram on right, with time on horizontal axis and frequency on vertical axis indicated by color intensity."}
|
||
|
||
### Idempotent Transformations {#sec-data-engineering-building-idempotent-data-transformations-7961}
|
||
|
||
\index{Idempotent Transformations!building}\index{Reliability!idempotency}Building on quality foundations, we turn to reliability. While quality focuses on what transformations produce, reliability ensures how consistently they operate. Processing reliability means transformations produce identical outputs given identical inputs, regardless of when, where, or how many times they execute. This property, called idempotency,\index{Idempotency!definition}[^fn-idempotency-retry-safety] proves essential for production ML systems where processing may be retried due to failures, where data may be reprocessed to fix bugs, or where the same data flows through multiple processing paths.
|
||
|
||
[^fn-idempotency-retry-safety]: **Idempotency**: From Latin *idem* ("the same") + *potens* ("having power") -- literally, "having the same power when applied again." In ML pipelines, idempotency enables safe retries after partial failures: a non-idempotent transformation (e.g., appending to a log) creates duplicates on retry, silently corrupting training data with repeated examples that bias the model. Idempotent transformations (e.g., upsert by key) guarantee identical output regardless of retry count, which is essential for reproducible training and debugging production accuracy drops. \index{Idempotency!retry safety}
|
||
|
||
Consider a light switch to build intuition. Flipping the switch to the "on" position turns the light on. Flipping it to "on" again leaves the light on; the operation can be repeated without changing the outcome. This is idempotent behavior. In contrast, a toggle switch that changes state with each press is not idempotent: pressing it repeatedly alternates between on and off states. In data processing, we want light switch behavior where reapplying the same transformation yields the same result, not toggle switch behavior where repeated application changes the outcome unpredictably.
|
||
|
||
Idempotent transformations enable reliable error recovery. When a processing job fails midway, the system can safely retry processing the same data without worrying about duplicate transformations or inconsistent state. A non-idempotent transformation might append data to existing records, so retrying would create duplicates. An idempotent transformation would upsert data (insert if not exists, update if exists), so retrying produces the same final state. This distinction becomes critical in distributed systems where partial failures are common and retries are the primary recovery mechanism.
|
||
|
||
Handling partial processing failures requires careful state management. Processing pipelines should be designed so that each stage can be retried independently without affecting other stages. Checkpoint-restart mechanisms enable recovery from the last successful processing state rather than restarting from scratch. For long-running data processing jobs operating on terabyte-scale datasets, checkpointing progress every few minutes means a failure near the end requires reprocessing only recent data rather than the entire dataset. The checkpoint logic must carefully track what data has been processed and what remains, ensuring no data is lost or processed twice.
|
||
|
||
Deterministic transformations\index{Deterministic Transformations!reproducibility} are those that always produce the same output for the same input, without dependence on external factors like time, random numbers, or mutable global state. Transformations that depend on current time (e.g., computing "days since event" based on current date) break determinism because reprocessing historical data would produce different results. The solution is to capture temporal reference points explicitly: instead of "days since event," compute "days from event to reference date" where reference date is fixed and persisted. Random operations should use seeded random number generators where the seed is derived deterministically from input data, ensuring reproducibility.
|
||
|
||
Reliability in the KWS pipeline requires reproducible feature extraction. Audio preprocessing must be deterministic: given the same raw audio file, the same MFCC features are always computed regardless of when processing occurs or which server executes it. This enables debugging model behavior (can always recreate exact features for a problematic example), reprocessing data when bugs are fixed (produces consistent results), and distributed processing (different workers produce identical features from the same input). The processing code captures all parameters (FFT window size, hop length, number of MFCC coefficients) in configuration versioned alongside the code, ensuring reproducibility across time and execution environments. However, even with rigorous design, production systems must implement runtime monitoring to detect skew if it emerges; @sec-ml-operations covers the operational infrastructure for shadow scoring and distribution monitoring.
|
||
|
||
### Distributed Processing {#sec-data-engineering-scaling-distributed-processing-cb9b}
|
||
|
||
\index{Distributed Processing!scaling}\index{Amdahl's Law!parallelization limits}With quality and reliability established, we face the challenge of scale. Quality ensures transformations produce correct outputs; reliability ensures they produce *consistent* outputs. Neither matters if processing cannot keep pace with data volume. As datasets grow larger and ML systems become more complex, the scalability of data processing becomes the limiting factor. Consider the data processing stages discussed earlier: cleaning, quality assessment, transformation, and feature engineering. When these operations must handle terabytes of data, a single machine becomes insufficient. The cleaning techniques that work on gigabytes of data in memory must be redesigned to work across distributed systems.
|
||
|
||
These challenges manifest when quality assessment must keep pace with incoming data, when feature engineering requires computing statistics across entire datasets before transforming individual records, and when transformation pipelines create bottlenecks at massive volumes. Processing must scale from development (gigabytes on laptops) through production (terabytes across clusters) while maintaining consistent behavior.
|
||
|
||
To address these scaling bottlenecks, data must be partitioned across multiple computing resources, which introduces coordination challenges. Distributed coordination is constrained by network round-trip times: local operations complete in microseconds while network coordination requires milliseconds, creating a 1,000$\times$ latency difference. This constraint explains why operations requiring global coordination (like computing normalization statistics across 100 machines) create bottlenecks. Each partition computes local statistics quickly, but combining them requires information from all partitions.
|
||
|
||
Data locality becomes critical at this scale. At 10 GB/s peak throughput, transferring one terabyte of training data across a network takes on the order of 100 seconds; reading the same amount from a 5 GB/s SSD takes on the order of 200 seconds. These are the same order of magnitude, which drives ML system design toward compute-follows-data architectures.[^fn-mapreduce-origin] When processing nodes access local data at RAM speeds (50–200 GB/s) but must coordinate over networks limited to 1–10 GB/s, the bandwidth mismatch creates severe bottlenecks. Geographic distribution amplifies these challenges: cross-datacenter coordination must handle network latency (50–200 ms between regions), partial failures, and regulatory constraints preventing data from crossing borders. Understanding which operations parallelize easily versus those requiring expensive coordination determines system architecture and performance characteristics. This overhead constitutes a *coordination tax* that limits distributed data processing.
|
||
|
||
[^fn-mapreduce-origin]: **MapReduce**: Designed by Jeffrey Dean and Sanjay Ghemawat at Google (2004) to process the company's multi-petabyte web index across thousands of commodity machines [@dean2004mapreduce]. The key design decision was making data locality the scheduling primitive: the scheduler assigns map tasks to nodes that already hold the input data, eliminating network transfers that would otherwise dominate wall-clock time. This compute-follows-data architecture became the template for Hadoop, Spark, and every subsequent distributed data processing framework. \index{MapReduce!data locality}
|
||
|
||
::: {.callout-notebook title="The Coordination Tax"}
|
||
**Problem**: You're computing mean normalization across 1TB of features distributed across 100 nodes. Is it faster to (A) gather all data to one node and compute, or (B) compute local means and aggregate?
|
||
|
||
**Option A (Centralized)**:
|
||
|
||
- Transfer 1TB at 10 GB/s network: **100 seconds**
|
||
- Compute mean on single node: **~1 second**
|
||
- Total: **~101 seconds**
|
||
|
||
**Option B (Distributed)**:
|
||
|
||
- Each node computes local mean: **~0.01 seconds** (10GB at RAM speed)
|
||
- Send 100 partial means (8 bytes each): **<1 ms**
|
||
- Aggregate: **negligible**
|
||
- Total: **~0.01 seconds** (10,000$\times$ faster)
|
||
|
||
**The Engineering Lesson**: Operations that *reduce* data (sum, mean, count) should always run locally first. Operations that *expand* data (joins, cross-products) face unavoidable network costs. Pipeline design should minimize data movement by pushing computation to where data resides, the compute-follows-data principle central to systems like MapReduce [@dean2004mapreduce], Spark [@zaharia2010spark], and modern ML frameworks.
|
||
:::
|
||
|
||
Single-machine processing\index{Single-Machine Processing!scalability} suffices for surprisingly large workloads when engineered carefully. Modern servers with 256 gigabytes RAM can process datasets of several terabytes using out-of-core processing that streams data from disk. Libraries like Dask\index{Dask!out-of-core processing} or Vaex\index{Lazy Evaluation!data processing} enable pandas-like APIs that automatically stream and parallelize computations across multiple cores. Before investing in distributed processing infrastructure, teams should exhaust single-machine optimization: using efficient data formats (Parquet[^fn-parquet-columnar-io] instead of CSV), minimizing memory allocations, leveraging vectorized operations, and exploiting multi-core parallelism. The operational simplicity of single-machine processing—no network coordination, no partial failures, simple debugging—makes it preferable when performance is adequate.
|
||
|
||
[^fn-parquet-columnar-io]: **Parquet**: A columnar storage format that organizes data by column, not by row. For the single-machine optimizations described, this is critical; instead of wastefully reading an entire CSV row to access a few columns, a Parquet reader loads only the specific data needed for a computation. This selective I/O can reduce data movement from disk by 5--10$\times$ for typical feature-selection workloads, enabling terabyte-scale analysis on a single machine. \index{Parquet!I/O reduction}
|
||
|
||
Distributed processing frameworks become necessary when data volumes or computational requirements exceed single-machine capacity, but the speedup achievable through parallelization faces inherent limits described by **Amdahl's Law**.[^fn-amdahls-law-data] Let $S$ be the serial fraction, $p$ the parallelizable fraction, and $N$ the number of workers. @eq-amdahl-data gives the bound:
|
||
|
||
[^fn-amdahls-law-data]: **Amdahl's Law**: Presented by Gene Amdahl at the 1967 AFIPS Spring Joint Computer Conference to argue that multiprocessor designs would yield diminishing returns. His original point -- that the serial fraction of a workload imposes a hard ceiling on parallelism -- applies directly to data pipelines: operations like computing global normalization statistics force serial aggregation phases that cap speedup regardless of how many workers process individual records in parallel. \index{Amdahl's Law!data pipeline ceiling}
|
||
|
||
$$\text{Speedup} \leq \frac{1}{S + \frac{p}{N}}$$ {#eq-amdahl-data}
|
||
|
||
where $S$ represents the serial fraction of work that cannot parallelize, $p$ the parallel fraction (with $S + p = 1$), and $N$ the number of processors. This explains why distributing our KWS feature extraction across 64 cores achieves only a 64$\times$ speedup when the work is embarrassingly parallel ($S \approx 0$), but coordination-heavy operations like computing global normalization statistics might achieve only 10$\times$ speedup even with 64 cores due to the serial aggregation phase. Understanding this relationship guides architectural decisions: operations with high serial fractions should run on fewer, faster cores rather than many slower cores, while highly parallel workloads benefit from maximum distribution. @sec-model-training examines distributed training architectures that apply these principles at cluster scale.
|
||
|
||
Apache Spark\index{Apache Spark!distributed processing} provides a distributed computing framework that parallelizes transformations across clusters of machines, handling data partitioning, task scheduling, and fault tolerance automatically. Beam provides a unified API for both batch and streaming processing, enabling the same transformation logic to run on multiple execution engines (Spark, Flink, Dataflow). TensorFlow's tf.data API optimizes data loading pipelines for ML training, supporting distributed reading, prefetching, and transformation. The choice of framework depends on whether processing is batch or streaming, how transformations parallelize, and what execution environment is available.
|
||
|
||
The feature computation placement trade-off introduced in @sec-data-engineering-feature-computation-placement-a998 takes on additional significance at scale. When distributed processing increases throughput, the cost of recomputing features across hundreds of workers per epoch must be weighed against the storage cost of materializing those features once. At terabyte scale, even small per-example compute costs multiply into significant overhead, reinforcing why production systems adopt hybrid patterns: precomputing expensive, stable features while computing cheap, time-sensitive features on-the-fly.
|
||
|
||
Scalability in the KWS pipeline manifests at multiple stages. Development uses single-machine processing on sample datasets to iterate rapidly. Training at scale requires distributed processing when dataset size (`{python} kws_dataset_size_m_round_str` million examples) exceeds single-machine capacity or when multiple experiments run concurrently. The processing pipeline parallelizes naturally: audio files are independent, so transforming them requires no coordination between workers. Each worker reads its assigned audio files from distributed storage, computes features, and writes results back—a trivially parallel pattern achieving near-linear scalability. Production deployment requires real-time processing on edge devices with severe resource constraints (our `{python} kws_memory_limit_edge_kb_str` kilobyte memory limit), necessitating careful optimization and quantization to fit processing within device capabilities.
|
||
|
||
### Transformation Lineage {#sec-data-engineering-tracking-data-transformation-lineage-3b09}
|
||
|
||
\index{Data Lineage!transformation tracking}\index{Reproducibility!lineage}Completing our four-pillar view of data processing, governance ensures accountability and reproducibility. The governance pillar requires tracking what transformations were applied, when they executed, which version of processing code ran, and what parameters were used. This transformation lineage enables reproducibility essential for debugging, compliance with regulations requiring explainability, and iterative improvement when transformation bugs are discovered. Without comprehensive lineage, teams cannot reproduce training data, cannot explain why models make specific predictions, and cannot safely fix processing bugs without risking inconsistency.
|
||
|
||
Transformation versioning\index{Transformation Versioning!reproducibility} captures which version of processing code produced each dataset. When transformation logic changes—fixing a bug, adding features, or improving quality—the version number increments. Datasets are tagged with the transformation version that created them, enabling identification of all data requiring reprocessing when bugs are fixed. This versioning extends beyond just code versions to capture the entire processing environment: library versions (different NumPy versions may produce slightly different numerical results), runtime configurations (environment variables affecting behavior), and execution infrastructure (CPU architecture affecting floating-point precision).
|
||
|
||
Parameter tracking maintains the specific values used during transformation. For normalization, this means storing the mean and standard deviation computed on training data. For categorical encoding, this means storing the vocabulary (set of all observed categories). For feature engineering, this means storing any constants, thresholds, or parameters used in feature computation. These parameters are typically serialized alongside model artifacts, ensuring serving uses identical parameters to training. Modern ML frameworks like TensorFlow and PyTorch provide mechanisms for bundling preprocessing parameters with models, simplifying deployment and ensuring consistency.
|
||
|
||
Processing lineage\index{Data Lineage!processing history} for reproducibility tracks the complete transformation history from raw data to final features. This includes which raw data files were read, what transformations were applied in what order, what parameters were used, and when processing occurred. Lineage systems like Apache Atlas, Amundsen, or commercial offerings instrument pipelines to automatically capture this flow. When model predictions prove incorrect, engineers can trace back through lineage: which training data contributed to this behavior, what quality scores did that data have, what transformations were applied, and can we recreate this exact scenario to investigate?
|
||
|
||
Code version ties processing results to the exact code that produced them. When processing code lives in version control (Git), each dataset should record the commit hash of the code that created it. This enables recreating the exact processing environment: checking out the specific code version, installing dependencies listed at that version, and running processing with identical parameters. Container technologies like Docker\index{Containerization!reproducible environments} simplify this by capturing the entire processing environment (code, dependencies, system libraries) in an immutable image that can be rerun months or years later with identical results.
|
||
|
||
The governance pillar in the KWS pipeline tracks audio processing parameters that critically affect model behavior. When audio is normalized to standard volume, the reference volume level is persisted. When FFT transforms audio to frequency domain, the window size, hop length, and window function (Hamming, Hanning, etc.) are recorded. When MFCCs are computed, the number of coefficients, frequency range, and mel filterbank parameters are captured. This comprehensive parameter tracking enables several critical capabilities: reproducing training data exactly when debugging model failures, validating that serving uses identical preprocessing to training, and systematically studying how preprocessing choices affect model accuracy. Without this governance infrastructure, teams resort to manual documentation that inevitably becomes outdated or incorrect, leading to subtle training-serving skew that degrades production performance.
|
||
|
||
With raw inputs cleaned, normalized, and transformed into usable features, the remaining question is how we assign meaning to those features. Labels provide that meaning, and they introduce human judgment into the pipeline.
|
||
|
||
## Data Labeling {#sec-data-engineering-data-labeling-6836}
|
||
|
||
\index{Data Labeling!systems engineering}\index{Ground Truth!establishing}The processing pipelines examined above transform raw data into structured features, but one critical input remains: the labels that tell our models what patterns to learn. Consider our KWS system: the ingestion and processing stages have produced millions of clean, standardized audio spectrograms, but these feature vectors are meaningless to a model until someone—or something—declares which ones contain the wake word and which are background noise. This declaration is the **ground truth**\index{Ground Truth!definition}[^fn-ground-truth-proxy], and producing it at scale is the most expensive, most human-dependent, and most error-prone stage of the entire pipeline.
|
||
|
||
[^fn-ground-truth-proxy]: **Ground Truth**: From remote sensing, where orbital measurements are verified by sending a team to the physical location -- the "ground" -- to establish the "truth." The etymology carries a systems warning: ML labels are proxies for reality, not reality itself. When a crowdsourced annotator labels an image as "cat," that label reflects the annotator's judgment, not an objective fact. Every downstream metric -- accuracy, precision, recall -- is measured against this proxy, meaning label quality errors propagate silently into every evaluation of the model. \index{Ground Truth!proxy limitation}
|
||
|
||
Unlike automated transformations that can be parallelized across machines, labeling introduces human judgment into the pipeline, creating unique engineering challenges. A crowdsourced annotator might mislabel a whispered "Alexa" as background noise. An expert radiologist might disagree with a colleague about a borderline diagnosis. These disagreements are not bugs—they are irreducible ambiguity that the labeling system must measure, manage, and mitigate. The infrastructure supporting labeling operations must therefore handle not just throughput (millions of examples) but also quality control (inter-annotator agreement), cost management (the 1,000$\times$ Rule from our data engineering constants), and governance (privacy, consent, and bias monitoring).
|
||
|
||
### Label Types and System Requirements {#sec-data-engineering-label-types-system-requirements-4c33}
|
||
|
||
Building effective labeling systems requires understanding how different label types affect system architecture and resource requirements. Consider a practical example: building a smart city system that needs to detect and track various objects like vehicles, pedestrians, and traffic signs from video feeds. Labels capture information about key tasks or concepts, with each label type imposing distinct storage, computation, and validation requirements.
|
||
|
||
Classification labels\index{Classification Labels!storage requirements} represent the simplest form, categorizing images with a specific tag or (in multi-label classification) tags such as labeling an image as "car" or "pedestrian." While conceptually straightforward, a production system processing millions of video frames must efficiently store and retrieve these labels. Storage requirements are modest (a single integer or string per image), but retrieval patterns matter: training often samples random subsets while validation requires sequential access to all labels, driving different indexing strategies.
|
||
|
||
Bounding boxes\index{Bounding Box Annotation!localization}\index{Object Detection!bounding boxes} extend beyond simple classification by identifying object locations, drawing a box around each object of interest. Our system now needs to track not just what objects exist, but where they are in each frame. This spatial information introduces new storage and processing challenges, especially when tracking moving objects across video frames. Each bounding box requires storing four coordinates (x, y, width, height) plus the object class, multiplying storage by 5$\times$ compared to classification. Bounding box annotation requires pixel-precise positioning that takes 10--20$\times$ longer than classification, dramatically affecting labeling throughput and cost.
|
||
|
||
Segmentation maps\index{Semantic Segmentation!pixel-level labels} provide the most comprehensive information by classifying objects at the pixel level, highlighting each object in a distinct color. For our traffic monitoring system, this might mean precisely outlining each vehicle, pedestrian, and road sign. These detailed annotations significantly increase our storage and processing requirements. A segmentation mask for a $1920\times1080$ image requires 2 million labels (one per pixel), compared to perhaps 10 bounding boxes or a single classification label. This ~200,000$\times$ storage increase relative to bounding boxes—and the hours required per image for manual segmentation—make this approach suitable only when pixel-level precision is essential.
|
||
|
||
{#fig-labels width=90% fig-alt="Three versions of same street scene showing increasing annotation detail: simple classification label, bounding boxes around vehicles and pedestrians, and pixel-level semantic segmentation with distinct colors."}
|
||
|
||
Compare the three annotation levels in @fig-labels. The choice depends on system requirements and resource constraints [@10.1109/ICRA.2017.7989092]: classification suffices for traffic counting, but autonomous vehicles need segmentation maps for precise navigation. Production systems often maintain hybrid annotations—a single camera frame might carry classification labels (scene type), bounding boxes (obstacle detection), and segmentation masks (path planning)—with each label type serving distinct downstream models.
|
||
|
||
\index{Common Voice!speech dataset}
|
||
Beyond these geometric labels, production systems must also manage rich metadata essential for quality control and debugging. The Common Voice dataset [@ardila2020common] exemplifies this in speech recognition: tracking speaker demographics for fairness, recording quality metrics for filtering, and language information for multilingual support. If our traffic monitoring system fails in rainy conditions, weather metadata captured during collection pinpoints the coverage gap. This metadata requirement demonstrates how label type choice cascades through entire system design—the infrastructure must optimize storage for the chosen format, implement appropriate retrieval patterns, and track which model versions used which label versions to correlate quality improvements with performance gains.
|
||
|
||
### Label Accuracy and Consensus {#sec-data-engineering-achieving-label-accuracy-consensus-7190}
|
||
|
||
\index{Label Quality!consensus mechanisms}\index{Inter-Annotator Agreement}In the labeling domain, quality centers on ensuring label accuracy despite the inherent subjectivity and ambiguity in many labeling tasks. Even with clear guidelines and careful system design, some fraction of labels will inevitably be incorrect [@northcutt2021pervasive, @thyagarajan2023multilabel]. The challenge is not eliminating labeling errors entirely—an impossible goal—but systematically measuring and managing error rates to keep them within bounds that do not degrade model performance.
|
||
|
||
Labeling failures arise from two distinct sources requiring different engineering responses. @fig-hard-labels presents concrete examples of both failure modes: data quality issues where the underlying data is genuinely ambiguous or corrupted, such as blurred images where even expert annotators cannot determine the species with certainty. Other errors require deep domain expertise where the correct label is determinable but only by specialists with domain knowledge, as with rare species identification. These different failure modes drive architectural decisions about annotator qualification, task routing, and consensus mechanisms.
|
||
|
||
![**Labeling Ambiguity**: How subjective or difficult examples, such as blurry images or rare species, can introduce errors during data labeling, highlighting the need for careful quality control and potentially expert annotation. Source: [@northcutt2021pervasive].](images/png/label-errors-examples_new.png){#fig-hard-labels fig-alt="Grid of example images showing labeling challenges: blurred animal photos where species is unclear, rare specimens requiring expert knowledge, and ambiguous object boundaries causing annotator disagreement."}
|
||
|
||
Given these inherent quality challenges, production ML systems implement multiple layers of quality control. Systematic quality checks\index{Label Quality!systematic checks} continuously monitor the labeling pipeline through random sampling of labeled data for expert review and statistical methods to flag potential errors. The infrastructure must efficiently process these checks across millions of examples without creating bottlenecks. Sampling strategies typically validate 1-10% of labels, balancing detection sensitivity against review costs. Higher-risk applications like medical diagnosis or autonomous vehicles may validate 100% of labels through multiple independent reviews, while lower-stakes applications like product recommendations may validate only 1% through spot checks.
|
||
|
||
Beyond random sampling approaches, collecting multiple labels per data point, often referred to as "consensus labeling,"\index{Consensus Labeling!quality control} can help identify controversial or ambiguous cases. Professional labeling companies have developed sophisticated infrastructure for this process. For example, [Labelbox](https://labelbox.com/) has consensus tools that track inter-annotator agreement rates and automatically route controversial cases for expert review. [Scale AI](https://scale.com) implements tiered quality control, where experienced annotators verify the work of newer team members. The consensus infrastructure typically collects 3-5 labels per example, computing inter-annotator agreement using metrics like Fleiss' kappa—a generalization of the Cohen's kappa statistic introduced in @sec-data-engineering-detecting-responding-data-drift-509a from two raters to any number of annotators—which measures agreement beyond what would occur by chance. Examples with low agreement (kappa below 0.4) route to expert review rather than forcing consensus from genuinely ambiguous cases.
|
||
|
||
The consensus approach reflects an economic trade-off essential for scalable systems. Expert review costs 10--50$\times$ more per example than crowdsourced labeling, but forcing agreement on ambiguous examples through majority voting of non-experts produces systematically biased labels. By routing only genuinely ambiguous cases to experts—often 5-15% of examples identified through low inter-annotator agreement—systems balance cost against quality. This tiered approach enables processing millions of examples economically while maintaining quality standards through targeted expert intervention.
|
||
|
||
While technical infrastructure provides the foundation for quality control, successful labeling systems must also consider human factors. When working with annotators, organizations need reliable systems for training and guidance. This includes good documentation with clear examples of correct labeling, visual demonstrations of edge cases and how to handle them, regular feedback mechanisms showing annotators their accuracy on gold standard examples, and calibration sessions where annotators discuss ambiguous cases to develop shared understanding. For complex or domain-specific tasks, the system might implement tiered access levels, routing challenging cases to annotators with appropriate expertise based on their demonstrated accuracy on similar examples.
|
||
|
||
Quality monitoring generates substantial data that must be efficiently processed and tracked. The most informative signals span several dimensions. Inter-annotator agreement rates reveal whether multiple annotators converge on the same example, while label confidence scores capture how certain annotators feel about their decisions. Time per annotation serves as a dual-sided indicator: annotations completed too quickly suggest carelessness, while those taking too long suggest confusion or unclear guidelines. Error patterns expose systematic biases or misunderstandings in the annotator pool, and annotator performance on gold standard examples provides ground-truth calibration. Finally, demographic analysis of annotator behavior detects whether certain groups systematically label differently, which could introduce unintended bias into the training data. These metrics must be computed and updated efficiently across millions of examples, often requiring dedicated analytics pipelines that process labeling data in near real-time to catch quality issues before they affect large volumes of data.
|
||
|
||
### Scaling with AI-Assisted Labeling {#sec-data-engineering-scaling-aiassisted-labeling-9360}
|
||
|
||
\index{AI-Assisted Labeling!scalability}\index{Pre-annotation!model-assisted}The scalability pillar drives AI assistance as a force multiplier for human labeling rather than a replacement. Manual annotation alone cannot keep pace with modern ML systems' data needs, while fully automated labeling lacks the nuanced judgment that humans provide. AI-assisted labeling finds the sweet spot: using automation to handle clear cases and accelerate annotation while preserving human judgment for ambiguous or high-stakes decisions. Explore the decision hierarchy in @fig-weak-supervision to see the paths AI assistance offers for scaling labeling operations—each branch requires careful system design to balance speed, quality, and resource usage.
|
||
|
||
::: {#fig-weak-supervision fig-env="figure" fig-pos="htb" fig-cap="**AI-Augmented Labeling Decision Hierarchy**: A top-level question about obtaining labeled data branches into four paths: traditional supervision, semi-supervised learning, weak supervision, and transfer learning, with active learning as a cost-saving alternative. Lower-cost strategies trade labeling precision for throughput. Source: Stanford AI Lab." fig-alt="Hierarchical diagram with question about getting labeled data at top. Four branches: traditional supervision, semi-supervised, weak supervision, and transfer learning. Active learning branches as cost-saving alternative."}
|
||
```{.tikz}
|
||
\begin{tikzpicture}[font=\small\usefont{T1}{phv}{m}{n}]
|
||
%
|
||
\tikzset{%
|
||
Line/.style={line width=1.0pt,black!50,text=black},
|
||
Box/.style={align=flush center,
|
||
inner sep=5pt,
|
||
node distance=0.75,
|
||
draw=GreenLine,
|
||
line width=0.75pt,
|
||
fill=GreenL,
|
||
text width=55mm,
|
||
minimum width=53mm, minimum height=9mm
|
||
},
|
||
Box1/.style={Box,
|
||
node distance=0.35,
|
||
draw=OrangeLine,
|
||
line width=0.75pt,
|
||
fill=OrangeL,
|
||
text width=31mm,
|
||
minimum width=31mm, minimum height=8.2mm
|
||
},
|
||
Box2/.style={Box,
|
||
node distance=0.5,
|
||
draw=BlueLine,
|
||
line width=0.75pt,
|
||
fill=BlueL,
|
||
text width=42mm,
|
||
minimum width=42mm, minimum height=9mm
|
||
},
|
||
Text/.style={%
|
||
inner sep=2pt,
|
||
draw=none,
|
||
line width=0.75pt,
|
||
fill=TextColor,
|
||
text=black,
|
||
font=\footnotesize\usefont{T1}{phv}{m}{n},
|
||
align=flush center,
|
||
minimum width=7mm, minimum height=5mm
|
||
},
|
||
}
|
||
%
|
||
\node[Box,text width=59mm,minimum width=59mm, minimum height=10mm,
|
||
fill=RedL,draw=RedLine](B1){\textbf{How to get more labeled training data?}};
|
||
\node[Box, node distance=1.05,below=of B1,xshift=9mm](B2){\textbf{Traditional Supervision:} Have subject
|
||
matter experts (SMEs) hand-label more training data};
|
||
\node[Box,below=of B2](B3){\textbf{Semi-supervised Learning:} Use structural
|
||
assumptions to automatically leverage unlabeled data};
|
||
\node[Box,below=of B3](B4){\textbf{Weak Supervision:}\\ Get lower-quality
|
||
labels more efficiently and/or at a higher abstraction level};
|
||
\node[Box,below=of B4](B5){\textbf{Transfer Learning:}\\ Use models
|
||
already trained on a different task};
|
||
%
|
||
\node[Box2,above right=0.7 and 1.75 of B2,fill=BrownL,draw=BrownLine](2B1){Too expensive!};
|
||
\node[Box2,below = of 2B1,fill=BrownL,draw=BrownLine](2B2){\textbf{Active Learning:} Estimate
|
||
which points are most valuable to solicit labels for};
|
||
%
|
||
\node[Box2,above right=1.1 and 1.75 of B4](2B3){Get cheaper, lower-quality labels from non-experts};
|
||
\node[Box2,below =of 2B3](2B4){Get higher-level supervision
|
||
over unlabeled data from SMEs};
|
||
\node[Box2,below =of 2B4](2B5){Use one or more
|
||
(noisy/biased) pre-trained models to provide supervision};
|
||
%
|
||
\node[Box1,above right=0.25 and 1.45 of 2B3](3B1){Heuristics};
|
||
\node[Box1,below =of 3B1](3B2){Distant Supervision};
|
||
\node[Box1,below =of 3B2](3B3){Constraints};
|
||
\node[Box1,below =of 3B3](3B4){Expected distributions};
|
||
\node[Box1,below =of 3B4](3B5){Invariances};
|
||
%%
|
||
\foreach \x in{2,3,4,5}{
|
||
\draw[-latex,Line](B1.191)|-(B\x);
|
||
}
|
||
|
||
\foreach \x in{1,2}{
|
||
\draw[-latex,Line](B2.east)--++(0:1.1)|-(2B\x);
|
||
}
|
||
|
||
\foreach \x in{3,4,5}{
|
||
\draw[-latex,Line](B4.355)--++(0:1.1)|-(2B\x);
|
||
}
|
||
|
||
\foreach \x in{1,2,3,4,5}{
|
||
\draw[-latex,Line](2B4.east)--++(0:0.8)|-(3B\x);
|
||
}
|
||
\draw[-latex,Line](2B2)--++(270:1.35)--++(180:3.5)|-(B4.05);
|
||
\draw[-latex,Line](B5.355)-|(2B5);
|
||
\end{tikzpicture}
|
||
```
|
||
:::
|
||
|
||
The key insight behind AI-assisted labeling is that human and machine intelligence excel at different aspects of the task. Humans provide judgment on ambiguous cases, catch subtle errors, and encode domain knowledge that models lack. Machines provide speed, consistency, and tireless attention to clear-cut cases. Modern systems orchestrate these complementary strengths through three primary approaches.
|
||
|
||
Pre-annotation\index{Pre-annotation!workflow}\index{AI-Assisted Labeling!pre-annotation} uses AI models to generate preliminary labels that humans then review and correct—transforming the task from "label from scratch" to "verify and fix." This approach, often employing semi-supervised learning techniques\index{Semi-supervised Learning!labeling efficiency} [@chapelle2009semisupervised], reduces manual effort by 50-80% for many computer vision tasks. Programmatic labeling frameworks like Snorkel [@ratner2018snorkel] extend this further through weak supervision\index{Weak Supervision!programmatic labeling}[^fn-weak-supervision-cost], automatically generating initial labels at scale through rule-based heuristics, knowledge bases, and existing model outputs. In autonomous driving, pre-trained object detection models label vehicles and pedestrians that human annotators verify and refine, handling the majority of clear cases automatically.
|
||
|
||
[^fn-weak-supervision-cost]: **Weak Supervision**: Introduced as the "data programming" paradigm by Ratner et al. at Stanford (2016), motivated by the observation that domain experts write heuristic rules faster than they label examples. This method exchanges manual annotation labor for the upfront effort of writing programmatic "labeling functions," each of which can then label millions of data points at near-zero marginal cost, whereas manual labeling costs scale linearly with dataset size. \index{Weak Supervision!cost trade-off}
|
||
|
||
Large Language Models (LLMs)\index{LLM!labeling assistance} have further transformed labeling pipelines by generating rich text descriptions, creating labeling guidelines from examples, and explaining their reasoning for label assignments. Content moderation systems, for instance, use LLMs for initial content classification with explanations that human reviewers validate. However, LLM integration introduces systems challenges: inference costs (\$0.01–\$1 per example), API rate limits (100–10,000 requests per minute), and the need for systematic output validation since LLMs occasionally produce confident but incorrect labels. Many organizations adopt tiered approaches, using smaller specialized models for routine cases while reserving larger LLMs for complex scenarios requiring nuanced judgment.
|
||
|
||
[^fn-active-learning-budget]: **Active Learning**: Inverts the traditional labeling paradigm: instead of randomly selecting examples to label, the model queries for the examples it needs most, typically those where prediction uncertainty is highest. This converts labeling from a data-proportional cost into a model-proportional cost, achieving target accuracy with 50--90% fewer labels than random sampling. The infrastructure trade-off is compute for labels: active learning adds inference cost (\~\$0.01/image) to select hard examples, but this 10$\times$ reduction in labeling spend makes otherwise budget-infeasible projects viable. \index{Active Learning!budget multiplier}
|
||
|
||
Methods such as active learning\index{Active Learning!label prioritization}[^fn-active-learning-budget] complement these approaches by intelligently prioritizing which examples need human attention [@coleman2022similarity]. These systems continuously analyze model uncertainty to identify valuable labeling candidates. Rather than labeling a random sample of unlabeled data, active learning selects examples where the current model is most uncertain or where labels would most improve model performance. The infrastructure must efficiently compute uncertainty metrics (often prediction entropy or disagreement between ensemble models), maintain task queues ordered by informativeness, and adapt prioritization strategies based on incoming labels. Consider a medical imaging system: active learning might identify unusual pathologies for expert review while handling routine cases through pre-annotation that experts merely verify. This approach can reduce required annotations by 50-90% compared to random sampling, though it requires careful engineering to prevent feedback loops where the model's uncertainty biases which data gets labeled. The following analysis quantifies this *active learning multiplier* in concrete budget terms.
|
||
|
||
::: {.callout-notebook title="The Active Learning Multiplier"}
|
||
**Problem**: You have a 10M image dataset and a \$50K labeling budget. Random sampling achieves 85% accuracy with 100K images. You need 95% accuracy.
|
||
|
||
**The Physics**:
|
||
|
||
1. **Sample Efficiency**: Active learning typically achieves target accuracy with **5--10$\times$ fewer samples** than random selection.
|
||
2. **Cost per Point**: Random sampling = \$0.50/label. Active learning adds compute cost (~\$0.01/image for inference) to find hard examples.
|
||
3. **The Multiplier**:
|
||
* **Random**: To reach 95%, you might need 1M labels (\$500K). **Budget Exceeded.**
|
||
* **Active**: You need ~100K–200K *hard* examples (\$50K–\$100K). **Feasible.**
|
||
|
||
**The Engineering Conclusion**: Algorithm choice is a 10$\times$ lever on data budget. Spending 10% of your budget on compute to select data saves 90% of your budget on human labeling.
|
||
:::
|
||
|
||
Quality control becomes increasingly important as these AI components interact. The system must monitor both AI and human performance through systematic metrics. Model confidence calibration matters: if the AI reports 95% confidence but achieves only 75% accuracy at that confidence level, pre-annotations mislead human reviewers. Human-AI agreement rates reveal whether AI assistance helps or hinders: when humans frequently override AI suggestions, the pre-annotations may be introducing bias rather than accelerating work. These metrics require careful instrumentation throughout the labeling pipeline, tracking not just final labels but the interaction between human and AI at each stage.
|
||
|
||
These principles manifest at scale across safety-critical domains. Autonomous vehicle labeling infrastructure processes millions of sensor frames daily, using AI pre-annotation to label common objects while routing unusual scenarios (construction zones, emergency vehicles) to human experts—a distributed architecture where pre-annotation runs on GPU clusters while human review scales horizontally across thousands of annotators. Medical imaging systems [@krishnan2022selfsupervised] combine pre-annotation for common conditions with active learning for rare pathologies, all under strict privacy constraints with comprehensive audit trails. The common pattern across domains is *tiered escalation*: AI handles clear cases, humans handle ambiguous ones, and monitoring ensures the boundary between "clear" and "ambiguous" adapts as both AI capability and deployment conditions evolve.
|
||
|
||
### Automated Labeling in KWS {#sec-data-engineering-case-study-automated-labeling-kws-systems-976d}
|
||
|
||
\index{Keyword Spotting (KWS)!automated labeling}Our KWS case study has now progressed through problem definition (@sec-data-engineering-framework-application-keyword-spotting-case-study-c8e9), data collection, ingestion, and processing. At the labeling stage, we confront a challenge unique to speech systems at scale. Generating millions of labeled wake word samples without proportional human annotation cost requires moving beyond the manual and crowdsourced approaches we examined earlier. The Multilingual Spoken Words Corpus (MSWC) [@mazumder2021multilingual] demonstrates how automated labeling addresses this challenge through its innovative approach to generating labeled wake word data, containing over `{python} kws_dataset_size_m_str` million one-second spoken examples across 340,000 keywords in `{python} kws_languages_str` different languages.
|
||
|
||
This scale makes manual annotation infeasible: `{python} kws_dataset_size_m_str` million examples at even 10 seconds per label would require approximately `{python} kws_label_hours_str` person-hours—roughly `{python} kws_label_years_str` person-years of full-time effort. Achieving `{python} kws_accuracy_target_str`% accuracy across diverse environments requires millions of training examples covering acoustic variations (background noises, speaking styles, recording environments), and transparent sourcing across `{python} kws_languages_str` languages ensures the technology serves diverse speaker populations.
|
||
|
||
Walk through the pipeline in @fig-mswc to see how this automated system works. It begins with paired sentence audio recordings and corresponding transcriptions from projects like [Common Voice](https://commonvoice.mozilla.org/en) or multilingual captioned content platforms, processing these inputs through forced alignment\index{Forced Alignment!speech labeling}\index{Speech Recognition!forced alignment}[^fn-forced-alignment-kws] to identify precise word boundaries within continuous speech.
|
||
|
||
[^fn-forced-alignment-kws]: **Forced Alignment**: Given a known transcription, the algorithm aligns specific words to audio frames with millisecond precision using dynamic programming (Viterbi algorithm), bypassing the harder problem of recognizing what was said. This distinction is what makes automated KWS corpus construction feasible: because the transcription is already known from paired text, forced alignment converts sentence-level audio into word-level training samples at negligible marginal cost -- enabling datasets of millions of labeled keywords without proportional human annotation effort. \index{Forced Alignment!automated labeling}
|
||
|
||
{#fig-mswc fig-alt="Pipeline showing audio waveform and text transcript inputs processed through forced alignment stage, then segmented into individual one-second labeled keyword samples for KWS training."}
|
||
|
||
Building on these precise timing markers, the extraction system generates clean keyword samples while handling engineering challenges our problem definition anticipated: background noise interfering with word boundaries, speakers stretching or compressing words unexpectedly beyond our target `{python} kws_sample_duration_ms_low_str`-`{python} kws_sample_duration_ms_high_str` millisecond duration, and longer words exceeding the one-second boundary. MSWC provides automated quality assessment that analyzes audio characteristics to identify potential issues with recording quality, speech clarity, or background noise, which is essential for maintaining consistent standards across `{python} kws_dataset_size_m_round_str` million samples without the manual review expenses that would make this scale prohibitive.
|
||
|
||
Modern voice assistant developers often build on this automated labeling foundation. While automated corpora may not contain the specific wake words a product requires, they provide starting points for KWS prototyping, particularly in underserved languages where commercial datasets do not exist. Production systems typically layer targeted human recording and verification for challenging cases (unusual accents, rare words, or difficult acoustic environments), coordinating between automated processing and human expertise.
|
||
|
||
With data acquired, ingested, processed, and labeled, the pipeline has produced its compilation artifacts: millions of feature vectors paired with ground truth labels. The question now shifts from *what* data we have to *where* it lives and *how fast* it reaches the accelerators. Storage architecture determines whether expensive GPUs spend their time computing or waiting.
|
||
|
||
## Storage Architecture {#sec-data-engineering-strategic-storage-architecture-1a6b}
|
||
|
||
\index{Storage Architecture!ML systems}\index{Data Storage!tiered approach}The labeled datasets from our pipeline (23 million audio samples spanning 50 languages for KWS) now require strategic storage decisions that determine training efficiency, serving latency, and long-term maintainability. Storage architecture addresses a core tension: batch training requires sequential scans across millions of examples, while real-time serving demands millisecond lookups of individual feature vectors. These competing access patterns shape every storage decision.
|
||
|
||
ML storage requirements diverge from those of transactional systems. Rather than optimizing for frequent small writes and point lookups that characterize e-commerce or banking, ML workloads prioritize high-throughput sequential reads, large-scale scans, and schema flexibility. A database serving an e-commerce application performs well with millions of individual product lookups per second, but an ML training job scanning that entire catalog repeatedly across epochs requires completely different storage optimization.
|
||
|
||
### Storage System Options {#sec-data-engineering-ml-storage-systems-architecture-options-67fa}
|
||
|
||
\index{IOPS!storage performance}\index{Throughput!storage bandwidth}Storage system selection extends beyond capacity planning. The goal is minimizing the **Data Term** ($\frac{\text{Data}}{\text{Bandwidth}}$) in the **Iron Law** of ML Systems. Every storage medium imposes physical constraints on bandwidth that determine the maximum speed of your training and serving pipelines.
|
||
|
||
Optimizing this term requires understanding two core storage performance metrics:
|
||
|
||
1. **IOPS (Input/Output Operations Per Second)**: The number of distinct read/write requests a device can handle per second. This limits performance for *random access* workloads (e.g., fetching small batches of images or individual user profiles).
|
||
2. **Throughput (Bandwidth)**: The volume of data transferred per second, typically $\text{IOPS} \times \text{Block Size}$. This limits performance for *sequential access* workloads (e.g., scanning a Parquet file for training).
|
||
|
||
The choice between databases, data warehouses, and data lakes is fundamentally a choice about which of these metrics to optimize. Databases (OLTP systems) optimize for high IOPS with small block sizes, making them suited for serving individual feature vectors in real-time where per-request latency dominates. Data warehouses (OLAP systems) optimize for high throughput with large block sizes and sequential access, making them ideal for feature engineering and batch analytics. Data lakes prioritize capacity and throughput for unstructured data, essential for training jobs where the "Data" numerator is measured in petabytes and aggregate bandwidth must scale to thousands of GPUs.
|
||
|
||
Each storage architecture exhibits distinct strengths when applied to specific ML tasks. For online feature serving, the high-IOPS characteristics of databases enable millisecond lookups of individual records. The DLRM recommendation lighthouse introduced in @sec-data-engineering-four-foundational-pillars-c119 exemplifies this challenge at its most extreme: its terabyte-scale embedding tables must serve billions of sparse lookups per second, requiring storage architectures that optimize IOPS over sequential throughput. More generally, a recommendation system looking up a user's profile during real-time inference requires random access optimized for per-request latency.
|
||
|
||
For model training on structured data, the throughput-optimized design of data warehouses enables high-speed sequential scans over large, clean tables. Training a fraud detection model that processes millions of transactions with hundreds of features per transaction benefits from columnar storage that reads only relevant features efficiently, directly reducing the Data Term by minimizing bytes transferred.
|
||
|
||
For exploratory analysis and training on unstructured data (images, audio, text), data lakes provide the flexibility and low-cost storage needed for massive volumes. A computer vision system storing terabytes of raw images alongside metadata, annotations, and intermediate processing results requires the schema flexibility and cost efficiency that only data lakes provide, where the sheer scale of the Data numerator demands the highest aggregate bandwidth.
|
||
|
||
Databases\index{Database!OLTP for ML}\index{OLTP (Online Transaction Processing)} excel at operational and transactional purposes, maintaining product catalogs, user profiles, or transaction histories with strong consistency guarantees and low-latency point lookups. For ML workflows, databases serve specific roles well: storing feature metadata that changes frequently, managing experiment tracking where transactional consistency matters, or maintaining model registries that require atomic updates. A PostgreSQL database handling structured user attributes (user_id, age, country, preferences) provides millisecond lookups for serving systems that need individual user features in real-time. However, databases struggle when ML training requires scanning millions of records repeatedly across multiple epochs. The row-oriented storage that optimizes transactional lookups becomes inefficient when training needs only 20 of 100 columns from each record but must read entire rows to extract those columns.
|
||
|
||
Data warehouses\index{Data Warehouse!analytical queries}\index{Data Warehouse!OLAP}\index{OLAP (Online Analytical Processing)}\index{Columnar Storage!ML training} fill this analytical gap, optimized for complex queries across integrated datasets transformed into standardized schemas. Modern warehouses like Google BigQuery, Amazon Redshift, and Snowflake use columnar storage formats [@stonebraker2005cstore] that enable reading specific features without loading entire records, essential when tables contain hundreds of columns but training needs only a subset. This columnar organization delivers five to ten times I/O reduction compared to row-based formats for typical ML workloads. Consider a fraud detection dataset with 100 columns where models typically use 20 features—columnar storage reads only needed columns, achieving 80% I/O reduction before even considering compression. Many successful ML systems draw training data from warehouses because the structured environment simplifies exploratory analysis and iterative development. Data analysts can quickly compute aggregate statistics, identify correlations between features, and validate data quality using familiar SQL interfaces.
|
||
|
||
However, warehouses assume relatively stable schemas and struggle with truly unstructured data (images, audio, free-form text) or rapidly evolving formats common in experimental ML pipelines. When a computer vision team wants to store raw images alongside extracted features, multiple annotation formats from different labeling vendors, intermediate model predictions, and embedding vectors, forcing all these into rigid warehouse schemas creates more friction than value. Schema evolution becomes painful: adding new feature types requires ALTER TABLE operations that may take hours on large datasets, blocking other operations and slowing iteration velocity.
|
||
|
||
Data lakes\index{Data Lake!unstructured storage}\index{Data Lake!schema flexibility}\index{Schema-on-Read!flexibility} address these limitations by storing structured, semi-structured, and unstructured data in native formats, deferring schema definitions until the point of reading, a pattern called schema-on-read.[^fn-schema-on-read-flexibility]
|
||
|
||
[^fn-schema-on-read-flexibility]: **Schema-on-Read**: Applies data structure definitions at query time rather than during ingestion, contrasting with schema-on-write (traditional databases) where data must conform to a predefined structure before storage. For ML pipelines in early development, schema-on-read enables rapid experimentation -- teams can store raw sensor data, images, and logs without committing to a feature schema upfront. The trade-off is governance: without enforced schemas, data lakes degrade into "data swamps" where finding and validating training data becomes the bottleneck instead. \index{Schema-on-Read!governance trade-off}
|
||
|
||
This flexibility proves valuable during early ML development when teams experiment with diverse data sources and are not yet certain which features will prove useful. A recommendation system might store in the same data lake: transaction logs as JSON, product images as JPEGs, user reviews as text files, clickstream data as Parquet, and model embeddings as NumPy arrays. Rather than forcing these heterogeneous types into a common schema upfront, the data lake preserves them in their native formats. Applications impose schema only when reading, enabling different consumers to interpret the same data differently: one team extracts purchase amounts from transaction logs while another analyzes temporal patterns, each applying schemas suited to their analysis.
|
||
|
||
This flexibility comes with serious governance challenges. Without disciplined metadata management and cataloging, data lakes degrade into "data swamps,"\index{Data Swamp!governance failure} disorganized repositories where finding relevant data becomes nearly impossible, undermining the productivity benefits that motivated their adoption. A data lake might contain thousands of datasets across hundreds of directories with names like "userdata_v2_final" and "userdata_v2_final_ACTUALLY_FINAL", where only the original authors (who have since left the company) understand what distinguishes them. Successful data lake implementations maintain searchable metadata about data lineage, quality metrics, update frequencies, ownership, and access patterns, essentially providing warehouse-like discoverability over lake-scale data. Tools like AWS Glue Data Catalog, Apache Atlas, or Databricks Unity Catalog provide this metadata layer, enabling teams to discover and understand data before investing effort in processing it.
|
||
|
||
@tbl-storage summarizes these essential trade-offs, comparing databases, warehouses, and data lakes across purpose, data types, scale, and performance optimization.
|
||
|
||
| **Attribute** | **Conventional Database** | **Data Warehouse** | **Data Lake** |
|
||
|:-----------------------------|:-------------------------------------------|:----------------------------------------------------------|:-------------------------------------------------------|
|
||
| **Purpose** | Operational and transactional | Analytical and reporting | Storage for raw and diverse data for future processing |
|
||
| **Data type** | Structured | Structured | Structured, semi-structured, and unstructured |
|
||
| **Scale** | Small to medium volumes | Medium to large volumes | Large volumes of diverse data |
|
||
| **Performance Optimization** | Optimized for transactional queries (OLTP) | Optimized for analytical queries (OLAP) | Optimized for scalable storage and retrieval |
|
||
| **Examples** | MySQL, PostgreSQL, Oracle DB | Google BigQuery, Amazon Redshift, Microsoft Azure Synapse | Google Cloud Storage, AWS S3, Azure Data Lake Storage |
|
||
|
||
: **Storage System Characteristics**: Each storage architecture optimizes for a different access pattern that maps to a distinct ML workflow stage. Databases excel at the high-IOPS random access needed for real-time feature serving; data warehouses deliver the sequential throughput that batch training and feature engineering demand; and data lakes provide the schema flexibility and cost efficiency required for petabyte-scale raw data retention. Choosing the wrong system for a workload creates order-of-magnitude performance penalties that no software optimization can overcome. {#tbl-storage}
|
||
|
||
Choosing appropriate storage requires evaluating workload requirements rather than following technology trends. The decision typically follows a maturity trajectory: early-stage projects start with databases (familiar SQL, existing infrastructure), migrate to warehouses when analytical queries overwhelm transactional performance, and adopt data lakes when unstructured data types (images, audio, text) or petabyte-scale cost optimization become critical. Mature ML organizations typically employ all three, orchestrated through unified data catalogs: databases for operational data and real-time serving, warehouses for curated analytical data and feature engineering, and data lakes for raw heterogeneous data and large-scale training. Consider a self-driving car system: vehicle telemetry lives in a database for real-time monitoring, aggregated driving statistics reside in a warehouse for batch analytics, and terabytes of raw camera images and lidar point clouds occupy a data lake for model training—each storage tier optimized for its access pattern.
|
||
|
||
### Storage Performance and Cost {#sec-data-engineering-ml-storage-requirements-performance-1b0d}
|
||
|
||
Beyond the functional differences between storage systems, cost and performance characteristics directly impact ML system economics and iteration speed. Understanding these quantitative trade-offs enables informed architectural decisions based on workload requirements.
|
||
|
||
| **Storage Tier** | **Cost ($/TB/month)** | **Sequential Read** **Throughput** | **Random Read** **Latency** | **Typical ML Use Case** |
|
||
|:-------------------------|----------------------:|:-----------------------------------|:----------------------------|:----------------------------------------------|
|
||
| **NVME SSD (local)** | \$100–300 | 5–7 GB/s | 10–100 μs | Training data loading, active feature serving |
|
||
| **Object Storage** | \$20–25 | 100–500 MB/s | 10–50 ms | Data lake raw storage, |
|
||
| **(S3, GCS)** | | (per connection) | | model artifacts |
|
||
| **Data Warehouse** | \$20–40 | 1–5 GB/s | 100–500 ms | Training data queries, |
|
||
| **(BigQuery, Redshift)** | | (columnar scan) | (query startup) | feature engineering |
|
||
| **In-Memory Cache** | \$500–1000 | 20–50 GB/s | 1–10 μs | Online feature serving, |
|
||
| **(Redis, Memcached)** | | | | real-time inference |
|
||
| **Archival Storage** | \$1–4 | 10–50 MB/s | Hours (retrieval) | Historical retention, |
|
||
| **(Glacier, Nearline)** | | (after retrieval) | | compliance archives |
|
||
|
||
: **Storage Cost-Performance Trade-offs**: Different storage tiers provide distinct cost-performance characteristics that determine their suitability for specific ML workloads. Training data loading requires high-throughput sequential access—an I/O pattern that must align with the **Accelerator Memory Hierarchy** (@sec-hardware-acceleration)—while online serving needs low-latency random reads, while archival storage prioritizes cost over access speed for compliance and historical data. {#tbl-storage-performance}
|
||
|
||
@tbl-storage-performance reveals why ML systems employ tiered storage architectures. Consider the economics of storing our KWS training dataset (`{python} kws_dataset_size_gb_str` GB): object storage costs \$`{python} kws_storage_s3_cost_str`/month, enabling affordable long-term retention of raw audio, while maintaining working datasets on NVMe[^fn-nvme-gpu-utilization] for active training costs \$`{python} kws_storage_nvme_low_cost_str`–`{python} kws_storage_nvme_high_cost_str`/month but provides 50$\times$ faster data loading.
|
||
|
||
[^fn-nvme-gpu-utilization]: **NVMe (Non-Volatile Memory Express)**: A storage protocol connecting directly to the PCIe bus with 64K command queues, delivering 5--7 GB/s sequential throughput and microsecond-scale latency. The contrast with SATA SSD (500 MB/s, single queue) is a 10$\times$ bandwidth gap that directly determines GPU utilization: at SATA speeds, a training pipeline reading 100 GB datasets spends more time waiting for storage than computing gradients, converting a \$15,000 accelerator into an expensive space heater. \index{NVMe!GPU utilization}
|
||
|
||
```{python}
|
||
#| label: storage-loading-calc
|
||
#| echo: false
|
||
# ┌─────────────────────────────────────────────────────────────────────────────
|
||
# │ STORAGE LOADING CALCULATION
|
||
# ├─────────────────────────────────────────────────────────────────────────────
|
||
# │ Context: Storage performance discussion, preceding @tbl-ml-latencies
|
||
# │
|
||
# │ Goal: Contrast dataset loading times across storage tiers.
|
||
# │ Show: The 50× speedup available when moving from object storage to NVMe.
|
||
# │ How: Calculate total load time for a 736 GB dataset.
|
||
# │
|
||
# │ Imports: mlsys.formatting (fmt)
|
||
# │ Exports: nvme_load_str, obj_load_str, load_speedup_str
|
||
# └─────────────────────────────────────────────────────────────────────────────
|
||
from mlsys.formatting import fmt
|
||
|
||
# ┌── P.I.C.O. ISOLATED SCENARIO ───────────────────────────────────────────────
|
||
class StorageLoading:
|
||
"""KWS dataset load time comparison: NVMe vs. object storage."""
|
||
|
||
# ┌── 1. PARAMETERS (Inputs) ──────────────────────────────────────────────
|
||
kws_dataset_gb = 736
|
||
nvme_bw_gbs = 5 # effective NVMe throughput
|
||
obj_bw_gbs = 0.1 # typical object storage throughput
|
||
|
||
# ┌── 2. CALCULATION (The Physics) ────────────────────────────────────────
|
||
nvme_load_s = kws_dataset_gb / nvme_bw_gbs
|
||
obj_load_s = kws_dataset_gb / obj_bw_gbs
|
||
load_speedup = obj_load_s / nvme_load_s
|
||
|
||
# ┌── 4. OUTPUTS (Formatting) ─────────────────────────────────────────────
|
||
nvme_load_str = fmt(nvme_load_s, precision=0, commas=False)
|
||
obj_load_str = fmt(obj_load_s, precision=0, commas=True)
|
||
load_speedup_str = fmt(load_speedup, precision=0, commas=False)
|
||
nvme_bw_gbs_str = fmt(nvme_bw_gbs, precision=0, commas=False)
|
||
|
||
# ┌── EXPORTS (Bridge to Text) ─────────────────────────────────────────────────
|
||
nvme_load_str = StorageLoading.nvme_load_str
|
||
obj_load_str = StorageLoading.obj_load_str
|
||
load_speedup_str = StorageLoading.load_speedup_str
|
||
nvme_bw_gbs_str = StorageLoading.nvme_bw_gbs_str
|
||
```
|
||
|
||
The performance difference directly impacts iteration velocity. Training that loads data at `{python} nvme_bw_gbs_str` GB/s completes dataset loading in `{python} nvme_load_str` seconds, compared to `{python} obj_load_str` seconds at typical object storage speeds. This `{python} load_speedup_str`$\times$ difference determines whether teams can iterate multiple times daily or must wait hours between experiments.
|
||
|
||
To build engineering judgment, practitioners must internalize the orders of magnitude separating these tiers. @tbl-ml-latencies translates these disparities into human-scale analogies that build intuition for system design: if a CPU cycle were one second, fetching from local SSD would take two days, while a cross-country network request would span six years. Internalizing these ratios—three orders of magnitude between L1 cache and DRAM, another three between DRAM and SSD—explains why seemingly small architectural choices cascade into large performance differences.
|
||
|
||
| **Operation** | **Latency (ns)** | **Human Scale** | **ML System Impact** |
|
||
|:-------------------------|-----------------:|----------------:|:-----------------------------|
|
||
| **L1 Cache Reference** | 0.5 | 1 second | Immediate |
|
||
| **L2 Cache Reference** | 7 | 14 seconds | Fast computation |
|
||
| **Main Memory (DRAM)** | 100 | 3 minutes | The "Memory Wall" threshold |
|
||
| **SSD (local NVMe)** | 100,000 | 2 days | Data loading bottleneck |
|
||
| **Network (same DC)** | 500,000 | 1 week | Distributed coordination lag |
|
||
| **SSD (remote network)** | 2,000,000 | 1 month | Training-serving skew source |
|
||
| **Object Store (S3)** | 20,000,000 | 1 year | Archival access |
|
||
| **Internet (CA to VA)** | 100,000,000 | 6 years | Global user experience |
|
||
|
||
: **Latency Numbers Every ML Systems Engineer Should Know**: Understanding the quantitative disparities in the storage hierarchy is essential for diagnosing bottlenecks. If a CPU cycle were 1 second, fetching data from local SSD would be like waiting 2 days, while a cross-country network request would take 6 years. Source: Adapted from Jeff Dean's "Numbers Every Programmer Should Know". For the full hardware reference with bandwidth and energy ratios, see @sec-machine-foundations-numbers-know-b531. {#tbl-ml-latencies}
|
||
|
||
These latency numbers originated from Jeff Dean's influential 2009 talk at Stanford,[^fn-jeff-dean-latency-numbers] which established the quantitative culture that distinguishes systems engineering from programming. Understanding them is foundational to ML systems engineering: they explain why a poorly designed storage architecture can idle a \$15,000 GPU, and why distributed training requires careful attention to data locality.
|
||
|
||
[^fn-jeff-dean-latency-numbers]: **Jeff Dean**: Google Senior Fellow, architect of MapReduce, BigTable, Spanner, and TensorFlow. His 2009 Stanford talk distilled the numbers in @tbl-ml-latencies into the engineering heuristic that an L1 cache reference (0.5 ns) and a cross-datacenter round trip (150 ms) span a $3 \times 10^{8}$ ratio -- eight orders of magnitude that explain why a training pipeline reading features from remote storage instead of local NVMe can convert a \$15,000 accelerator into an idle heater. \index{Dean, Jeff!latency numbers}
|
||
|
||
Beyond the core storage capabilities we have examined, ML workloads introduce unique requirements that conventional databases and warehouses were not designed to handle. Understanding these ML-specific needs and their performance implications shapes infrastructure decisions that cascade through the entire development lifecycle, from experimental notebooks to production serving systems handling millions of requests per second.
|
||
|
||
Modern ML models contain millions to billions of parameters\index{Model Parameters!storage requirements} requiring storage and retrieval patterns quite different from traditional data. GPT-3 [@brown2020language] requires approximately `{python} gpt3_fp32_str` gigabytes for model weights when stored in FP32 format (`{python} gpt3_params_str` billion parameters times 4 bytes), though practical deployments typically use FP16 (`{python} gpt3_fp16_str` GB) or quantized formats for reduced storage and faster inference (see @sec-model-compression for quantization techniques). Even at FP16 precision, this exceeds many organizations' entire operational databases. The trajectory reveals accelerating scale: from AlexNet's `{python} alexnet_params_str` million parameters in 2012 [@alexnet2012] to GPT-3's `{python} gpt3_params_str` billion parameters in 2020, model size grew approximately `{python} model_growth_str`-fold in eight years (`{python} alexnet_params_str` M to `{python} gpt3_params_str` B parameters). Storage systems must handle these dense numerical arrays efficiently for both capacity and access speed. Unlike typical files where sequential organization matters for readability, model weights benefit from block-aligned storage enabling parallel reads across parameter groups. When multiple GPUs need to read model data from shared storage, whether during training initialization or checkpoint loading, storage systems must deliver aggregate bandwidth approaching network interface limits, often 25 Gbps or higher, without introducing bottlenecks that would idle expensive compute resources.
|
||
|
||
The iterative nature of ML development introduces versioning requirements\index{Model Versioning!requirements} qualitatively different from traditional software. Git excels at tracking code changes where files are predominantly text with small incremental modifications, but it fails for large binary files where even small model changes result in entirely new checkpoints. Storing 10 versions of a 10 GB model naively would consume 100 GB, but most ML versioning systems store only deltas between versions, reducing storage proportionally to how much models actually change. Tools like DVC (Data Version Control) and MLflow maintain pointers to model artifacts rather than storing copies, enabling efficient versioning while preserving the ability to reproduce any historical model. A typical ML project generates hundreds of model versions during hyperparameter tuning—one version per training run as engineers explore learning rates, batch sizes, architectures, and regularization strategies. Without systematic versioning capturing training configuration, accuracy metrics, and training data version alongside model weights, reproducing results becomes impossible when yesterday's model performed better than today's but teams cannot identify which configuration produced it. This reproducibility challenge connects directly to governance requirements: regulatory compliance often requires demonstrating exactly which data and process produced specific model predictions, a concern addressed by the data debt and remediation strategies in @sec-data-engineering-data-debt-hidden-liability-3335.
|
||
|
||
Large-scale training generates substantial intermediate data requiring storage systems to handle concurrent read/write operations efficiently. When training jobs use multiple GPUs, each processing unit works on different portions of data, requiring storage systems to handle many simultaneous reads and writes. The specific patterns depend on the parallelization strategy employed, which @sec-model-training examines in detail. From a storage perspective, systems must handle concurrent I/O at rates proportional to the number of processing units, with each potentially writing tens to hundreds of megabytes of intermediate results during model updates. Memory optimization strategies like *activation checkpointing* (also called gradient checkpointing) trade computation for storage by discarding intermediate activations during the forward pass and recomputing them during backpropagation—this reduces GPU memory requirements but increases storage I/O as intermediate values write to disk. Storage systems must provide low-latency access to support efficient coordination. If workers spend more time waiting for storage than performing computations, parallel processing becomes counterproductive regardless of the specific training approach used.
|
||
|
||
The bandwidth hierarchy constrains ML system design at every level, creating bottlenecks that no amount of compute optimization can overcome. While RAM delivers 50 to 200 gigabytes per second bandwidth on modern servers, network storage systems typically provide only one to 10 gigabytes per second, and even high-end NVMe SSDs max out at one to seven gigabytes per second sequential throughput. Modern GPUs can process data faster than storage can supply it, creating scenarios where expensive accelerators idle waiting for data. Consider training an image classification model: loading 1,000 images per second at 150 KB each requires 150 MB/s sustained throughput from storage. When the GPU can process images faster than storage delivers them, the data pipeline—not the model—becomes the bottleneck. A 10-fold mismatch between GPU processing speed and storage bandwidth means expensive accelerators sit idle 90% of the time waiting for data. No amount of GPU optimization can overcome this I/O constraint.
|
||
|
||
Understanding these quantitative relationships enables informed architectural decisions about storage system selection and data pipeline optimization, which become even more critical during distributed training as examined in @sec-model-training. The training throughput equation reveals the critical dependencies:
|
||
|
||
```{python}
|
||
#| label: storage-bandwidth-calc
|
||
#| echo: false
|
||
# ┌─────────────────────────────────────────────────────────────────────────────
|
||
# │ STORAGE BANDWIDTH CALCULATION
|
||
# ├─────────────────────────────────────────────────────────────────────────────
|
||
# │ Context: Callout "Storage Bandwidth Budget"
|
||
# │
|
||
# │ Goal: Derive the storage bandwidth required to saturate a training GPU.
|
||
# │ Show: That SATA SSDs leave the A100 at 13% utilization for ResNet-50.
|
||
# │ How: Calculate required MB/s from image consumption rate and file size.
|
||
# │
|
||
# │ Imports: mlsys.constants (KB, GB, flop, GFLOPs, TFLOPs),
|
||
# │ mlsys.formatting (fmt)
|
||
# │ Exports: max_img_per_sec_str, required_bw_gbs_str, a100_flops_t_str,
|
||
# │ resnet_flops_g_str, image_size_kb_str, sata_bw_mbs_str,
|
||
# │ sata_max_img_sec_str, sata_utilization_pct_str, s3_bw_mbs_str,
|
||
# │ s3_workers_str
|
||
# └─────────────────────────────────────────────────────────────────────────────
|
||
from mlsys.constants import KB, GB, flop, GFLOPs, TFLOPs
|
||
from mlsys.formatting import fmt
|
||
|
||
# ┌── P.I.C.O. ISOLATED SCENARIO ───────────────────────────────────────────────
|
||
class StorageBandwidth:
|
||
"""Storage bandwidth budget for saturating an A100 training ResNet-50."""
|
||
|
||
# ┌── 1. PARAMETERS (Inputs) ──────────────────────────────────────────────
|
||
h_a100 = Hardware.Cloud.A100
|
||
m_resnet = Models.Vision.ResNet50
|
||
image_size_kb = 150
|
||
sata_bw_mbs = 500
|
||
s3_bw_mbs = 100
|
||
|
||
# ┌── 2. CALCULATION (The Physics) ────────────────────────────────────────
|
||
a100_flops_val = h_a100.peak_flops.m_as(flop/second)
|
||
resnet_flops_per_img = (m_resnet.inference_flops * 3).m_as(flop) # forward + backward
|
||
max_img_per_sec = int(a100_flops_val / resnet_flops_per_img)
|
||
required_bw_gbs = (max_img_per_sec * image_size_kb * KB).m_as(GB)
|
||
sata_max_img_sec = int(sata_bw_mbs * MILLION / (image_size_kb * 1e3))
|
||
sata_utilization_pct = int(round(sata_max_img_sec / max_img_per_sec * 100))
|
||
s3_workers = int(round(required_bw_gbs * 1e3 / s3_bw_mbs))
|
||
|
||
# ┌── 4. OUTPUTS (Formatting) ─────────────────────────────────────────────
|
||
max_img_per_sec_str = f"{max_img_per_sec:,}"
|
||
required_bw_gbs_str = fmt(required_bw_gbs, precision=1, commas=False)
|
||
a100_flops_t_str = str(int(h_a100.peak_flops.m_as(TFLOPs/second)))
|
||
resnet_flops_g_str = str(int(resnet_flops_per_img / 1e9))
|
||
image_size_kb_str = fmt(image_size_kb, precision=0, commas=False)
|
||
sata_bw_mbs_str = fmt(sata_bw_mbs, precision=0, commas=False)
|
||
sata_max_img_sec_str = f"{sata_max_img_sec:,}"
|
||
sata_utilization_pct_str = f"{sata_utilization_pct}"
|
||
s3_bw_mbs_str = fmt(s3_bw_mbs, precision=0, commas=False)
|
||
s3_workers_str = f"{s3_workers}"
|
||
|
||
# ┌── EXPORTS (Bridge to Text) ─────────────────────────────────────────────────
|
||
max_img_per_sec_str = StorageBandwidth.max_img_per_sec_str
|
||
required_bw_gbs_str = StorageBandwidth.required_bw_gbs_str
|
||
a100_flops_t_str = StorageBandwidth.a100_flops_t_str
|
||
resnet_flops_g_str = StorageBandwidth.resnet_flops_g_str
|
||
image_size_kb_str = StorageBandwidth.image_size_kb_str
|
||
sata_bw_mbs_str = StorageBandwidth.sata_bw_mbs_str
|
||
sata_max_img_sec_str = StorageBandwidth.sata_max_img_sec_str
|
||
sata_utilization_pct_str = StorageBandwidth.sata_utilization_pct_str
|
||
s3_bw_mbs_str = StorageBandwidth.s3_bw_mbs_str
|
||
s3_workers_str = StorageBandwidth.s3_workers_str
|
||
```
|
||
|
||
Designing for high-throughput training requires calculating a *storage bandwidth budget*.
|
||
|
||
::: {.callout-notebook title="Storage Bandwidth Budget"}
|
||
**The Problem:** Design a storage system that ensures an NVIDIA A100 is never starved for data while training ResNet-50.
|
||
|
||
**1. The Compute Term (Throughput Ceiling)**
|
||
|
||
* **Hardware Peak:** NVIDIA A100 = `{python} a100_tflops_fp16_str` TFLOPS (FP16 Tensor Core, dense operations).
|
||
* **Model Cost:** ResNet-50 requires approximately 4 GFLOPs for the forward pass per image; including the backward pass, total is approximately `{python} resnet_flops_g_str` GFLOPs per training step.
|
||
* **Maximum Theoretical Throughput**:
|
||
`{python} a100_flops_t_str` $\times$ $10^{12}$ FLOPs/sec / `{python} resnet_flops_g_str` $\times$ $10^{9}$ FLOPs/img = **`{python} max_img_per_sec_str` images/sec**
|
||
|
||
**2. The Data Term (Bandwidth Requirement)**
|
||
|
||
* **Image Size:** `{python} image_size_kb_str` KB (JPEG compressed).
|
||
* **Required Bandwidth**:
|
||
`{python} max_img_per_sec_str` img/sec$\times$ `{python} image_size_kb_str` KB/img ≈ **`{python} required_bw_gbs_str` GB/s**
|
||
|
||
**The Systems Conclusion:**
|
||
To saturate a *single* A100, your storage must deliver **`{python} required_bw_gbs_str` GB/s**.
|
||
|
||
* **S3 Standard**: ~`{python} s3_bw_mbs_str` MB/s per thread. You need **`{python} s3_workers_str` concurrent worker threads**.
|
||
* **SATA SSD**: ~`{python} sata_bw_mbs_str` MB/s sequential. Totally insufficient (bottleneck).
|
||
* **NVMe SSD**: ~3-7 GB/s. **Required.**
|
||
|
||
**Iron Law Implication:** If you use SATA SSDs, your maximum throughput is capped at `{python} sata_bw_mbs_str` MB/s / `{python} image_size_kb_str` KB ≈ `{python} sata_max_img_sec_str` img/s. Your USD 15,000 GPU will run at **`{python} sata_utilization_pct_str`% utilization** (`{python} sata_max_img_sec_str`/`{python} max_img_per_sec_str`). Storage physics dictates training speed.
|
||
:::
|
||
|
||
The `{python} sata_bw_mbs_str` MB/s figure represents SATA III sequential read throughput (the interface maximum is 600 MB/s). Real-world random read performance with small files can be significantly lower.
|
||
|
||
This calculation illustrates the general principle governing data pipelines, as formalized in @eq-training-throughput and @eq-data-supply:
|
||
|
||
$$\text{Training Throughput} = \min(\text{Compute Capacity}, \text{Data Supply Rate})$$ {#eq-training-throughput}
|
||
\index{Training Throughput!compute vs data}
|
||
|
||
$$\text{Data Supply Rate} = \text{Storage Bandwidth} \times (1 - \text{Overhead})$$ {#eq-data-supply}
|
||
\index{Data Supply Rate!bandwidth constraint}
|
||
|
||
When storage bandwidth becomes the limiting factor, teams must either improve storage performance through faster media, parallelization, or caching, or reduce data movement requirements through compression, quantization, or architectural changes. Large language model training may require processing hundreds of gigabytes of text per hour, while computer vision models processing high-resolution imagery can demand sustained data rates exceeding 50 gigabytes per second across distributed clusters. These requirements explain the rise of specialized ML storage systems optimizing data loading pipelines: PyTorch DataLoader with multiple worker processes parallelizing I/O, TensorFlow tf.data API with prefetching and caching, and frameworks like NVIDIA DALI (Data Loading Library) that offload data augmentation to GPUs rather than loading pre-augmented data from storage.
|
||
|
||
File format selection dramatically impacts the **Data Term** ($\frac{D_{vol}}{BW}$) of the Iron Law. We can quantify this impact as *format efficiency* ($\eta_{format}$), which acts as a multiplier on effective bandwidth.
|
||
|
||
```{python}
|
||
#| label: format-efficiency-calc
|
||
#| echo: false
|
||
# ┌─────────────────────────────────────────────────────────────────────────────
|
||
# │ FORMAT EFFICIENCY CALCULATION
|
||
# ├─────────────────────────────────────────────────────────────────────────────
|
||
# │ Context: Callout "Format Efficiency"
|
||
# │
|
||
# │ Goal: Quantify the I/O waste of row-oriented vs. columnar formats.
|
||
# │ Show: That Parquet yields a 5× throughput gain by skipping unused columns.
|
||
# │ How: Model effective data transfer for 20 columns out of a 100-column dataset.
|
||
# │
|
||
# │ Imports: mlsys.formatting (fmt)
|
||
# │ Exports: eta_csv_str, eta_parquet_str, waste_pct_str,
|
||
# │ throughput_ratio_str, needed_cols_str, total_cols_str
|
||
# └─────────────────────────────────────────────────────────────────────────────
|
||
from mlsys.formatting import fmt
|
||
|
||
# ┌── P.I.C.O. ISOLATED SCENARIO ───────────────────────────────────────────────
|
||
class FormatEfficiency:
|
||
"""CSV vs. Parquet I/O efficiency for a 20-of-100-column fraud model."""
|
||
|
||
# ┌── 1. PARAMETERS (Inputs) ──────────────────────────────────────────────
|
||
needed_cols = 20
|
||
total_cols = 100
|
||
eta_parquet = 1.0
|
||
|
||
# ┌── 2. CALCULATION (The Physics) ────────────────────────────────────────
|
||
eta_csv = needed_cols / total_cols
|
||
waste_pct = (1 - eta_csv) * 100
|
||
throughput_ratio = eta_parquet / eta_csv
|
||
|
||
# ┌── 4. OUTPUTS (Formatting) ─────────────────────────────────────────────
|
||
eta_csv_str = fmt(eta_csv, precision=1, commas=False)
|
||
eta_parquet_str = fmt(eta_parquet, precision=1, commas=False)
|
||
waste_pct_str = fmt(waste_pct, precision=0, commas=False)
|
||
throughput_ratio_str = fmt(throughput_ratio, precision=0, commas=False)
|
||
needed_cols_str = f"{needed_cols}"
|
||
total_cols_str = f"{total_cols}"
|
||
|
||
# ┌── EXPORTS (Bridge to Text) ─────────────────────────────────────────────────
|
||
eta_csv_str = FormatEfficiency.eta_csv_str
|
||
eta_parquet_str = FormatEfficiency.eta_parquet_str
|
||
waste_pct_str = FormatEfficiency.waste_pct_str
|
||
throughput_ratio_str = FormatEfficiency.throughput_ratio_str
|
||
needed_cols_str = FormatEfficiency.needed_cols_str
|
||
total_cols_str = FormatEfficiency.total_cols_str
|
||
```
|
||
|
||
::: {.callout-notebook title="Format Efficiency"}
|
||
**The Concept**: Storage formats determine how much "waste" data you move to get the signal you need, as captured in @eq-effective-bandwidth:
|
||
|
||
$$ \text{Effective Bandwidth} = \text{Physical Bandwidth} \times \eta_{format} $$ {#eq-effective-bandwidth}
|
||
|
||
**Scenario**: Training a fraud model using `{python} needed_cols_str` features from a `{python} total_cols_str`-column table.
|
||
|
||
- **Row-Oriented (CSV)**: Must read all `{python} total_cols_str` columns to get the `{python} needed_cols_str` needed.
|
||
- eta_format = `{python} needed_cols_str`/`{python} total_cols_str` = `{python} eta_csv_str`.
|
||
- **Result**: You waste `{python} waste_pct_str`% of your disk bandwidth.
|
||
- **Column-Oriented (Parquet)**: Reads only the `{python} needed_cols_str` columns needed.
|
||
- eta_format ≈ `{python} eta_parquet_str` (ignoring metadata overhead).
|
||
- **Result**: You get **`{python} throughput_ratio_str`$\times$ higher effective throughput**.
|
||
|
||
**The Systems Conclusion**: Switching from CSV to Parquet is not just a file change; it is mathematically equivalent to buying a **`{python} throughput_ratio_str`$\times$ faster hard drive**. The serialization overhead of different formats is quantified in @tbl-serialization-cost in @sec-data-foundations. For a deeper treatment of row vs. columnar storage layouts and the algebra of data operations (selection, projection, join), see @sec-data-foundations-row-vs-columnar-formats-bca2.
|
||
:::
|
||
|
||
Format choice is not a software preference. It is a direct consequence of the Data Gravity Invariant: when data is too massive to move, you must minimize the bytes read per training step, and columnar formats achieve this by reading only the columns the model requires.
|
||
|
||
Columnar storage formats\index{Parquet!columnar format}\index{ORC!columnar format} like Parquet or ORC deliver this five to 10 times I/O reduction compared to row-based formats like CSV or JSON for typical ML workloads. The reduction comes from two mechanisms: reading only required columns rather than entire records, and column-level compression exploiting value patterns within columns. Consider a fraud detection dataset with 100 columns where models typically use 20 features—columnar formats read only needed columns, achieving 80% I/O reduction before compression. Column compression proves particularly effective for categorical features with limited cardinality: a country code column with 200 unique values in 100 million records compresses 20 to 50 times through dictionary encoding, while run-length encoding compresses sorted columns by storing only value changes. The combination can achieve total I/O reduction of 20 to 100 times compared to uncompressed row formats, directly translating to faster training iterations and reduced infrastructure costs.
|
||
|
||
```{python}
|
||
#| label: compression-tradeoff-calc
|
||
#| echo: false
|
||
# ┌─────────────────────────────────────────────────────────────────────────────
|
||
# │ COMPRESSION TRADEOFF CALCULATION
|
||
# ├─────────────────────────────────────────────────────────────────────────────
|
||
# │ Context: Compression tradeoff discussion following format efficiency
|
||
# │
|
||
# │ Goal: Contrast the impacts of decompression throughput on training time.
|
||
# │ Show: That Snappy saves hours of wall-clock time over Gzip across 50 epochs.
|
||
# │ How: Calculate total decompression latency for a 100 GB dataset.
|
||
# │
|
||
# │ Imports: mlsys.formatting (fmt)
|
||
# │ Exports: decompress_speedup_str, gzip_min_str, snappy_min_str,
|
||
# │ time_diff_str, total_diff_hours_str, snappy_decompress_mbs_str,
|
||
# │ gzip_decompress_mbs_str, dataset_gb_str, n_epochs_str
|
||
# └─────────────────────────────────────────────────────────────────────────────
|
||
from mlsys.formatting import fmt
|
||
|
||
# ┌── P.I.C.O. ISOLATED SCENARIO ───────────────────────────────────────────────
|
||
class CompressionTradeoff:
|
||
"""Snappy vs. Gzip decompression time over 50 training epochs."""
|
||
|
||
# ┌── 1. PARAMETERS (Inputs) ──────────────────────────────────────────────
|
||
snappy_decompress_mbs = 500 # MB/s
|
||
gzip_decompress_mbs = 120 # MB/s
|
||
dataset_gb = 100
|
||
n_epochs = 50
|
||
|
||
# ┌── 2. CALCULATION (The Physics) ────────────────────────────────────────
|
||
decompress_speedup = snappy_decompress_mbs / gzip_decompress_mbs
|
||
dataset_mb = dataset_gb * 1000
|
||
gzip_decompress_min = dataset_mb / gzip_decompress_mbs / 60
|
||
snappy_decompress_min = dataset_mb / snappy_decompress_mbs / 60
|
||
time_diff_min = gzip_decompress_min - snappy_decompress_min
|
||
total_diff_hours = time_diff_min * n_epochs / 60
|
||
|
||
# ┌── 4. OUTPUTS (Formatting) ─────────────────────────────────────────────
|
||
decompress_speedup_str = fmt(decompress_speedup, precision=0, commas=False)
|
||
gzip_min_str = fmt(gzip_decompress_min, precision=0, commas=False)
|
||
snappy_min_str = fmt(snappy_decompress_min, precision=0, commas=False)
|
||
time_diff_str = fmt(time_diff_min, precision=0, commas=False)
|
||
total_diff_hours_str = fmt(total_diff_hours, precision=0, commas=False)
|
||
snappy_decompress_mbs_str = f"{snappy_decompress_mbs}"
|
||
gzip_decompress_mbs_str = f"{gzip_decompress_mbs}"
|
||
compress_dataset_gb_str = f"{dataset_gb}"
|
||
n_epochs_str = f"{n_epochs}"
|
||
|
||
# ┌── EXPORTS (Bridge to Text) ─────────────────────────────────────────────────
|
||
decompress_speedup_str = CompressionTradeoff.decompress_speedup_str
|
||
gzip_min_str = CompressionTradeoff.gzip_min_str
|
||
snappy_min_str = CompressionTradeoff.snappy_min_str
|
||
time_diff_str = CompressionTradeoff.time_diff_str
|
||
total_diff_hours_str = CompressionTradeoff.total_diff_hours_str
|
||
snappy_decompress_mbs_str = CompressionTradeoff.snappy_decompress_mbs_str
|
||
gzip_decompress_mbs_str = CompressionTradeoff.gzip_decompress_mbs_str
|
||
compress_dataset_gb_str = CompressionTradeoff.compress_dataset_gb_str
|
||
n_epochs_str = CompressionTradeoff.n_epochs_str
|
||
```
|
||
|
||
Compression algorithm\index{Compression!algorithm selection} selection involves trade-offs between compression ratio and decompression speed. While gzip achieves higher compression ratios of six to eight times, Snappy achieves only two to three times compression but decompresses at `{python} snappy_decompress_mbs_str` megabytes per second—roughly `{python} decompress_speedup_str` times faster than gzip's `{python} gzip_decompress_mbs_str` megabytes per second. For ML training where throughput matters more than storage costs, Snappy's speed advantage often outweighs gzip's space savings. Training on a `{python} compress_dataset_gb_str` gigabyte dataset compressed with gzip requires `{python} gzip_min_str` minutes of decompression time, while Snappy requires only `{python} snappy_min_str` minutes. When training iterates over data for `{python} n_epochs_str` epochs, this `{python} time_diff_str`-minute difference per epoch compounds to `{python} total_diff_hours_str` hours total—potentially the difference between running experiments overnight versus waiting multiple days for results. The choice cascades through the system: faster decompression enables higher batch sizes (fitting more examples in memory after decompression), reduced buffering requirements (less decompressed data needs staging), and better GPU utilization (less time idle waiting for data).
|
||
|
||
Storage performance optimization extends beyond format and compression to data layout strategies. Data partitioning\index{Data Partitioning!query optimization} based on frequently used query parameters dramatically improves retrieval efficiency. A recommendation system processing user interactions might partition data by date and user demographic attributes, enabling training on recent data subsets or specific user segments without scanning the entire dataset. Partitioning strategies interact with distributed training patterns: range partitioning by user ID enables data parallel training where each worker processes a consistent user subset, while random partitioning ensures workers see diverse data distributions. The partitioning granularity matters—too few partitions limit parallelism, while too many partitions increase metadata overhead and reduce efficiency of sequential reads within partitions.
|
||
|
||
### Storage Across the ML Lifecycle {#sec-data-engineering-storage-across-ml-lifecycle-2b22}
|
||
|
||
\index{ML Lifecycle!storage requirements}Storage requirements evolve substantially as ML systems progress from development through deployment. The same dataset is accessed very differently during exploratory analysis (random sampling for visualization), model training (sequential scanning for epochs), and production serving (random access for individual predictions)—requiring storage architectures that accommodate these diverse patterns.
|
||
|
||
During development, flexibility matters more than raw performance. The key challenge is managing dataset versions without overwhelming storage capacity: 10 experiments on a 100 GB dataset would naively require 1 TB of copies. Tools like DVC address this by tracking versions through pointers and storing only deltas. Governance considerations demand tiered access controls where synthetic or anonymized datasets are broadly available for experimentation, while production data containing sensitive information requires approval and audit trails.
|
||
|
||
Training phase requirements shift dramatically toward throughput. Modern deep learning processes massive datasets repeatedly across dozens or hundreds of epochs, making I/O efficiency critical. Training ResNet-50 on ImageNet across 8 GPUs requires loading approximately 40,000 images per second—roughly 500 MB/s of decompressed data. Storage unable to sustain this throughput idles GPUs, directly increasing infrastructure costs. The feature computation placement trade-off (@sec-data-engineering-feature-computation-placement-a998) is especially acute here: precomputing features achieves 30-fold storage reduction (150 KB images to 5 KB vectors) but introduces staleness risk when extraction logic changes.
|
||
|
||
Deployment and serving requirements prioritize low-latency random access.\index{Real-time Inference!storage requirements} A recommendation system serving 10,000 requests per second with 10 ms latency budgets requires 100,000 random reads per second—achievable only through in-memory databases like Redis\index{In-memory Database!feature serving}\index{In-memory Database!ML serving} or aggressive caching. Edge deployment\index{Edge Deployment!storage} adds further constraints: limited device storage, intermittent connectivity, and the need for model updates without disrupting inference, typically addressed through tiered storage\index{Tiered Storage!edge deployment} where models cache locally while reference data pulls from the cloud. Model versioning\index{Model Versioning!deployment patterns} must support smooth transitions between versions, rapid rollback, and serving multiple versions simultaneously for A/B testing—operational patterns examined in @sec-ml-operations.
|
||
|
||
### Data Versioning for ML Reproducibility {#sec-data-engineering-data-versioning-ml-reproducibility-16d0}
|
||
|
||
\index{Data Versioning!reproducibility}\index{DVC (Data Version Control)!workflow}A recommendation model's click-through rate dropped 3% after a routine weekly retraining. The code had not changed. Was it a data change? A labeling shift? A corrupted upstream table? Without a record linking the model to the exact dataset that produced it, the team spent two weeks bisecting possibilities. With data versioning, they would have diffed the training snapshots in minutes and identified the root cause: an upstream provider had silently backfilled six months of historical records, shifting the label distribution.
|
||
|
||
Data versioning\index{Data Versioning!model reproducibility} exists to prevent exactly this scenario. It connects model versions to exact training data, enabling debugging and reproducibility. Without it, teams cannot answer essential questions like "what exact data trained model v47?"
|
||
|
||
@lst-dvc-workflow shows how DVC\index{DVC (Data Version Control)!git semantics} provides Git-like semantics for data versioning, while @lst-delta-time-travel demonstrates querying historical data states directly in SQL.\index{Delta Lake!time travel}
|
||
|
||
::: {#lst-dvc-workflow lst-cap="**DVC Workflow**: Git-like semantics for data versioning. DVC tracks large data files alongside code commits, enabling exact reproduction of any historical training dataset through paired `git checkout` and `dvc checkout` commands."}
|
||
```{.bash}
|
||
# Add data to version control
|
||
dvc add data/training.csv
|
||
git add data/training.csv.dvc
|
||
git commit -m "Add training data v1"
|
||
dvc push # Upload to remote storage
|
||
|
||
# Later: retrieve exact data for any historical commit
|
||
git checkout abc123
|
||
dvc checkout # Restores exact data from that commit
|
||
```
|
||
:::
|
||
|
||
::: {#lst-delta-time-travel lst-cap="**Delta Lake Time Travel**: Querying historical data states directly in SQL. Delta Lake maintains a transaction log enabling point-in-time queries by date or version number, eliminating the need for separate snapshot management."}
|
||
```{.sql}
|
||
-- Query data as it existed on a specific date
|
||
SELECT * FROM training_data VERSION AS OF '2024-01-15'
|
||
|
||
-- Or by version number for programmatic access
|
||
SELECT * FROM training_data VERSION AS OF 47
|
||
```
|
||
:::
|
||
|
||
\index{Feature Store!point-in-time retrieval}\index{Model Registry!provenance tracking}Two complementary capabilities complete the versioning infrastructure. Feature store point-in-time retrieval maintains historical feature values, enabling training with features "as they existed" at prediction time and preventing label leakage where training inadvertently uses feature values computed after the prediction timestamp. Model registry integration links each model registry entry to its complete provenance: Git commit hash (code), data version (DVC commit or Delta version), feature store snapshot timestamp, and training configuration file. This complete lineage enables rapid debugging when production issues arise.
|
||
|
||
A common failure mode illustrates the importance: Model v47 performed 3% worse than v46 with identical code. Without data versioning, the team spent two weeks investigating the accuracy drop. With proper versioning, they would have immediately seen that the data version was inadvertently updated mid-experiment and identified the root cause within hours.
|
||
|
||
Long-term maintenance introduces a final storage consideration: retaining enough data to debug issues and satisfy compliance requirements. A recommendation system serving 10 million users generates terabytes of interaction logs daily, necessitating tiered retention\index{Tiered Retention!hot/warm/cold}: hot storage retains the past week for rapid analysis, warm storage keeps the past quarter for periodic review, and cold archive storage\index{Cold Storage!archival} retains years of data for compliance and rare deep investigations. Regulated industries often require immutable storage demonstrating complete data provenance—which training data and model version produced each prediction—potentially for years or decades.
|
||
|
||
The storage architectures we have examined address where data resides and how it is retrieved, but a critical challenge remains: ensuring that features computed during training match exactly those computed during serving. This consistency requirement, which we emphasized throughout the processing section, demands specialized infrastructure that bridges the gap between batch training environments and real-time serving systems. Feature stores have emerged as the architectural solution to this challenge.
|
||
|
||
### Feature Stores {#sec-data-engineering-feature-stores-bridging-training-serving-55c8}
|
||
|
||
\index{Feature Store!training-serving bridge}\index{Point-in-Time Correctness}Feature stores have emerged as critical infrastructure components addressing the unique challenge of maintaining consistency between training and serving environments while enabling feature reuse across models and teams. Traditional ML architectures often compute features differently offline during training versus online during serving, creating training-serving skew that silently degrades model performance.
|
||
|
||
::: {.callout-definition title="Feature Store"}
|
||
|
||
***Feature Store***\index{Feature Store!definition} is the architectural layer that centralizes the management of **Machine Learning Features**, decoupling feature computation from consumption.
|
||
|
||
1. **Significance (Quantitative):** It enforces **Point-in-Time Correctness**, ensuring that historical data used for training ($x_{t-\Delta}$) is computed with identical logic to the real-time data served at inference ($x_t$), eliminating **Training-Serving Skew** by design.
|
||
2. **Distinction (Durable):** Unlike a **General-Purpose Database**, a Feature Store is designed for **Dual Storage Modes**: an *Offline Store* (columnar/batch) for training and an *Online Store* (key-value/low-latency) for serving.
|
||
3. **Common Pitfall:** A frequent misconception is that a Feature Store is just "a place to store data." In reality, it is a **Transformation Engine**: it stores the *logic* to compute features consistently across the entire ML lifecycle.
|
||
|
||
:::
|
||
|
||
The core problem feature stores address becomes clear when examining typical ML development workflows. During model development, data scientists write feature engineering logic in notebooks or scripts, often using different libraries and languages than production serving systems. Training might compute a user's "total purchases last 30 days" using SQL aggregating historical data, while serving computes the same feature using a microservice that incrementally updates cached values. These implementations should produce identical results, but subtle differences—handling timezone conversions, dealing with missing data, or rounding numerical values—cause training and serving features to diverge. A study of production ML systems found that 30% to 40% of initial deployments at Uber suffered from training-serving skew, motivating development of their Michelangelo platform with integrated feature stores.
|
||
|
||
Feature stores (discussed architecturally in @sec-ml-operations-feature-stores-c01c) provide a single source of truth for feature definitions, ensuring consistency across all stages of the ML lifecycle. When data scientists define a feature like "user_purchase_count_30d", the feature store maintains both the definition (SQL query, transformation logic, or computation graph) and executes it consistently whether providing historical feature values for training or real-time values for serving. This architectural pattern eliminates an entire class of subtle bugs that prove notoriously difficult to debug because models train successfully but perform poorly in production without obvious errors. The same centralized approach enables feature reuse across models and teams: when multiple teams build models requiring similar features, the feature store prevents each team from reimplementing identical computations with subtle variations. A recommendation system might compute user embedding vectors across hundreds of dimensions, aggregating months of interaction history. Rather than each model team recomputing these expensive embeddings, the feature store computes them once and serves them to all consumers.
|
||
|
||
The architectural pattern typically implements dual storage modes\index{Feature Store!offline vs online stores} optimized for different access patterns. The offline store uses columnar formats like Parquet on object storage, optimized for batch access during training where sequential scanning of millions of examples is common. The online store uses key-value systems like Redis, optimized for random access during serving where individual feature vectors must be retrieved in milliseconds. Synchronization between stores becomes critical. As training generates new models using current feature values, those models deploy to production expecting the online store to serve consistent features. Feature stores typically implement scheduled batch updates propagating new feature values from offline to online stores, with update frequencies depending on feature freshness requirements.
|
||
|
||
Time-travel capabilities\index{Time Travel!feature stores}\index{Point-in-Time Correctness!time travel} distinguish sophisticated feature stores from simple caching layers. Training requires accessing feature values as they existed at specific points in time, not just current values. Consider training a churn prediction model: for users who churned on January 15th, the model should use features computed on January 14th, not current features reflecting their churned status. Point-in-time correctness ensures training data matches production conditions where predictions use currently-available features to forecast future outcomes. Implementing time-travel requires storing feature history, not just current values, substantially increasing storage requirements but enabling correct training on historical data.
|
||
|
||
Feature store performance characteristics directly impact both training throughput and serving latency. The offline store must support high-throughput batch reads (millions of feature vectors per minute) using columnar formats that enable efficient reads of specific features from wide tables. The online store must support thousands to millions of reads per second with single-digit millisecond latency. In production, feature freshness adds further pressure: when users add items to shopping carts, recommendation systems need updated features within seconds, not hours. Streaming feature computation\index{Feature Store!streaming updates} pipelines address this by updating online stores continuously rather than through periodic batch jobs, though streaming introduces complexity around exactly-once processing semantics\index{Exactly-Once Semantics!streaming} and handling late-arriving events.
|
||
|
||
With the pipeline fully assembled—acquisition, ingestion, processing, labeling, and storage—a tempting conclusion is that data engineering work is "done." But production systems do not stand still. User behavior drifts, upstream schemas evolve, labeling guidelines change, and the careful engineering described in this chapter gradually erodes unless actively maintained. The final section addresses the ongoing health of these systems after deployment.
|
||
|
||
## Operational Data Health {#sec-data-engineering-data-debt-hidden-liability-3335}
|
||
|
||
\index{Data Debt!hidden liability}\index{Technical Debt!data systems}Imagine a KWS system that launched with 98% accuracy and excellent training-serving consistency. Six months later, accuracy has slipped to 94%, not because anyone changed the model, but because new smartphone microphone hardware shifted the audio distribution, labeling guidelines were updated without retraining, and a schema change in the user metadata pipeline went undocumented. Each is a form of *data debt*: accumulated compromises in data quality, documentation, and infrastructure that compound silently.\index{Technical Debt!data engineering} Unlike code debt, which manifests as slower development velocity, data debt directly degrades model performance and can remain invisible until failures become catastrophic. This section addresses the categories of data debt, how to measure them, and the *debugging techniques* for diagnosing pipeline failures when debt comes due.
|
||
|
||
### Categories of Data Debt {#sec-data-engineering-categories-data-debt-ee90}
|
||
|
||
The KWS scenario above illustrates each of the four categories of data debt. The microphone hardware shift is *freshness debt*; the updated labeling guidelines are *quality debt*; the undocumented schema change is both *schema debt* and *documentation debt*. Each category requires different detection and remediation strategies, but all share the property captured in the following definition.
|
||
|
||
::: {.callout-definition title="Data Debt"}
|
||
|
||
***Data Debt***\index{Data Debt!definition} is the **Compound Interest** of implicit coupling and missing documentation across the Data Stack.
|
||
|
||
1. **Significance (Quantitative):** It manifests as **Silent Degradation**, where the cost of maintenance scales superlinearly with system age due to unmanaged dependencies and distribution shifts ($D(P_t \| P_0)$).
|
||
2. **Distinction (Durable):** Unlike **Technical Debt** in code, which manifests as **Slower Development**, Data Debt manifests as **Lower Accuracy** even when the code is perfectly maintained.
|
||
3. **Common Pitfall:** A frequent misconception is that Data Debt is just "bad data." In reality, it is a **Systems Architecture Failure**: it occurs when the assumptions of the training distribution are no longer enforced at the system boundary.
|
||
|
||
:::
|
||
|
||
#### Documentation Debt {#sec-data-engineering-documentation-debt-57fa}
|
||
|
||
\index{Documentation Debt!data provenance}Documentation debt accumulates when data provenance, meaning, and quality characteristics go unrecorded. Datasets without data cards [@gebru2021datasheets], unlabeled columns, and undocumented transformations create debt that compounds when original authors leave organizations. A survey of production ML systems found that 40% of data quality incidents traced to misunderstanding data semantics due to missing documentation [@sambasivan2021everyone]. The symptoms are unmistakable: columns whose meanings must be reverse-engineered from usage patterns, missing provenance records that block compliance audits, undocumented assumptions that cause silent failures when those assumptions change, and absent quality metrics that prevent informed dataset selection.
|
||
|
||
#### Schema Debt {#sec-data-engineering-schema-debt-e420}
|
||
|
||
\index{Schema Debt!workarounds}Schema debt emerges from accumulated schema workarounds and migrations. When upstream systems change data formats, quick fixes—string parsing instead of proper type handling, NULL coercion instead of error handling—accumulate into fragile transformation logic. Teams can diagnose schema debt by looking for telltale patterns: multiple date format handlers for the same logical field, defensive null checks scattered throughout pipeline code, version-specific parsing branches that grow with each upstream change, and undocumented enum value mappings that break when new values appear.
|
||
|
||
#### Quality Debt {#sec-data-engineering-quality-debt-ce63}
|
||
|
||
\index{Quality Debt!uncorrected errors}Quality debt consists of known data errors that remain uncorrected due to time or resource constraints. A dataset with 3% known label errors represents quality debt—each training run on this data produces models carrying those errors. Quality debt compounds through a particularly dangerous feedback loop: models trained on erroneous data make predictions that become training data for downstream systems, amplifying the original errors across the ecosystem. Concrete indicators include known label error rates not yet corrected, documented biases not yet mitigated, identified duplicates not yet deduplicated, and detected drift not yet addressed through retraining.
|
||
|
||
#### Freshness Debt {#sec-data-engineering-freshness-debt-0e54}
|
||
|
||
\index{Freshness Debt!distribution divergence}Freshness debt arises when training data diverges from production distributions over time. A model trained on 2023 user behavior deployed in 2025 carries freshness debt—the distribution shift between training and serving degrades performance continuously. Freshness debt is particularly dangerous because it accumulates silently. Unlike code that breaks obviously, stale models degrade gradually until performance drops below acceptable thresholds. Warning signs include time since last retraining exceeding the distribution shift rate, feature staleness from cached features not updated at the required frequency, and reference data lag where lookup tables no longer reflect current state.
|
||
|
||
### Measuring and Projecting Data Debt {#sec-data-engineering-measuring-data-debt-973c}
|
||
|
||
Unlike technical debt, which can be assessed through code complexity metrics, data debt requires specialized measurement approaches.
|
||
|
||
@tbl-data-debt-metrics provides quantitative indicators for each debt category:
|
||
|
||
| **Debt Category** | **Metric** | **Warning Threshold** | **Critical Threshold** |
|
||
|:------------------|:---------------------------------|----------------------:|---------------------------:|
|
||
| **Documentation** | % datasets with data cards | < 80% | < 50% |
|
||
| **Documentation** | % columns with descriptions | < 90% | < 70% |
|
||
| **Schema** | Schema version branches | > 3 | > 10 |
|
||
| **Schema** | % transformations with try/catch | > 20% | > 50% |
|
||
| **Quality** | Known label error rate | > 1% | > 5% |
|
||
| **Quality** | Documented bias metrics | Not measured | Measured but not mitigated |
|
||
| **Freshness** | Days since last retraining | > 90 days | > 365 days |
|
||
| **Freshness** | PSI vs training baseline | > 0.1 | > 0.25 |
|
||
|
||
: **Data Debt Metrics**: Quantitative thresholds for detecting data debt accumulation across four categories. Warning thresholds indicate debt requiring attention in upcoming planning cycles; critical thresholds indicate debt requiring immediate remediation to prevent system degradation. These thresholds are calibrated from industry experience—a PSI above 0.25 against the training baseline, for instance, typically correlates with measurable accuracy loss within one to two retraining cycles. {#tbl-data-debt-metrics}
|
||
|
||
Data debt compounds through feedback loops unique to ML systems:
|
||
|
||
1. **Training Amplification**: Models trained on erroneous data learn incorrect patterns. When these models generate predictions used as features or pseudo-labels, errors propagate to downstream systems.
|
||
|
||
2. **Documentation Decay**: As original authors leave and systems evolve, undocumented datasets become increasingly opaque. Each year without documentation makes future documentation exponentially harder.
|
||
|
||
3. **Schema Entropy**: Quick fixes to handle schema changes create brittle code. Each additional workaround increases the probability of the next schema change causing failures.
|
||
|
||
4. **Distribution Drift**: Without continuous monitoring and retraining, the gap between training and serving distributions widens, causing accuracy degradation that accelerates as models become less calibrated.
|
||
|
||
The compound nature means that data debt left unaddressed for *n* periods grows superlinearly. Let $\text{Debt}_0$ be the initial debt level, $r$ the accumulation rate per period, and $n$ the number of periods. The growth follows @eq-debt-compound:
|
||
|
||
$$\text{Debt}_n \approx \text{Debt}_0 \times (1 + r)^n$$ {#eq-debt-compound}
|
||
|
||
where *r* is the debt accumulation rate (typically 10–30% per period for undocumented systems).
|
||
|
||
### Remediation Strategies {#sec-data-engineering-remediation-strategies-e457}
|
||
|
||
Addressing data debt requires systematic investment, not heroic one-time efforts. Each debt category calls for a distinct remediation approach, but all share a common pattern: regular, budgeted effort rather than crisis-driven scrambles.
|
||
|
||
Documentation debt responds best to dedicated sprints—one week per quarter reserved exclusively for documenting data provenance, column semantics, and transformation logic. Prioritizing by dataset usage (document the 20% of datasets serving 80% of models first) ensures maximum impact from limited documentation effort.
|
||
|
||
Schema debt\index{Schema Contracts!data producers} requires preventive infrastructure rather than reactive fixes. Explicit schema contracts between data producers and consumers, codified through tools like Great Expectations or Pandera, fail fast when schemas drift rather than allowing workarounds to accumulate. The investment in contract enforcement pays dividends by preventing the brittle parsing logic that characterizes mature systems with unmanaged schema evolution.
|
||
|
||
Quality debt\index{Quality Budget!debt remediation} demands allocated capacity—typically 10% of data engineering effort dedicated to remediating known label errors, documented biases, and identified duplicates. The known error backlog should be tracked and its burn-down rate measured like any engineering metric, making quality remediation visible alongside feature development rather than perpetually deferred.
|
||
|
||
Freshness debt\index{Continuous Retraining!drift-triggered} cannot be addressed through periodic heroics; it requires automated retraining triggers based on drift detection rather than calendar schedules, connecting directly to the PSI and KL divergence monitoring infrastructure established in @sec-data-engineering-detecting-responding-data-drift-509a.
|
||
|
||
The key insight is that data debt,\index{Data Debt!strategic vs unconscious} like technical debt, is not inherently bad. Strategic debt (knowingly accepting documentation shortcuts to meet a deadline) can be rational. The danger lies in *unconscious* debt that accumulates untracked until remediation costs exceed system value. Managing this trade-off requires treating data debt not as a moral failing, but as a systems parameter to be optimized alongside latency and throughput.
|
||
|
||
### Debugging Data Pipelines {#sec-data-engineering-debugging-data-pipelines-2f26}
|
||
|
||
\index{Debugging!data pipelines}\index{Pipeline!failure diagnosis}When data debt comes due, it surfaces as model accuracy degradation, pipeline failures, or subgroup performance disparities. Effective debugging applies the diagnostic principles established throughout this chapter: data cascades (@sec-data-engineering-data-cascades-systematic-foundations-matter-2efe) remind us that root causes lie upstream of symptoms, training-serving skew (@sec-data-engineering-ensuring-trainingserving-consistency-c683) explains many deployment failures, and drift detection (@sec-data-engineering-detecting-responding-data-drift-509a) surfaces gradual degradation. @fig-debug-flowchart synthesizes these concepts into an actionable diagnostic sequence:
|
||
|
||
::: {#fig-debug-flowchart fig-env="figure" fig-pos="htb" fig-cap="**Data Pipeline Debugging Flowchart**: Four sequential decision nodes guide root cause diagnosis: (1) accuracy degrades over time leads to Data Drift, (2) training accuracy exceeds validation leads to Overfitting, (3) validation exceeds production accuracy leads to Training-Serving Skew, and (4) subgroup inconsistency leads to Bias. If all answers are no, the issue points to Model Architecture." fig-alt="Vertical flowchart with four blue diamond decision nodes and red result boxes. Top diamond asks if accuracy degrades over time, leading to Data Drift result. Second asks if training accuracy exceeds validation, leading to Overfitting. Third asks if validation exceeds production accuracy, leading to Training-Serving Skew. Fourth asks about subgroup inconsistency, leading to Bias. Gray box at bottom shows Model Architecture issue if all answers are no."}
|
||
```{.tikz}
|
||
\begin{tikzpicture}[font=\small\usefont{T1}{phv}{m}{n},line width=0.75pt]
|
||
\tikzset{
|
||
Diamond/.style={aspect=2, inner sep=2pt,
|
||
draw=BlueLine, line width=0.65pt,
|
||
fill=BlueL, minimum height=11mm,
|
||
text width=40mm,align=flush center
|
||
},
|
||
Box/.style={inner xsep=2pt,
|
||
draw=GreenLine, line width=0.65pt,
|
||
fill=GreenL,
|
||
% text width=40mm,
|
||
align=flush center,
|
||
minimum width=40mm, minimum height=11mm,
|
||
},
|
||
Result/.style={inner xsep=2pt,
|
||
draw=RedLine, line width=0.65pt,
|
||
fill=RedL!20,
|
||
% text width=45mm,
|
||
align=flush center,
|
||
minimum width=55mm, minimum height=15mm,
|
||
},
|
||
Line/.style={line width=1.0pt,black!50,text=black,->,>=Latex},
|
||
Text/.style={inner sep=1pt,
|
||
font=\footnotesize\usefont{T1}{phv}{m}{n},
|
||
text=black
|
||
}
|
||
}
|
||
|
||
%target
|
||
\tikzset{
|
||
pics/target/.style = {
|
||
code = {
|
||
\pgfkeys{/channel/.cd, #1}
|
||
\begin{scope}[shift={($(0,0)+(0,0)$)},scale=\scalefac,every node/.append style={transform shape}]
|
||
\definecolor{col1}{RGB}{62,100,125}
|
||
\definecolor{col2}{RGB}{219,253,166}
|
||
\colorlet{col1}{\filllcolor}
|
||
\colorlet{col2}{\filllcirclecolor}
|
||
\foreach\i/\col [count=\k]in {22mm/col1,17mm/col2,12mm/col1,7mm/col2,2.5mm/col1}{
|
||
\node[circle,inner sep=0pt,draw=\drawcolor,fill=\col,minimum size=\i,line width=\Linewidth](C\k){};
|
||
}
|
||
\draw[thick,fill=brown,xscale=-1](0,0)--++(111:0.13)--++(135:1)--++(225:0.1)--++(315:1)--cycle;
|
||
\path[green,xscale=-1](0,0)--(135:0.85)coordinate(XS1);
|
||
\draw[thick,fill=yellow,xscale=-1](XS1)--++(80:0.2)--++(135:0.37)--++(260:0.2)--++(190:0.2)--++(315:0.37)--cycle;
|
||
\end{scope}
|
||
}
|
||
}
|
||
}
|
||
%graph
|
||
\tikzset{pics/graph/.style = {
|
||
code = {
|
||
\pgfkeys{/channel/.cd, #1}
|
||
\begin{scope}[local bounding box=GRAPH,scale=\scalefac, every node/.append style={transform shape}]
|
||
\draw[line width=1.5*\Linewidth,draw = \drawcolor](-0.20,0)--(2,0);
|
||
\draw[line width=1.5*\Linewidth,draw = \drawcolor](-0.20,0)--(-0.20,2);
|
||
\foreach \i/\vi in {0/10,0.5/17,1/9,1.5/5}{
|
||
\node[draw, minimum width =4mm, minimum height = \vi mm, inner sep = 0pt,
|
||
draw = \filllcolor, fill=\filllcolor!20, line width=\Linewidth,anchor=south west](COM)at(\i,0.2){};
|
||
}
|
||
\end{scope}
|
||
}
|
||
}
|
||
}
|
||
%square
|
||
\tikzset{
|
||
pics/square/.style = {
|
||
code = {
|
||
\pgfkeys{/channel/.cd, #1}
|
||
\begin{scope}[local bounding box=SQUARE,scale=\scalefac,every node/.append style={transform shape}]
|
||
% Right Face
|
||
\draw[fill=\filllcolor!70,line width=\Linewidth]
|
||
(\Depth,0,0)coordinate(\picname-ZDD)--(\Depth,\Width,0)--(\Depth,\Width,\Height)--(\Depth,0,\Height)--cycle;
|
||
% Front Face
|
||
\draw[fill=\filllcolor!40,line width=\Linewidth]
|
||
(0,0,\Height)coordinate(\picname-DL)--(0,\Width,\Height)coordinate(\picname-GL)--
|
||
(\Depth,\Width,\Height)coordinate(\picname-GD)--(\Depth,0,\Height)coordinate(\picname-DD)--(0,0,\Height);
|
||
% Top Face
|
||
\draw[fill=\filllcolor!20,line width=\Linewidth]
|
||
(0,\Width,0)coordinate(\picname-ZGL)--(0,\Width,\Height)coordinate(\picname-ZGL)--
|
||
(\Depth,\Width,\Height)--(\Depth,\Width,0)coordinate(\picname-ZGD)--cycle;
|
||
\end{scope}
|
||
}
|
||
}
|
||
}
|
||
% #1 number of teeths
|
||
% #2 radius intern
|
||
% #3 radius extern
|
||
% #4 angle from start to end of the first arc
|
||
% #5 angle to decale the second arc from the first
|
||
% #6 inner radius to cut off
|
||
\tikzset{
|
||
pics/gear/.style args={#1/#2/#3/#4/#5/#6/#7}{
|
||
code={
|
||
\pgfkeys{/channel/.cd, #7}
|
||
\begin{scope}[shift={($(0,0)+(0,0)$)},scale=\scalefac,every node/.append style={transform shape}]
|
||
\pgfmathtruncatemacro{\N}{#1}%
|
||
\def\rin{#2}\def\rout{#3}\def\aA{#4}\def\aOff{#5}\def\rcut{#6}%
|
||
\path[rounded corners=1.5pt,draw=\drawcolor,fill=\filllcolor]
|
||
(0:\rin)
|
||
\foreach \i [evaluate=\i as \n using (\i-1)*360/\N] in {1,...,\N}{%
|
||
arc (\n:\n+\aA:\rin)
|
||
-- (\n+\aA+\aOff:\rout)
|
||
arc (\n+\aA+\aOff:\n+360/\N-\aOff:\rout)
|
||
-- (\n+360/\N:\rin)
|
||
} -- cycle;
|
||
\draw[draw=none,fill=white](0,0) circle[radius=\rcut];
|
||
\end{scope}
|
||
}}
|
||
}
|
||
%puzzle
|
||
\tikzset{pics/puzzle/.style = {
|
||
code = {
|
||
\pgfkeys{/channel/.cd, #1}
|
||
\begin{scope}[scale=\scalefac, every node/.append style={transform shape}]
|
||
\fill[fill=\filllcolor] (-2,-0.35) to[out=90,in=135] (-1.5,-0.45) arc(-135:135:0.6 and
|
||
{0.45*sqrt(2)}) to[out=-135,in=-90] (-2,0.35) |- (-0.35,2)
|
||
to[out=0,in=-45] (-0.45,2.5) arc(225:-45:{0.45*sqrt(2)} and 0.6)
|
||
to[out=-135,in=180] (0.35,2) -| (2,0.35)
|
||
to[out=-90,in=225] (2.5,0.45) arc(135:-135:0.6 and {0.45*sqrt(2)})
|
||
to[out=135,in=90] (2,-0.35) |- (0.35,-2)
|
||
to[out=180,in=-135] (0.45,-1.5) arc(-45:225:{0.45*sqrt(2)} and 0.6)
|
||
to[out=-45,in=0] (-0.35,-2) -| cycle;
|
||
\end{scope}
|
||
}}}
|
||
\pgfkeys{
|
||
/channel/.cd,
|
||
Dual/.store in=\Dual,
|
||
Depth/.store in=\Depth,
|
||
Height/.store in=\Height,
|
||
Width/.store in=\Width,
|
||
Smile/.store in=\Smile,
|
||
Level/.store in=\Level,
|
||
filllcirclecolor/.store in=\filllcirclecolor,
|
||
filllcolor/.store in=\filllcolor,
|
||
drawcolor/.store in=\drawcolor,
|
||
drawcircle/.store in=\drawcircle,
|
||
scalefac/.store in=\scalefac,
|
||
Linewidth/.store in=\Linewidth,
|
||
picname/.store in=\picname,
|
||
filllcolor=BrownLine,
|
||
filllcirclecolor=cyan,
|
||
drawcolor=black,
|
||
drawcircle=violet,
|
||
scalefac=1,
|
||
Dual=adual,
|
||
Smile=smile,
|
||
Level=0.52,
|
||
Linewidth=0.5pt,
|
||
Depth=1.3,
|
||
Height=0.8,
|
||
Width=1.1,
|
||
picname=C
|
||
}
|
||
|
||
% Nodes
|
||
\node[Diamond](D1){Accuracy degrading over time?};
|
||
% Branch 1: Yes (Drift)
|
||
\node[Result, right=1.5 of D1](R1){};
|
||
\pic[shift={(0.37,-0.55)}] at (R1.west){graph={scalefac=0.55,picname=1,filllcolor=OrangeLine, Linewidth=0.7pt}};
|
||
\node[Result,draw=none,anchor=south east,minimum width=42mm, fill=none](R3T)
|
||
at (R1.south east){\textbf{Data Drift}\\[-2pt]\footnotesize (Freshness Debt)\\[-2pt] \footnotesize Check PSI, Distributions};
|
||
\draw[Line] (D1.east) -- node[Text,above=1pt]{Yes} (R1.west);
|
||
|
||
% Branch 1: No -> D2
|
||
\node[Diamond, below=1.15 of D1](D2){Training Acc $\gg$ Validation Acc?};
|
||
\draw[Line] (D1.south) -- node[Text,right=1pt]{No/Constant} (D2.north);
|
||
|
||
% Branch 2: Yes (Overfitting/Quality)
|
||
\node[Result, right=1.5 of D2](R2){};
|
||
|
||
\begin{scope}[local bounding box=PUZZLE1,shift={($(0.42,-0.28)+(R2.west)$)},
|
||
scale=0.58, every node/.append style={transform shape}]
|
||
\pic[shift={(0,0)}] at (0,0){puzzle={scalefac=0.2,picname=1,filllcolor=orange!80}};
|
||
\pic[shift={(0,0)}] at (0,0.8){puzzle={scalefac=0.2,picname=1,filllcolor=red!80}};
|
||
\pic[shift={(0,0)}] at (0.8,0){puzzle={scalefac=0.2,picname=1,filllcolor=green!60!black}};
|
||
\pic[shift={(0,0)}] at (0.8,0.8){puzzle={scalefac=0.2,picname=1,filllcolor=cyan!70}};
|
||
\end{scope}
|
||
\node[Result,draw=none,anchor=south east,minimum width=42mm, fill=none](R3T)
|
||
at (R2.south east){\textbf{Overfitting/Artifacts}\\[-1pt] \footnotesize Check Label Noise,
|
||
\\[-1pt]
|
||
Duplicates};
|
||
\draw[Line] (D2.east) -- node[Text,above=1pt]{Yes} (R2.west);
|
||
|
||
% Branch 2: No -> D3
|
||
\node[Diamond, below=1.15 of D2](D3){Validation Acc $\gg$ Production Acc?};
|
||
\draw[Line] (D2.south) -- node[Text,right=1pt]{No ($\approx$)} (D3.north);
|
||
|
||
% Branch 3: Yes (Skew)
|
||
\node[Result, right=1.5 of D3](R3){};
|
||
\draw[Line] (D3.east) -- node[Text,above=1pt]{Yes} (R3.west);
|
||
\pic[shift={(0.65,0)}] at (R3.west) {gear={9/1.6/2.1/5/2/1.0/scalefac=0.25,drawcolor=GreenD,filllcolor=GreenD}};
|
||
\node[Result,draw=none,anchor=south east,minimum width=45mm, fill=none](R3T)
|
||
at (R3.south east){\textbf{Training-Serving Skew}\\[-1pt] \footnotesize Check Transformations,\\[-1pt] Schema};
|
||
|
||
% Branch 3: No -> D4
|
||
\node[Diamond, below=1.15 of D3](D4){Performance inconsistent across subgroups?};
|
||
\draw[Line] (D3.south) -- node[Text,right=1pt]{No ($\approx$)} (D4.north);
|
||
|
||
% Branch 4: Yes (Bias)
|
||
\node[Result, right=1.5 of D4](R4){};
|
||
\pic[shift={(0.65,0)}] at (R4.west){target={scalefac=0.45,picname=1,drawcolor=BlueLine,
|
||
filllcolor=cyan!90!,Linewidth=0.7pt, filllcirclecolor=cyan!20}};
|
||
\node[Result,draw=none,anchor=south east,minimum width=45mm, fill=none](R5T)
|
||
at (R4.south east){\textbf{Bias / Coverage Gap}\\ \footnotesize Check Slice Metrics};
|
||
\draw[Line] (D4.east) -- node[Text,above=1pt]{Yes} (R4.west);
|
||
|
||
% Branch 4: No -> Model Issue
|
||
\node[Box, below=1.15 of D4,fill=gray!20, draw=black!60](R5){};
|
||
\pic[rotate=20,shift={(0.27,-0.35)}] at (R5.west){square={scalefac=0.4,picname=1,
|
||
filllcolor=VioletLine, Linewidth=0.5pt}};
|
||
\node[Box,draw=none,anchor=south east,minimum width=32mm, fill=none](R5T)
|
||
at (R5.south east){\textbf{Model Architecture}\\or Capacity Issue};
|
||
\draw[Line] (D4.south) -- node[Text,right=1pt]{No} (R5.north);
|
||
|
||
% Context Labels
|
||
\node[text width=45mm, align=center, font=\footnotesize\usefont{T1}{phv}{m}{n}, anchor=north] at (R1.south) {Fix: Retraining, Monitoring};
|
||
\node[text width=45mm, align=center, font=\footnotesize\usefont{T1}{phv}{m}{n}, anchor=north] at (R2.south) {Fix: Deduplication, Audit};
|
||
\node[text width=45mm, align=center, font=\footnotesize\usefont{T1}{phv}{m}{n}, anchor=north] at (R3.south) {Fix: Feature Store, Consistency};
|
||
\node[text width=45mm, align=center, font=\footnotesize\usefont{T1}{phv}{m}{n}, anchor=north] at (R4.south) {Fix: Targeted Collection};
|
||
\end{tikzpicture}
|
||
```
|
||
|
||
:::
|
||
|
||
Most production ML failures trace to data, not models.[^fn-failure-data-dominance] Industry experience suggests a consistent pattern: training-serving skew accounts for 30–40% of failures, data drift for another 20–30%, and label quality issues for 15–25%. Model architecture—the component engineers most naturally investigate first—accounts for only 10–15%. Debugging the model before verifying data consistency wastes engineering cycles on the wrong tenth of the problem space.
|
||
|
||
[^fn-failure-data-dominance]: **ML Failure Distribution**: The 65--95% data-dominance ratio is synthesized from industry reports at Uber, Google, and Meta and holds across domains despite variation in absolute proportions. The invariant has a structural explanation: model architecture is tested before deployment, but data distributions shift continuously after deployment. This asymmetry means model bugs are caught by validation; data bugs are caught by users -- making data infrastructure the primary reliability investment, not model sophistication. \index{ML Failures!data dominance}
|
||
|
||
The patterns in this section—data debt accumulation, diagnostic debugging, and systematic remediation—close the loop on the data engineering lifecycle. From acquisition through storage, every pipeline stage we have examined introduces opportunities for both excellence and failure. The following fallacies and pitfalls distill the most consequential misconceptions that lead teams astray.
|
||
|
||
## Fallacies and Pitfalls {#sec-data-engineering-fallacies-pitfalls-886b}
|
||
|
||
**Fallacy:** *More data always improves model performance.*
|
||
|
||
Beyond a threshold, additional data yields diminishing returns while costs scale linearly. Empirical studies across image classification, translation, and language modeling confirm that test loss follows a power law in dataset size, with exponents so small that a 10$\times$ increase in data often reduces error by less than one percentage point, while labeling and storage costs scale proportionally [@hestness2017deep]. The Information Entropy concept from @sec-data-engineering-physics-data-cdcb explains why: if new examples are redundant (low entropy), they add mass without information. Smart data selection, including active learning, deduplication, and curriculum design, often outperforms naive data accumulation.
|
||
|
||
**Pitfall:** *Treating data preprocessing as a one-time task.*
|
||
|
||
Data distributions drift continuously as user behavior, market conditions, and upstream systems evolve. A preprocessing pipeline validated at launch degrades silently as the world changes around it. Production systems require continuous monitoring (PSI, KL divergence) and automated retraining triggers, not periodic manual audits.
|
||
|
||
**Fallacy:** *High training accuracy indicates production readiness.*
|
||
|
||
Training accuracy measures fit to historical data; production performance measures generalization to future data under real-world conditions. Training-serving skew, distribution drift, and coverage gaps cause models with 99% validation accuracy to fail catastrophically in deployment. The debugging flowchart in @fig-debug-flowchart exists precisely because this fallacy wastes engineering cycles.
|
||
|
||
**Pitfall:** *Ignoring training-serving skew until deployment.*
|
||
|
||
Feature computation differences between training and serving environments are the leading cause of ML deployment failures. By the time skew manifests in production metrics, debugging becomes archaeological work. Feature stores and consistency contracts should be architectural requirements from project inception, not retrofits after deployment incidents.
|
||
|
||
**Fallacy:** *Synthetic data can fully replace real-world data collection.*
|
||
|
||
Synthetic data excels at augmenting real data—generating rare edge cases, increasing diversity, reducing costs—but cannot replace it entirely. Synthetic generation inherits the biases and limitations of its generative models. A KWS system trained purely on synthesized speech will fail on accent patterns, background noises, and pronunciation variations that the generator never modeled. The optimal strategy combines real data for coverage with synthetic data for scale.
|
||
|
||
**Pitfall:** *Neglecting data versioning until model debugging requires it.*
|
||
|
||
Teams treat data versioning as optional infrastructure to add "when needed," then discover the need only after a deployed model produces unexpected results. Without versioning, reproducing a training run requires re-executing the entire data pipeline from scratch, a process that can consume days of compute and weeks of engineering time. Consider a model that performed well three months ago but degrades after retraining on updated data. Without versioned snapshots, the team cannot determine whether the regression stems from a labeling policy change, a schema migration error, or genuine distribution shift. The data lineage principles established in @sec-data-engineering-tracking-data-transformation-lineage-3b09 formalize this requirement: every training artifact must trace back to a specific, immutable dataset version. Organizations that defer versioning report 2--4$\times$ longer debugging cycles when production issues arise, because each investigation begins with the question "what data did this model actually train on?" and no system can answer it.
|
||
|
||
## Summary {#sec-data-engineering-summary-4ac6}
|
||
|
||
Data engineering provides the foundational infrastructure that transforms raw information into the basis of machine learning systems, determining model performance, system reliability, ethical compliance, and long-term maintainability. The Four Pillars framework—Quality, Reliability, Scalability, and Governance (@fig-four-pillars)—organizes design choices across acquisition, ingestion, validation, and storage, while the cascading nature of data quality failures (@fig-cascades) reveals why every pipeline stage requires careful engineering decisions. The task of "getting data ready" encompasses complex trade-offs quantified throughout this chapter: data engineering cost constants for budgeting, storage performance hierarchies (@tbl-storage-performance, @tbl-ml-latencies), and drift detection thresholds (PSI and KL divergence) that operationalize the Degradation Equation (@eq-degradation) into production monitoring infrastructure.
|
||
|
||
The technical architecture of data systems demonstrates how engineering decisions compound across the pipeline, creating either reliable, scalable foundations or brittle, maintenance-heavy technical debt. Data acquisition strategies must navigate the reality that perfect datasets rarely exist in nature, requiring approaches from crowdsourcing and synthetic generation to careful curation and active learning. Storage architectures from traditional databases to modern data lakes and feature stores represent consequential choices about how data flows through the system, affecting everything from training speed to serving latency.
|
||
|
||
::: {.callout-takeaways title="Data Is the Source Code"}
|
||
|
||
* **Data cascades make upstream quality the highest-leverage investment.** Errors introduced at collection amplify through every pipeline stage (@fig-cascades). The Four Pillars framework—Quality, Reliability, Scalability, Governance—provides the diagnostic structure for preventing these failures before they compound.
|
||
|
||
* **Data is code: version it, test it, review it.** The Data as Code Invariant established in Part I defines the engineering mindset: the dataset is the source code of an ML system. Apply the same rigor: version control, unit tests (validation), and code review (data review).
|
||
|
||
* **Training-serving consistency is non-negotiable.** Any transformation applied during training must be applied identically during serving. This is a mathematical requirement, not a best practice.
|
||
|
||
* **Pipeline architecture choices have large cost implications.** Real-time streaming increases operational cost and complexity relative to batch, while ETL reduces storage footprint at the expense of higher engineering overhead during schema changes. Select ingestion patterns based on the value of latency, not the appeal of real-time.
|
||
|
||
* **Labeling costs dominate and require substantial resource allocation.** Labeling typically costs 1,000--3,000$\times$ more than model training compute. Labeling is the serial bottleneck that parallelization cannot solve.
|
||
|
||
* **Storage hierarchy determines iteration speed.** The 70$\times$ throughput gap between local NVMe (7 GB/s) and cloud object storage (100 MB/s) determines whether iterations occur daily or weekly.
|
||
|
||
* **Data debt compounds and requires continuous remediation.** Documentation, schema, quality, and freshness debt accumulate with compound interest. Allocate sustained engineering capacity to prevent remediation from overwhelming new feature work.
|
||
|
||
* **The Degradation Equation becomes actionable through drift detection.** The divergence term $D(P_t \| P_0)$ from the Introduction is exactly what PSI and KL divergence measure. Data engineering operationalizes this theoretical equation into monitoring infrastructure that catches silent model degradation before users are affected.
|
||
|
||
:::
|
||
|
||
Our KWS case study demonstrates these principles in action: multi-source acquisition combining curated datasets with crowdsourcing and synthetic generation; pipeline architecture with consistency validation; tiered storage handling `{python} kws_dataset_size_m_round_str` million audio samples across `{python} kws_dataset_size_gb_str` GB of raw data; and lineage tracking essential for always-listening devices in users' homes. Data engineering is not a preprocessing step to be completed before "real" ML work begins—it is the foundation upon which model performance, user trust, and regulatory compliance rest. While this chapter focuses on *building* reliable data infrastructure, @sec-data-selection examines *optimizing* data usage through active learning, data pruning, and intelligent sampling strategies.
|
||
|
||
::: {.callout-chapter-connection title="From Source Code to Executable"}
|
||
|
||
The dataset compiler has produced its output: a clean, versioned, optimized training set ready for consumption. We have lexed raw streams into records, parsed and validated schemas, applied optimization passes that preserve signal while removing noise, and linked labeled annotations to produce a complete compilation unit. But a compiled binary does nothing until it runs on hardware. We turn next to the mathematical foundations of learning in @sec-neural-computation, which transforms neural networks from opaque components into engineerable systems whose behavior we can predict, debug, and optimize.
|
||
|
||
:::
|
||
|
||
::: { .quiz-end }
|
||
:::
|
||
|
||
```{=latex}
|
||
\part{key:vol1_build}
|
||
```
|