github-starred/cs249r_book
2
0
Fork 0
mirror of https://github.com/harvard-edge/cs249r_book.git synced 2026-07-21 00:23:30 -05:00
Code Issues 276 Packages Projects Releases 26 Wiki Activity

Merge PR #1826: Refine PDF layout for chapter 14 (ml_ops.qmd)

Browse Source 
Resolved conflict in ml_ops.qmd: kept PR #1826's relocation of the
'Optimal retraining interval' subsection to after the fraud-detection
example, and carried local dev's margin figure (vol1_ml_ops_margin_002.svg)
into the relocated subsection.
This commit is contained in:
Vijay Janapa Reddi
2026-06-01 08:18:34 -04:00
parent ca14b4ca5d bae934f76a
commit df7f7b855f
3 changed files with 78 additions and 55 deletions
Download Patch File Download Diff File Expand all files Collapse all files
book/quarto/config/_quarto-pdf-vol1.yml
+1
View File
@@ -193,6 +193,7 @@ format:
lot: false
lol: false
toc-depth: 2
toc-title: "Table of Contents"
number-depth: 3
keep-tex: true # Preserve .tex intermediate for inspection and debugging
reference-location: document
book/quarto/contents/vol1/ml_ops/ml_ops.qmd
+71 -49
View File
@@ -673,6 +673,11 @@ Recognizing debt patterns, however, is only half the battle. The organizations i
*Every technical debt pattern diagnosed earlier points to a missing infrastructure component.* The mapping in @tbl-mlops-infrastructure-debt is direct: each component implements a foundational principle (@sec-ml-operations-foundational-principles-44e6) and addresses a specific failure mode:
```{=latex}
\begingroup
\renewcommand{\arraystretch}{1.25}
```
| **Infrastructure Component** | **Principle Implemented** | **Debt Pattern Addressed** |
|:-----------------------------|:-----------------------------------|:--------------------------------------------|
| **Feature stores** | Consistency Imperative | Data dependency debt, training-serving skew |
@@ -682,6 +687,10 @@ Recognizing debt patterns, however, is only half the battle. The organizations i
: **MLOps Infrastructure as Debt Remediation**: Each infrastructure component is the engineering response to a specific class of technical debt observed in production ML systems. Feature stores enforce the consistency imperative that eliminates training-serving skew; versioning systems preserve reproducibility against correction cascades; CI/CD pipelines automate the rollout discipline that prevents boundary erosion; monitoring systems surface silent degradation before users do. {#tbl-mlops-infrastructure-debt}
```{=latex}
\endgroup
```
Examine the layered architecture in @fig-ops-layers, which organizes these components across ML models, frameworks, orchestration, infrastructure, and hardware. Understanding how these layers interact enables practitioners to design systems that systematically address the technical debt patterns identified earlier while maintaining operational sustainability.
::: {#fig-ops-layers fig-env="figure" fig-pos="htb" fig-cap="**MLOps Stack Layers**: Five tiers organize the ML system stack: ML Models at the top, followed by Frameworks, Orchestration, Infrastructure, and Hardware. MLOps spans orchestration tasks (data management through model serving) and infrastructure tasks (job scheduling through monitoring), enabling automation, reproducibility, and scalable deployment." fig-alt="Layered architecture diagram. Top row: ML models, frameworks, orchestration, infrastructure, hardware . MLOps section spans orchestration tasks (data management through model serving) and infrastructure tasks (job scheduling through monitoring)."}
@@ -703,15 +712,15 @@ Examine the layered architecture in @fig-ops-layers, which organizes these compo
Box2/.style={Box,text width=40mm,minimum width=40mm,fill=OrangeFill,draw=OrangeLine
},
Box3/.style={Box, fill=GreenFill,draw=GreenLine},
Box31/.style={Box3, node distance=0.5, minimum height=8mm},
Box31/.style={Box3, node distance=0.5, minimum height=9mm},
Box4/.style={Box, fill=RedFill,draw=RedLine,text width=34mm, minimum width=34mm},
Box41/.style={Box4, node distance=0.5, minimum height=8mm},
Box41/.style={Box4, node distance=0.5, minimum height=9mm},
}
%
\node[Box,text width=37mm, minimum width=37mm](B1){\textbf{ML Models/Applications} (e.g., BERT)};
\node[Box2,right=of B1](B2){\textbf{ML Frameworks/Platforms} (e.g., PyTorch)};
\node[Box2,right=0.35 of B1](B2){\textbf{ML Frameworks/Platforms} (e.g., PyTorch)};
\node[Box3,right=of B2](B3){\textbf{Model Orchestration} (e.g., Ray)};
\node[Box4,right=of B3](B4){\textbf{Infrastructure}\\ (e.g., Kubernetes)};
\node[Box4,right=0.4of B3](B4){\textbf{Infrastructure}\\ (e.g., Kubernetes)};
\node[Box,right=of B4,fill=VioletFill,draw=VioletLine](B5){\textbf{Hardware}\\ (e.g., a GPU cluster)};
%
\node[Box31,below=of B3](B31){Data Management};
@@ -1164,7 +1173,6 @@ Testing difficulties compound this challenge. Traditional unit testing framework
Several mitigation strategies address these operational concerns. Papermill\index{Papermill!notebook parameterization} enables parameterization and programmatic execution of notebooks, treating them as configurable pipeline stages. The nbconvert tool converts validated notebooks to Python scripts for production execution. Cell execution order enforcement tools execute all cells top-to-bottom, rejecting out-of-order dependencies.
The recommended practice is to use notebooks for exploration and rapid iteration, then refactor validated logic into tested Python modules for production pipelines. The overhead of refactoring pays off in maintainability and reliability. Failing to robustify these pipelines can lead to silent failures with massive economic consequences. A short cost model quantifies why reducing time-to-detection pays for the extra infrastructure.
```{python}
#| echo: false
@@ -1245,6 +1253,9 @@ class SilentFailureCost:
:::
The recommended practice is to use notebooks for exploration and rapid iteration, then refactor validated logic into tested Python modules for production pipelines. The overhead of refactoring pays off in maintainability and reliability. Failing to robustify these pipelines can lead to silent failures with massive economic consequences. A short cost model quantifies why reducing time-to-detection pays for the extra infrastructure.
Beyond reproducibility, automation enhances model training by reducing manual effort and standardizing critical steps. MLOps workflows incorporate techniques such as hyperparameter tuning [@ranjit2019; @li2017hyperband], neural architecture search [@elsken2019], and automatic feature selection [@scikit_learn_feature_selection] to explore the design space efficiently. These tasks are orchestrated using CI/CD pipelines, which automate data preprocessing, model training, evaluation, registration, and deployment. For instance, a Jenkins pipeline triggers a retraining job when new labeled data becomes available. The resulting model is evaluated against baseline metrics, and if performance thresholds are met, it is deployed automatically.
The increasing availability of cloud-based infrastructure has expanded the reach of model training [@gartner2024cloud]. This connects to the workflow orchestration patterns explored in @sec-ml-workflow, which provide the foundation for managing complex, multi-stage training processes across distributed systems. Cloud providers offer managed services that provision high-performance computing resources, including GPU and Tensor Processing Unit (TPU) accelerators, on demand[^fn-cloud-training-economics]. Depending on the platform, teams construct their own training workflows or rely on fully managed services such as Vertex AI Fine Tuning [@vertex_ai_fine_tuning], which support automated adaptation of foundation models to new tasks. Hardware availability, regional access restrictions, and cost constraints remain important considerations when designing cloud-based training systems [@oecd_ai_2021].
@@ -1423,14 +1434,13 @@ class RetrainingInterval:
**Formula**: The approximation in @eq-optimal-retrain gives the optimal retraining interval $(T^*)$ that minimizes the sum of staleness losses and training costs:
$$ T^* \approx \sqrt{\frac{2 \cdot C}{Q \cdot V \cdot A_0 \cdot \gamma}} $$ {#eq-optimal-retrain}
**Math**: Consider a Lighthouse Fraud Model ($A_0$ = `{python} RetrainingInterval.A0_str`):
* **Traffic** $(Q)$: `{python} RetrainingInterval.Q_str` transactions/day.
```{=latex}
\tcbbreak
```
**Math**: Consider a Lighthouse Fraud Model ($A_0$ = `{python} RetrainingInterval.A0_str`):
* **Traffic** $(Q)$: `{python} RetrainingInterval.Q_str` transactions/day.
* **Utility** $(V)$: \$`{python} RetrainingInterval.V_str` per query for a unit accuracy change.
* **Retraining Cost** $(C)$: \$`{python} RetrainingInterval.retrain_cost_str`.
* **Drift Rate** $(\gamma)$: `{python} RetrainingInterval.gamma_pct_str` per day.
@@ -1459,6 +1469,17 @@ Each retraining incurs fixed costs including compute, validation, and deployment
$$\text{Retraining Cost} = C_{\text{compute}} + C_{\text{validation}} + C_{\text{deployment}} + C_{\text{risk}}$$ {#eq-retrain-cost}
where $C_{\text{risk}}$ represents the expected cost of potential regression from the new model.
::: {#exmp-ml-ops-fraud-detection-retraining .callout-example title="Fraud detection retraining"}
Consider a fraud detection model with the parameters in @tbl-retraining-parameters that captures the high query volume and rapid drift rate characteristic of financial fraud detection.
**Applying the formula**:
`{python} RetrainingInterval.T_star_eq`
This analysis suggests daily retraining is economically optimal for this high-volume, high-stakes fraud detection scenario.
:::
###### Optimal retraining interval {.unnumbered}
The optimal retraining interval $T^*$ minimizes total cost per unit time, as @eq-optimal-interval shows:
@@ -1472,17 +1493,6 @@ $$T^* = \operatorname{arg\,min}_T \frac{\text{Staleness Cost}(T) + \text{Retrain
For exponential decay, this yields the square-root law used in our earlier napkin math calculation. In fraud detection, these formulas translate directly into a retraining schedule.
::: {#exmp-ml-ops-fraud-detection-retraining .callout-example title="Fraud detection retraining"}
Consider a fraud detection model with the parameters in @tbl-retraining-parameters that captures the high query volume and rapid drift rate characteristic of financial fraud detection.
**Applying the formula**:
`{python} RetrainingInterval.T_star_eq`
This analysis suggests daily retraining is economically optimal for this high-volume, high-stakes fraud detection scenario.
:::
| **Parameter** | **Value** | **Description** |
|:--------------------|:------------------------------------------------:|:-----------------------------------------------------------------------|
| **$Q$** | `{python} RetrainingInterval.Q_str` | Transactions per day |
@@ -1826,7 +1836,12 @@ A PyTorch model that achieves top accuracy on a benchmark may serve predictions
##### Optimization frameworks {#sec-ml-operations-optimization-frameworks-d013}
\index{Model Optimization!framework comparison}The choice of optimization framework determines which hardware targets are available and which performance ceiling is reachable—@tbl-model-optimization-frameworks reveals that broader hardware compatibility (ONNX Runtime) comes at the cost of peak performance, while maximum throughput (TensorRT) locks the deployment to a single vendor:
\index{Model Optimization!framework comparison}The choice of optimization framework determines which hardware targets are available and which performance ceiling is reachable—@tbl-model-optimization-frameworks reveals that broader hardware compatibility (ONNX Runtime) comes at the cost of peak performance, while maximum throughput (TensorRT) locks the deployment to a single vendor.
##### ONNX as interchange format {#sec-ml-operations-onnx-interchange-format-6a76}
ONNX (Open Neural Network Exchange)\index{ONNX!model interchange format} has emerged as the standard interchange format for model portability. A typical workflow exports a PyTorch model to ONNX, runs graph-level cleanup (constant folding, dead-code elimination), applies operator fusion (such as Conv+BN+ReLU collapsed into a single op), and quantizes weights (FP32 to INT8) before deploying to the target runtime. Numerical equivalence to the source model must be validated at each step.
| **Framework** | **Source Formats** | **Target Hardware** | **Key Optimizations** |
|:------------------------------------------------|:--------------------------------|:------------------------------|:--------------------------------------------------------|
@@ -1839,10 +1854,6 @@ A PyTorch model that achieves top accuracy on a benchmark may serve predictions
: **Model Optimization Frameworks**: Different optimization tools target different deployment scenarios. TensorRT provides maximum performance on NVIDIA GPUs but locks deployments into that hardware. ONNX Runtime offers broader compatibility across hardware targets. OpenVINO optimizes for Intel hardware ecosystems. {#tbl-model-optimization-frameworks tbl-colwidths="[15,23,30,32]"}
##### ONNX as interchange format {#sec-ml-operations-onnx-interchange-format-6a76}
ONNX (Open Neural Network Exchange)\index{ONNX!model interchange format} has emerged as the standard interchange format for model portability. A typical workflow exports a PyTorch model to ONNX, runs graph-level cleanup (constant folding, dead-code elimination), applies operator fusion (such as Conv+BN+ReLU collapsed into a single op), and quantizes weights (FP32 to INT8) before deploying to the target runtime. Numerical equivalence to the source model must be validated at each step.
##### Quantization strategies {#sec-ml-operations-quantization-strategies-940b}
\index{Quantization!serving optimization}Quantization reduces model size and increases throughput by using lower-precision arithmetic. The mechanics of PTQ, QAT, and mixed-precision strategies are covered in @sec-model-compression, with serving-specific precision selection (including dynamic per-request precision) detailed in @sec-model-serving-precision-selection-serving-55ba. From an operational perspective, the key deployment consideration is validating that quantized models maintain accuracy under production traffic distributions, not merely calibration datasets.
@@ -2033,7 +2044,7 @@ GPU utilization metrics can mislead operators. A GPU reporting 90 percent utiliz
| **I/O-bound** | <50 percent | <50 percent | Improve data pipeline, prefetch inputs, use SSDs |
| **Batch-starved** | Variable (spiky) | Variable | Dynamic batching, request queuing on single server |
: **GPU Utilization Patterns**: Different utilization signatures require different optimizations. High GPU utilization with low memory bandwidth suggests compute-bound workloads that benefit from parallelism. High memory bandwidth with moderate GPU utilization indicates memory-bound workloads requiring model optimization. {#tbl-gpu-utilization-patterns}
: **GPU Utilization Patterns**: Different utilization signatures require different optimizations. High GPU utilization with low memory bandwidth suggests compute-bound workloads that benefit from parallelism. High memory bandwidth with moderate GPU utilization indicates memory-bound workloads requiring model optimization. {#tbl-gpu-utilization-patterns tbl-colwidths="[15,15,23,47]"}
##### Utilization targets by workload {.unnumbered}
@@ -2186,6 +2197,9 @@ Infrastructure management provisions resources; monitoring\index{Model Monitorin
Effective monitoring spans both model behavior and infrastructure performance. On the model side, teams track metrics such as accuracy, precision, recall, and the confusion matrix [@scikit_learn_confusion_matrix] using live or sampled predictions to detect whether performance remains stable or begins to drift. A critical constraint is the *drift detection delay*\index{Drift Detection Delay!statistical monitoring}, which determines how quickly statistical monitoring can confirm that degradation has occurred. The speed of detection depends on traffic volume. A short sample-rate calculation makes that constraint visible.
The detection delay calculated above exposes a fundamental asymmetry: statistical tests require enough labeled samples to achieve power, and low-traffic systems may wait days or weeks before accumulating sufficient evidence. This latency gap is not an engineering shortcut that better tooling can close; it is a consequence of finite sample rates colliding with the statistical power requirements of hypothesis testing. The practical implication is that monitoring systems must distinguish between drift that alters the input distribution (detectable without labels) and drift that changes the decision boundary itself (detectable only after ground truth arrives).
```{python}
#| echo: false
#| label: drift-detection-delay-calc
@@ -2255,8 +2269,6 @@ class DriftDetectionDelay:
:::
The detection delay calculated above exposes a fundamental asymmetry: statistical tests require enough labeled samples to achieve power, and low-traffic systems may wait days or weeks before accumulating sufficient evidence. This latency gap is not an engineering shortcut that better tooling can close; it is a consequence of finite sample rates colliding with the statistical power requirements of hypothesis testing. The practical implication is that monitoring systems must distinguish between drift that alters the input distribution (detectable without labels) and drift that changes the decision boundary itself (detectable only after ground truth arrives).
Production ML systems face two distinct forms of model drift[^fn-drift-types-ops] that monitoring must distinguish. *Concept drift*\index{Concept Drift!changing relationships}[^fn-covid-concept-drift] occurs when the underlying relationship between features and targets evolves: the function $p(y \mid x)$ changes even though the inputs look similar. During the COVID-19 pandemic, for example, purchasing behavior shifted dramatically, invalidating many previously accurate recommendation models. *Data drift*[^fn-drift-covariate-shift], by contrast, refers to shifts in the input distribution $p(x)$ itself. In applications such as self-driving cars, this may result from seasonal changes in weather, lighting, or road conditions, all of which alter the model's inputs without changing the underlying physics of driving.
[^fn-drift-types-ops]: **Drift Detection Lag**: Feature drift (covariate shift on $p(x)$) is detectable immediately from input distributions, with no labels needed. Concept drift ($p(y \mid x)$ changing) is invisible until ground truth arrives, which in high-stakes domains (medical diagnosis, fraud detection, legal decisions) can take days, weeks, or months. This asymmetry means the most dangerous drift is also the slowest to detect, requiring proxy metrics (prediction confidence distributions, output entropy) as imperfect early warning systems that trade false alarm rate for detection speed. \index{Drift!operational distinction}
@@ -2473,7 +2485,7 @@ $$ \text{Cost} \approx \text{Frequency} \times \text{Metrics} \times (\text{Inge
| **`{python} ObservabilitySampling.high_seconds_str`** | Micro-bursts | ~`{python} ObservabilitySampling.high_gb_s_str` | High (Requires dedicated cluster) |
| **`{python} ObservabilitySampling.low_seconds_str`** | Trends | ~`{python} ObservabilitySampling.low_mb_s_str` | Low (Standard sidecar) |
: **Sampling Frequency vs. Observability Cost**: Roughly `{python} ObservabilitySampling.sampling_ratio_str`$\times$ data-volume swing between `{python} ObservabilitySampling.high_seconds_str` and `{python} ObservabilitySampling.low_seconds_str` sampling at `{python} ObservabilitySampling.request_rate_m_str` M requests per second, assuming about `{python} ObservabilitySampling.telemetry_kb_str` of telemetry per request. {#tbl-observability-sampling-cost tbl-colwidths="[14,28,28,30]"}
: **Sampling Frequency vs. Observability Cost**: Roughly `{python} ObservabilitySampling.sampling_ratio_str`$\times$ data-volume swing between `{python} ObservabilitySampling.high_seconds_str` and `{python} ObservabilitySampling.low_seconds_str` sampling at `{python} ObservabilitySampling.request_rate_m_str` M requests per second, assuming about `{python} ObservabilitySampling.telemetry_kb_str` of telemetry per request. {#tbl-observability-sampling-cost tbl-colwidths="[14,21,28,37]"}
**Recommendation**: Use dynamic sampling. Sample `{python} ObservabilitySampling.success_sample_pct_str` of successful requests but `{python} ObservabilitySampling.error_sample_pct_str` of errors. Use high-frequency (`{python} ObservabilitySampling.high_seconds_str`) monitoring only for aggregate counters (like error rate), but low-frequency (`{python} ObservabilitySampling.low_seconds_str`) for high-cardinality data (like user-level distribution sketches). @Sec-ml-operations-monitoring-cost-model-7fe3 provides worked examples for budgeting monitoring infrastructure.
@@ -2614,6 +2626,11 @@ class PSIAgeExample:
warning_threshold_str = fmt(warning_threshold, precision=1, commas=False)
```
```{=latex}
\begingroup
\renewcommand{\arraystretch}{1.3}
```
| **Age Bin** | **Training %** | **Serving %** | **Difference** | **ln(Serving/Training)** | **Contribution** |
|:-----------:|:--------------------------------------------:|:-------------------------------------------:|:------------------------------------------:|:-----------------------------------------:|:--------------------------------------------:|
| **18–25** | `{python} PSIAgeExample.training_pct_str[0]` | `{python} PSIAgeExample.serving_pct_str[0]` | `{python} PSIAgeExample.difference_str[0]` | `{python} PSIAgeExample.log_ratio_str[0]` | `{python} PSIAgeExample.contribution_str[0]` |
@@ -2623,7 +2640,11 @@ class PSIAgeExample:
| **56–65** | `{python} PSIAgeExample.training_pct_str[4]` | `{python} PSIAgeExample.serving_pct_str[4]` | `{python} PSIAgeExample.difference_str[4]` | `{python} PSIAgeExample.log_ratio_str[4]` | `{python} PSIAgeExample.contribution_str[4]` |
| **66+** | `{python} PSIAgeExample.training_pct_str[5]` | `{python} PSIAgeExample.serving_pct_str[5]` | `{python} PSIAgeExample.difference_str[5]` | `{python} PSIAgeExample.log_ratio_str[5]` | `{python} PSIAgeExample.contribution_str[5]` |
: **PSI Worked Example**: User age distribution drift from training to serving, decomposed across six age bins. The total PSI is `{python} PSIAgeExample.total_psi_str`, well below the `{python} PSIAgeExample.warning_threshold_str` warning threshold, even though several bins shifted by 3 percentage points. The contribution column shows why aggregate PSI is more reliable than per-bin inspection: individual bin movement can look noticeable while the summed evidence remains below the alert threshold. {#tbl-psi-age-example tbl-colwidths="[14,17,17,17,18,17]"}
: **PSI Worked Example**: User age distribution drift from training to serving, decomposed across six age bins. The total PSI is `{python} PSIAgeExample.total_psi_str`, well below the `{python} PSIAgeExample.warning_threshold_str` warning threshold, even though several bins shifted by 3 percentage points. The contribution column shows why aggregate PSI is more reliable than per-bin inspection: individual bin movement can look noticeable while the summed evidence remains below the alert threshold. {#tbl-psi-age-example tbl-colwidths="[13,15,15,16,24,17]"}
```{=latex}
\endgroup
```
The total PSI is `{python} PSIAgeExample.total_psi_str` (Stable). The training and serving columns are shown as percentages, while the difference column is expressed in proportion units, so a 3 percentage-point shift appears as $\pm 0.03$. Even though specific bins shifted by 3 percentage points, the aggregate drift is well below the `{python} PSIAgeExample.warning_threshold_str` warning threshold. This calculation prevents false alarms from minor fluctuations while remaining sensitive to systematic shifts.
@@ -2979,7 +3000,7 @@ class SliceAnalysisCalc:
| **Tablet users** | **5 percent** | **62 percent** | **Severe—investigate** |
| **Overall** | **`{python} SliceAnalysisCalc.total_traffic_str` percent** | **`{python} SliceAnalysisCalc.overall_accuracy_str` percent** | **Masks tablet problem** |
: **Slice Analysis Example**: Overall accuracy of `{python} SliceAnalysisCalc.overall_accuracy_str` percent appears acceptable, but tablet users (5 percent of traffic) experience 62 percent accuracy, a severe degradation masked by aggregation. Effective debugging requires systematic slice analysis across key dimensions. {#tbl-slice-analysis-example}
: **Slice Analysis Example**: Overall accuracy of `{python} SliceAnalysisCalc.overall_accuracy_str` percent appears acceptable, but tablet users (5 percent of traffic) experience 62 percent accuracy, a severe degradation masked by aggregation. Effective debugging requires systematic slice analysis across key dimensions. {#tbl-slice-analysis-example tbl-colwidths="[28,22,22,28]"}
##### Feature attribution for debugging {#sec-ml-operations-feature-attribution-debugging-f4ab}
@@ -3047,6 +3068,8 @@ This ambiguity compounds with delayed impact visibility. Unlike latency spikes t
These challenges motivate tiered escalation structures\index{Tiered Escalation!incident response} that match expertise to incident complexity. @Tbl-oncall-structure illustrates a recommended on-call structure for ML teams, where primary responders handle routine issues using standardized runbooks while escalation paths connect to specialists capable of deeper investigation.
The parallel data on-call role deserves particular attention. Since data issues cause the majority of ML incidents, having a data engineer available alongside the ML on-call dramatically reduces time-to-resolution for upstream problems. Without this parallel structure, ML engineers waste hours investigating model behavior only to discover that the root cause lies in a data pipeline they cannot access or modify.
| **Tier** | **Responder** | **Responsibility** |
|:----------------------------|:----------------------------------|:----------------------------------------------------------------------|
| **Tier 1 (Primary)** | ML Engineer | Initial triage, standard runbooks, escalation decisions |
@@ -3056,8 +3079,6 @@ These challenges motivate tiered escalation structures\index{Tiered Escalation!i
: **ML On-Call Structure**: Tiered escalation with parallel data on-call enables efficient incident response. Tier 1 handles routine issues using runbooks; Tier 2 addresses complex debugging; Tier 3 manages critical incidents requiring architectural decisions. {#tbl-oncall-structure tbl-colwidths="[25,33,41]"}
The parallel data on-call role deserves particular attention. Since data issues cause the majority of ML incidents, having a data engineer available alongside the ML on-call dramatically reduces time-to-resolution for upstream problems. Without this parallel structure, ML engineers waste hours investigating model behavior only to discover that the root cause lies in a data pipeline they cannot access or modify.
Effective on-call depends heavily on runbook quality. Every production ML model should have documentation covering the model's purpose, ownership, and business criticality alongside its normal operating parameters—expected latency, throughput, and accuracy ranges that define healthy behavior. Historical incidents and their resolutions provide templates for common failure patterns, while diagnostic commands enable rapid health assessment: how to check recent predictions, feature distributions, and model confidence scores. Critically, runbooks must specify escalation criteria (when to wake up Tier 2 vs. when to rollback without approval) and rollback procedures with step-by-step instructions and expected recovery times. Runbooks written during calm periods save critical minutes during 3:00 AM incidents.
Even well-designed monitoring can generate excessive alerts that erode on-call effectiveness. Alert fatigue\index{Alert Fatigue!operational risk}, the tendency to ignore or dismiss alerts after experiencing too many false positives, represents a significant operational risk. Teams combat fatigue through consolidation, grouping related alerts so that multiple features drifting simultaneously generate a single notification rather than dozens. Adaptive thresholds that account for weekly and seasonal patterns prevent predictable variations from triggering unnecessary pages. Measuring alert actionability provides empirical guidance: alerts acted upon less than 10 percent of the time should be retired or recalibrated. When temporary silencing is necessary, accountability mechanisms (requiring a follow-up ticket before snoozing) prevent alerts from being permanently ignored.
@@ -3251,7 +3272,7 @@ Before examining system design and maturity frameworks, @tbl-technical-debt-summ
| **Configuration Debt** | Fragmented settings, poor versioning | Irreproducible results, silent failures, tuning opacity | Version control, validation, structured formats, automation |
| **Early-Stage Debt** | Rapid prototyping shortcuts, tight code-logic coupling | Inflexibility as systems scale, difficult team collaboration | Flexible foundations, intentional debt tracking, planned refactoring |
: **Technical Debt Patterns**: Machine learning systems accumulate distinct forms of technical debt from data dependencies, model interactions, and evolving operational contexts. Primary debt patterns, their causes, symptoms, and recommended mitigation strategies guide practitioners in recognizing and addressing these challenges systematically. {#tbl-technical-debt-summary}
: **Technical Debt Patterns**: Machine learning systems accumulate distinct forms of technical debt from data dependencies, model interactions, and evolving operational contexts. Primary debt patterns, their causes, symptoms, and recommended mitigation strategies guide practitioners in recognizing and addressing these challenges systematically. {#tbl-technical-debt-summary tbl-colwidths="[21,22,28,29]"}
### ML test score {#sec-ml-operations-assessing-technical-debt-ml-test-score-0099}
@@ -3259,6 +3280,11 @@ Before examining system design and maturity frameworks, @tbl-technical-debt-summ
The ML Test Score\index{ML Test Score!production readiness} [@breck2017] provides a systematic rubric for evaluating production readiness across four categories: data tests, model tests, ML infrastructure tests, and monitoring tests. The paper defines 28 tests in total, seven per section, with partial or full credit for each test. Readiness is tracked by section rather than by a simple grand-total maturity band: a system with strong model tests but weak monitoring still carries production risk. @Tbl-ml-test-score summarizes representative tests practitioners should implement:
- **Data section**: Validates feature expectations, privacy controls, and whether each feature is beneficial relative to its operational cost.
- **Model section**: Validates reviewed model specifications, hyperparameter discipline, staleness limits, and offline-online metric alignment.
- **Infrastructure section**: Validates reproducible training, rollback, training-serving consistency, and deployment gates.
- **Monitoring section**: Validates alerts for dependency changes, data invariants, skew, and model staleness.
| **Category** | **Test** | **Implementation Example** |
|:-------------------------|:-----------------------------------------------------------|:-------------------------------------------------|
| **Data Tests** | Feature expectations are captured in schema | Great Expectations, TFX Data Validation |
@@ -3280,10 +3306,6 @@ The ML Test Score\index{ML Test Score!production readiness} [@breck2017] provide
: **ML Test Score Checklist**: Representative production-readiness tests from the ML Test Score rubric. The original rubric contains 28 tests grouped into four sections of seven tests each; section scores expose whether production risk comes from data validation, model validation, infrastructure, or monitoring rather than hiding the weakness in a single total. Based on @breck2017. {#tbl-ml-test-score}
- **Data section**: Validates feature expectations, privacy controls, and whether each feature is beneficial relative to its operational cost.
- **Model section**: Validates reviewed model specifications, hyperparameter discipline, staleness limits, and offline-online metric alignment.
- **Infrastructure section**: Validates reproducible training, rollback, training-serving consistency, and deployment gates.
- **Monitoring section**: Validates alerts for dependency changes, data invariants, skew, and model staleness.
Quarterly audits against this rubric, prioritizing tests that address the most frequent incident types, reveal where operational investments will yield the highest reliability gains.
@@ -3441,7 +3463,7 @@ The operational benefits of MLOps are substantial, but implementing mature pract
For a single production ML system, establishing solid MLOps practices typically incurs the costs summarized in @tbl-mlops-single-model-investment:
| **Component** | **Typical Cost** | **Justification** |
|:----------------------------|---------------------:|:-------------------------------------------|
|:----------------------------|:--------------------:|:-------------------------------------------|
| **CI/CD pipeline setup** | \$10–30K one-time | Reduces deployment time from days to hours |
| **Monitoring and alerting** | \$2–10K/year | Catches degradation before user impact |
| **Feature store (basic)** | \$5–20K/year | Eliminates training-serving skew |
@@ -3534,15 +3556,15 @@ A battery-powered sleep-tracking ring and AI/ML-based medical software governed
:::
| | **Principle** | **Oura Ring** | **ClinAIOps** |
|:----|:---------------------------|:---------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------|
| | **Reproducibility** | Versioned synchronized wearable and PSG datasets | Audit trails, decision provenance |
| | **Separation of concerns** | Independent data, training, and serving layers with edge-specific deployment pipeline | Distinct clinical validation and deployment stages with regulatory compliance isolation |
| | **Consistency** | PSG-aligned preprocessing across training and on-device inference | Standardized clinical data pipelines ensuring training-serving parity |
| | **Observable degradation** | On-device anomaly detection, limited telemetry | Cohort-specific monitoring, outcome tracking |
| | **Cost-aware automation** | Battery-aware retraining triggers, CI/CD for edge balancing accuracy and resource cost | Automated model updates with human-in-loop gates balancing update cost and patient risk |
| **Principle** | **Oura Ring** | **ClinAIOps** |
|:---------------------------|:---------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------|
| **Reproducibility** | Versioned synchronized wearable and PSG datasets | Audit trails, decision provenance |
| **Separation of concerns** | Independent data, training, and serving layers with edge-specific deployment pipeline | Distinct clinical validation and deployment stages with regulatory compliance isolation |
| **Consistency** | PSG-aligned preprocessing across training and on-device inference | Standardized clinical data pipelines ensuring training-serving parity |
| **Observable degradation** | On-device anomaly detection, limited telemetry | Cohort-specific monitoring, outcome tracking |
| **Cost-aware automation** | Battery-aware retraining triggers, CI/CD for edge balancing accuracy and resource cost | Automated model updates with human-in-loop gates balancing update cost and patient risk |
: **MLOps Principles by Case Study**: Side-by-side implementation of the five foundational MLOps principles in the Oura Ring and ClinAIOps deployments, showing how domain constraints reshape how each principle is realized without changing which principles apply. {#tbl-ml-ops-principle-mapping tbl-colwidths="[7,17,38,38]"}
: **MLOps Principles by Case Study**: Side-by-side implementation of the five foundational MLOps principles in the Oura Ring and ClinAIOps deployments, showing how domain constraints reshape how each principle is realized without changing which principles apply. {#tbl-ml-ops-principle-mapping tbl-colwidths="[16,41,43]"}
### Oura Ring case study {#sec-ml-operations-oura-ring-case-study-2444}
@@ -3733,7 +3755,7 @@ Three interlocking feedback loops enable safe, adaptive integration of machine l
::: {#fig-clinaiops fig-env="figure" fig-pos="htb" fig-cap="**ClinAIOps Feedback Loops**: The cyclical framework coordinates patients, clinicians, and AI developers to support continuous model improvement and safe clinical integration. Patients and clinicians use AI outputs in care workflows, while AI developers receive feedback from both groups to refine models and operations. Source: [@chen2023]." fig-alt="Circular diagram with three nodes: patient, clinician, and AI developer. Arrows form cyclic flow: patient provides monitoring data, clinician sets therapy regimen and approvals, and AI developer receives feedback to improve AI models and workflows."}
```{.tikz}
\scalebox{0.8}{%
\scalebox{0.85}{%
\begin{tikzpicture}[line join=round,font=\small\usefont{T1}{phv}{m}{n}]
%radius
\def\ra{53mm}
@@ -4142,7 +4164,7 @@ ClinAIOps extends beyond technical infrastructure to support complex sociotechni
| **Model validation** | Testing model performance metrics | Clinical evaluation of recommendations |
| **Implementation** | Focuses on technical integration | Aligns incentives of human stakeholders |
: **Clinical AI Operations**: Traditional MLOps focuses on model performance, while ClinAIOps integrates technical systems with clinical workflows, ethical considerations, and ongoing feedback loops to ensure safe, trustworthy AI assistance in healthcare settings. ClinAIOps prioritizes human oversight and accountability alongside automation, addressing unique challenges in clinical decision-making that standard MLOps pipelines often overlook. {#tbl-clinical_ops}
: **Clinical AI Operations**: Traditional MLOps focuses on model performance, while ClinAIOps integrates technical systems with clinical workflows, ethical considerations, and ongoing feedback loops to ensure safe, trustworthy AI assistance in healthcare settings. ClinAIOps prioritizes human oversight and accountability alongside automation, addressing unique challenges in clinical decision-making that standard MLOps pipelines often overlook. {#tbl-clinical_ops tbl-colwidths="[18,36,46]"}
Successfully deploying AI in healthcare requires aligning systems with clinical workflows, human expertise, and patient needs. Technical performance alone is insufficient; deployment must account for ethical oversight and continuous adaptation to dynamic clinical contexts.
book/quarto/tex/header-includes.tex
+6 -6
View File
@@ -361,13 +361,13 @@ aboveskip=0pt
\renewcommand{\sidenote}[1]{%
\oldsidenote{%
\noindent
\color{crimson!100} % Crimson vertical line
\color{crimson!100}% % Crimson vertical line
\raisebox{0em}{%
\rule{0.5pt}{1.5em} % Thin vertical line
\rule{0.5pt}{1.2em}% % Thin vertical line
}
\hspace{0.3em} % Space after line
\color{black} % Reset text color
\footnotesize #1 % Sidenote content
\hspace{0.3em}% % Space after line
\color{black}% % Reset text color
\footnotesize\ignorespaces#1\unskip% % Sidenote content
}%
}
@@ -1459,7 +1459,7 @@ align=right,font={\fontsize{40pt}{40}\selectfont}]
% a visibly different typeface from body text in the chapter opener.
\etocifnumbered{%
\noindent
\makebox[2em][l]{%
\makebox[2.3em][l]{%
\rmfamily\fontsize{6.5}{8}\selectfont\bfseries\color{accentcolor}%
\etocnumber
}%