github-starred/cs249r_book
2
0
Fork 0
mirror of https://github.com/harvard-edge/cs249r_book.git synced 2026-07-16 06:07:17 -05:00
Code Issues 276 Packages Projects Releases 26 Wiki Activity

[PR #1931] [MERGED] fix(mlperf-edu): quality-target check silently skipped for real submissions #32195

New Issue
Closed
opened 2026-07-12 08:51:25 -05:00 by GiteaMirror · 0 comments
GiteaMirror commented 2026-07-12 08:51:25 -05:00
Owner
Copy Link

📋 Pull Request Information

Original PR: https://github.com/harvard-edge/cs249r_book/pull/1931
Author: @Shashank-Tripathi-07
Created: 7/7/2026
Status: ✅ Merged
Merged: 7/10/2026
Merged by: @profvjreddi

Base: dev ← Head: fix/compliance-quality-target-lookup

📝 Commits (1)

  • 0500781 fix(mlperf-edu): quality-target check silently skipped for real submissions

📊 Changes

1 file changed (+16 additions, -5 deletions)

View changed files

📝 mlperf-edu/scripts/compliance_checker.py (+16 -5)

📄 Description

Summary

  • check_from_submission() looked up QUALITY_TARGETS by the full workload id (e.g. resnet18-train, matching every registry/suites/*.yaml id field), but QUALITY_TARGETS is keyed by the short form (resnet18). The lookup always missed, so the quality-target check never ran at all for any real submission, regardless of reported accuracy.
  • Separately, metrics.get(metric_name) or metrics.get("loss") or metrics.get("accuracy") used Python truthiness, so a genuine 0.0 accuracy value fell through to the loss value and could pass the quality-target check.

Test plan

  • Reproduced locally: built a resnet18-train submission with top1_accuracy: 0.0, confirmed the "Quality target reached" check was entirely absent from check_from_submission()'s output before the fix.
  • After the fix: same submission now correctly reports FAIL | Quality target reached | 0.0000 < 0.75.
  • Sanity check: a legitimate top1_accuracy: 0.80 submission still reports PASS | Quality target reached | 0.8000 >= 0.75.
  • pre-commit run passes on the changed file.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/harvard-edge/cs249r_book/pull/1931 **Author:** [@Shashank-Tripathi-07](https://github.com/Shashank-Tripathi-07) **Created:** 7/7/2026 **Status:** ✅ Merged **Merged:** 7/10/2026 **Merged by:** [@profvjreddi](https://github.com/profvjreddi) **Base:** `dev` ← **Head:** `fix/compliance-quality-target-lookup` --- ### 📝 Commits (1) - [`0500781`](https://github.com/harvard-edge/cs249r_book/commit/05007815ced9c7b1043261c37bb00c68fed33095) fix(mlperf-edu): quality-target check silently skipped for real submissions ### 📊 Changes **1 file changed** (+16 additions, -5 deletions) <details> <summary>View changed files</summary> 📝 `mlperf-edu/scripts/compliance_checker.py` (+16 -5) </details> ### 📄 Description ## Summary - `check_from_submission()` looked up `QUALITY_TARGETS` by the full workload id (e.g. `resnet18-train`, matching every `registry/suites/*.yaml` `id` field), but `QUALITY_TARGETS` is keyed by the short form (`resnet18`). The lookup always missed, so the quality-target check never ran at all for any real submission, regardless of reported accuracy. - Separately, `metrics.get(metric_name) or metrics.get("loss") or metrics.get("accuracy")` used Python truthiness, so a genuine `0.0` accuracy value fell through to the loss value and could pass the quality-target check. ## Test plan - [x] Reproduced locally: built a `resnet18-train` submission with `top1_accuracy: 0.0`, confirmed the "Quality target reached" check was entirely absent from `check_from_submission()`'s output before the fix. - [x] After the fix: same submission now correctly reports `FAIL | Quality target reached | 0.0000 < 0.75`. - [x] Sanity check: a legitimate `top1_accuracy: 0.80` submission still reports `PASS | Quality target reached | 0.8000 >= 0.75`. - [x] `pre-commit run` passes on the changed file. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
GiteaMirror added the pull-request label 2026-07-12 08:51:25 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area: book
area: collabs
area: kits
area: labs
area: socratiq
area: staffml
area: tinytorch
area: tools
area: website
bug
dependencies
format: epub
format: pdf
javascript
link-health
link-rot
priority-high
pull-request
Mirrored from GitHub Pull Request
 staffml
type: bug
type: citation
type: code
type: errata
type: improvement
type: new
type: question
vault-sli
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/cs249r_book#32195
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?