github-starred/cs249r_book
2
0
Fork 0
mirror of https://github.com/harvard-edge/cs249r_book.git synced 2026-07-16 23:24:55 -05:00
Code Issues 276 Packages Projects Releases 26 Wiki Activity

[PR #1938] [MERGED] fix(mlperf-edu): plateau detection mixes losses across NAS attempts #34662

New Issue
Closed
opened 2026-07-14 19:28:00 -05:00 by GiteaMirror · 0 comments
GiteaMirror commented 2026-07-14 19:28:00 -05:00
Owner
Copy Link

📋 Pull Request Information

Original PR: https://github.com/harvard-edge/cs249r_book/pull/1938
Author: @Shashank-Tripathi-07
Created: 7/7/2026
Status: ✅ Merged
Merged: 7/10/2026
Merged by: @profvjreddi

Base: dev ← Head: fix/auto-trainer-plateau-history-reset

📝 Commits (1)

  • de57faa fix(mlperf-edu): plateau detection mixes losses across NAS attempts

📊 Changes

1 file changed (+5 additions, -1 deletions)

View changed files

📝 mlperf-edu/scripts/orchestration/auto_trainer.py (+5 -1)

📄 Description

Summary

  • loss_history/val_loss_history accumulate across the whole outer NAS loop, but _load_model() reloads a fresh (randomly-initialized) model after each LLM-driven rewrite.
  • _detect_plateau() slides a PLATEAU_WINDOW-entry window over this cumulative history, so right after a rewrite the window mixes the discarded architecture's plateaued tail with the new model's high initial losses -- falsely re-triggering NAS almost immediately on an architecture that never got real training, or conversely masking a genuine plateau.

Fix

Added attempt_loss_history, reset at the start of each NAS attempt (right after the model reload) and used only for plateau detection, while loss_history/val_loss_history keep accumulating the full trajectory used for the saved results.json and convergence plot.

Test plan

  • Simulated the control flow in isolation (pure list logic, no torch needed): with the old code, 3 epochs of a freshly-reloaded model immediately re-triggered _detect_plateau() because the sliding window still contained the old model's plateaued tail. With the fix, the same 3 epochs correctly report no plateau (not enough data yet for the new attempt).
  • python -m py_compile passes on the edited file.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/harvard-edge/cs249r_book/pull/1938 **Author:** [@Shashank-Tripathi-07](https://github.com/Shashank-Tripathi-07) **Created:** 7/7/2026 **Status:** ✅ Merged **Merged:** 7/10/2026 **Merged by:** [@profvjreddi](https://github.com/profvjreddi) **Base:** `dev` ← **Head:** `fix/auto-trainer-plateau-history-reset` --- ### 📝 Commits (1) - [`de57faa`](https://github.com/harvard-edge/cs249r_book/commit/de57faada03c3f2d19564a12ba6d649ef20fdcba) fix(mlperf-edu): plateau detection mixes losses across NAS attempts ### 📊 Changes **1 file changed** (+5 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `mlperf-edu/scripts/orchestration/auto_trainer.py` (+5 -1) </details> ### 📄 Description ## Summary - `loss_history`/`val_loss_history` accumulate across the whole outer NAS loop, but `_load_model()` reloads a fresh (randomly-initialized) model after each LLM-driven rewrite. - `_detect_plateau()` slides a `PLATEAU_WINDOW`-entry window over this cumulative history, so right after a rewrite the window mixes the discarded architecture's plateaued tail with the new model's high initial losses -- falsely re-triggering NAS almost immediately on an architecture that never got real training, or conversely masking a genuine plateau. ## Fix Added `attempt_loss_history`, reset at the start of each NAS attempt (right after the model reload) and used only for plateau detection, while `loss_history`/`val_loss_history` keep accumulating the full trajectory used for the saved `results.json` and convergence plot. ## Test plan - [x] Simulated the control flow in isolation (pure list logic, no torch needed): with the old code, 3 epochs of a freshly-reloaded model immediately re-triggered `_detect_plateau()` because the sliding window still contained the old model's plateaued tail. With the fix, the same 3 epochs correctly report no plateau (not enough data yet for the new attempt). - [x] `python -m py_compile` passes on the edited file. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
GiteaMirror added the pull-request label 2026-07-14 19:28:00 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area: book
area: collabs
area: kits
area: labs
area: socratiq
area: staffml
area: tinytorch
area: tools
area: website
bug
dependencies
format: epub
format: pdf
javascript
link-health
link-rot
priority-high
pull-request
Mirrored from GitHub Pull Request
 staffml
type: bug
type: citation
type: code
type: errata
type: improvement
type: new
type: question
vault-sli
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/cs249r_book#34662
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?