github-starred/cs249r_book
2
0
Fork 0
mirror of https://github.com/harvard-edge/cs249r_book.git synced 2026-07-15 21:28:33 -05:00
Code Issues 276 Packages Projects Releases 26 Wiki Activity

[PR #1936] [MERGED] fix(mlperf-edu): p99 decode-latency formula wrong for small sample counts #36546

New Issue
Closed
opened 2026-07-16 00:27:37 -05:00 by GiteaMirror · 0 comments
GiteaMirror commented 2026-07-16 00:27:37 -05:00
Owner
Copy Link

📋 Pull Request Information

Original PR: https://github.com/harvard-edge/cs249r_book/pull/1936
Author: @Shashank-Tripathi-07
Created: 7/7/2026
Status: ✅ Merged
Merged: 7/10/2026
Merged by: @profvjreddi

Base: dev ← Head: fix/nanogpt-decode-p99-percentile

📝 Commits (1)

  • a1be485 fix(mlperf-edu): p99 decode-latency formula wrong for small sample counts

📊 Changes

1 file changed (+2 additions, -1 deletions)

View changed files

📝 mlperf-edu/reference/cloud/nanogpt_decode.py (+2 -1)

📄 Description

Summary

  • sorted(per_step)[int(len(per_step) * 0.99) - 1] picks the wrong index for small n. For the min/smoke profile's default of 4 decode steps (n_loop=3), int(3*0.99)-1 = 1, i.e. the ~66th percentile (the middle value), not p99 (which for 3 samples should be the max).
  • mlperf/harness.py's own percentile() already implements this correctly with a ceil-based rank (rank = max(1, ceil(p/100*n))), matching the same pattern already used elsewhere in reference/cloud/ (e.g. from mlperf.roofline import measure_roofline). Reused it instead of duplicating the broken formula.

Test plan

  • Verified the old formula against per_step=[10.0, 20.0, 30.0]: returned 20.0 (the median), not p99.
  • Verified harness.percentile([10.0, 20.0, 30.0], 99) returns 30.0 (the correct max/p99 for 3 samples).
  • python -m py_compile passes on the edited file.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/harvard-edge/cs249r_book/pull/1936 **Author:** [@Shashank-Tripathi-07](https://github.com/Shashank-Tripathi-07) **Created:** 7/7/2026 **Status:** ✅ Merged **Merged:** 7/10/2026 **Merged by:** [@profvjreddi](https://github.com/profvjreddi) **Base:** `dev` ← **Head:** `fix/nanogpt-decode-p99-percentile` --- ### 📝 Commits (1) - [`a1be485`](https://github.com/harvard-edge/cs249r_book/commit/a1be485c19d0ce22cfd5df25c6c25b34c7ee8aa7) fix(mlperf-edu): p99 decode-latency formula wrong for small sample counts ### 📊 Changes **1 file changed** (+2 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `mlperf-edu/reference/cloud/nanogpt_decode.py` (+2 -1) </details> ### 📄 Description ## Summary - `sorted(per_step)[int(len(per_step) * 0.99) - 1]` picks the wrong index for small `n`. For the min/smoke profile's default of 4 decode steps (`n_loop=3`), `int(3*0.99)-1 = 1`, i.e. the ~66th percentile (the middle value), not p99 (which for 3 samples should be the max). - `mlperf/harness.py`'s own `percentile()` already implements this correctly with a ceil-based rank (`rank = max(1, ceil(p/100*n))`), matching the same pattern already used elsewhere in `reference/cloud/` (e.g. `from mlperf.roofline import measure_roofline`). Reused it instead of duplicating the broken formula. ## Test plan - [x] Verified the old formula against `per_step=[10.0, 20.0, 30.0]`: returned `20.0` (the median), not p99. - [x] Verified `harness.percentile([10.0, 20.0, 30.0], 99)` returns `30.0` (the correct max/p99 for 3 samples). - [x] `python -m py_compile` passes on the edited file. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
GiteaMirror added the pull-request label 2026-07-16 00:27:37 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area: book
area: collabs
area: kits
area: labs
area: socratiq
area: staffml
area: tinytorch
area: tools
area: website
bug
dependencies
format: epub
format: pdf
javascript
link-health
link-rot
priority-high
pull-request
Mirrored from GitHub Pull Request
 staffml
type: bug
type: citation
type: code
type: errata
type: improvement
type: new
type: question
vault-sli
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/cs249r_book#36546
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?