mirror of
https://github.com/harvard-edge/cs249r_book.git
synced 2026-05-08 18:01:20 -05:00
[GH-ISSUE #1090] Consider Discussion of Goodhart's Law in Benchmarking Chapter #4308
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: github-starred/cs249r_book#4308
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @profvjreddi on GitHub (Dec 31, 2025).
Original GitHub issue: https://github.com/harvard-edge/cs249r_book/issues/1090
Originally assigned to: @profvjreddi on GitHub.
I should possibly, in the benchmarking chapter or somewhere else in the book, discuss Goodhart's Law:
"When a measure becomes a target, it ceases to be a good measure."
Evaluate how and where this important concept fits best in the context of benchmarking, and whether it deserves a dedicated discussion or a mention alongside other potential pitfalls.
Reference: Goodhart's Law, often cited in policy analysis and evaluation disciplines, but highly relevant to performance metrics and benchmarking.
@profvjreddi commented on GitHub (Jan 11, 2026):
After review, Goodhart's Law is already comprehensively discussed in the benchmarking chapter 🤦
Benchmark Engineering" (Section 18.6.4.1) directly addresses how benchmark optimization exemplifies Goodhart's Law