[GH-ISSUE #1090] Consider Discussion of Goodhart's Law in Benchmarking Chapter #5650

New Issue

GiteaMirror · 2026-04-21T21:37:51-05:00

GiteaMirror commented

2026-04-21 21:37:51 -05:00

Originally created by @profvjreddi on GitHub (Dec 31, 2025).
Original GitHub issue: https://github.com/harvard-edge/cs249r_book/issues/1090

Originally assigned to: @profvjreddi on GitHub.

I should possibly, in the benchmarking chapter or somewhere else in the book, discuss Goodhart's Law:

"When a measure becomes a target, it ceases to be a good measure."

Evaluate how and where this important concept fits best in the context of benchmarking, and whether it deserves a dedicated discussion or a mention alongside other potential pitfalls.

Reference: Goodhart's Law, often cited in policy analysis and evaluation disciplines, but highly relevant to performance metrics and benchmarking.

Originally created by @profvjreddi on GitHub (Dec 31, 2025). Original GitHub issue: https://github.com/harvard-edge/cs249r_book/issues/1090 Originally assigned to: @profvjreddi on GitHub. I should possibly, in the benchmarking chapter or somewhere else in the book, discuss Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure." Evaluate how and where this important concept fits best in the context of benchmarking, and whether it deserves a dedicated discussion or a mention alongside other potential pitfalls. Reference: Goodhart's Law, often cited in policy analysis and evaluation disciplines, but highly relevant to performance metrics and benchmarking.

GiteaMirror added the area: book type: improvement labels 2026-04-21 21:37:51 -05:00

GiteaMirror closed this issue

2026-04-21 21:37:52 -05:00

GiteaMirror commented

2026-04-21 21:37:54 -05:00

@profvjreddi commented on GitHub (Jan 11, 2026):

After review, Goodhart's Law is already comprehensively discussed in the benchmarking chapter 🤦

Benchmark Engineering" (Section 18.6.4.1) directly addresses how benchmark optimization exemplifies Goodhart's Law

@profvjreddi commented on GitHub (Jan 11, 2026): After review, Goodhart's Law is already comprehensively discussed in the benchmarking chapter 🤦 Benchmark Engineering" (Section 18.6.4.1) directly addresses how benchmark optimization exemplifies Goodhart's Law

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/cs249r_book#5650