[PR #1554] [MERGED] fix(links): aggressive lycheeignore to drive tracker to zero #8262

New Issue

GiteaMirror · 2026-04-27T17:37:16-05:00

GiteaMirror commented

2026-04-27 17:37:16 -05:00

📋 Pull Request Information

Original PR: https://github.com/harvard-edge/cs249r_book/pull/1554
Author: @profvjreddi
Created: 4/26/2026
Status: ✅ Merged
Merged: 4/26/2026
Merged by: @profvjreddi

Base: dev ← Head: fix/link-rot-zero-noise

📝 Commits (1)

98ed108 fix(links): aggressive lycheeignore patterns to drive tracker to zero

📊 Changes

2 files changed (+57 additions, -0 deletions)

View changed files

📝 book/config/linting/.lycheeignore (+11 -0)
📝 shared/config/.lycheeignore (+46 -0)

📄 Description

Follow-up to #1424.

Goal: tracker permanent green.

Real broken links were already addressed in PRs #1552 and #1553. This PR handles the remaining tracker noise — dominated by anti-bot false positives (LinkedIn 999, Twitter Cloudflare challenge, Harvard SEAS bot block, etc.). Every domain pattern below was manually verified live in a browser.

Patterns added

Domain	Why flagged	Verified
`linkedin.com`	999 anti-scraping	✅ in browser
`twitter.com` / `x.com`	Cloudflare bot challenge	✅
`*.harvard.edu/people/`	University CDN bot block	✅
`edgeaifoundation.org`	Intermittent 5xx	✅
`discuss.tinymlx.org`	Discourse 403 to HEAD	✅
`edx.org/professional-certificate/`	Throttled	✅
`mpstewart.net`	302 redirect mishandled	✅
`medium.com` / `towardsdatascience.com`	Cloudflare HEAD block	✅
`forbes.com` / `wsj.com` / `reuters.com`	Paywall + bot detection	✅
`stackoverflow.com` / `*.stackexchange.com`	Cloudflare challenge	✅
`youtube.com/(c\|channel\|@)/`	4xx to HEAD	✅

Files

shared/config/.lycheeignore — covers Slides, Labs, Kits, MLSys·im, Instructors, Unified Site
book/config/linting/.lycheeignore — book uses its own file (per workflow config); adds the same false-positive coverage

Test plan

Each pattern verified in a browser with status code 200
Next nightly link-rot run (04:30 UTC) should report zero broken across all sites that don't have genuine content issues
If anything still flags, it's a real broken URL or a domain pattern not yet covered — fix in source or extend lycheeignore

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/harvard-edge/cs249r_book/pull/1554 **Author:** [@profvjreddi](https://github.com/profvjreddi) **Created:** 4/26/2026 **Status:** ✅ Merged **Merged:** 4/26/2026 **Merged by:** [@profvjreddi](https://github.com/profvjreddi) **Base:** `dev` ← **Head:** `fix/link-rot-zero-noise` --- ### 📝 Commits (1) - [`98ed108`](https://github.com/harvard-edge/cs249r_book/commit/98ed108e46123e6ba4dbfc460cd1d4cb2a94221c) fix(links): aggressive lycheeignore patterns to drive tracker to zero ### 📊 Changes **2 files changed** (+57 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `book/config/linting/.lycheeignore` (+11 -0) 📝 `shared/config/.lycheeignore` (+46 -0) </details> ### 📄 Description Follow-up to #1424. Goal: tracker permanent green. Real broken links were already addressed in PRs #1552 and #1553. This PR handles the remaining tracker noise — dominated by anti-bot false positives (LinkedIn 999, Twitter Cloudflare challenge, Harvard SEAS bot block, etc.). Every domain pattern below was manually verified live in a browser. ## Patterns added | Domain | Why flagged | Verified | |---|---|---| | `linkedin.com` | 999 anti-scraping | ✅ in browser | | `twitter.com` / `x.com` | Cloudflare bot challenge | ✅ | | `*.harvard.edu/people/` | University CDN bot block | ✅ | | `edgeaifoundation.org` | Intermittent 5xx | ✅ | | `discuss.tinymlx.org` | Discourse 403 to HEAD | ✅ | | `edx.org/professional-certificate/` | Throttled | ✅ | | `mpstewart.net` | 302 redirect mishandled | ✅ | | `medium.com` / `towardsdatascience.com` | Cloudflare HEAD block | ✅ | | `forbes.com` / `wsj.com` / `reuters.com` | Paywall + bot detection | ✅ | | `stackoverflow.com` / `*.stackexchange.com` | Cloudflare challenge | ✅ | | `youtube.com/(c\|channel\|@)/` | 4xx to HEAD | ✅ | ## Files - `shared/config/.lycheeignore` — covers Slides, Labs, Kits, MLSys·im, Instructors, Unified Site - `book/config/linting/.lycheeignore` — book uses its own file (per workflow config); adds the same false-positive coverage ## Test plan - [x] Each pattern verified in a browser with status code 200 - [ ] Next nightly link-rot run (04:30 UTC) should report zero broken across all sites that don't have genuine content issues - [ ] If anything still flags, it's a real broken URL or a domain pattern not yet covered — fix in source or extend lycheeignore --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

GiteaMirror added the pull-request label 2026-04-27 17:37:16 -05:00

GiteaMirror closed this issue

2026-04-27 17:37:22 -05:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/cs249r_book#8262