From 39eeab7dff302e8eedf7ba7b1f6a429d707bce16 Mon Sep 17 00:00:00 2001 From: Vinta Chen Date: Sun, 16 Aug 2026 15:57:15 +0800 Subject: [PATCH] docs: warn about wrong-package PyPI download counts The pypi downloads sweep looks up counts by README display name; when the display name differs from the canonical package, the row silently measures an unrelated squatter or a dead predecessor. Document the failure mode in both the fetcher's docstring and the audit skill so famous entries with off-looking counts get identity-verified before being cited. Co-Authored-By: Claude --- .claude/skills/audit-the-list/SKILL.md | 2 +- website/fetch_pypi_downloads_via_clickpy.py | 6 +++++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/.claude/skills/audit-the-list/SKILL.md b/.claude/skills/audit-the-list/SKILL.md index 961b3c7f..4c2b576c 100644 --- a/.claude/skills/audit-the-list/SKILL.md +++ b/.claude/skills/audit-the-list/SKILL.md @@ -18,7 +18,7 @@ Fetch live evidence for every entry in scope before judging anything (CLAUDE.md - **Downloads/month**: `cd website && UV_PYTHON=3.13 uv run python fetch_pypi_downloads_via_clickpy.py` — free keyless ClickPy sweep of the full README, sole writer of `data/pypi_downloads.tsv` (rewritten from scratch each run; header row, every row stamped with its `fetched_at` date). A cache whose `fetched_at` is within the last 7 days is current enough for verdicts — skip the sweep; older than that, re-run it (costs ~1s). Cross-checks print to stdout, take explicit names, and never touch the cache: `fetch_pypi_downloads_via_bigquery.py ...` (canonical source, maintainer's own GCP account, `--dry-run` first — the docstring carries the cost constraints; full-README sweeps exceed the free tier, keep name lists small), `fetch_pypi_downloads_via_pepy.py ...` (needs `PEPY_TECH_API_KEY` in repo-root `.env`, throttled to 5 requests/minute), or `https://pypistats.org/api/packages/{name}/recent` paced 8s or slower. pypistats excludes mirror/CI traffic; ClickPy, BigQuery, and pepy include it — never mix sources within one comparison. - **Repo state**: archived flag, last push, created date, stars, description — `gh api repos/{owner}/{repo}`, GitLab API for GitLab-hosted projects. -- **PyPI metadata** (`https://pypi.org/pypi/{name}/json`) wherever a name might not be the canonical package — ownership collisions and wrong display names surface here. +- **PyPI metadata** (`https://pypi.org/pypi/{name}/json`) wherever a name might not be the canonical package — ownership collisions and wrong display names surface here. Wrong-package cache rows are common: the sweep looks up README display names, so when the display name differs from the canonical package, the row silently measures an unrelated squatter or a dead predecessor. When a famous entry shows a count that looks off (too small, or absurdly round), verify identity before citing it, and fetch the canonical package's count via pepy. Done when every entry in scope has downloads (or a stated no-signal reason), repo state, and a confirmed PyPI name. diff --git a/website/fetch_pypi_downloads_via_clickpy.py b/website/fetch_pypi_downloads_via_clickpy.py index e384b365..1e7c1a26 100644 --- a/website/fetch_pypi_downloads_via_clickpy.py +++ b/website/fetch_pypi_downloads_via_clickpy.py @@ -12,7 +12,11 @@ excludes mirrors by default. This script is the sole writer of data/pypi_downloads.tsv and rewrites it from scratch each run, so entries removed from README.md drop out -naturally; names with no PyPI rows are written as NOT_FOUND. The file +naturally; names with no PyPI rows are written as NOT_FOUND. Counts are +looked up by README display name: when the display name differs from the +canonical PyPI package, the row silently measures the wrong package (a +squatter or a dead predecessor) — verify identity at +pypi.org/pypi/{name}/json before citing a count. The file starts with a header row (name, downloads, fetched_at) and every row carries the sweep date: a cache fetched within the last 7 days is current enough for audit verdicts, so only re-run when older. Cross-checks