diff --git a/.claude/skills/audit-the-list/SKILL.md b/.claude/skills/audit-the-list/SKILL.md new file mode 100644 index 00000000..6ebce16c --- /dev/null +++ b/.claude/skills/audit-the-list/SKILL.md @@ -0,0 +1,40 @@ +--- +name: audit-the-list +description: Audit README.md sections against the shortlist rules — re-verify every entry's verdict with live data, promote or demote challengers, restructure oversized use cases, prune, and evaluate proposed additions. Fire when the maintainer asks to audit, sweep, prune, re-check, or maintain sections, or asks whether an entry (or a proposed one) deserves its slot. +argument-hint: [all or specific sections] +--- + +# Audit the list + +An Audit re-runs the admission test over live sections of README.md: every entry re-verified against current data, tiers reassessed, structure reconsidered, then the maintainer adjudicates and the result is committed. The rules live in CONTRIBUTING.md (Admission, Evidence, Entry Ordering) and CONTEXT.md (vocabulary) — this skill is the process that applies them, not a second copy of them. Entry changes land only on the maintainer's explicit go. + +## 1. Scope + +Resolve the scope from the arguments. Named sections mean exactly those, whether or not they were audited before (a re-run is how a past verdict gets rechecked). `all` is ambiguous — `AskUserQuestion` whether it means every section or only never-audited ones; prior audits are recorded in git history (`git log --oneline --grep="sweep\|audit"`). With no arguments, AskUserQuestion listing the never-audited Thematic Groups. Batch the work one Thematic Group per sitting. Done when the section list is settled. + +## 2. Evidence + +Fetch live evidence for every entry in scope before judging anything (CLAUDE.md verification rule): + +- **Downloads/month**: `cd website && uv run python fetch_pypi_downloads_via_clickpy.py` — free keyless ClickPy sweep of the full README, sole writer of `data/pypi_downloads.tsv` (rewritten from scratch each run; header row, every row stamped with its `fetched_at` date). A cache whose `fetched_at` is within the last 7 days is current enough for verdicts — skip the sweep; older than that, re-run it (costs ~1s). Cross-checks print to stdout, take explicit names, and never touch the cache: `fetch_pypi_downloads_via_bigquery.py ...` (canonical source, maintainer's own GCP account, `--dry-run` first — the docstring carries the cost constraints; full-README sweeps exceed the free tier, keep name lists small), `fetch_pypi_downloads_via_pepy.py ...` (needs `PEPY_TECH_API_KEY` in repo-root `.env`, throttled to 5 requests/minute), or `https://pypistats.org/api/packages/{name}/recent` paced 8s or slower. pypistats excludes mirror/CI traffic; ClickPy, BigQuery, and pepy include it — never mix sources within one comparison. +- **Repo state**: archived flag, last push, created date, stars, description — `gh api repos/{owner}/{repo}`, GitLab API for GitLab-hosted projects. +- **PyPI metadata** (`https://pypi.org/pypi/{name}/json`) wherever a name might not be the canonical package — ownership collisions and wrong display names surface here. Wrong-package cache rows are common: the sweep looks up README display names, so when the display name differs from the canonical package, the row silently measures an unrelated squatter or a dead predecessor. When a famous entry shows a count that looks off (too small, or absurdly round), verify identity before citing it, and fetch the canonical package's count via pepy. +- **Renaming an entry can silently expose it to the sweep, and no automatic check catches that.** The sweep only queries names matching `PYPI_NAME_RE`, so a display name carrying a space or other non-PyPI shape is skipped outright. Rename it into a PyPI-shaped one and it starts being queried: `uv audit` became `uv-audit`, and PyPI `uv-audit` is an unrelated third-party tool by rocshers, not Astral's subcommand. After any rename, fetch `https://pypi.org/pypi/{new-name}/json` and confirm the package is the project the entry links to; when it is not, add a null entry to `website/data/pypi_name_overrides.json` with the reason. Only one case is guarded automatically: `test_bundled_entries_are_never_queried_on_pypi` requires a null override for every `(part of X)` entry. Two broader checks were measured against the full list and rejected — do not re-propose either. Requiring a package's PyPI metadata to link back to the entry's GitHub repo misses this case completely, since `uv-audit` declares no repo URL at all, as do 26 legitimate entries including numba, selenium and pyglet; it also flags 10 benign org moves and binding splits (gensim, msgspec, pendulum, instructor, duckdb). Flagging entries whose display name differs from their repo name returns 46 hits, every one legitimate, and `uv-build` sits among them with the shape identical to `uv-audit` despite being a real Astral package. + +Done when every entry in scope has downloads (or a stated no-signal reason), repo state, and a confirmed PyPI name. + +## 3. Verdicts + +Draft a verdict and reason for every entry, restructure before cap: decide Splits, mints, and re-homes first — cutting before restructuring destroys entries — then apply the cap tier by tier, including promotions and demotions between obvious choice and challenger. Ground every reason in the fetched evidence; label anything unverifiable as a judgment call. Where fresh evidence contradicts the standing verdict, say so in the reason instead of silently keeping the seed. + +## 4. Review and go + +Run the `preview-verdicts` skill: it generates the interactive review page and defines how the maintainer's feedback JSON comes back and gets processed. Their verdicts are final. Before touching README.md, surface what the feedback implies but does not decide — cap overflows, homeless entries after a Split, tier for a tierless flip — as named assumptions or questions, then get an explicit go. + +## 5. Execute + +One commit per section: body lists each removal with its reason and downloads figure, naming the figure's source and fetch date so a later audit can tell whether a trend compares like with like (ClickPy counts mirror traffic, pypistats does not); restructures, tier moves, and reorders ride the same commit. A restructure that renames or dissolves a category or subcategory adds `"old path": "new path"` to `website/data/redirects.json` in that commit, so the old URL keeps its search ranking; the build fails when a redirect target no longer exists, so a later rename also updates the entries pointing at the renamed path. Format-only outcomes (no removals) are a single style commit. `make test` before every commit. After the last add or rename, re-run the ClickPy sweep, since new names have no cache row and the build would count them without downloads, then `make build`. Generic commit helpers tend to split a section audit into structural and per-subcategory commits — if that happens, squash back to one commit per section. Done when the tree is clean, tests passed before each commit, and the build count reconciles with the adjudicated changes. + +## 6. Record + +A conclusion that outlives the sitting goes into the repo before the audit ends: admission or evidence rules into CONTRIBUTING.md, repo process and environment facts into CLAUDE.md and AGENTS.md (kept in sync), vocabulary into CONTEXT.md. Done when nothing the next audit needs is stranded in conversation. diff --git a/.claude/skills/preview-verdicts/SKILL.md b/.claude/skills/preview-verdicts/SKILL.md new file mode 100644 index 00000000..2f44ff11 --- /dev/null +++ b/.claude/skills/preview-verdicts/SKILL.md @@ -0,0 +1,24 @@ +--- +name: preview-verdicts +description: Generate the interactive keep/drop/add verdict preview (HTML page with per-row feedback controls) whenever a prune sweep, batch entry edit, or restructure needs maintainer review before touching README.md — and process the feedback JSON the maintainer pastes back. +--- + +# Verdict preview + +Maintainer review happens through an interactive HTML page: one row per entry with your seeded verdict and reason, Keep/Drop buttons (Add/Drop on proposed new entries) and a reason field for the maintainer, and a **Copy feedback** button that exports only changed or commented rows as JSON. Generate the page, wait for the pasted JSON, then apply it. Entry changes land in README.md only after the review — and only on an explicit go. + +## Generate the preview + +1. Build the `DATA` array. A group is `[section, subcategory, rows]`; a row is `[entry, url, downloads, verdict, reason]`. + - `subcategory` may carry a note after ` — ` (rendered muted): use it for proposed splits, re-homes, or anything the maintainer should weigh for the whole group. + - `downloads` is PyPI last-month as a comma-formatted string; use `—` when no signal exists (e.g. agent skill packs), `stdlib` for standard-library modules, `fetch failed` when the lookup failed. State the fetch date in the sub-header. + - `verdict` is `keep`, `drop`, or `add`, seeded from the current adjudication or dry-run. `add` marks a proposed new entry; its row gets Add/Drop buttons, where Drop means not added. Show a replacement as two rows: the old entry `drop`, the new entry `add`. + - `reason` is plain language the maintainer reads cold — no invented shorthand. When fresh evidence contradicts the seeded verdict (a big download count on a drop, a dead repo on a keep), say so in that row's reason instead of silently changing the seed. +2. Copy `template.html` (sibling of this file) and replace the placeholders: `__TITLE__` (page title), `__SUB__` (sub-header: scope, seed provenance, fetch date, and the standing instruction to flip/comment then Copy feedback), `__KEY__` (localStorage key), `__DATA__` (the array). `__KEY__` must be unique per review — slug plus date, e.g. `awesome-python-science-2026-09-01` — because saved state under a reused key bleeds a previous review's flips into rows with the same section and entry name. +3. Write the page to `tmp/awesome-python--preview.html` in the repo root (git-ignored; create the directory if needed), `open` it, and tell the maintainer the path and the return path: flip or comment rows (they highlight yellow), press **Copy feedback**, paste the JSON into the chat. Done when the page is open and the return path is stated. + +## Process the pasted feedback + +Each JSON row is `{section, subcategory, entry, my_verdict, your_verdict, reason}`. On a row seeded `add`, `your_verdict: "drop"` means do not add it. The maintainer's verdict is final — apply it, never re-argue it. An empty reason means the verdict stands unexplained; that is enough. + +Before executing, surface anything the flips imply that the maintainer has not decided: a use case pushed past its cap, an entry left homeless by a proposed split, a request that is already satisfied (a no-op). Ask, then execute on their go. Done when every pasted row is either applied or surfaced back — none silently dropped. diff --git a/.claude/skills/preview-verdicts/template.html b/.claude/skills/preview-verdicts/template.html new file mode 100644 index 00000000..980fbc7e --- /dev/null +++ b/.claude/skills/preview-verdicts/template.html @@ -0,0 +1,163 @@ + + + + + +__TITLE__ + + + +
+

__TITLE__

+

__SUB__

+
+ +
+
+ + + Copies only rows you changed or commented, as JSON. +
+ + + diff --git a/.claude/skills/review-prs/SKILL.md b/.claude/skills/review-prs/SKILL.md new file mode 100644 index 00000000..f5c3c890 --- /dev/null +++ b/.claude/skills/review-prs/SKILL.md @@ -0,0 +1,35 @@ +--- +name: review-prs +description: Triage open PRs — screen from the diff, delegate the admission judgment to audit-the-list, then merge or close on GitHub. Fire when the maintainer asks to review PRs, process the PR queue, or judge whether a specific PR should be merged. +argument-hint: [PR numbers] +--- + +# Review PRs + +A PR review turns the open queue into terminal states: merged, closed, or explicitly parked. The rules live in CONTRIBUTING.md (Quality Requirements, Admission, Review Process, Automatic Rejection) — this skill is the workflow that applies them, not a second copy. Every PR named in chat is a PR link, `[#](https://github.com/vinta/awesome-python/pull/)`, so the maintainer can open it from the terminal. + +## 1. Fetch + +Arguments name specific PRs; otherwise take the queue: `gh pr list --repo vinta/awesome-python --limit 10 --json number,title,author,url,body,files,mergeable,mergeStateStatus`. Fetch all diffs in parallel: `gh pr diff --repo vinta/awesome-python`. + +Sort the batch: entry additions continue; anything else (typo fixes, website changes, docs) is out of scope — report it under "needs human" and touch nothing, so it resurfaces every run until a human acts. Done when every PR is sorted and has its diff. + +## 2. Screen + +Apply the Automatic Rejection rules that the diff and PR metadata answer without judgment. The recently-closed-duplicate rule needs a lookup: `gh pr list --repo vinta/awesome-python --state closed --search "" --limit 10`. A screened-out PR goes straight to step 4 as a close, with the rule it broke as its reason. + +Resolve each survivor's target use case against the current README — a diff's context lines show the base the PR was written on, which may have changed since. A merge conflict rides to the Merge arm, which absorbs it. Done when every surviving PR names its target use case. + +## 3. Judge + +Group survivors by target use case — PRs proposing entries for the same use case compete for the same slots, so they ride one invocation. Per group, invoke the `audit-the-list` skill with arguments in this shape: "Judge proposed entries for the
use case. Evidence and Verdicts steps only; report the verdicts back. No preview page, no README changes, no commits." The verdict for each PR is merge or close, grounded in the fetched evidence; when the use case is at cap, a merge verdict names the entry that leaves. An entry no current use case fits is a third outcome — a structure question: carry it to Act with evidence attached, and the maintainer decides there (mint the subcategory and merge, close, or leave open). Done when every survivor holds a verdict with a reason. + +## 4. Act + +- **Close**: draft each closing comment — it states the reason and links CONTRIBUTING.md. Per batch of up to 4 closing PRs: print each draft in chat, its PR link as a label above a fenced block holding only the comment bytes, then AskUserQuestion over that batch — arms: close with the draft as printed, close without a comment, keep open. The chat print is the review copy; an option description only points at it. Keep a checklist of which verdicts have been asked; answers often arrive as custom text, and that text is the decision — an edited comment replaces the draft. Then `gh pr close --repo vinta/awesome-python --comment ""`, or a plain close. +- **Merge**: a clean PR merges with `gh pr merge --repo vinta/awesome-python --merge`. A conflicted one merges locally: `git fetch origin pull//head`, `git merge FETCH_HEAD` with the standard `Merge pull request # from /` message, resolving the conflict by placing the entry correctly — GitHub still marks the PR merged and the contributor keeps credit. Either way, reconcile the section per CONTRIBUTING before pushing: remove the entry the verdict displaced, fix the new entry's display name and Entry Ordering position, `make test`, commit. An add-only diff that displaces is the normal case — the removal is this step's job, not the contributor's. +Done when every verdict has been adjudicated by the maintainer and its action executed, with every posted comment byte-identical to a printed draft or to the custom text the maintainer supplied. A kept-open PR resurfaces next run — that is its point. + +## 5. Report + +Summary table: PR link, verdict, action taken, plus the needs-human list. Done when every fetched PR ends in exactly one state — merged with its section reconciled, closed, or left open (kept open, needs-human, or structure question pending). diff --git a/.github/FUNDING.yml b/.github/FUNDING.yml index aa22b9d0..7d0bb19f 100644 --- a/.github/FUNDING.yml +++ b/.github/FUNDING.yml @@ -1 +1,2 @@ -# These are supported funding model platforms +open_collective: awesome-python +custom: https://awesome-python.com/sponsorship/ diff --git a/.github/ISSUE_TEMPLATE/add-new-library-to-this-list.md b/.github/ISSUE_TEMPLATE/add-new-library-to-this-list.md deleted file mode 100644 index eba33ff6..00000000 --- a/.github/ISSUE_TEMPLATE/add-new-library-to-this-list.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -name: Add new library to this list -about: '' -title: '' -labels: '' -assignees: '' - ---- - -**Please open a Pull Request instead.** diff --git a/.github/ISSUE_TEMPLATE/add-xxx.md b/.github/ISSUE_TEMPLATE/add-xxx.md deleted file mode 100644 index 585f4919..00000000 --- a/.github/ISSUE_TEMPLATE/add-xxx.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -name: Add XXX -about: Add new item to the list -title: '' -labels: '' -assignees: '' - ---- - -**Please open a Pull Request instead.** diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index fdacb788..41890f46 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -1,11 +1,26 @@ -## What is this Python project? +## Project -Describe features. +[project-name](url) -## What's the difference between this Python project and similar ones? +## Checklist -Enumerate comparisons. +- [ ] I read [CONTRIBUTING.md](https://github.com/vinta/awesome-python/blob/master/CONTRIBUTING.md) - awesome-python is a shortlist, not a catalog +- [ ] One project per PR +- [ ] PR title format: `Add project-name` +- [ ] Entry format: `- [pypi-name](https://github.com/owner/repo) - Description ending with period.` +- [ ] Display name is the PyPI package name +- [ ] Placed in an existing use case (new sections and subcategories are maintainer-only) +- [ ] Meets all Quality Requirements: active, stable, documented, at least 1 month old --- +## Which Tier -Anyone who agrees with this pull request could submit an *Approve* review to it. +Pick one: + +- [ ] **Obvious choice** - a tool an experienced Python developer would name when asked "what do I use for this?" +- [ ] **Challenger** - not yet the obvious choice, but a credible successor to one. Give adoption-trajectory evidence, not popularity alone. + +Explain: + +## Displacement + +If the use case is at its cap, name the entry your project replaces and argue that yours does that entry's job better. One in, one out. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 00000000..fc8084bc --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,33 @@ +name: CI + +on: + push: + branches: [master] + pull_request: + +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }} + cancel-in-progress: true + +jobs: + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + + - name: Install uv + uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0 + with: + enable-cache: true + + - name: Install dependencies + run: uv sync --group build --locked + + - name: Run tests + run: make test + + - name: Build website + run: make build diff --git a/.github/workflows/deploy-website.yml b/.github/workflows/deploy-website.yml new file mode 100644 index 00000000..4f5f28c7 --- /dev/null +++ b/.github/workflows/deploy-website.yml @@ -0,0 +1,105 @@ +name: Deploy Website + +on: + push: + branches: [master] + schedule: + - cron: "0 0 * * *" + +permissions: + contents: read + pages: write + id-token: write + +concurrency: + group: pages + cancel-in-progress: false + +jobs: + deploy: + if: github.repository == 'vinta/awesome-python' + runs-on: ubuntu-latest + environment: + name: github-pages + url: https://awesome-python.com/ + steps: + - uses: actions/checkout@v7 + with: + # build.py reads each page's last change from git history for the sitemap's lastmod + fetch-depth: 0 + + - name: Install uv + uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0 + with: + enable-cache: true + + - name: Install dependencies + run: uv sync --group build --locked + + - name: Run tests + if: github.event_name == 'schedule' + run: make test + + - name: Get date + id: date + run: echo "today=$(date -u +%Y-%m-%d)" >> "$GITHUB_OUTPUT" + + - name: Restore star data cache + id: cache-stars + uses: actions/cache/restore@v6 + with: + path: website/data/github_stars.json + key: github-stars-${{ steps.date.outputs.today }} + restore-keys: github-stars- + + - name: Fetch GitHub stars + id: fetch-stars + continue-on-error: true + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: make fetch_github_stars + + - name: Save star data cache + if: steps.fetch-stars.outcome == 'success' + uses: actions/cache/save@v6 + with: + path: website/data/github_stars.json + key: github-stars-${{ steps.date.outputs.today }} + + - name: Verify star data exists + run: | + if [ ! -f website/data/github_stars.json ]; then + echo "::error::github_stars.json not found. No cache and fetch failed or was skipped." + exit 1 + fi + python -m json.tool website/data/github_stars.json > /dev/null + + - name: Restore PyPI download data cache + uses: actions/cache/restore@v6 + with: + path: website/data/pypi_downloads.tsv + key: pypi-downloads-${{ steps.date.outputs.today }} + restore-keys: pypi-downloads- + + - name: Fetch PyPI downloads + id: fetch-downloads + continue-on-error: true + run: make fetch_pypi_downloads + + - name: Save PyPI download data cache + if: steps.fetch-downloads.outcome == 'success' + uses: actions/cache/save@v6 + with: + path: website/data/pypi_downloads.tsv + key: pypi-downloads-${{ steps.date.outputs.today }} + + - name: Build website + run: make build + + - name: Upload artifact + uses: actions/upload-pages-artifact@v5 + with: + path: website/output/ + + - name: Deploy to GitHub Pages + uses: actions/deploy-pages@v5 diff --git a/.gitignore b/.gitignore index 096c327d..de692147 100644 --- a/.gitignore +++ b/.gitignore @@ -1,9 +1,21 @@ +# secrets +.env* + +# macOS .DS_Store +# python +.venv/ +__pycache__/ *.py[co] -docs/index.md -site/ +# website +website/output/ +website/data/* +!website/data/pypi_name_overrides.json +!website/data/redirects.json +!website/data/category_intros/ -# PyCharm IDE -.idea +# agents +.playwright-cli/ +tmp/ diff --git a/.travis.yml b/.travis.yml deleted file mode 100644 index e1c0a23d..00000000 --- a/.travis.yml +++ /dev/null @@ -1,18 +0,0 @@ -language: python - -python: - - "3.6" - -script: - - python sort.py - - cp README.md docs/index.md - - mkdocs build - -deploy: - provider: pages - local-dir: site - skip-cleanup: true - keep-history: true - github-token: $GITHUB_TOKEN - on: - branch: master diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..01e007a9 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,23 @@ +# AGENTS.md + +An opinionated guide to the best Python frameworks, libraries, and tools. + +[README.md](README.md) is the single source of content truth; `website/` renders it into the static site: [awesome-python.com](https://awesome-python.com/). + +## Entry Rules + +[CONTRIBUTING.md](CONTRIBUTING.md) holds the admission rules, quality requirements, rejection rules, entry format, and ordering. Apply it whenever adding or removing an entry — direct commits included, not only PR reviews. + +- Every keep/drop reason must be verified against current online data at decision time — download counts, repo activity and archived status, PyPI metadata, project docs. Judging tiers — obvious choice vs challenger — also requires WebSearch evidence (adoption trajectory, community sentiment), not download counts alone. Training-data recollections are not evidence; label anything unverifiable as a judgment call. +- Checks stay read-only: judge whether a listed project installs or works from its PyPI metadata (`requires_python`, classifiers, wheel tags) and issue tracker, never by installing or running it, since that runs untrusted code. +- A download count is not automatically independent demand. When one listed entry depends on another, check `requires_dist` on PyPI before citing the depended-on entry's count: mkdocs-material hard-depends on mkdocs, so mkdocs' figure exceeded mkdocs-material's by only about one percent and nearly all of it was mkdocs-material pulling it in. +- One entry per commit when adding or deleting entries. Exceptions: a prune sweep is one commit per section, its body listing each removal with its reason; format, wording, or categorization changes may be bundled. Cross-section re-homes ride the originating audit's commit (both sides of the move in one diff). +- Resources sections are not project entries: out of audit scope, and the website never parses them. +- Sponsor placement never influences which projects get listed — see [SPONSORSHIP.md](SPONSORSHIP.md). + +## Website + +- Model the layout on https://www.placestoread.xyz: the whole list on one page, dense rows, a row that expands inline with its content aligned to the Name column, sorting by column headers, full-text search, tags that filter, a divider under the header row instead of a strong top border, a solid dark footer, and minimal decoration. Keep that model (no card grid, modal details, or pagination) unless the maintainer asks, and keep green out of the palette. +- `.section-shell` (`--shell-max: 84rem`) is the only width cap: sections, tables, rows, CTAs, and paragraphs stay uncapped, since the maintainer reads on wide screens and prose-width advice like 65-75ch doesn't apply. Remove inner `max-width` rules you come across, and ask with a concrete reason before adding one. +- Pick font sizes one step larger than feels right (`--text-base` over `--text-sm`, 1.75rem over 1.5rem for a heading), and shrink an existing size only when the maintainer asks: they asked for bigger type 11 times across 8 sessions and never for smaller. +- When you style one link, tag, or label, give its peers the same style in the same change: hero, nav, footer, project, and sponsor links share hover styles, and tag variants build on `.tag`. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..22541c01 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,23 @@ +# CLAUDE.md + +An opinionated guide to the best Python frameworks, libraries, and tools. + +[README.md](README.md) is the single source of content truth; `website/` renders it into the static site: [awesome-python.com](https://awesome-python.com/). + +## Entry Rules + +[CONTRIBUTING.md](CONTRIBUTING.md) holds the admission rules, quality requirements, rejection rules, entry format, and ordering. Apply it whenever adding or removing an entry — direct commits included, not only PR reviews. + +- Every keep/drop reason must be verified against current online data at decision time — download counts, repo activity and archived status, PyPI metadata, project docs. Judging tiers — obvious choice vs challenger — also requires WebSearch evidence (adoption trajectory, community sentiment), not download counts alone. Training-data recollections are not evidence; label anything unverifiable as a judgment call. +- Checks stay read-only: judge whether a listed project installs or works from its PyPI metadata (`requires_python`, classifiers, wheel tags) and issue tracker, never by installing or running it, since that runs untrusted code. +- A download count is not automatically independent demand. When one listed entry depends on another, check `requires_dist` on PyPI before citing the depended-on entry's count: mkdocs-material hard-depends on mkdocs, so mkdocs' figure exceeded mkdocs-material's by only about one percent and nearly all of it was mkdocs-material pulling it in. +- One entry per commit when adding or deleting entries. Exceptions: a prune sweep is one commit per section, its body listing each removal with its reason; format, wording, or categorization changes may be bundled. Cross-section re-homes ride the originating audit's commit (both sides of the move in one diff). +- Resources sections are not project entries: out of audit scope, and the website never parses them. +- Sponsor placement never influences which projects get listed — see [SPONSORSHIP.md](SPONSORSHIP.md). + +## Website + +- Model the layout on https://www.placestoread.xyz: the whole list on one page, dense rows, a row that expands inline with its content aligned to the Name column, sorting by column headers, full-text search, tags that filter, a divider under the header row instead of a strong top border, a solid dark footer, and minimal decoration. Keep that model (no card grid, modal details, or pagination) unless the maintainer asks, and keep green out of the palette. +- `.section-shell` (`--shell-max: 84rem`) is the only width cap: sections, tables, rows, CTAs, and paragraphs stay uncapped, since the maintainer reads on wide screens and prose-width advice like 65-75ch doesn't apply. Remove inner `max-width` rules you come across, and ask with a concrete reason before adding one. +- Pick font sizes one step larger than feels right (`--text-base` over `--text-sm`, 1.75rem over 1.5rem for a heading), and shrink an existing size only when the maintainer asks: they asked for bigger type 11 times across 8 sessions and never for smaller. +- When you style one link, tag, or label, give its peers the same style in the same change: hero, nav, footer, project, and sponsor links share hover styles, and tag variants build on `.tag`. diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 00000000..f13d736d --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,40 @@ +# Contributor Covenant Code of Conduct + +## Our Pledge + +We as members, contributors, and leaders pledge to make participation in our +community a harassment-free experience for everyone, regardless of age, body +size, visible or invisible disability, ethnicity, sex characteristics, gender +identity and expression, level of experience, education, socio-economic status, +nationality, personal appearance, race, religion, or sexual identity and +orientation. + +## Our Standards + +Examples of behavior that contributes to a positive environment: + +- Using welcoming and inclusive language +- Being respectful of differing viewpoints and experiences +- Gracefully accepting constructive criticism +- Focusing on what is best for the community +- Showing empathy towards other community members + +Examples of unacceptable behavior: + +- The use of sexualized language or imagery +- Trolling, insulting or derogatory comments +- Public or private harassment +- Publishing others' private information without permission +- Other conduct which could reasonably be considered inappropriate + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be +reported to the project maintainers. All complaints will be reviewed and +investigated and will result in a response that is deemed necessary and +appropriate to the circumstances. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant](https://www.contributor-covenant.org), +version 2.0. diff --git a/CONTEXT.md b/CONTEXT.md new file mode 100644 index 00000000..8d489c10 --- /dev/null +++ b/CONTEXT.md @@ -0,0 +1,67 @@ +# Awesome Python Curation + +The editorial domain of awesome-python: which projects the list carries, how they are admitted, and how they are displaced. The list is a curated shortlist of obvious choices, not a catalog of everything good. + +## Language + +### List structure + +**Entry**: +A single listed project: `- [name](url) - Description.` The unit that is admitted, displaced, or pruned. Named by PyPI package name when one exists, else by repository name. + +**Sub-item**: +An indented awesome-* also-see link under an Entry (awesome-django under django, awesome-pytest under pytest). Not an Entry: it holds no slot, does not count against the Cap, and rides its parent — re-homed or removed with it. Maintainer-only placement, and only for awesome-* lists: a companion project either earns a full Entry in its proper Use Case or is not listed. + +**Bundled Entry**: +An Entry for a feature that ships inside a larger library, framework, or tool rather than as its own PyPI package (django.db.models, geodjango, httpx.URL, uv audit). Marked in README.md by a `(part of X)` description prefix, which the website reads to render a "Bundled" badge where a download count would go. It holds a normal slot and is audited like any other Entry; only its download signal is absent. +_Avoid_: Pointer entry (reform-era name) + +**Thematic Group**: +A bold group line (e.g. "AI & ML", "Web Development") that clusters Sections in the TOC and the Projects body. + +**Section**: +A `###` heading in README.md (e.g. "Testing", "AI and Agents"). Sections group entries and live under a Thematic Group. +_Avoid_: Category (overloaded — the TOC calls them categories, but rules bind to Use Cases, not Sections) + +**Subcategory**: +A named bullet inside a Section with indented entries under it (e.g. "Mock" inside "Testing"). Where present, each Subcategory is one Use Case. + +**Use Case**: +The unit of "it" in "one obvious way to do it": one distinct job a reader needs done. Defined by the list's structure — each Subcategory is a Use Case; a flat Section is a single Use Case. Submitters cannot define Use Cases; only the maintainer changes the structure (see Split). +_Avoid_: Niche, micro-category + +### Admission + +**Serves Python Developers**: +The scope test for what belongs on the list at all: Python developers use it in their Python work. Implementation language and packaging are irrelevant — uv and ty are Rust and belong; a pure-Python library nobody uses in Python work does not. +_Avoid_: Python-first, written-in-Python (old requirement — removed) + +**Obvious Choice**: +An entry an experienced Python developer would name unprompted when asked "what do I use for [Use Case]?". Certified by maintainer editorial judgment, informed primarily by PyPI download counts rather than GitHub stars; judgment overrides the signal's known failure modes (CI-inflated counts, model releases consumed as weights rather than pip installs — extended to any project consumed outside pip: SDK downloads like renpy, deployed services like thumbor — and large-but-specific audiences misread as "niche"). A standard-library module holds a slot only when it is itself the Obvious Choice for the Use Case, not merely relevant to it. +_Avoid_: Industry Standard (old lane name), awesome (unfalsifiable) + +**Cap**: +The per-Use-Case entry limit: up to 3 Obvious Choices plus up to 2 Challengers, hard maximum 5. A qualitative bar first, a numeric backstop second — and a ceiling, not a floor: a minted Use Case may hold a single entry. + +**Displacement**: +The only admission path into a full Use Case: the PR names the entry it replaces and argues the newcomer does that entry's job better. One in, one out. +_Avoid_: One-in-one-out (informal alias) + +**Challenger**: +An entry (at most two per Use Case) that is not yet the Obvious Choice but is a credible successor to one. Marked by ordering, not description: within a Use Case, Obvious Choices are listed first, then Challengers, each tier by PyPI downloads/month descending (standard-library modules first in the Use Case outright — always Obvious Choices by the stdlib rule — other no-signal entries last in tier, alphabetically). Admission requires adoption-trajectory evidence, not popularity alone. +_Avoid_: Rising Star (old lane name), Hidden Gem (old lane name — concept removed entirely) + +**Second Tier**: +A Challenger slot holding a demoted incumbent instead of a rising successor — a former Obvious Choice or fading community standard an Audit kept below the current Obvious Choices (clickhouse-driver behind the official client, dogpile.cache and django-cacheops behind cachetools and diskcache, django-haystack in Search). It counts against the two Challenger slots and is marked by position like any Challenger; the adoption-trajectory bar gates only new admissions, not demotions. + +**Override**: +A maintainer decision, made in an Audit and explicitly recorded, that keeps an entry or use-case shape past a written limit — the activity line, the stability bar, a cap, or a structure rule. Case-by-case: an Override never loosens the written rule for any other entry, and submitters cannot cite one. + +**Split**: +Maintainer-only restructuring of an oversized Use Case into finer Use Cases (new Subcategories), considered before any trimming when the size reflects genuinely distinct jobs. A submitter's entry PR can never perform a Split. + +### Maintenance + +**Audit**: +The recurring maintenance pass over one or more Sections: every Entry's verdict re-verified against current evidence, Challengers promoted or demoted, oversized Use Cases Split, drops and admissions re-decided. The maintainer adjudicates through an interactive preview, and entry changes land only on their explicit go. The 2026-08 shortlist-reform sweeps were the first Audits. +_Avoid_: Sweep (reform-era name for a Section's first Audit) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 943f93cb..1ea381d1 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,20 +1,128 @@ # Contributing -Your contributions are always welcome! +awesome-python is a shortlist, not a catalog. Each use case lists only its obvious choices, and most rejections mean "the use case is full", not "your project is bad". Read this whole page before opening a PR. -## Guidelines +## Quality Requirements -* Add one link per Pull Request. - * Make sure the PR title is in the format of `Add project-name`. - * Write down the reason why the library is awesome. -* Add the link: `* [project-name](http://example.com/) - A short description ends with a period.` - * Keep descriptions concise and **short**. -* Add a section if needed. - * Add the section description. - * Add the section title to Table of Contents. -* Search previous Pull Requests or Issues before making a new one, as yours may be a duplicate. -* Don't mention `Python` in the description as it's implied. -* Check your spelling and grammar. -* Remove any trailing whitespace. +All submissions must satisfy **ALL** of these: -Just a gentle reminder: **Try not to submit your own project. Instead, wait for someone finds it useful and submits it for you.** +1. **Serves Python Developers**: Python developers use it in their Python work. Implementation language and packaging are irrelevant — uv and ty are written in Rust, and agent skill packs are markdown, yet all belong; a pure-Python project nobody uses in Python work does not. +2. **Active**: Commits within the last 12 months +3. **Stable**: Production-ready, not alpha/beta/experimental. A PyPI "Development Status" classifier alone does not decide this; judge by releases, documentation, and production use. +4. **Documented**: Clear README with examples and use cases +5. **Established**: Repository at least 1 month old + +## Admission + +A **use case** is one distinct job a reader needs done. Use cases are defined by the list's structure: each subcategory is a use case, and a section without subcategories is a single use case. Structure changes — new sections, new subcategories, splitting an oversized use case into finer ones — are made only by the maintainer; an entry PR can never create the subcategory it needs. + +Each use case lists at most: + +- **Up to 3 obvious choices** — tools an experienced Python developer would name when asked "what do I use for this?" +- **Up to 2 challengers** — tools that are not yet the obvious choice but are credible successors to one. Admission as a challenger requires adoption-trajectory evidence, not popularity alone. + +Hard maximum: 5 entries per use case. This is a qualitative bar first and a numeric backstop second — most use cases should carry fewer. + +**Overrides**: the maintainer may exceed any limit on this page — the caps, the activity requirement, the stability requirement — for a specific entry or use case by explicit decision. An override is case-by-case; it does not loosen these rules for submissions, and citing one in a PR carries no weight. + +**Displacement**: once a use case is at its cap, the only way in is to name the entry your project replaces and argue that yours does that entry's job better. One in, one out. + +**Dual-listing**: a tool may hold entries in multiple use cases, but only when it is an obvious choice in each. List the full entry in each home with identical lines; never a "see X above" note, since the website only renders list items. Description edits update every copy in the same commit. Each slot is audited on its own: dropping one home keeps the other, and dropping the tool entirely removes all copies in one commit. Dual-listing is a maintainer decision; a PR adding a second home for an existing entry is treated as a duplicate. + +**Standard library**: a standard-library module is listed only where the stdlib is itself the obvious choice for the use case (tomllib yes, unittest no). + +**Python versions**: missing support for the newest Python releases is not by itself grounds for removal while the entry still has a large user base; it can place the entry in the Second Tier instead. + +**Evidence**: admission is decided by maintainer editorial judgment, informed primarily by PyPI download counts rather than GitHub stars. Judgment overrides the signal's known failure modes (CI-inflated counts, model releases consumed as weights rather than pip installs, large-but-specific audiences misread as "niche"). The maintainer's decision is final. + +Looking for an exhaustive catalog instead? Follow the awesome-\* lists linked under individual entries (for example awesome-python-testing) — they exist precisely so this list doesn't have to be one. + +## Entry Format Reference + +**Use GitHub repository URLs** whenever possible. Projects linked to a GitHub repo are ranked higher on [awesome-python.com](https://awesome-python.com/). + +### Naming Convention + +Use the **PyPI package name** as the display name so developers can copy it directly to `pip install`. Check the canonical name at `https://pypi.org/pypi/{package}/json`. If the project is not on PyPI, use the GitHub repository name instead. + +### Standard Entry + +```markdown +- [pypi-name](https://github.com/owner/repo) - Description ending with period. +``` + +### Standard Library Module + +```markdown +- [module](https://docs.python.org/3/library/module.html) - (Python standard library) Description. +``` + +### Bundled Feature + +A feature that ships inside a larger library, framework, or tool rather than as its own PyPI package. The `(part of X)` prefix is parsed by the website, which renders a "Bundled" badge in place of a download count, so write it exactly in that form and name the parent as developers say it. + +```markdown +- [name](url) - (part of Parent) Description ending with period. +``` + +### Fork of Another Project + +```markdown +- [new-name](https://github.com/owner/new-name) - Description ([original-name](original-url) fork). +``` + +### Entry with Related Awesome List + +```markdown +- [project](https://github.com/owner/project) - Description. + - [awesome-project](https://github.com/someone/awesome-project) +``` + +### Subcategory Format + +```markdown +- Subcategory Name + - [project](url) - Description. +``` + +### Entry Ordering + +Within a use case, the obvious choices are listed first, ordered by PyPI downloads per month from high to low; challengers follow, in the same order. Standard-library modules sort first in the use case — built-ins lead, before everything else — alphabetically when there are several. (This never conflicts with tier order: a standard-library module is only listed where it is itself the obvious choice, so it always belongs to the first tier.) Other entries without a download signal (agent skill packs, projects distributed outside PyPI) sort last within their tier, alphabetically. There is no marker in the entry text — position is the marker, so the last entries of a use case may be its challengers. + +## Changing the Structure + +Adding sections or subcategories is maintainer-only (see Admission). For maintainer reference: + +1. Add the section description in italics: `*Libraries for doing X.*` +2. Add the section under the appropriate thematic group (e.g., **AI & ML**, **Web**, **Data & Science**) +3. Add the section title to the Table of Contents under its group +4. Order entries per Entry Ordering above + +A subcategory never repeats its section's name: one named like its section competes with the section page in search results. Name the section's catch-all or main use case General instead. + +## Review Process + +PRs are reviewed by automated tools and maintainers: + +1. **Format Check**: Entry follows the correct format +2. **Category Check**: Placed in the appropriate use case +3. **Duplicate Check**: Not already listed or previously rejected +4. **Activity Check**: Project shows recent activity +5. **Admission Check**: Meets the Admission rules above, including Displacement when the use case is at its cap + +Search previous Pull Requests and Issues before submitting, as yours may be a duplicate. + +## Automatic Rejection + +PRs will be **closed** if: + +- Adding multiple projects in one PR +- The use case is at its cap and the PR makes no Displacement argument +- The PR creates a new section or subcategory and fills it (structure changes are maintainer-only) +- Coordinated multi-entry self-promotion: multiple related projects from the same organization or author, across one or several PRs +- Duplicate of existing entry or recently-closed PR +- Empty or placeholder PR descriptions +- Placed under an inappropriate category +- Project is archived or abandoned (no commits in 12+ months) +- No documentation or unclear use case +- Repository less than 1 month old diff --git a/DESIGN.md b/DESIGN.md new file mode 100644 index 00000000..56ea32bc --- /dev/null +++ b/DESIGN.md @@ -0,0 +1,226 @@ +--- +version: alpha +name: awesome-python.com +description: Warm editorial Python index. Light cream canvas, brown-red interactive accent, Cormorant Garamond plus Manrope, table-driven single-page reference. +--- + +# awesome-python.com DESIGN.md + +awesome-python.com is a searchable, filterable index of ~650 curated Python projects. It is a reference tool, not a landing page and not a GitHub README mirror. + +This file follows the [Google Stitch DESIGN.md format](https://stitch.withgoogle.com/docs/design-md/overview/). The source of truth for token values lives in `website/static/style.css`. Color tokens here are written in OKLCH because the project mandates OKLCH over hex, which is a deliberate divergence from the spec's hex-only token requirement. + +## Overview + +Three words: **opinionated, confident, dense**. + +Working Python developers (mid to senior) are the target reader. They write Python daily and arrive with a specific question in mind: "what's a good HTTP client these days", "is there still a maintained ORM for X", "what are people using for task queues now". Secondary readers: polyglot developers evaluating Python's ecosystem, and curious browsers. + +Jobs to be done: + +1. Find a library for a specific need fast (search + tag filter). +2. Compare candidates at a glance (stars, last commit, tags, one-line description). +3. Confirm a project is alive before clicking through. + +These users skim. They reward density and terse copy. They penalize marketing fluff. + +Voice: + +- Editorial. Every word earns its place. +- Confident, not combative. "This is the list" energy, not "check out these cool projects". +- No hype. The content is what's interesting. +- Calm authority. Closer to a well-edited technical reference (O'Reilly index, The Economist briefing, a good man page) than a blog or product site. + +Emotional goals: trust, efficiency, craft. The reader should feel the list was edited by someone with taste, find what they need in seconds, and notice the typographic care as a signal that the curation is careful too. + +Reference points (stay close to these): + +- **https://www.placestoread.xyz** is the primary visual model for the table, expand row, sorting, and footer. "Like placestoread" means dense single-page list, inline click-to-expand rows that indent under the Name column, sortable headers, minimum decoration. When in doubt about a table or row treatment, check placestoread first. +- Magazine reference pages (The Economist, FT Weekend, Monocle). +- Field-guide books. Curated, functional, hand-made. +- Library card catalogs. Dense tabular information, excellent typography, no decoration for decoration's sake. + +Anti-references (avoid strictly): + +- Generic dark developer-tool look. No cyan on near-black, neon gradients, VSCode-palette dashboards, terminal-green monospace branding. +- Other awesome-\* sites. No plain README dumps, bare lists of links, no voice. +- SaaS marketing pages. No big metric counters, testimonial cards, feature grids, pricing tiers, or "join 10,000+ developers" social proof bands. + +Design principles: + +1. **The list is the hero.** Hero, sponsor band, and CTA exist, but they must not compete with the table for attention. +2. **Density is a feature.** Prefer tables and tight rhythm over giant cards with one fact each. Mid-senior developers want to see more at once. +3. **Editorial typography over decoration.** Visual interest comes from the serif/sans pairing, type hierarchy, and whitespace. Not from gradients, shadows, badges, or icon boxes above headings. +4. **Warm, not cool.** Neutrals tint toward warm hues (roughly 55 to 80 in OKLCH). Pure grays and cool blues do not belong. +5. **One point of view.** No dark mode, no theme picker, no alternate palettes. Consistency signals curation. + +## Colors + +Warm editorial palette. Light theme only (`color-scheme: light`). OKLCH only. + +Each token below shows the OKLCH value (canonical, lives in `style.css`) followed by an approximate hex sRGB equivalent for spec linters and any tool that expects hex. + +Surfaces: + +- `--bg-page` `oklch(96.8% 0.018 80)` ≈ `#FBF3E7`. Cream/ivory canvas, the body floor. The body uses a vertical gradient between `--bg-page-top` `oklch(95.2% 0.018 78)` ≈ `#F7EFE3`, `--bg-page` at 24rem, and `--bg-page-end` `oklch(98.4% 0.01 80)` ≈ `#FCF8F0`, with a soft radial highlight in the top-left corner. +- `--bg-paper` `oklch(98.6% 0.01 80)` ≈ `#FEFAF3`. Warm white for the content shell. +- `--bg-paper-strong` `oklch(95.7% 0.016 76)` ≈ `#F7F0E5`. Tinted paper for sponsor band, CTA backgrounds, static decoration. +- `--hero-bg-start` `oklch(14% 0.03 32)` ≈ `#130503` through `--hero-bg-mid` `oklch(19% 0.035 35)` ≈ `#22120B` to `--hero-bg-end` `oklch(28% 0.05 42)` ≈ `#3D2014`. Dark earthy hero gradient. +- `--footer-bg` `oklch(16% 0.025 35)` ≈ `#170906`. Dark warm charcoal footer, part of the same system. + +Ink: + +- `--ink` `oklch(22% 0.02 55)` ≈ `#221812`. Body text. +- `--ink-soft` `oklch(38% 0.018 55)` ≈ `#4A4039`. Secondary copy. +- `--ink-muted` `oklch(52% 0.02 55)` ≈ `#72665E`. Meta rows, captions, static labels. +- `--line` / `--line-strong`. Hairlines and dividers. + +Accent (warm brown-red, reserved for interactive): + +- `--accent` `oklch(58% 0.16 45)` ≈ `#C4530F`. Primary accent. +- `--accent-deep` `oklch(44% 0.15 42)` ≈ `#922900`. Link text, hover. +- `--accent-soft` `oklch(92% 0.045 55)` ≈ `#FDDDC9`. Tinted background for filter tags. +- `--accent-underline` `oklch(58% 0.16 45 / 0.4)` ≈ `#C4530F66`. Subtle text-decoration-color. + +Rules: + +- Use OKLCH for any new color. Not HSL, not hex. +- Accent tokens (`--accent`, `--accent-deep`, `--accent-soft`) are reserved for interactive elements. Clickable filter tags (`.tag`) correctly use `--accent-soft` background with `--accent-deep` text. Interactive link states (`.col-name > a:hover`, `.sponsor-link:hover`, `.hero-action-primary`, `.back-to-top`, CTAs) use accent tokens. +- Non-interactive elements (inline code, `.source-badge`, static labels, decorative pills) must use ink tokens (`--ink`, `--ink-soft`, `--ink-muted`) on `--bg-paper-strong` or `--bg-paper`, never the accent. `.source-badge` uses `--ink-soft`; `.sponsorship-body code` uses `--ink`. Users should not mistake static decoration for something clickable. +- Same role uses the same token everywhere. No one-off inline `color: oklch(...)` buried in a rule. + +Aversions: + +- **No green.** The user rejected it when picking the palette. Warm brown-red, ivory, and dark earthy tones are the established system. Do not introduce green even for success states or ancillary accents. +- No cyan, no neon gradients, no pure grays, no cool blues. + +## Typography + +Pairing (do not swap): + +- **Display**: `Cormorant Garamond` (serif, 600 only). +- **Body**: `Manrope` (sans, 400 / 600 / 700 / 800). + +Scale: + +| Role | Token | Size | Family | Weight | Notes | +| ---------------- | ----------------- | ----------------------------- | ------------------ | --------- | ---------------------------------------------- | +| Hero headline | (literal `clamp`) | `clamp(4.5rem, 11vw, 8.5rem)` | Cormorant Garamond | 600 | Magazine-cover scale, single use on the hero | +| Body large | `--text-lg` | `1.125rem` | Manrope | 400 | Content-heavy passages | +| Body | `--text-base` | `1rem` (16px) | Manrope | 400 | Body floor, do not go smaller | +| Meta / secondary | `--text-sm` | `0.95rem` | Manrope | 400 / 600 | Meta rows, secondary copy | +| Caption / pill | `--text-xs` | `0.8rem` (12.8px) | Manrope | 600 / 700 | Smallest token, pills, badges, tags, footnotes | + +Hard-won sizing rules (do not relax): + +- **Body type floor is 16px.** Do not go smaller. +- **Absolute minimum font size is 12px (`0.75rem`) for ANY text**, including pills, badges, tags, captions, footnotes. Anything smaller hits Chrome's default minimum-font-size floor and renders inconsistently across browsers and user accessibility settings. Use `var(--text-xs)` (`0.8rem`) as the smallest token in code. +- **When in doubt, pick one step larger** than what generic scale references suggest. The user has corrected sizes upward 11+ times across 8 sessions. Footer, meta rows, expand content, labels, and headings all trend too small by default. **Never reduce an existing size unprompted.** +- Adjacent heading levels differ by at least 0.25rem of rendered size. +- Row numbers in the table: left-align, no leading zeros. Zero-padding was tried and rejected. +- **Never `text-transform`.** Write the casing in the markup. + +## Layout + +- **Single width cap: `--shell-max: 84rem` (~1344px) applied via `.section-shell`.** This is the ONLY width cap in the project. Widescreen monitors are the default viewing context. +- **Do NOT add `max-width`** to sections, cards, table cells, table rows, expanded rows, CTA backgrounds, sponsor descriptions, hero subcopy, paragraphs, or list items. The user has removed narrow inner caps repeatedly (`56ch`, `65-75ch`, etc.). Default is no inner cap. +- The "cap line length at ~65-75ch" rule does NOT apply here. Ignore it. Readability at wide widths is carried by vertical rhythm, leading, and the modular type scale instead. +- If a width cap is genuinely necessary for a specific element, ask first with a concrete reason before adding it. +- Shell padding: `--shell-pad: clamp(1.25rem, 3vw, 2.5rem)`. Symmetric gutters: logo left-gap equals logo right-gap, column paddings match across header and body. +- `gap` over child margins in flex and grid. +- Logical properties (`margin-inline`, `padding-block`) over physical (`margin-left`, `padding-top`). +- `rem` for spacing and type. `px` only for borders and shadows. +- CSS custom properties for all colors and repeated values. +- Sibling components (card lists, grid items) share identical spacing. +- Use flexbox or grid for layout. Avoid floats and absolute positioning except for genuine overlay cases (focus rings, sticky headers). +- Never `!important`. Fix specificity instead. + +## Elevation & Depth + +Depth comes from **tonal layers**, not heavy shadows. + +- The page is a quiet warm canvas (`--bg-page`). The content shell is slightly brighter paper (`--bg-paper`). The sponsor band, CTA backgrounds, and inline decorative blocks step up to `--bg-paper-strong`. +- The hero and the category guide band are the only places with real atmosphere: warm gradients on a dark earthy ground (`--hero-bg-start` → `--hero-bg-mid` → `--hero-bg-end`). Only the hero adds the subtle grid and slow sheen. The sheen and any other motion respect `prefers-reduced-motion`. +- The footer is a single tonal block in `--footer-bg`, no internal gradients. +- Two depth treatments are allowed and only these two. The search input combines a 1px inset highlight (`--search-inset`) with a soft warm drop shadow (`--search-shadow`, intensified by `--search-focus-shadow` on focus). The primary CTA button (`.hero-action-primary`) carries a warm drop shadow for press affordance. Both shadows are soft, warm-tinted, and tied to interactive elements. No new drop shadows on cards, panels, rows, or static decoration. +- No glassmorphism as default decoration. +- No bounce or elastic easing. Real objects decelerate smoothly. + +## Shapes + +The shape language is overwhelmingly **pill on small, zero radius on large**. + +- **Pills** (`border-radius: 999px`) for tags, search, sponsor logo chip, source badges, back-to-top, and primary CTA buttons. +- **`0.4rem`** is used in exactly one place: inline `` inside `.sponsorship-body`. Do not introduce a tokenized radius scale. The project does not need one. +- Containers use the page surface itself, not rounded panels. When a panel is needed, prefer pill on small chips and zero radius on large surfaces. +- **No `border-left` or `border-right` greater than 1px as a colored accent stripe** on cards, list items, callouts, or alerts. Use a different structure. + +## Components + +The component vocabulary is small and table-led. Source of truth: `website/static/style.css`. + +- **Table-driven index** (the hero of the page). Sticky header, sortable columns, click-to-expand rows that indent under the Name column. Modeled on placestoread.xyz. Not a card grid. +- **Group rows**. Section, subcategory, and group pages list rows in README order. Section pages add one H2 row per use case; group pages add one per section, linking to that section's page. A note replaces the sort arrow until someone sorts a column, which flattens the list and hides the group rows. +- **Filter tags** (`.tag`). `--accent-soft` background with `--accent-deep` text. Pill shape. Hover swaps to `--highlight` background with `--tag-hover-border` border and ink text. Active state uses the warm `--tag-active-start` → `--tag-active-end` gradient with hero-ink text. Tag variants (`tag-group`, `tag-source`) inherit the base `.tag` style today and differ only at narrow widths (`tag-group` hides under 960px). Add a new variant only when a real visual difference is needed. +- **Hero**. Magazine-cover headline, dark earthy ground, kicker and proof microcopy, primary CTA button using `--hero-btn-start` / `--hero-btn-end`. Subtle grid plus slow sheen. Respects `prefers-reduced-motion`. +- **Jump links**. One per use case in a category hero, plus one for the guide. Underlined like the intro's links, with a trailing ↓. Never tag-styled: tags filter the table or open another page, jump links only scroll. +- **Guide band**. Everything in a category intro after its "How to choose" list. It sits below the table on the hero's dark gradient, with its H2 above the text. +- **Sponsor band**. Sits in the README header on `--bg-paper-strong`. Editorial layout, not a logo wall. Sponsor links share the global accent treatment. +- **CTA**. Warm `--cta-bg`, full-bleed within shell. The button itself uses accent tokens. +- **Footer**. Dark warm charcoal, part of the same system. Footer links share the global hover and focus treatment. +- **Search**. Pill input with `--search-inset` interior and `--search-focus-ring` focus ring. Focus shadow uses `--search-focus-shadow`. +- **Source badge / inline code**. Static decoration on `--bg-paper-strong`. `.source-badge` uses `--ink-soft` text in pill shape; `.sponsorship-body code` uses `--ink` text with the lone `0.4rem` radius. Never the accent. + +Peer-consistency check (run before shipping any visual change): + +- Hover and focus states: if one link type gets a treatment, peer links (hero topbar, footer, project names, sponsor names, expand-meta) share it. +- Tag variants inherit the base `.tag` style. Differ only where a real difference is needed. +- Typography tiers: labels that play the same role share size, weight, and letter-spacing. +- Symmetric gutters: logo left-gap equals logo right-gap, column paddings match across header and body. +- Role-based color tokens: same role uses the same token everywhere. + +## Do's and Don'ts + +- **Do** keep the table the focal point. Hero, sponsor band, and CTA must not compete. +- **Do** use accent tokens only on interactive elements. +- **Do** prefer density over whitespace expansion. +- **Do** check peer elements before shipping a visual change. +- **Do** use OKLCH for every new color. +- **Don't** add inner `max-width` to anything. The shell handles width. +- **Don't** introduce green, cyan, neon, pure gray, or cool blue. +- **Don't** add a dark mode, theme picker, or alternate palette. +- **Don't** use gradient text (`background-clip: text` on gradients). Solid color only. +- **Don't** use `!important`. Fix specificity instead. +- **Don't** use `text-transform`. Write the casing in markup. +- **Don't** use a `border-left` or `border-right` greater than 1px as an accent stripe. +- **Don't** use bounce or elastic easing. +- **Don't** use glassmorphism as default decoration. +- **Don't** mimic generic dark developer-tool sites, other awesome-\* sites, or SaaS marketing pages. + +## Narrow-Screen Behavior + +The user actively tests `< 960px` and `< 680px`. Narrow screens must stay functional. + +- Do not drop features the user might want (sort affordance, filter chips, sticky header where reasonable). Hiding is a last resort and requires justification. +- Always run the `playwright-cli` skill at a narrow viewport after any layout change. + +## Iteration Guide + +Run this audit after generating or modifying a screen. Failure on any item means revise before moving on. + +1. **Width caps.** Inspect every section, card, paragraph, table cell, expanded row, CTA, sponsor description, hero subcopy. Only `.section-shell` (`--shell-max: 84rem`) may cap width. Anything else with a `max-width` is wrong. +2. **Accent reservation.** Grep the changed CSS for `--accent`, `--accent-deep`, `--accent-soft`. Each match must back an interactive element (link, button, focus ring, filter tag). Static decoration must use ink tokens (`--ink`, `--ink-soft`, `--ink-muted`) on `--bg-paper-strong` or `--bg-paper`. +3. **Shape language.** Containers are square or pill. Anything in the 4px-to-16px radius range is suspect. The lone `0.4rem` on `.sponsorship-body code` is the only allowed exception. +4. **Type sizes.** Confirm no rendered text falls below 12px. If a size feels small to a mid-senior reader on a 27-inch display, bump one step up. Never reduce an existing size. +5. **Peer consistency.** Compare against the closest peer element (sibling link type, sibling tag variant, sibling label). Hover, focus, color token, and gutter must match unless there is a stated reason to differ. +6. **Narrow viewport.** Run the `playwright-cli` skill at `< 960px` and `< 680px`. Sort affordance, filter chips, and sticky header must remain functional. + +## Known Gaps + +- **Color format diverges from the Stitch spec.** The official linter requires hex sRGB (`/^#([0-9a-fA-F]{3,8})$/`). The project mandates OKLCH in `style.css`. The Colors section above resolves this by showing both: OKLCH is canonical, hex is the linter-friendly approximation. +- **YAML frontmatter is minimal.** Only `version`, `name`, and `description` are encoded. The project has no JSON / Figma export pipeline that would consume token-level frontmatter, so the prose-led format is preferred for everything else. +- **No formal spacing or radius scale.** The codebase uses `clamp()` and ad-hoc rem values rather than a tokenized scale. Adding one would be invention, not documentation. + +## Verification + +After any frontend change, use the `playwright-cli` skill to visually verify in a real browser. Check layout, responsiveness, and interactive behavior. Do not claim a UI change works based on code alone. diff --git a/Makefile b/Makefile index eda7a8ff..5a07ad91 100644 --- a/Makefile +++ b/Makefile @@ -1,14 +1,32 @@ -site_install: - pip install -r requirements.txt +-include .env +export -site_link: - ln -sf $(CURDIR)/README.md $(CURDIR)/docs/index.md +install: + uv sync --locked -site_preview: site_link - mkdocs serve +fetch_github_stars: + uv run python website/fetch_github_stars.py -site_build: site_link - mkdocs build +fetch_pypi_downloads: + uv run python website/fetch_pypi_downloads_via_clickpy.py -site_deploy: site_link - mkdocs gh-deploy --clean +test: + uv run pytest website/tests/ -v + +lint: + uv run ruff check . + +format: + uv run ruff format . + +typecheck: + uv run ty check website + +build: + uv run python website/build.py + +preview: build + uv run watchfiles \ + 'uv run python website/build.py' \ + README.md website/templates website/static website/data & \ + python -m http.server -b 127.0.0.1 -d website/output/ 8000 diff --git a/README.md b/README.md index b56fdefe..11548a76 100644 --- a/README.md +++ b/README.md @@ -1,1216 +1,1207 @@ - -
- -# Awesome Python [![Awesome](https://cdn.rawgit.com/sindresorhus/awesome/d7305f38d29fed78fa85652e3a63e154dd8e8829/media/badge.svg)](https://github.com/sindresorhus/awesome) - -An opinionated list of awesome Python frameworks, libraries, software and resources. - -Inspired by [awesome-php](https://github.com/ziadoz/awesome-php). - -- [Awesome Python](#awesome-python) - - [Admin Panels](#admin-panels) - - [Algorithms and Design Patterns](#algorithms-and-design-patterns) - - [ASGI Servers](#asgi-servers) - - [Asynchronous Programming](#asynchronous-programming) - - [Audio](#audio) - - [Authentication](#authentication) - - [Build Tools](#build-tools) - - [Built-in Classes Enhancement](#built-in-classes-enhancement) - - [Caching](#caching) - - [ChatOps Tools](#chatops-tools) - - [CMS](#cms) - - [Code Analysis](#code-analysis) - - [Command-line Interface Development](#command-line-interface-development) - - [Command-line Tools](#command-line-tools) - - [Computer Vision](#computer-vision) - - [Configuration Files](#configuration-files) - - [Cryptography](#cryptography) - - [Data Analysis](#data-analysis) - - [Data Validation](#data-validation) - - [Data Visualization](#data-visualization) - - [Database Drivers](#database-drivers) - - [Database](#database) - - [Date and Time](#date-and-time) - - [Debugging Tools](#debugging-tools) - - [Deep Learning](#deep-learning) - - [DevOps Tools](#devops-tools) - - [Distributed Computing](#distributed-computing) - - [Distribution](#distribution) - - [Documentation](#documentation) - - [Downloader](#downloader) - - [Editor Plugins and IDEs](#editor-plugins-and-ides) - - [Email](#email) - - [Environment Management](#environment-management) - - [File Manipulation](#file-manipulation) - - [Functional Programming](#functional-programming) - - [Game Development](#game-development) - - [Geolocation](#geolocation) - - [GUI Development](#gui-development) - - [Hardware](#hardware) - - [HTML Manipulation](#html-manipulation) - - [HTTP Clients](#http-clients) - - [Image Processing](#image-processing) - - [Implementations](#implementations) - - [Interactive Interpreter](#interactive-interpreter) - - [Internationalization](#internationalization) - - [Job Scheduler](#job-scheduler) - - [Logging](#logging) - - [Machine Learning](#machine-learning) - - [Miscellaneous](#miscellaneous) - - [Natural Language Processing](#natural-language-processing) - - [Network Virtualization](#network-virtualization) - - [News Feed](#news-feed) - - [ORM](#orm) - - [Package Management](#package-management) - - [Package Repositories](#package-repositories) - - [Penetration testing](#penetration-testing) - - [Permissions](#permissions) - - [Processes](#processes) - - [Recommender Systems](#recommender-systems) - - [Refactoring](#refactoring) - - [RESTful API](#restful-api) - - [Robotics](#robotics) - - [RPC Servers](#rpc-servers) - - [Science](#science) - - [Search](#search) - - [Serialization](#serialization) - - [Serverless Frameworks](#serverless-frameworks) - - [Shell](#shell) - - [Specific Formats Processing](#specific-formats-processing) - - [Static Site Generator](#static-site-generator) - - [Tagging](#tagging) - - [Task Queues](#task-queues) - - [Template Engine](#template-engine) - - [Testing](#testing) - - [Text Processing](#text-processing) - - [Third-party APIs](#third-party-apis) - - [URL Manipulation](#url-manipulation) - - [Video](#video) - - [Web Asset Management](#web-asset-management) - - [Web Content Extracting](#web-content-extracting) - - [Web Crawling](#web-crawling) - - [Web Frameworks](#web-frameworks) - - [WebSocket](#websocket) - - [WSGI Servers](#wsgi-servers) -- [Resources](#resources) - - [Newsletters](#newsletters) - - [Podcasts](#podcasts) -- [Contributing](#contributing) - ---- - -## Admin Panels - -*Libraries for administrative interfaces.* - -* [ajenti](https://github.com/ajenti/ajenti) - The admin panel your servers deserve. -* [django-grappelli](https://github.com/sehmaschine/django-grappelli) - A jazzy skin for the Django Admin-Interface. -* [flask-admin](https://github.com/flask-admin/flask-admin) - Simple and extensible administrative interface framework for Flask. -* [flower](https://github.com/mher/flower) - Real-time monitor and web admin for Celery. -* [jet-bridge](https://github.com/jet-admin/jet-bridge) - Admin panel framework for any application with nice UI (ex Jet Django). -* [wooey](https://github.com/wooey/wooey) - A Django app which creates automatic web UIs for Python scripts. -* [streamlit](https://github.com/streamlit/streamlit) - A framework which lets you build dashboards, generate reports, or create chat apps in minutes. - -## Algorithms and Design Patterns - -*Python implementation of data structures, algorithms and design patterns. Also see [awesome-algorithms](https://github.com/tayllan/awesome-algorithms).* - -* Algorithms - * [algorithms](https://github.com/keon/algorithms) - Minimal examples of data structures and algorithms. - * [python-ds](https://github.com/prabhupant/python-ds) - A collection of data structure and algorithms for coding interviews. - * [sortedcontainers](https://github.com/grantjenks/python-sortedcontainers) - Fast and pure-Python implementation of sorted collections. - * [thealgorithms](https://github.com/TheAlgorithms/Python) - All Algorithms implemented in Python. -* Design Patterns - * [pypattyrn](https://github.com/tylerlaberge/PyPattyrn) - A simple yet effective library for implementing common design patterns. - * [python-patterns](https://github.com/faif/python-patterns) - A collection of design patterns in Python. - * [transitions](https://github.com/pytransitions/transitions) - A lightweight, object-oriented finite state machine implementation. - -## ASGI Servers - -*[ASGI](https://asgi.readthedocs.io/en/latest/)-compatible web servers.* - -* [daphne](https://github.com/django/daphne) - A HTTP, HTTP2 and WebSocket protocol server for ASGI and ASGI-HTTP. -* [uvicorn](https://github.com/encode/uvicorn) - A lightning-fast ASGI server implementation, using uvloop and httptools. -* [hypercorn](https://github.com/pgjones/hypercorn) - An ASGI and WSGI Server based on Hyper libraries and inspired by Gunicorn. - -## Asynchronous Programming - -*Libraries for asynchronous, concurrent and parallel execution. Also see [awesome-asyncio](https://github.com/timofurrer/awesome-asyncio).* - -* [asyncio](https://docs.python.org/3/library/asyncio.html) - (Python standard library) Asynchronous I/O, event loop, coroutines and tasks. - - [awesome-asyncio](https://github.com/timofurrer/awesome-asyncio) -* [concurrent.futures](https://docs.python.org/3/library/concurrent.futures.html) - (Python standard library) A high-level interface for asynchronously executing callables. -* [multiprocessing](https://docs.python.org/3/library/multiprocessing.html) - (Python standard library) Process-based parallelism. -* [trio](https://github.com/python-trio/trio) - A friendly library for async concurrency and I/O. -* [twisted](https://github.com/twisted/twisted) - An event-driven networking engine. -* [uvloop](https://github.com/MagicStack/uvloop) - Ultra fast asyncio event loop. -* [eventlet](https://github.com/eventlet/eventlet) - Asynchronous framework with WSGI support. -* [gevent](https://github.com/gevent/gevent) - A coroutine-based Python networking library that uses [greenlet](https://github.com/python-greenlet/greenlet). - -## Audio - -*Libraries for manipulating audio and its metadata.* - -* Audio - * [audioread](https://github.com/beetbox/audioread) - Cross-library (GStreamer + Core Audio + MAD + FFmpeg) audio decoding. - * [audioFlux](https://github.com/libAudioFlux/audioFlux) - A library for audio and music analysis, feature extraction. - * [dejavu](https://github.com/worldveil/dejavu) - Audio fingerprinting and recognition. - * [kapre](https://github.com/keunwoochoi/kapre) - Keras Audio Preprocessors. - * [librosa](https://github.com/librosa/librosa) - Python library for audio and music analysis. - * [matchering](https://github.com/sergree/matchering) - A library for automated reference audio mastering. - * [mingus](http://bspaans.github.io/python-mingus/) - An advanced music theory and notation package with MIDI file and playback support. - * [pyaudioanalysis](https://github.com/tyiannak/pyAudioAnalysis) - Audio feature extraction, classification, segmentation and applications. - * [pydub](https://github.com/jiaaro/pydub) - Manipulate audio with a simple and easy high level interface. - * [timeside](https://github.com/Parisson/TimeSide) - Open web audio processing framework. -* Metadata - * [beets](https://github.com/beetbox/beets) - A music library manager and [MusicBrainz](https://musicbrainz.org/) tagger. - * [eyed3](https://github.com/nicfit/eyeD3) - A tool for working with audio files, specifically MP3 files containing ID3 metadata. - * [mutagen](https://github.com/quodlibet/mutagen) - A Python module to handle audio metadata. - * [tinytag](https://github.com/devsnd/tinytag) - A library for reading music meta data of MP3, OGG, FLAC and Wave files. - -## Authentication - -*Libraries for implementing authentications schemes.* - -* OAuth - * [authlib](https://github.com/lepture/authlib) - JavaScript Object Signing and Encryption draft implementation. - * [django-allauth](https://github.com/pennersr/django-allauth) - Authentication app for Django that "just works." - * [django-oauth-toolkit](https://github.com/jazzband/django-oauth-toolkit) - OAuth 2 goodies for Django. - * [oauthlib](https://github.com/oauthlib/oauthlib) - A generic and thorough implementation of the OAuth request-signing logic. -* JWT - * [pyjwt](https://github.com/jpadilla/pyjwt) - JSON Web Token implementation in Python. - * [python-jose](https://github.com/mpdavis/python-jose/) - A JOSE implementation in Python. - -## Build Tools - -*Compile software from source code.* - -* [bitbake](https://github.com/openembedded/bitbake) - A make-like build tool for embedded Linux. -* [buildout](https://github.com/buildout/buildout) - A build system for creating, assembling and deploying applications from multiple parts. -* [platformio](https://github.com/platformio/platformio-core) - A console tool to build code with different development platforms. -* [pybuilder](https://github.com/pybuilder/pybuilder) - A continuous build tool written in pure Python. -* [scons](https://github.com/SCons/scons) - A software construction tool. - -## Built-in Classes Enhancement - -*Libraries for enhancing Python built-in classes.* - -* [attrs](https://github.com/python-attrs/attrs) - Replacement for `__init__`, `__eq__`, `__repr__`, etc. boilerplate in class definitions. -* [bidict](https://github.com/jab/bidict) - Efficient, Pythonic bidirectional map data structures and related functionality.. -* [box](https://github.com/cdgriffith/Box) - Python dictionaries with advanced dot notation access. -* [dataclasses](https://docs.python.org/3/library/dataclasses.html) - (Python standard library) Data classes. -* [dotteddict](https://github.com/carlosescri/DottedDict) - A library that provides a method of accessing lists and dicts with a dotted path notation. - -## CMS - -*Content Management Systems.* - -* [feincms](https://github.com/feincms/feincms) - One of the most advanced Content Management Systems built on Django. -* [indico](https://github.com/indico/indico) - A feature-rich event management system, made @ [CERN](https://en.wikipedia.org/wiki/CERN). -* [wagtail](https://github.com/wagtail/wagtail) - A Django content management system. - -## Caching - -*Libraries for caching data.* - -* [beaker](https://github.com/bbangert/beaker) - A WSGI middleware for sessions and caching. -* [django-cache-machine](https://github.com/django-cache-machine/django-cache-machine) - Automatic caching and invalidation for Django models. -* [django-cacheops](https://github.com/Suor/django-cacheops) - A slick ORM cache with automatic granular event-driven invalidation. -* [dogpile.cache](https://github.com/sqlalchemy/dogpile.cache) - dogpile.cache is a next generation replacement for Beaker made by the same authors. -* [hermescache](https://pypi.org/project/HermesCache/) - Python caching library with tag-based invalidation and dogpile effect prevention. -* [pylibmc](https://github.com/lericson/pylibmc) - A Python wrapper around the [libmemcached](https://libmemcached.org/libMemcached.html) interface. -* [python-diskcache](https://github.com/grantjenks/python-diskcache) - SQLite and file backed cache backend with faster lookups than memcached and redis. - -## ChatOps Tools - -*Libraries for chatbot development.* - -* [errbot](https://github.com/errbotio/errbot/) - The easiest and most popular chatbot to implement ChatOps. - -## Code Analysis - -*Tools of static analysis, linters and code quality checkers. Also see [awesome-static-analysis](https://github.com/mre/awesome-static-analysis).* - -* Code Analysis - * [code2flow](https://github.com/scottrogowski/code2flow) - Turn your Python and JavaScript code into DOT flowcharts. - * [prospector](https://github.com/PyCQA/prospector) - A tool to analyse Python code. - * [vulture](https://github.com/jendrikseipp/vulture) - A tool for finding and analysing dead Python code. -* Code Linters - * [flake8](https://github.com/PyCQA/flake8) - A wrapper around `pycodestyle`, `pyflakes` and McCabe. - * [awesome-flake8-extensions](https://github.com/DmytroLitvinov/awesome-flake8-extensions) - * [pylint](https://github.com/pylint-dev/pylint) - A fully customizable source code analyzer. - * [ruff](https://github.com/astral-sh/ruff) - An extremely fast Python linter and code formatter. -* Code Formatters - * [black](https://github.com/psf/black) - The uncompromising Python code formatter. - * [isort](https://github.com/timothycrosley/isort) - A Python utility / library to sort imports. - * [yapf](https://github.com/google/yapf) - Yet another Python code formatter from Google. -* Static Type Checkers, also see [awesome-python-typing](https://github.com/typeddjango/awesome-python-typing) - * [mypy](https://github.com/python/mypy) - Check variable types during compile time. - * [pyre-check](https://github.com/facebook/pyre-check) - Performant type checking. - * [typeshed](https://github.com/python/typeshed) - Collection of library stubs for Python, with static types. -* Static Type Annotations Generators - * [monkeytype](https://github.com/Instagram/MonkeyType) - A system for Python that generates static type annotations by collecting runtime types. - * [pytype](https://github.com/google/pytype) - Pytype checks and infers types for Python code - without requiring type annotations. - -## Command-line Interface Development - -*Libraries for building command-line applications.* - -* Command-line Application Development - * [cement](https://github.com/datafolklabs/cement) - CLI Application Framework for Python. - * [click](https://github.com/pallets/click/) - A package for creating beautiful command line interfaces in a composable way. - * [cliff](https://github.com/openstack/cliff) - A framework for creating command-line programs with multi-level commands. - * [python-fire](https://github.com/google/python-fire) - A library for creating command line interfaces from absolutely any Python object. - * [python-prompt-toolkit](https://github.com/prompt-toolkit/python-prompt-toolkit) - A library for building powerful interactive command lines. -* Terminal Rendering - * [alive-progress](https://github.com/rsalmei/alive-progress) - A new kind of Progress Bar, with real-time throughput, eta and very cool animations. - * [asciimatics](https://github.com/peterbrittain/asciimatics) - A package to create full-screen text UIs (from interactive forms to ASCII animations). - * [bashplotlib](https://github.com/glamp/bashplotlib) - Making basic plots in the terminal. - * [colorama](https://github.com/tartley/colorama) - Cross-platform colored terminal text. - * [rich](https://github.com/Textualize/rich) - Python library for rich text and beautiful formatting in the terminal. Also provides a great `RichHandler` log handler. - * [tqdm](https://github.com/tqdm/tqdm) - Fast, extensible progress bar for loops and CLI. - -## Command-line Tools - -*Useful CLI-based tools for productivity.* - -* Productivity Tools - * [copier](https://github.com/copier-org/copier) - A library and command-line utility for rendering projects templates. - * [cookiecutter](https://github.com/cookiecutter/cookiecutter) - A command-line utility that creates projects from cookiecutters (project templates). - * [doitlive](https://github.com/sloria/doitlive) - A tool for live presentations in the terminal. - * [howdoi](https://github.com/gleitz/howdoi) - Instant coding answers via the command line. - * [invoke](https://github.com/pyinvoke/invoke) - A tool for managing shell-oriented subprocesses and organizing executable Python code into CLI-invokable tasks. - * [pathpicker](https://github.com/facebook/PathPicker) - Select files out of bash output. - * [thefuck](https://github.com/nvbn/thefuck) - Correcting your previous console command. - * [tmuxp](https://github.com/tmux-python/tmuxp) - A [tmux](https://github.com/tmux/tmux) session manager. - * [try](https://github.com/timofurrer/try) - A dead simple CLI to try out python packages - it's never been easier. -* CLI Enhancements - * [httpie](https://github.com/httpie/cli) - A command line HTTP client, a user-friendly cURL replacement. - * [iredis](https://github.com/laixintao/iredis) - Redis CLI with autocompletion and syntax highlighting. - * [litecli](https://github.com/dbcli/litecli) - SQLite CLI with autocompletion and syntax highlighting. - * [mycli](https://github.com/dbcli/mycli) - MySQL CLI with autocompletion and syntax highlighting. - * [pgcli](https://github.com/dbcli/pgcli) - PostgreSQL CLI with autocompletion and syntax highlighting. - -## Computer Vision - -*Libraries for Computer Vision.* - -* [easyocr](https://github.com/JaidedAI/EasyOCR) - Ready-to-use OCR with 40+ languages supported. -* [kornia](https://github.com/kornia/kornia/) - Open Source Differentiable Computer Vision Library for PyTorch. -* [opencv](https://opencv.org/) - Open Source Computer Vision Library. -* [pytesseract](https://github.com/madmaze/pytesseract) - A wrapper for [Google Tesseract OCR](https://github.com/tesseract-ocr). -* [tesserocr](https://github.com/sirfz/tesserocr) - Another simple, Pillow-friendly, wrapper around the `tesseract-ocr` API for OCR. - -## Configuration Files - -*Libraries for storing and parsing configuration options.* - -* [configparser](https://docs.python.org/3/library/configparser.html) - (Python standard library) INI file parser. -* [configobj](https://github.com/DiffSK/configobj) - INI file parser with validation. -* [hydra](https://github.com/facebookresearch/hydra) - Hydra is a framework for elegantly configuring complex applications. -* [python-decouple](https://github.com/HBNetwork/python-decouple) - Strict separation of settings from code. - -## Cryptography - -* [cryptography](https://github.com/pyca/cryptography) - A package designed to expose cryptographic primitives and recipes to Python developers. -* [paramiko](https://github.com/paramiko/paramiko) - The leading native Python SSHv2 protocol library. -* [pynacl](https://github.com/pyca/pynacl) - Python binding to the Networking and Cryptography (NaCl) library. - -## Data Analysis - -*Libraries for data analyzing.* - -* [pandas](http://pandas.pydata.org/) - A library providing high-performance, easy-to-use data structures and data analysis tools. -* [aws-sdk-pandas](https://github.com/aws/aws-sdk-pandas) - Pandas on AWS. -* [datasette](https://github.com/simonw/datasette) - An open source multi-tool for exploring and publishing data. -* [desbordante](https://github.com/desbordante/desbordante-core/) - An open source data profiler for complex pattern discovery. -* [optimus](https://github.com/hi-primus/optimus) - Agile Data Science Workflows made easy with PySpark. - -## Data Validation - -*Libraries for validating data. Used for forms in many cases.* - -* [cerberus](https://github.com/pyeve/cerberus) - A lightweight and extensible data validation library. -* [colander](https://github.com/Pylons/colander) - Validating and deserializing data obtained via XML, JSON, an HTML form post. -* [jsonschema](https://github.com/python-jsonschema/jsonschema) - An implementation of [JSON Schema](http://json-schema.org/) for Python. -* [schema](https://github.com/keleshev/schema) - A library for validating Python data structures. -* [schematics](https://github.com/schematics/schematics) - Data Structure Validation. -* [voluptuous](https://github.com/alecthomas/voluptuous) - A Python data validation library. -* [pydantic](https://github.com/pydantic/pydantic) - Data validation using Python type hints. - -## Data Visualization - -*Libraries for visualizing data. Also see [awesome-javascript](https://github.com/sorrycc/awesome-javascript#data-visualization).* - -* [altair](https://github.com/altair-viz/altair) - Declarative statistical visualization library for Python. -* [bokeh](https://github.com/bokeh/bokeh) - Interactive Web Plotting for Python. -* [bqplot](https://github.com/bloomberg/bqplot) - Interactive Plotting Library for the Jupyter Notebook. -* [cartopy](https://github.com/SciTools/cartopy) - A cartographic python library with matplotlib support. -* [diagrams](https://github.com/mingrammer/diagrams) - Diagram as Code. -* [matplotlib](https://github.com/matplotlib/matplotlib) - A Python 2D plotting library. -* [plotnine](https://github.com/has2k1/plotnine) - A grammar of graphics for Python based on ggplot2. -* [pygal](https://github.com/Kozea/pygal) - A Python SVG Charts Creator. -* [pygraphviz](https://github.com/pygraphviz/pygraphviz/) - Python interface to [Graphviz](http://www.graphviz.org/). -* [pyqtgraph](https://github.com/pyqtgraph/pyqtgraph) - Interactive and realtime 2D/3D/Image plotting and science/engineering widgets. -* [seaborn](https://github.com/mwaskom/seaborn) - Statistical data visualization using Matplotlib. -* [vispy](https://github.com/vispy/vispy) - High-performance scientific visualization based on OpenGL. - -## Database - -*Databases implemented in Python.* - -* [pickleDB](https://github.com/patx/pickledb) - A simple and lightweight key-value store for Python. -* [tinydb](https://github.com/msiemens/tinydb) - A tiny, document-oriented database. -* [zodb](https://github.com/zopefoundation/ZODB) - A native object database for Python. A key-value and object graph database. - -## Database Drivers - -*Libraries for connecting and operating databases.* - -* MySQL - [awesome-mysql](http://shlomi-noach.github.io/awesome-mysql/) - * [mysqlclient](https://github.com/PyMySQL/mysqlclient) - MySQL connector with Python 3 support ([mysql-python](https://sourceforge.net/projects/mysql-python/) fork). - * [pymysql](https://github.com/PyMySQL/PyMySQL) - A pure Python MySQL driver compatible to mysql-python. -* PostgreSQL - [awesome-postgres](https://github.com/dhamaniasad/awesome-postgres) - * [psycopg](https://github.com/psycopg/psycopg) - The most popular PostgreSQL adapter for Python. -* SQlite - [awesome-sqlite](https://github.com/planetopendata/awesome-sqlite) - * [sqlite3](https://docs.python.org/3/library/sqlite3.html) - (Python standard library) SQlite interface compliant with DB-API 2.0. - * [sqlite-utils](https://github.com/simonw/sqlite-utils) - Python CLI utility and library for manipulating SQLite databases. -* Other Relational Databases - * [pymssql](https://github.com/pymssql/pymssql) - A simple database interface to Microsoft SQL Server. - * [clickhouse-driver](https://github.com/mymarilyn/clickhouse-driver) - Python driver with native interface for ClickHouse. -* NoSQL Databases - * [cassandra-driver](https://github.com/datastax/python-driver) - The Python Driver for Apache Cassandra. - * [happybase](https://github.com/python-happybase/happybase) - A developer-friendly library for Apache HBase. - * [kafka-python](https://github.com/dpkp/kafka-python) - The Python client for Apache Kafka. - * [pymongo](https://github.com/mongodb/mongo-python-driver) - The official Python client for MongoDB. - * [motor](https://github.com/mongodb/motor) - The async Python driver for MongoDB. - * [redis-py](https://github.com/redis/redis-py) - The Python client for Redis. - -## Date and Time - -*Libraries for working with dates and times.* - -* [arrow](https://github.com/arrow-py/arrow) - A Python library that offers a sensible and human-friendly approach to creating, manipulating, formatting and converting dates, times and timestamps. -* [dateutil](https://github.com/dateutil/dateutil) - Extensions to the standard Python [datetime](https://docs.python.org/3/library/datetime.html) module. -* [pendulum](https://github.com/sdispater/pendulum) - Python datetimes made easy. -* [pytz](https://pypi.org/project/pytz/) - World timezone definitions, modern and historical. Brings the [tz database](https://en.wikipedia.org/wiki/Tz_database) into Python. - -## Debugging Tools - -*Libraries for debugging code.* - -* pdb-like Debugger - * [ipdb](https://github.com/gotcha/ipdb) - IPython-enabled [pdb](https://docs.python.org/3/library/pdb.html). - * [pudb](https://github.com/inducer/pudb) - A full-screen, console-based Python debugger. -* Tracing - * [manhole](https://github.com/ionelmc/python-manhole) - Debugging UNIX socket connections and present the stacktraces for all threads and an interactive prompt. - * [python-hunter](https://github.com/ionelmc/python-hunter) - A flexible code tracing toolkit. -* Profiler - * [py-spy](https://github.com/benfred/py-spy) - A sampling profiler for Python programs. Written in Rust. - * [vprof](https://github.com/nvdv/vprof) - Visual Python profiler. -* Others - * [django-debug-toolbar](https://github.com/jazzband/django-debug-toolbar) - Display various debug information for Django. - * [flask-debugtoolbar](https://github.com/pallets-eco/flask-debugtoolbar) - A port of the django-debug-toolbar to flask. - * [icecream](https://github.com/gruns/icecream) - Inspect variables, expressions, and program execution with a single, simple function call. - * [pyelftools](https://github.com/eliben/pyelftools) - Parsing and analyzing ELF files and DWARF debugging information. - -## Deep Learning - -*Frameworks for Neural Networks and Deep Learning. Also see [awesome-deep-learning](https://github.com/ChristosChristofidis/awesome-deep-learning).* - -* [keras](https://github.com/keras-team/keras) - A high-level neural networks library and capable of running on top of either TensorFlow or Theano. -* [pytorch](https://github.com/pytorch/pytorch) - Tensors and Dynamic neural networks in Python with strong GPU acceleration. -* [pytorch-lightning](https://github.com/Lightning-AI/pytorch-lightning) - Deep learning framework to train, deploy, and ship AI products Lightning fast. -* [stable-baselines3](https://github.com/DLR-RM/stable-baselines3) - PyTorch implementations of Stable Baselines (deep) reinforcement learning algorithms. -* [tensorflow](https://github.com/tensorflow/tensorflow) - The most popular Deep Learning framework created by Google. -* [theano](https://github.com/Theano/Theano) - A library for fast numerical computation. - -## DevOps Tools - -*Software and libraries for DevOps.* - -* Configuration Management - * [ansible](https://github.com/ansible/ansible) - A radically simple IT automation platform. - * [cloudinit](https://github.com/canonical/cloud-init) - A multi-distribution package that handles early initialization of a cloud instance. - * [openstack](https://www.openstack.org/) - Open source software for building private and public clouds. - * [pyinfra](https://github.com/pyinfra-dev/pyinfra) - A versatile CLI tools and python libraries to automate infrastructure. - * [saltstack](https://github.com/saltstack/salt) - Infrastructure automation and management system. -* SSH-style Deployment - * [cuisine](https://github.com/sebastien/cuisine) - Chef-like functionality for Fabric. - * [fabric](https://github.com/fabric/fabric) - A simple, Pythonic tool for remote execution and deployment. -* Process Management - * [supervisor](https://github.com/Supervisor/supervisor) - Supervisor process control system for UNIX. -* Monitoring - * [psutil](https://github.com/giampaolo/psutil) - A cross-platform process and system utilities module. -* Backup - * [borg](https://github.com/borgbackup/borg) - A deduplicating archiver with compression and encryption. - -## Distributed Computing - -*Frameworks and libraries for Distributed Computing.* - -* Batch Processing - * [dask](https://github.com/dask/dask) - A flexible parallel computing library for analytic computing. - * [luigi](https://github.com/spotify/luigi) - A module that helps you build complex pipelines of batch jobs. - * [PySpark](https://github.com/apache/spark) - [Apache Spark](https://spark.apache.org/) Python API. - * [Ray](https://github.com/ray-project/ray/) - A system for parallel and distributed Python that unifies the machine learning ecosystem. -* Stream Processing - * [faust](https://github.com/robinhood/faust) - A stream processing library, porting the ideas from [Kafka Streams](https://kafka.apache.org/documentation/streams/) to Python. - * [streamparse](https://github.com/Parsely/streamparse) - Run Python code against real-time streams of data via [Apache Storm](http://storm.apache.org/). - -## Distribution - -*Libraries to create packaged executables for release distribution.* - -* [py2app](https://github.com/ronaldoussoren/py2app) - Freezes Python scripts (Mac OS X). -* [py2exe](https://github.com/py2exe/py2exe) - Freezes Python scripts (Windows). -* [pyarmor](https://github.com/dashingsoft/pyarmor) - A tool used to obfuscate python scripts, bind obfuscated scripts to fixed machine or expire obfuscated scripts. -* [pyinstaller](https://github.com/pyinstaller/pyinstaller) - Converts Python programs into stand-alone executables (cross-platform). -* [shiv](https://github.com/linkedin/shiv) - A command line utility for building fully self-contained zipapps (PEP 441), but with all their dependencies included. - -## Documentation - -*Libraries for generating project documentation.* - -* [sphinx](https://github.com/sphinx-doc/sphinx/) - Python Documentation generator. - * [awesome-sphinxdoc](https://github.com/yoloseem/awesome-sphinxdoc) -* [pdoc](https://github.com/mitmproxy/pdoc) - Epydoc replacement to auto generate API documentation for Python libraries. - -## Downloader - -*Libraries for downloading.* - -* [akshare](https://github.com/jindaxiang/akshare) - A financial data interface library, built for human beings! -* [s3cmd](https://github.com/s3tools/s3cmd) - A command line tool for managing Amazon S3 and CloudFront. -* [youtube-dl](https://github.com/ytdl-org/youtube-dl/) - A command-line program to download videos from YouTube and other video sites. - -## Editor Plugins and IDEs - -* Emacs - * [elpy](https://github.com/jorgenschaefer/elpy) - Emacs Python Development Environment. -* Vim - * [jedi-vim](https://github.com/davidhalter/jedi-vim) - Vim bindings for the Jedi auto-completion library for Python. - * [python-mode](https://github.com/python-mode/python-mode) - An all in one plugin for turning Vim into a Python IDE. - * [YouCompleteMe](https://github.com/Valloric/YouCompleteMe) - Includes [Jedi](https://github.com/davidhalter/jedi)-based completion engine for Python. -* Visual Studio - * [PTVS](https://github.com/Microsoft/PTVS) - Python Tools for Visual Studio. -* Visual Studio Code - * [Python](https://marketplace.visualstudio.com/items?itemName=ms-python.python) - The official VSCode extension with rich support for Python. -* IDE - * [PyCharm](https://www.jetbrains.com/pycharm/) - Commercial Python IDE by JetBrains. Has free community edition available. - * [spyder](https://github.com/spyder-ide/spyder) - Open Source Python IDE. - -## Email - -*Libraries for sending and parsing email.* - -* Mail Servers - * [modoboa](https://github.com/modoboa/modoboa) - A mail hosting and management platform including a modern Web UI. - * [salmon](https://github.com/moggers87/salmon) - A Python Mail Server. -* Clients - * [imbox](https://github.com/martinrusev/imbox) - Python IMAP for Humans. - * [yagmail](https://github.com/kootenpv/yagmail) - Yet another Gmail/SMTP client. -* Others - * [flanker](https://github.com/mailgun/flanker) - An email address and Mime parsing library. - * [mailer](https://github.com/marrow/mailer) - High-performance extensible mail delivery framework. +# [Awesome Python](https://awesome-python.com/) -## Environment Management - -*Libraries for Python version and virtual environment management.* - -* [pyenv](https://github.com/pyenv/pyenv) - Simple Python version management. -* [virtualenv](https://github.com/pypa/virtualenv) - A tool to create isolated Python environments. - -## File Manipulation +An opinionated guide to the best Python frameworks, libraries, and tools. -*Libraries for file manipulation.* +**Visit the [website](https://awesome-python.com/) to search and filter projects more easily.** -* [mimetypes](https://docs.python.org/3/library/mimetypes.html) - (Python standard library) Map filenames to MIME types. -* [pathlib](https://docs.python.org/3/library/pathlib.html) - (Python standard library) An cross-platform, object-oriented path library. -* [path.py](https://github.com/jaraco/path.py) - A module wrapper for [os.path](https://docs.python.org/3/library/os.path.html). -* [python-magic](https://github.com/ahupp/python-magic) - A Python interface to the libmagic file type identification library. -* [watchdog](https://github.com/gorakhargosh/watchdog) - API and shell utilities to monitor file system events. +## **Sponsors** -## Functional Programming +- [iPulse AI](https://ipulseai.com) - Open Agentic Investment Research Platform. Inspect deep multi-agent market forecasts and stock picks. -*Functional Programming with Python.* +> The **#10 most-starred repo on GitHub**. Put your product in front of Python developers. [Become a sponsor](SPONSORSHIP.md). -* [coconut](https://github.com/evhub/coconut) - A variant of Python built for simple, elegant, Pythonic functional programming. -* [funcy](https://github.com/Suor/funcy) - A fancy and practical functional tools. -* [more-itertools](https://github.com/erikrose/more-itertools) - More routines for operating on iterables, beyond `itertools`. -* [returns](https://github.com/dry-python/returns) - A set of type-safe monads, transformers, and composition utilities. -* [cytoolz](https://github.com/pytoolz/cytoolz/) - Cython implementation of `Toolz`: High performance functional utilities. -* [toolz](https://github.com/pytoolz/toolz) - A collection of functional utilities for iterators, functions, and dictionaries. +## Categories -## GUI Development +**AI & ML** -*Libraries for working with graphical user interface applications.* +- [AI and Agents](#ai-and-agents) +- [Deep Learning](#deep-learning) +- [Machine Learning](#machine-learning) +- [Natural Language Processing](#natural-language-processing) +- [Computer Vision](#computer-vision) +- [Recommender Systems](#recommender-systems) -* [curses](https://docs.python.org/3/library/curses.html) - Built-in wrapper for [ncurses](http://www.gnu.org/software/ncurses/) used to create terminal GUI applications. -* [Eel](https://github.com/ChrisKnott/Eel) - A library for making simple Electron-like offline HTML/JS GUI apps. -* [enaml](https://github.com/nucleic/enaml) - Creating beautiful user-interfaces with Declarative Syntax like QML. -* [Flet](https://flet.dev) - Cross-platform GUI framework for building modern apps in pure Python. Run on Windows, macOS, Linux, Android, iOS, and the Web. -* [Flexx](https://github.com/zoofIO/flexx) - Flexx is a pure Python toolkit for creating GUI's, that uses web technology for its rendering. -* [Gooey](https://github.com/chriskiehl/Gooey) - Turn command line programs into a full GUI application with one line. -* [kivy](https://kivy.org/) - A library for creating NUI applications, running on Windows, Linux, Mac OS X, Android and iOS. -* [pyglet](https://github.com/pyglet/pyglet) - A cross-platform windowing and multimedia library for Python. -* [PyGObject](https://pygobject.readthedocs.io/) - Python Bindings for GLib/GObject/GIO/GTK+ (GTK+3). -* [PyQt](https://doc.qt.io/qtforpython/) - Python bindings for the [Qt](https://www.qt.io/) cross-platform application and UI framework. -* [PySimpleGUI](https://github.com/PySimpleGUI/PySimpleGUI) - Wrapper for tkinter, Qt, WxPython and Remi. -* [pywebview](https://github.com/r0x0r/pywebview/) - A lightweight cross-platform native wrapper around a webview component. -* [Tkinter](https://wiki.python.org/moin/TkInter) - Tkinter is Python's de-facto standard GUI package. -* [Toga](https://github.com/pybee/toga) - A Python native, OS native GUI toolkit. -* [urwid](http://urwid.org/) - A library for creating terminal GUI applications with strong support for widgets, events, rich colors, etc. -* [wxPython](https://wxpython.org/) - A blending of the wxWidgets C++ class library with the Python. -* [DearPyGui](https://github.com/RaylockLLC/DearPyGui/) - A Simple GPU accelerated Python GUI framework +**Web Development** -## GraphQL +- [Web Frameworks](#web-frameworks) +- [Web APIs](#web-apis) +- [Web Servers](#web-servers) +- [WebSocket](#websocket) +- [Template Engines](#template-engines) +- [Web Asset Management](#web-asset-management) +- [Authentication](#authentication) +- [Admin Panels](#admin-panels) +- [CMS](#cms) +- [ERP](#erp) +- [Static Site Generators](#static-site-generators) -*Libraries for working with GraphQL.* +**HTTP & Scraping** -* [graphene](https://github.com/graphql-python/graphene/) - GraphQL framework for Python. +- [HTTP Clients](#http-clients) +- [Web Scraping](#web-scraping) +- [Email](#email) -## Game Development +**Database & Storage** -*Awesome game development libraries.* +- [ORM](#orm) +- [Database Drivers](#database-drivers) +- [Database](#database) +- [Caching](#caching) +- [Search](#search) +- [Serialization](#serialization) -* [Arcade](https://api.arcade.academy/en/latest/) - Arcade is a modern Python framework for crafting games with compelling graphics and sound. -* [Cocos2d](https://www.cocos.com/en/cocos2d-x) - cocos2d is a framework for building 2D games, demos, and other graphical/interactive applications. -* [Harfang3D](http://www.harfang3d.com) - Python framework for 3D, VR and game development. -* [Panda3D](https://www.panda3d.org/) - 3D game engine developed by Disney. -* [Pygame](http://www.pygame.org/news.html) - Pygame is a set of Python modules designed for writing games. -* [PyOgre](http://www.ogre3d.org/tikiwiki/PyOgre) - Python bindings for the Ogre 3D render engine, can be used for games, simulations, anything 3D. -* [PyOpenGL](http://pyopengl.sourceforge.net/) - Python ctypes bindings for OpenGL and it's related APIs. -* [PySDL2](https://pysdl2.readthedocs.io) - A ctypes based wrapper for the SDL2 library. -* [RenPy](https://www.renpy.org/) - A Visual Novel engine. +**Data & Science** -## Geolocation - -*Libraries for geocoding addresses and working with latitudes and longitudes.* - -* [django-countries](https://github.com/SmileyChris/django-countries) - A Django app that provides a country field for models and forms. -* [geodjango](https://docs.djangoproject.com/en/dev/ref/contrib/gis/) - A world-class geographic web framework. -* [geojson](https://github.com/jazzband/geojson) - Python bindings and utilities for GeoJSON. -* [geopy](https://github.com/geopy/geopy) - Python Geocoding Toolbox. - -## HTML Manipulation - -*Libraries for working with HTML and XML.* - -* [beautifulsoup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) - Providing Pythonic idioms for iterating, searching, and modifying HTML or XML. -* [bleach](https://github.com/mozilla/bleach) - A whitelist-based HTML sanitization and text linkification library. -* [cssutils](https://pypi.org/project/cssutils/) - A CSS library for Python. -* [html5lib](https://github.com/html5lib/html5lib-python) - A standards-compliant library for parsing and serializing HTML documents and fragments. -* [lxml](http://lxml.de/) - A very fast, easy-to-use and versatile library for handling HTML and XML. -* [markupsafe](https://github.com/pallets/markupsafe) - Implements a XML/HTML/XHTML Markup safe string for Python. -* [pyquery](https://github.com/gawel/pyquery) - A jQuery-like library for parsing HTML. -* [untangle](https://github.com/stchris/untangle) - Converts XML documents to Python objects for easy access. -* [WeasyPrint](http://weasyprint.org) - A visual rendering engine for HTML and CSS that can export to PDF. -* [xmldataset](https://xmldataset.readthedocs.io/en/latest/) - Simple XML Parsing. -* [xmltodict](https://github.com/martinblech/xmltodict) - Working with XML feel like you are working with JSON. - -## HTTP Clients - -*Libraries for working with HTTP.* - -* [httpx](https://github.com/encode/httpx) - A next generation HTTP client for Python. -* [requests](https://github.com/psf/requests) - HTTP Requests for Humans. -* [treq](https://github.com/twisted/treq) - Python requests like API built on top of Twisted's HTTP client. -* [urllib3](https://github.com/urllib3/urllib3) - A HTTP library with thread-safe connection pooling, file post support, sanity friendly. - -## Hardware - -*Libraries for programming with hardware.* - -* [keyboard](https://github.com/boppreh/keyboard) - Hook and simulate global keyboard events on Windows and Linux. -* [mouse](https://github.com/boppreh/mouse) - Hook and simulate global mouse events on Windows and Linux. -* [pynput](https://github.com/moses-palmer/pynput) - A library to control and monitor input devices. -* [scapy](https://github.com/secdev/scapy) - A brilliant packet manipulation library. - -## Image Processing - -*Libraries for manipulating images.* - -* [pillow](https://github.com/python-pillow/Pillow) - Pillow is the friendly [PIL](http://www.pythonware.com/products/pil/) fork. -* [python-barcode](https://github.com/WhyNotHugo/python-barcode) - Create barcodes in Python with no extra dependencies. -* [pymatting](http://github.com/pymatting/pymatting) - A library for alpha matting. -* [python-qrcode](https://github.com/lincolnloop/python-qrcode) - A pure Python QR Code generator. -* [pywal](https://github.com/dylanaraps/pywal) - A tool that generates color schemes from images. -* [pyvips](https://github.com/libvips/pyvips) - A fast image processing library with low memory needs. -* [quads](https://github.com/fogleman/Quads) - Computer art based on quadtrees. -* [scikit-image](http://scikit-image.org/) - A Python library for (scientific) image processing. -* [thumbor](https://github.com/thumbor/thumbor) - A smart imaging service. It enables on-demand crop, re-sizing and flipping of images. -* [wand](https://github.com/emcconville/wand) - Python bindings for [MagickWand](http://www.imagemagick.org/script/magick-wand.php), C API for ImageMagick. - -## Implementations - -*Implementations of Python.* - -* [cpython](https://github.com/python/cpython) - **Default, most widely used implementation of the Python programming language written in C.** -* [cython](https://github.com/cython/cython) - Optimizing Static Compiler for Python. -* [clpython](https://github.com/metawilm/cl-python) - Implementation of the Python programming language written in Common Lisp. -* [ironpython](https://github.com/IronLanguages/ironpython3) - Implementation of the Python programming language written in C#. -* [micropython](https://github.com/micropython/micropython) - A lean and efficient Python programming language implementation. -* [numba](https://github.com/numba/numba) - Python JIT compiler to LLVM aimed at scientific Python. -* [peachpy](https://github.com/Maratyszcza/PeachPy) - x86-64 assembler embedded in Python. -* [pypy](https://foss.heptapod.net/pypy/pypy) - A very fast and compliant implementation of the Python language. -* [pyston](https://github.com/pyston/pyston/) - A Python implementation using JIT techniques. - -## Interactive Interpreter - -*Interactive Python interpreters (REPL).* - -* [bpython](https://github.com/bpython/bpython) - A fancy interface to the Python interpreter. -* [Jupyter Notebook (IPython)](https://jupyter.org) - A rich toolkit to help you make the most out of using Python interactively. - * [awesome-jupyter](https://github.com/markusschanta/awesome-jupyter) -* [ptpython](https://github.com/jonathanslenders/ptpython) - Advanced Python REPL built on top of the [python-prompt-toolkit](https://github.com/jonathanslenders/python-prompt-toolkit). - -## Internationalization - -*Libraries for working with i18n.* - -* [Babel](http://babel.pocoo.org/en/latest/) - An internationalization library for Python. -* [PyICU](https://github.com/ovalhub/pyicu) - A wrapper of International Components for Unicode C++ library ([ICU](http://site.icu-project.org/)). - -## Job Scheduler - -*Libraries for scheduling jobs.* - -* [Airflow](https://airflow.apache.org/) - Airflow is a platform to programmatically author, schedule and monitor workflows. -* [APScheduler](http://apscheduler.readthedocs.io/en/latest/) - A light but powerful in-process task scheduler that lets you schedule functions. -* [django-schedule](https://github.com/thauber/django-schedule) - A calendaring app for Django. -* [doit](http://pydoit.org/) - A task runner and build tool. -* [gunnery](https://github.com/gunnery/gunnery) - Multipurpose task execution tool for distributed systems with web-based interface. -* [Joblib](https://joblib.readthedocs.io/) - A set of tools to provide lightweight pipelining in Python. -* [Plan](https://github.com/fengsp/plan) - Writing crontab file in Python like a charm. -* [Prefect](https://github.com/PrefectHQ/prefect) - A modern workflow orchestration framework that makes it easy to build, schedule and monitor robust data pipelines. -* [schedule](https://github.com/dbader/schedule) - Python job scheduling for humans. -* [Spiff](https://github.com/knipknap/SpiffWorkflow) - A powerful workflow engine implemented in pure Python. -* [TaskFlow](https://docs.openstack.org/developer/taskflow/) - A Python library that helps to make task execution easy, consistent and reliable. - -## Logging - -*Libraries for generating and working with logs.* - -* [logbook](http://logbook.readthedocs.io/en/stable/) - Logging replacement for Python. -* [logging](https://docs.python.org/3/library/logging.html) - (Python standard library) Logging facility for Python. -* [loguru](https://github.com/Delgan/loguru) - Library which aims to bring enjoyable logging in Python. -* [sentry-python](https://github.com/getsentry/sentry-python) - Sentry SDK for Python. -* [structlog](https://www.structlog.org/en/stable/) - Structured logging made easy. - -## Machine Learning - -*Libraries for Machine Learning. Also see [awesome-machine-learning](https://github.com/josephmisiti/awesome-machine-learning#python).* - -* [gym](https://github.com/openai/gym) - A toolkit for developing and comparing reinforcement learning algorithms. -* [H2O](https://github.com/h2oai/h2o-3) - Open Source Fast Scalable Machine Learning Platform. -* [Metrics](https://github.com/benhamner/Metrics) - Machine learning evaluation metrics. -* [NuPIC](https://github.com/numenta/nupic) - Numenta Platform for Intelligent Computing. -* [scikit-learn](http://scikit-learn.org/) - The most popular Python library for Machine Learning. -* [Spark ML](http://spark.apache.org/docs/latest/ml-guide.html) - [Apache Spark](http://spark.apache.org/)'s scalable Machine Learning library. -* [vowpal_porpoise](https://github.com/josephreisinger/vowpal_porpoise) - A lightweight Python wrapper for [Vowpal Wabbit](https://github.com/JohnLangford/vowpal_wabbit/). -* [xgboost](https://github.com/dmlc/xgboost) - A scalable, portable, and distributed gradient boosting library. -* [MindsDB](https://github.com/mindsdb/mindsdb) - MindsDB is an open source AI layer for existing databases that allows you to effortlessly develop, train and deploy state-of-the-art machine learning models using standard queries. - -## Microsoft Windows - -*Python programming on Microsoft Windows.* - -* [Python(x,y)](http://python-xy.github.io/) - Scientific-applications-oriented Python Distribution based on Qt and Spyder. -* [pythonlibs](http://www.lfd.uci.edu/~gohlke/pythonlibs/) - Unofficial Windows binaries for Python extension packages. -* [PythonNet](https://github.com/pythonnet/pythonnet) - Python Integration with the .NET Common Language Runtime (CLR). -* [PyWin32](https://github.com/mhammond/pywin32) - Python Extensions for Windows. -* [WinPython](https://winpython.github.io/) - Portable development environment for Windows 7/8. - -## Miscellaneous - -*Useful libraries or tools that don't fit in the categories above.* - -* [blinker](https://github.com/jek/blinker) - A fast Python in-process signal/event dispatching system. -* [boltons](https://github.com/mahmoud/boltons) - A set of pure-Python utilities. -* [itsdangerous](https://github.com/pallets/itsdangerous) - Various helpers to pass trusted data to untrusted environments. -* [magenta](https://github.com/magenta/magenta) - A tool to generate music and art using artificial intelligence. -* [pluginbase](https://github.com/mitsuhiko/pluginbase) - A simple but flexible plugin system for Python. -* [tryton](http://www.tryton.org/) - A general purpose business framework. - -## Natural Language Processing - -*Libraries for working with human languages.* +- [Data Analysis](#data-analysis) +- [Data Ingestion / ETL](#data-ingestion--etl) +- [Data Validation](#data-validation) +- [Data Visualization](#data-visualization) +- [Geolocation](#geolocation) +- [Science](#science) +- [Quantum Computing](#quantum-computing) + +**Developer Tools** + +- [Algorithms and Design Patterns](#algorithms-and-design-patterns) +- [Interactive Interpreter](#interactive-interpreter) +- [Code Analysis](#code-analysis) +- [Testing](#testing) +- [Debugging Tools](#debugging-tools) +- [Build Tools](#build-tools) +- [Documentation](#documentation) + +**DevOps** + +- [DevOps Tools](#devops-tools) +- [Distributed Computing](#distributed-computing) +- [Task Queues](#task-queues) +- [Messaging](#messaging) +- [Job Schedulers](#job-schedulers) +- [Logging](#logging) +- [Network Virtualization](#network-virtualization) + +**CLI & GUI** + +- [CLI Development](#cli-development) +- [CLI Tools](#cli-tools) +- [GUI Development](#gui-development) + +**Text & Documents** + +- [Text Processing](#text-processing) +- [HTML Manipulation](#html-manipulation) +- [File Format Processing](#file-format-processing) +- [File Manipulation](#file-manipulation) + +**Media** + +- [Image Processing](#image-processing) +- [Audio & Video Processing](#audio--video-processing) +- [Game Development](#game-development) + +**Python Language** + +- [Implementations](#implementations) +- [Built-in Classes Enhancement](#built-in-classes-enhancement) +- [Functional Programming](#functional-programming) +- [Asynchronous Programming](#asynchronous-programming) +- [Date and Time](#date-and-time) + +**Python Toolchain** + +- [Environment Management](#environment-management) +- [Package Management](#package-management) +- [Package Repositories](#package-repositories) +- [Distribution](#distribution) +- [Configuration Files](#configuration-files) + +**Security** + +- [Cryptography](#cryptography) +- [Penetration Testing](#penetration-testing) +- [Supply Chain Security](#supply-chain-security) +- [Web Security](#web-security) + +**Other** + +- [Hardware](#hardware) +- [Microsoft Windows](#microsoft-windows) +- [Miscellaneous](#miscellaneous) + +## Projects + +**AI & ML** + +### AI and Agents + +_Libraries for building AI applications, LLM integrations, and autonomous agents._ + +- Agent Skills + - [django-ai-plugins](https://github.com/vintasoftware/django-ai-plugins) - Django backend agent skills for Django, DRF, Celery, and Django-specific code review. + - [sentry-skills](https://github.com/getsentry/skills) - Agent skills the Sentry team uses for code review, pull requests, and Django reviews. + - [trailofbits-skills](https://github.com/trailofbits/skills) - Security skills for vulnerability detection, auditing, and testing. +- Orchestration + - [langchain](https://github.com/langchain-ai/langchain) - A framework for building agents and LLM-powered applications. + - [langgraph](https://github.com/langchain-ai/langgraph) - Low-level orchestration framework for building stateful, long-running LLM agents. + - [pydantic-ai](https://github.com/pydantic/pydantic-ai) - A Python agent framework for building generative AI applications with structured schemas. + - [crewai](https://github.com/crewAIInc/crewAI) - A framework for orchestrating role-playing autonomous AI agents for collaborative task solving. +- Vendor Agent SDKs + - [claude-agent-sdk](https://github.com/anthropics/claude-agent-sdk-python) - Anthropic's Python SDK for building AI agents on Claude Code's harness — custom tools, in-process MCP servers, hooks. + - [openai-agents](https://github.com/openai/openai-agents-python) - OpenAI's framework for building and managing AI agents. + - [google-adk](https://github.com/google/adk-python) - Google's code-first toolkit for building, evaluating, and deploying AI agents. +- Model Context Protocol + - [mcp](https://github.com/modelcontextprotocol/python-sdk) - The official Python SDK for building Model Context Protocol servers and clients. + - [fastmcp](https://github.com/PrefectHQ/fastmcp) - A high-level, Pythonic framework for building MCP servers and clients. +- Personal Assistants + - [hermes-agent](https://github.com/NousResearch/hermes-agent) - An adaptive personal AI assistant that grows with you. + - [AstrBot](https://github.com/AstrBotDevs/AstrBot) - A multi-platform AI assistant that connects LLMs to chat apps like Telegram, Slack, and QQ, extensible with Python plugins. +- Prompt Optimization + - [dspy](https://github.com/stanfordnlp/dspy) - A framework for programming, not prompting, language models. +- Data Layer + - [instructor](https://github.com/567-labs/instructor) - A library for extracting structured data from LLMs, powered by Pydantic. + - [llama-index](https://github.com/run-llama/llama_index) - A toolkit for building RAG pipelines and agents over your data. + - [mem0](https://github.com/mem0ai/mem0) - An intelligent memory layer for AI agents enabling personalized interactions. + - [openviking](https://github.com/volcengine/OpenViking) - A context database for AI agents that unifies memory, resources, and skills. + - [semantica](https://github.com/semantica-agi/semantica) - A graph-native context and knowledge layer for AI agents with reasoning, provenance, and governance. +- Pre-trained Models + - [transformers](https://github.com/huggingface/transformers) - The model-definition framework for pretrained models in text, computer vision, audio, video, and multimodal tasks, for inference and training. +- LLM Inference and Serving + - [sglang](https://github.com/sgl-project/sglang) - A high-performance serving framework for large language models and multimodal models. + - [vllm](https://github.com/vllm-project/vllm) - A high-throughput and memory-efficient inference and serving engine for LLMs. + - [mlx-lm](https://github.com/ml-explore/mlx-lm) - Run and fine-tune large language models on Apple Silicon with MLX. +- LLM Gateways + - [litellm](https://github.com/BerriAI/litellm) - Call 100+ LLMs using OpenAI format. +- Image and Video Generation + - [diffusers](https://github.com/huggingface/diffusers) - A library that provides pre-trained diffusion models for generating and editing images, audio, and video. +- Fine-tuning + - [peft](https://github.com/huggingface/peft) - A library for parameter-efficient fine-tuning of large pretrained models. + - [trl](https://github.com/huggingface/trl) - A library for post-training transformer language models with SFT, DPO, GRPO, and other trainers. + - [unsloth](https://github.com/unslothai/unsloth) - Faster, lower-memory LLM fine-tuning, as a Python library or a desktop app. + - [axolotl](https://github.com/axolotl-ai-cloud/axolotl) - A framework for fine-tuning and post-training large language models. +- Speech + - [faster-whisper](https://github.com/SYSTRAN/faster-whisper) - A Whisper reimplementation on CTranslate2, up to 4 times faster than openai-whisper with less memory. + - [openai-whisper](https://github.com/openai/whisper) - A general-purpose automatic speech recognition model trained on 680k hours of multilingual and multitask supervised data. + - [gTTS](https://github.com/pndurette/gTTS) - Python library and CLI tool for converting text to speech using Google Translate TTS. + - [funasr](https://github.com/modelscope/FunASR) - Industrial-grade speech recognition toolkit with speaker diarization and emotion detection. + +### Deep Learning + +_Frameworks for Neural Networks and Deep Learning. Also see [awesome-deep-learning](https://github.com/ChristosChristofidis/awesome-deep-learning)._ + +- Frameworks + - [pytorch](https://github.com/pytorch/pytorch) - Tensors and Dynamic neural networks in Python with strong GPU acceleration. + - [pytorch-lightning](https://github.com/Lightning-AI/pytorch-lightning) - Deep learning framework to train, deploy, and ship AI products Lightning fast. + - [jax](https://github.com/jax-ml/jax) - A library for high-performance numerical computing with automatic differentiation and JIT compilation. + - [keras](https://github.com/keras-team/keras) - A high-level deep learning library with support for JAX, TensorFlow, and PyTorch backends. + - [tensorflow](https://github.com/tensorflow/tensorflow) - An end-to-end machine learning platform from Google. +- Reinforcement Learning + - [gymnasium](https://github.com/Farama-Foundation/Gymnasium) - A standard API for reinforcement learning environments with popular reference environments ([gym](https://github.com/openai/gym) successor). + - [stable-baselines3](https://github.com/DLR-RM/stable-baselines3) - PyTorch implementations of Stable Baselines (deep) reinforcement learning algorithms. + +### Machine Learning + +_Libraries for Machine Learning. Also see [awesome-machine-learning](https://github.com/josephmisiti/awesome-machine-learning#python)._ - General - * [gensim](https://github.com/RaRe-Technologies/gensim) - Topic Modeling for Humans. - * [langid.py](https://github.com/saffsd/langid.py) - Stand-alone language identification system. - * [nltk](http://www.nltk.org/) - A leading platform for building Python programs to work with human language data. - * [pattern](https://github.com/clips/pattern) - A web mining module. - * [polyglot](https://github.com/aboSamoor/polyglot) - Natural language pipeline supporting hundreds of languages. - * [pytext](https://github.com/facebookresearch/pytext) - A natural language modeling framework based on PyTorch. - * [PyTorch-NLP](https://github.com/PetrochukM/PyTorch-NLP) - A toolkit enabling rapid deep learning NLP prototyping for research. - * [spacy](https://spacy.io/) - A library for industrial-strength natural language processing in Python and Cython. - * [Stanza](https://github.com/stanfordnlp/stanza) - The Stanford NLP Group's official Python library, supporting 60+ languages. + - [scikit-learn](https://github.com/scikit-learn/scikit-learn) - The most popular Python library for Machine Learning with extensive documentation and community support. + - [pgmpy](https://github.com/pgmpy/pgmpy) - A Python library for causal and probabilistic reasoning with graphical models. + - [feature-engine](https://github.com/feature-engine/feature_engine) - sklearn compatible API with the widest toolset for feature engineering and selection. +- Gradient Boosting + - [xgboost](https://github.com/dmlc/xgboost) - A scalable, portable, and distributed gradient boosting library. + - [lightgbm](https://github.com/lightgbm-org/LightGBM) - A fast, distributed, high performance gradient boosting framework. + - [catboost](https://github.com/catboost/catboost) - A fast, scalable, high performance gradient boosting on decision trees library. +- Time Series Forecasting + - [prophet](https://github.com/facebook/prophet) - A tool for producing forecasts for time series with multiple seasonality and trend changes. + - [statsforecast](https://github.com/Nixtla/statsforecast) - Fast statistical forecasting models such as ARIMA, ETS, and Theta, compiled with numba. + - [sktime](https://github.com/sktime/sktime) - A unified scikit-learn-style framework for forecasting and other time-series learning tasks. + - [timesfm](https://github.com/google-research/timesfm) - A pretrained foundation model from Google Research for time-series forecasting, with non-commercial default weights. + +### Natural Language Processing + +_Libraries for working with human languages._ + +- General + - [nltk](https://github.com/nltk/nltk) - A leading platform for building Python programs to work with human language data. + - [spacy](https://github.com/explosion/spaCy) - A library for industrial-strength natural language processing in Python and Cython. + - [gensim](https://github.com/piskvorky/gensim) - Topic Modeling for Humans. + - [stanza](https://github.com/stanfordnlp/stanza) - The Stanford NLP Group's official Python library, supporting 60+ languages. - Chinese - * [funNLP](https://github.com/fighting41love/funNLP) - A collection of tools and datasets for Chinese NLP. - * [jieba](https://github.com/fxsjy/jieba) - The most popular Chinese text segmentation library. - * [pkuseg-python](https://github.com/lancopku/pkuseg-python) - A toolkit for Chinese word segmentation in various domains. - * [snownlp](https://github.com/isnowfy/snownlp) - A library for processing Chinese text. + - [jieba](https://github.com/fxsjy/jieba) - The most popular Chinese text segmentation library. + - [pypinyin](https://github.com/mozillazg/python-pinyin) - Convert Chinese hanzi (漢字) to pinyin (拼音). + - [pangu.py](https://github.com/vinta/pangu.py) - Paranoid text spacing. -## Network Virtualization +### Computer Vision -*Tools and libraries for Virtual Networking and SDN (Software Defined Networking).* +_Libraries for image and video analysis, object detection, and OCR._ -* [mininet](https://github.com/mininet/mininet) - A popular network emulator and API written in Python. -* [napalm](https://github.com/napalm-automation/napalm) - Cross-vendor API to manipulate network devices. -* [pox](https://github.com/noxrepo/pox) - A Python-based SDN control applications, such as OpenFlow SDN controllers. +- General + - [opencv-python](https://github.com/opencv/opencv-python) - Open Source Computer Vision Library. + - [ultralytics](https://github.com/ultralytics/ultralytics) - Ultralytics YOLO for object detection, segmentation, pose estimation, classification, and tracking. + - [kornia](https://github.com/kornia/kornia) - Open Source Differentiable Computer Vision Library for PyTorch. + - [fiftyone](https://github.com/voxel51/fiftyone) - The open-source tool for building high-quality datasets and computer vision models. +- OCR + - [pytesseract](https://github.com/madmaze/pytesseract) - A wrapper for the [Tesseract OCR](https://github.com/tesseract-ocr/tesseract) engine. + - [easyocr](https://github.com/JaidedAI/EasyOCR) - Ready-to-use OCR with 80+ languages supported. + - [paddleocr](https://github.com/PaddlePaddle/PaddleOCR) - Multilingual OCR and document parsing toolkit based on PaddlePaddle. -## News Feed +### Recommender Systems -*Libraries for building user's activities.* +_Libraries for building recommender systems._ -* [django-activity-stream](https://github.com/justquick/django-activity-stream) - Generating generic activity streams from the actions on your site. -* [Stream Framework](https://github.com/tschellenbach/Stream-Framework) - Building news feed and notification systems using Cassandra and Redis. +- [annoy](https://github.com/spotify/annoy) - Approximate Nearest Neighbors in C++/Python optimized for memory usage. +- [implicit](https://github.com/benfred/implicit) - A fast Python implementation of collaborative filtering for implicit datasets. +- [scikit-surprise](https://github.com/NicolasHug/Surprise) - A scikit for building and analyzing recommender systems. -## ORM +**Web Development** -*Libraries that implement Object-Relational Mapping or data mapping techniques.* +### Web Frameworks -* Relational Databases - * [Django Models](https://docs.djangoproject.com/en/dev/topics/db/models/) - The Django ORM. - * [SQLAlchemy](https://www.sqlalchemy.org/) - The Python SQL Toolkit and Object Relational Mapper. - * [awesome-sqlalchemy](https://github.com/dahlia/awesome-sqlalchemy) - * [dataset](https://github.com/pudo/dataset) - Store Python dicts in a database - works with SQLite, MySQL, and PostgreSQL. - * [orator](https://github.com/sdispater/orator) - The Orator ORM provides a simple yet beautiful ActiveRecord implementation. - * [orm](https://github.com/encode/orm) - An async ORM. - * [peewee](https://github.com/coleifer/peewee) - A small, expressive ORM. - * [pony](https://github.com/ponyorm/pony/) - ORM that provides a generator-oriented interface to SQL. - * [pydal](https://github.com/web2py/pydal/) - A pure Python Database Abstraction Layer. -* NoSQL Databases - * [hot-redis](https://github.com/stephenmcd/hot-redis) - Rich Python data types for Redis. - * [mongoengine](https://github.com/MongoEngine/mongoengine) - A Python Object-Document-Mapper for working with MongoDB. - * [PynamoDB](https://github.com/pynamodb/PynamoDB) - A Pythonic interface for [Amazon DynamoDB](https://aws.amazon.com/dynamodb/). - * [redisco](https://github.com/kiddouk/redisco) - A Python Library for Simple Models and Containers Persisted in Redis. +_Traditional full stack web frameworks. Also see [Web APIs](#web-apis)._ -## Package Management +- Synchronous + - [flask](https://github.com/pallets/flask) - A microframework for Python. + - [awesome-flask](https://github.com/humiaozuzu/awesome-flask) + - [django](https://github.com/django/django) - A high-level web framework that encourages rapid development and clean, pragmatic design. + - [awesome-django](https://github.com/wsvincent/awesome-django) + - [bottle](https://github.com/bottlepy/bottle) - A fast and simple micro-framework distributed as a single file with no dependencies. + - [pyramid](https://github.com/Pylons/pyramid) - A small, fast, down-to-earth, open source Python web framework. + - [fasthtml](https://github.com/AnswerDotAI/fasthtml) - The fastest way to create an HTML app. +- Asynchronous + - [starlette](https://github.com/Kludex/starlette) - A lightweight ASGI framework and toolkit for building high-performance async services. + - [tornado](https://github.com/tornadoweb/tornado) - A web framework and asynchronous networking library. + - [litestar](https://github.com/litestar-org/litestar) - Production-ready, capable and extensible ASGI Web framework. + - [reflex](https://github.com/reflex-dev/reflex) - A framework for building reactive, full-stack web applications entirely with Python. -*Libraries for package and dependency management.* +### Web APIs -* [pip](https://pip.pypa.io/en/stable/) - The package installer for Python. - * [pip-tools](https://github.com/jazzband/pip-tools) - A set of tools to keep your pinned Python dependencies fresh. - * [PyPI](https://pypi.org/) -* [conda](https://github.com/conda/conda/) - Cross-platform, Python-agnostic binary package manager. -* [poetry](https://github.com/sdispater/poetry) - Python dependency management and packaging made easy. -* [uv](https://github.com/astral-sh/uv) - An extremely fast Python package and project manager, written in Rust. +_Libraries for building RESTful, GraphQL, and RPC APIs._ -## Package Repositories +- Django + - [django-rest-framework](https://github.com/encode/django-rest-framework) - A powerful and flexible toolkit to build web APIs. + - [django-ninja](https://github.com/vitalik/django-ninja) - Fast, Django REST framework based on type hints and Pydantic. + - [strawberry-django](https://github.com/strawberry-graphql/strawberry-django) - Strawberry GraphQL integration with Django. + - [django-modern-rest](https://github.com/wemake-services/django-modern-rest) - Modern REST with speed, types, async, `msgspec`, `pydantic` and other goodies. +- Flask + - [flask-restx](https://github.com/python-restx/flask-restx) - Fully featured framework for fast, easy and documented API development with Flask. + - [flask-smorest](https://github.com/marshmallow-code/flask-smorest) - A Flask/Marshmallow-based REST API framework with automatic OpenAPI documentation. + - [apiflask](https://github.com/apiflask/apiflask) - A lightweight Python web API framework based on Flask, supporting marshmallow schemas and Pydantic models. +- Framework Agnostic + - [fastapi](https://github.com/fastapi/fastapi) - A modern, fast, web framework for building APIs with standard Python type hints. + - [strawberry](https://github.com/strawberry-graphql/strawberry) - A GraphQL library that leverages Python type annotations for schema definition. + - [connexion](https://github.com/spec-first/connexion) - A spec-first framework that automatically handles requests based on your OpenAPI specification. +- RPC + - [grpcio](https://github.com/grpc/grpc) - HTTP/2-based RPC framework with Python bindings, built by Google. -*Local PyPI repository server and proxies.* +### Web Servers -* [bandersnatch](https://github.com/pypa/bandersnatch/) - PyPI mirroring tool provided by Python Packaging Authority (PyPA). -* [devpi](https://github.com/devpi/devpi) - PyPI server and packaging/testing/release tool. -* [localshop](https://github.com/jazzband/localshop) - Local PyPI server (custom packages and auto-mirroring of pypi). -* [warehouse](https://github.com/pypa/warehouse) - Next generation Python Package Repository (PyPI). +_ASGI and WSGI compatible web servers._ -## Penetration Testing +- ASGI + - [uvicorn](https://github.com/Kludex/uvicorn) - A lightning-fast ASGI server implementation. + - [granian](https://github.com/emmett-framework/granian) - A Rust HTTP server for Python applications built on top of Hyper and Tokio, supporting WSGI/ASGI/RSGI. + - [hypercorn](https://github.com/pgjones/hypercorn) - An ASGI and WSGI Server based on Hyper libraries and inspired by Gunicorn. +- WSGI + - [gunicorn](https://github.com/benoitc/gunicorn) - A pre-fork WSGI server with a native ASGI worker, ported from Ruby's Unicorn project. + - [waitress](https://github.com/Pylons/waitress) - Multi-threaded, powers Pyramid. -*Frameworks and tools for penetration testing.* +### WebSocket -* [fsociety](https://github.com/Manisso/fsociety) - A Penetration testing framework. -* [setoolkit](https://github.com/trustedsec/social-engineer-toolkit) - A toolkit for social engineering. -* [sqlmap](https://github.com/sqlmapproject/sqlmap) - Automatic SQL injection and database takeover tool. +_Libraries for working with WebSocket._ -## Permissions +- [websockets](https://github.com/python-websockets/websockets) - A library for building WebSocket servers and clients with a focus on correctness and simplicity. +- [channels](https://github.com/django/channels) - Brings WebSocket, long-poll HTTP, and other async support to Django. +- [flask-socketio](https://github.com/miguelgrinberg/Flask-SocketIO) - Socket.IO integration for Flask applications. +- [autobahn-python](https://github.com/crossbario/autobahn-python) - WebSocket & WAMP for Python on Twisted and [asyncio](https://docs.python.org/3/library/asyncio.html). -*Libraries that allow or deny users access to data or functionality.* +### Template Engines -* [django-guardian](https://github.com/django-guardian/django-guardian) - Implementation of per object permissions for Django 1.2+ -* [django-rules](https://github.com/dfunckt/django-rules) - A tiny but powerful app providing object-level permissions to Django, without requiring a database. +_Libraries for rendering text and HTML from templates._ -## Processes +- [jinja](https://github.com/pallets/jinja) - A modern and designer friendly templating language. +- [mako](https://github.com/sqlalchemy/mako) - Hyperfast and lightweight templating for the Python platform. -*Libraries for starting and communicating with OS processes.* +### Web Asset Management -* [delegator.py](https://github.com/amitt001/delegator.py) - [Subprocesses](https://docs.python.org/3/library/subprocess.html) for Humans 2.0. -* [sarge](https://sarge.readthedocs.io/en/latest/) - Yet another wrapper for subprocess. -* [sh](https://github.com/amoffat/sh) - A full-fledged subprocess replacement for Python. +_Tools for managing, storing, compressing and minifying website assets._ -## Recommender Systems +- [whitenoise](https://github.com/evansd/whitenoise) - Radically simplified static file serving for WSGI applications, with compression and caching headers. +- [django-storages](https://github.com/jschneier/django-storages) - A collection of custom storage back ends for Django. +- [django-compressor](https://github.com/django-compressor/django-compressor) - Compresses linked and inline JavaScript or CSS into a single cached file. -*Libraries for building recommender systems.* +### Authentication -* [annoy](https://github.com/spotify/annoy) - Approximate Nearest Neighbors in C++/Python optimized for memory usage. -* [fastFM](https://github.com/ibayer/fastFM) - A library for Factorization Machines. -* [implicit](https://github.com/benfred/implicit) - A fast Python implementation of collaborative filtering for implicit datasets. -* [libffm](https://github.com/guestwalk/libffm) - A library for Field-aware Factorization Machine (FFM). -* [lightfm](https://github.com/lyst/lightfm) - A Python implementation of a number of popular recommendation algorithms. -* [spotlight](https://github.com/maciejkula/spotlight) - Deep recommender models using PyTorch. -* [Surprise](https://github.com/NicolasHug/Surprise) - A scikit for building and analyzing recommender systems. -* [tensorrec](https://github.com/jfkirk/tensorrec) - A Recommendation Engine Framework in TensorFlow. +_Libraries for implementing authentication schemes._ -## Refactoring +- OAuth + - [oauthlib](https://github.com/oauthlib/oauthlib) - A generic and thorough implementation of the OAuth request-signing logic. + - [authlib](https://github.com/authlib/authlib) - A comprehensive library for building OAuth, OpenID Connect, and JWT/JWS/JWE/JWK/JWA. + - [django-allauth](https://github.com/pennersr/django-allauth) - Authentication app for Django that "just works." + - [django-oauth-toolkit](https://github.com/django-oauth/django-oauth-toolkit) - An OAuth 2.0 authorization server for Django. +- JWT + - [pyjwt](https://github.com/jpadilla/pyjwt) - JSON Web Token implementation in Python. +- Permissions + - [django-guardian](https://github.com/django-guardian/django-guardian) - Implementation of per-object permissions for Django. + - [django-rules](https://github.com/dfunckt/django-rules) - A tiny but powerful app providing object-level permissions to Django, without requiring a database. -*Refactoring tools and libraries for Python* +### Admin Panels - * [Bicycle Repair Man](http://bicyclerepair.sourceforge.net/) - Bicycle Repair Man, a refactoring tool for Python. - * [Bowler](https://pybowler.io/) - Safe code refactoring for modern Python. - * [Rope](https://github.com/python-rope/rope) - Rope is a python refactoring library. +_Libraries for administrative interfaces._ -## RESTful API +- [flask-admin](https://github.com/pallets-eco/flask-admin) - Simple and extensible administrative interface framework for Flask. +- [django-unfold](https://github.com/unfoldadmin/django-unfold) - A modern Django admin theme for building dashboards, internal tools, and business applications. +- [sqladmin](https://github.com/smithyhq/sqladmin) - An admin interface for SQLAlchemy models in FastAPI and Starlette. +- [django-grappelli](https://github.com/sehmaschine/django-grappelli) - A jazzy skin for the Django Admin-Interface. -*Libraries for building RESTful APIs.* +### CMS -* Django - * [django-rest-framework](https://github.com/encode/django-rest-framework) - A powerful and flexible toolkit to build web APIs. - * [django-tastypie](https://github.com/django-tastypie/django-tastypie) - Creating delicious APIs for Django apps. -* Flask - * [eve](https://github.com/pyeve/eve) - REST API framework powered by Flask, MongoDB and good intentions. - * [flask-api](https://github.com/flask-api/flask-api) - Browsable Web APIs for Flask. - * [flask-restful](https://github.com/flask-restful/flask-restful) - Quickly building REST APIs for Flask. -* Pyramid - * [cornice](https://github.com/Cornices/cornice) - A RESTful framework for Pyramid. -* Framework agnostic - * [falcon](https://github.com/falconry/falcon) - A high-performance framework for building cloud APIs and web app backends. - * [fastapi](https://github.com/tiangolo/fastapi) - A modern, fast, web framework for building APIs with Python 3.6+ based on standard Python type hints. - * [hug](https://github.com/hugapi/hug) - A Python 3 framework for cleanly exposing APIs. - * [sandman2](https://github.com/jeffknupp/sandman2) - Automated REST APIs for existing database-driven systems. - * [sanic](https://github.com/sanic-org/sanic) - A Python 3.6+ web server and web framework that's written to go fast. - -## Robotics - -*Libraries for robotics.* - -* [PythonRobotics](https://github.com/AtsushiSakai/PythonRobotics) - This is a compilation of various robotics algorithms with visualizations. -* [rospy](http://wiki.ros.org/rospy) - This is a library for ROS (Robot Operating System). +_Content Management Systems._ -## RPC Servers +- [wagtail](https://github.com/wagtail/wagtail) - A Django content management system. +- [django-cms](https://github.com/django-cms/django-cms) - The easy-to-use and developer-friendly enterprise CMS powered by Django. -*RPC-compatible servers.* +### ERP -* [RPyC](https://github.com/tomerfiliba/rpyc) (Remote Python Call) - A transparent and symmetric RPC library for Python -* [zeroRPC](https://github.com/0rpc/zerorpc-python) - zerorpc is a flexible RPC implementation based on [ZeroMQ](http://zeromq.org/) and [MessagePack](http://msgpack.org/). +_Enterprise resource planning frameworks._ -## Science +- [odoo](https://github.com/odoo/odoo) - A suite of open source business apps: CRM, e-commerce, accounting, inventory, and thousands of community modules. -*Libraries for scientific computing. Also see [Python-for-Scientists](https://github.com/TomNicholas/Python-for-Scientists).* +### Static Site Generators -* [astropy](http://www.astropy.org/) - A community Python library for Astronomy. -* [bcbio-nextgen](https://github.com/chapmanb/bcbio-nextgen) - Providing best-practice pipelines for fully automated high throughput sequencing analysis. -* [bccb](https://github.com/chapmanb/bcbb) - Collection of useful code related to biological analysis. -* [Biopython](http://biopython.org/wiki/Main_Page) - Biopython is a set of freely available tools for biological computation. -* [cclib](http://cclib.github.io/) - A library for parsing and interpreting the results of computational chemistry packages. -* [Colour](http://colour-science.org/) - Implementing a comprehensive number of colour theory transformations and algorithms. -* [Karate Club](https://github.com/benedekrozemberczki/karateclub) - Unsupervised machine learning toolbox for graph structured data. -* [NetworkX](https://networkx.github.io/) - A high-productivity software for complex networks. -* [NIPY](http://nipy.org) - A collection of neuroimaging toolkits. -* [NumPy](http://www.numpy.org/) - A fundamental package for scientific computing with Python. -* [ObsPy](https://github.com/obspy/obspy/wiki/) - A Python toolbox for seismology. -* [Open Babel](https://open-babel.readthedocs.io/) - A chemical toolbox designed to speak the many languages of chemical data. -* [PyDy](http://www.pydy.org/) - Short for Python Dynamics, used to assist with workflow in the modeling of dynamic motion. -* [PyMC](https://github.com/pymc-devs/pymc3) - Markov Chain Monte Carlo sampling toolkit. -* [QuTiP](http://qutip.org/) - Quantum Toolbox in Python. -* [RDKit](http://www.rdkit.org/) - Cheminformatics and Machine Learning Software. -* [SciPy](https://www.scipy.org/) - A Python-based ecosystem of open-source software for mathematics, science, and engineering. -* [SimPy](https://gitlab.com/team-simpy/simpy) - A process-based discrete-event simulation framework. -* [statsmodels](https://github.com/statsmodels/statsmodels) - Statistical modeling and econometrics in Python. -* [SymPy](https://github.com/sympy/sympy) - A Python library for symbolic mathematics. -* [Zipline](https://github.com/quantopian/zipline) - A Pythonic algorithmic trading library. +_Static site generator is a software that takes some text + templates as input and produces HTML files on the output._ -## Search +- [pelican](https://github.com/getpelican/pelican) - Static site generator that supports Markdown and reST syntax. +- [nikola](https://github.com/getnikola/nikola) - A static website and blog generator. -*Libraries and software for indexing and performing search queries on data.* +**HTTP & Scraping** -* [django-haystack](https://github.com/django-haystack/django-haystack) - Modular search for Django. -* [elasticsearch-dsl-py](https://github.com/elastic/elasticsearch-dsl-py) - The official high-level Python client for Elasticsearch. -* [elasticsearch-py](https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/index.html) - The official low-level Python client for [Elasticsearch](https://www.elastic.co/products/elasticsearch). -* [pysolr](https://github.com/django-haystack/pysolr) - A lightweight Python wrapper for [Apache Solr](https://lucene.apache.org/solr/). -* [whoosh](http://whoosh.readthedocs.io/en/latest/) - A fast, pure Python search engine library. +### HTTP Clients -## Serialization +_Libraries for working with HTTP._ -*Libraries for serializing complex data types* - -* [marshmallow](https://github.com/marshmallow-code/marshmallow) - A lightweight library for converting complex objects to and from simple Python datatypes. -* [pysimdjson](https://github.com/TkTech/pysimdjson) - A Python bindings for [simdjson](https://github.com/lemire/simdjson). -* [python-rapidjson](https://github.com/python-rapidjson/python-rapidjson) - A Python wrapper around [RapidJSON](https://github.com/Tencent/rapidjson). -* [toonify](https://github.com/ScrapeGraphAI/toonify) - A compact, human-readable serialization format that reduces LLM token usage by 30-60% compared to JSON. -* [ultrajson](https://github.com/esnme/ultrajson) - A fast JSON decoder and encoder written in C with Python bindings. - -## Serverless Frameworks - -*Frameworks for developing serverless Python code.* - -* [python-lambda](https://github.com/nficano/python-lambda) - A toolkit for developing and deploying Python code in AWS Lambda. -* [Zappa](https://github.com/zappa/Zappa) - A tool for deploying WSGI applications on AWS Lambda and API Gateway. - -## Shell - -*Shells based on Python.* - -* [xonsh](https://github.com/xonsh/xonsh/) - A Python-powered, cross-platform, Unix-gazing shell language and command prompt. - -## Specific Formats Processing - -*Libraries for parsing and manipulating specific text formats.* - -* General - * [tablib](https://github.com/jazzband/tablib) - A module for Tabular Datasets in XLS, CSV, JSON, YAML. -* Office - * [docxtpl](https://github.com/elapouya/python-docx-template) - Editing a docx document by jinja2 template - * [openpyxl](https://openpyxl.readthedocs.io/en/stable/) - A library for reading and writing Excel 2010 xlsx/xlsm/xltx/xltm files. - * [pyexcel](https://github.com/pyexcel/pyexcel) - Providing one API for reading, manipulating and writing csv, ods, xls, xlsx and xlsm files. - * [python-docx](https://github.com/python-openxml/python-docx) - Reads, queries and modifies Microsoft Word 2007/2008 docx files. - * [python-pptx](https://github.com/scanny/python-pptx) - Python library for creating and updating PowerPoint (.pptx) files. - * [unoconv](https://github.com/unoconv/unoconv) - Convert between any document format supported by LibreOffice/OpenOffice. - * [XlsxWriter](https://github.com/jmcnamara/XlsxWriter) - A Python module for creating Excel .xlsx files. - * [xlwings](https://github.com/ZoomerAnalytics/xlwings) - A BSD-licensed library that makes it easy to call Python from Excel and vice versa. - * [xlwt](https://github.com/python-excel/xlwt) / [xlrd](https://github.com/python-excel/xlrd) - Writing and reading data and formatting information from Excel files. -* PDF - * [pdfminer.six](https://github.com/pdfminer/pdfminer.six) - Pdfminer.six is a community maintained fork of the original PDFMiner. - * [PyPDF2](https://github.com/mstamy2/PyPDF2) - A library capable of splitting, merging and transforming PDF pages. - * [ReportLab](https://www.reportlab.com/opensource/) - Allowing Rapid creation of rich PDF documents. -* Markdown - * [Mistune](https://github.com/lepture/mistune) - Fastest and full featured pure Python parsers of Markdown. - * [Python-Markdown](https://github.com/waylan/Python-Markdown) - A Python implementation of John Gruber’s Markdown. -* YAML - * [PyYAML](http://pyyaml.org/) - YAML implementations for Python. -* CSV - * [csvkit](https://github.com/wireservice/csvkit) - Utilities for converting to and working with CSV. -* Archive - * [unp](https://github.com/mitsuhiko/unp) - A command line tool that can unpack archives easily. - -## Static Site Generator - -*Static site generator is a software that takes some text + templates as input and produces HTML files on the output.* - -* [lektor](https://github.com/lektor/lektor) - An easy to use static CMS and blog engine. -* [mkdocs](https://github.com/mkdocs/mkdocs/) - Markdown friendly documentation generator. -* [makesite](https://github.com/sunainapai/makesite) - Simple, lightweight, and magic-free static site/blog generator (< 130 lines). -* [nikola](https://github.com/getnikola/nikola) - A static website and blog generator. -* [pelican](https://github.com/getpelican/pelican) - Static site generator that supports Markdown and reST syntax. - -## Tagging - -*Libraries for tagging items.* - -* [django-taggit](https://github.com/jazzband/django-taggit) - Simple tagging for Django. - -## Task Queues - -*Libraries for working with task queues.* - -* [celery](https://docs.celeryproject.org/en/stable/) - An asynchronous task queue/job queue based on distributed message passing. -* [dramatiq](https://github.com/Bogdanp/dramatiq) - A fast and reliable background task processing library for Python 3. -* [huey](https://github.com/coleifer/huey) - Little multi-threaded task queue. -* [mrq](https://github.com/pricingassistant/mrq) - A distributed worker task queue in Python using Redis & gevent. -* [rq](https://github.com/rq/rq) - Simple job queues for Python. - -## Template Engine - -*Libraries and tools for templating and lexing.* - -* [Genshi](https://genshi.edgewall.org/) - Python templating toolkit for generation of web-aware output. -* [Jinja2](https://github.com/pallets/jinja) - A modern and designer friendly templating language. -* [Mako](http://www.makotemplates.org/) - Hyperfast and lightweight templating for the Python platform. - -## Testing - -*Libraries for testing codebases and generating test data.* - -* Testing Frameworks - * [hypothesis](https://github.com/HypothesisWorks/hypothesis) - Hypothesis is an advanced Quickcheck style property based testing library. - * [nose2](https://github.com/nose-devs/nose2) - The successor to `nose`, based on `unittest2. - * [pytest](https://docs.pytest.org/en/latest/) - A mature full-featured Python testing tool. - * [Robot Framework](https://github.com/robotframework/robotframework) - A generic test automation framework. - * [unittest](https://docs.python.org/3/library/unittest.html) - (Python standard library) Unit testing framework. -* Test Runners - * [green](https://github.com/CleanCut/green) - A clean, colorful test runner. - * [mamba](http://nestorsalceda.github.io/mamba/) - The definitive testing tool for Python. Born under the banner of BDD. - * [tox](https://tox.readthedocs.io/en/latest/) - Auto builds and tests distributions in multiple Python versions -* GUI / Web Testing - * [locust](https://github.com/locustio/locust) - Scalable user load testing tool written in Python. - * [PyAutoGUI](https://github.com/asweigart/pyautogui) - PyAutoGUI is a cross-platform GUI automation Python module for human beings. - * [Schemathesis](https://github.com/kiwicom/schemathesis) - A tool for automatic property-based testing of web applications built with Open API / Swagger specifications. - * [Selenium](https://pypi.org/project/selenium/) - Python bindings for [Selenium](https://selenium.dev/) [WebDriver](https://selenium.dev/documentation/webdriver/). - * [sixpack](https://github.com/seatgeek/sixpack) - A language-agnostic A/B Testing framework. - * [splinter](https://github.com/cobrateam/splinter) - Open source tool for testing web applications. -* Mock - * [doublex](https://pypi.org/project/doublex/) - Powerful test doubles framework for Python. - * [freezegun](https://github.com/spulec/freezegun) - Travel through time by mocking the datetime module. - * [httmock](https://github.com/patrys/httmock) - A mocking library for requests for Python 2.6+ and 3.2+. - * [httpretty](https://github.com/gabrielfalcao/HTTPretty) - HTTP request mock tool for Python. - * [mock](https://docs.python.org/3/library/unittest.mock.html) - (Python standard library) A mocking and patching library. - * [mocket](https://github.com/mindflayer/python-mocket) - A socket mock framework with gevent/asyncio/SSL support. - * [responses](https://github.com/getsentry/responses) - A utility library for mocking out the requests Python library. - * [VCR.py](https://github.com/kevin1024/vcrpy) - Record and replay HTTP interactions on your tests. -* Object Factories - * [factory_boy](https://github.com/FactoryBoy/factory_boy) - A test fixtures replacement for Python. - * [mixer](https://github.com/klen/mixer) - Another fixtures replacement. Supports Django, Flask, SQLAlchemy, Peewee and etc. - * [model_mommy](https://github.com/vandersonmota/model_mommy) - Creating random fixtures for testing in Django. -* Code Coverage - * [coverage](https://pypi.org/project/coverage/) - Code coverage measurement. -* Fake Data - * [fake2db](https://github.com/emirozer/fake2db) - Fake database generator. - * [faker](https://github.com/joke2k/faker) - A Python package that generates fake data. - * [mimesis](https://github.com/lk-geimfari/mimesis) - is a Python library that help you generate fake data. - * [radar](https://pypi.org/project/radar/) - Generate random datetime / time. - -## Text Processing - -*Libraries for parsing and manipulating plain texts.* - -* General - * [chardet](https://github.com/chardet/chardet) - Python 2/3 compatible character encoding detector. - * [difflib](https://docs.python.org/3/library/difflib.html) - (Python standard library) Helpers for computing deltas. - * [ftfy](https://github.com/LuminosoInsight/python-ftfy) - Makes Unicode text less broken and more consistent automagically. - * [fuzzywuzzy](https://github.com/seatgeek/fuzzywuzzy) - Fuzzy String Matching. - * [Levenshtein](https://github.com/ztane/python-Levenshtein/) - Fast computation of Levenshtein distance and string similarity. - * [pangu.py](https://github.com/vinta/pangu.py) - Paranoid text spacing. - * [pyfiglet](https://github.com/pwaller/pyfiglet) - An implementation of figlet written in Python. - * [pypinyin](https://github.com/mozillazg/python-pinyin) - Convert Chinese hanzi (漢字) to pinyin (拼音). - * [textdistance](https://github.com/orsinium/textdistance) - Compute distance between sequences with 30+ algorithms. - * [unidecode](https://pypi.org/project/Unidecode/) - ASCII transliterations of Unicode text. -* Slugify - * [awesome-slugify](https://github.com/dimka665/awesome-slugify) - A Python slugify library that can preserve unicode. - * [python-slugify](https://github.com/un33k/python-slugify) - A Python slugify library that translates unicode to ASCII. - * [unicode-slugify](https://github.com/mozilla/unicode-slugify) - A slugifier that generates unicode slugs with Django as a dependency. -* Unique identifiers - * [hashids](https://github.com/davidaurelio/hashids-python) - Implementation of [hashids](http://hashids.org) in Python. - * [shortuuid](https://github.com/skorokithakis/shortuuid) - A generator library for concise, unambiguous and URL-safe UUIDs. -* Parser - * [ply](https://github.com/dabeaz/ply) - Implementation of lex and yacc parsing tools for Python. - * [pygments](http://pygments.org/) - A generic syntax highlighter. - * [pyparsing](https://github.com/pyparsing/pyparsing) - A general purpose framework for generating parsers. - * [python-nameparser](https://github.com/derek73/python-nameparser) - Parsing human names into their individual components. - * [python-phonenumbers](https://github.com/daviddrysdale/python-phonenumbers) - Parsing, formatting, storing and validating international phone numbers. - * [python-user-agents](https://github.com/selwin/python-user-agents) - Browser user agent parser. - * [sqlparse](https://github.com/andialbrecht/sqlparse) - A non-validating SQL parser. - -## Third-party APIs - -*Libraries for accessing third party services APIs. Also see [List of Python API Wrappers and Libraries](https://github.com/realpython/list-of-python-api-wrappers).* - -* [apache-libcloud](https://libcloud.apache.org/) - One Python library for all clouds. -* [boto3](https://github.com/boto/boto3) - Python interface to Amazon Web Services. -* [django-wordpress](https://github.com/istrategylabs/django-wordpress) - WordPress models and views for Django. -* [facebook-sdk](https://github.com/mobolic/facebook-sdk) - Facebook Platform Python SDK. -* [google-api-python-client](https://github.com/google/google-api-python-client) - Google APIs Client Library for Python. -* [gspread](https://github.com/burnash/gspread) - Google Spreadsheets Python API. -* [twython](https://github.com/ryanmcgrath/twython) - A Python wrapper for the Twitter API. - -## URL Manipulation - -*Libraries for parsing URLs.* - -* [furl](https://github.com/gruns/furl) - A small Python library that makes parsing and manipulating URLs easy. -* [purl](https://github.com/codeinthehole/purl) - A simple, immutable URL class with a clean API for interrogation and manipulation. -* [pyshorteners](https://github.com/ellisonleao/pyshorteners) - A pure Python URL shortening lib. -* [webargs](https://github.com/marshmallow-code/webargs) - A friendly library for parsing HTTP request arguments with built-in support for popular web frameworks. - -## Video - -*Libraries for manipulating video and GIFs.* - -* [moviepy](https://zulko.github.io/moviepy/) - A module for script-based movie editing with many formats, including animated GIFs. -* [scikit-video](https://github.com/aizvorski/scikit-video) - Video processing routines for SciPy. -* [vidgear](https://github.com/abhiTronix/vidgear) - Most Powerful multi-threaded Video Processing framework. - -## Web Asset Management - -*Tools for managing, compressing and minifying website assets.* - -* [django-compressor](https://github.com/django-compressor/django-compressor) - Compresses linked and inline JavaScript or CSS into a single cached file. -* [django-pipeline](https://github.com/jazzband/django-pipeline) - An asset packaging library for Django. -* [django-storages](https://github.com/jschneier/django-storages) - A collection of custom storage back ends for Django. -* [fanstatic](http://www.fanstatic.org/en/latest/) - Packages, optimizes, and serves static file dependencies as Python packages. -* [fileconveyor](http://wimleers.com/fileconveyor) - A daemon to detect and sync files to CDNs, S3 and FTP. -* [flask-assets](https://github.com/miracle2k/flask-assets) - Helps you integrate webassets into your Flask app. -* [webassets](https://github.com/miracle2k/webassets) - Bundles, optimizes, and manages unique cache-busting URLs for static resources. - -## Web Content Extracting - -*Libraries for extracting web contents.* - -* [html2text](https://github.com/Alir3z4/html2text) - Convert HTML to Markdown-formatted text. -* [lassie](https://github.com/michaelhelmick/lassie) - Web Content Retrieval for Humans. -* [micawber](https://github.com/coleifer/micawber) - A small library for extracting rich content from URLs. -* [newspaper](https://github.com/codelucas/newspaper) - News extraction, article extraction and content curation in Python. -* [python-readability](https://github.com/buriy/python-readability) - Fast Python port of arc90's readability tool. -* [requests-html](https://github.com/psf/requests-html) - Pythonic HTML Parsing for Humans. -* [sumy](https://github.com/miso-belica/sumy) - A module for automatic summarization of text documents and HTML pages. -* [textract](https://github.com/deanmalmgren/textract) - Extract text from any document, Word, PowerPoint, PDFs, etc. -* [toapi](https://github.com/gaojiuli/toapi) - Every web site provides APIs. - -## Web Crawling - -*Libraries to automate web scraping.* - -* [feedparser](https://github.com/kurtmckee/feedparser) - Universal feed parser. -* [grab](https://github.com/lorien/grab) - Site scraping framework. -* [mechanicalsoup](https://github.com/MechanicalSoup/MechanicalSoup) - A Python library for automating interaction with websites. -* [scrapy](https://github.com/scrapy/scrapy) - A fast high-level screen scraping and web crawling framework. - -## Web Frameworks - -*Traditional full stack web frameworks. Also see [RESTful API](https://github.com/vinta/awesome-python#restful-api).* - -* Synchronous - * [django](https://github.com/django/django) - The most popular web framework in Python. - * [awesome-django](https://github.com/shahraizali/awesome-django) - * [awesome-django](https://github.com/wsvincent/awesome-django) - * [flask](https://github.com/pallets/flask) - A microframework for Python. - * [awesome-flask](https://github.com/humiaozuzu/awesome-flask) - * [pyramid](https://pylonsproject.org/) - A small, fast, down-to-earth, open source Python web framework. - * [awesome-pyramid](https://github.com/uralbash/awesome-pyramid) - * [masonite](https://github.com/MasoniteFramework/masonite) - The modern and developer centric Python web framework. -* Asynchronous - * [tornado](https://github.com/tornadoweb/tornado) - A web framework and asynchronous networking library. - -## WebSocket - -*Libraries for working with WebSocket.* +- General + - [requests](https://github.com/psf/requests) - HTTP Requests for Humans. + - [aiohttp](https://github.com/aio-libs/aiohttp) - Asynchronous HTTP client/server framework for asyncio and Python. + - [httpx2](https://github.com/pydantic/httpx2) - HTTP/1.1 and HTTP/2 client with sync and async APIs, maintained by Pydantic ([httpx](https://github.com/encode/httpx) fork). + - [urllib3](https://github.com/urllib3/urllib3) - An HTTP library with thread-safe connection pooling, file post, and more. + - [httpx](https://github.com/encode/httpx) - A next generation HTTP client for Python. +- URL Manipulation + - [yarl](https://github.com/aio-libs/yarl) - Yet another URL library. -* [autobahn-python](https://github.com/crossbario/autobahn-python) - WebSocket & WAMP for Python on Twisted and [asyncio](https://docs.python.org/3/library/asyncio.html). -* [channels](https://github.com/django/channels) - Developer-friendly asynchrony for Django. -* [websockets](https://github.com/aaugustin/websockets) - A library for building WebSocket servers and clients with a focus on correctness and simplicity. +### Web Scraping -## WSGI Servers +_Libraries to automate web scraping and extract web content._ -*WSGI-compatible web servers.* +- Frameworks + - [browser-use](https://github.com/browser-use/browser-use) - Make websites accessible for AI agents with easy browser automation. + - [scrapy](https://github.com/scrapy/scrapy) - A fast high-level web crawling and scraping framework. + - [crawl4ai](https://github.com/unclecode/crawl4ai) - An open-source, LLM-friendly web crawler that provides lightning-fast, structured data extraction specifically designed for AI agents. + - [stagehand](https://github.com/browserbase/stagehand) - A fast and token-efficient browser automation SDK to extract data and perform self-healing actions on web pages. + - [jev-ultrafast](https://github.com/browser-use/jev-ultrafast) - A fast browser agent that picks actions from an indexed table of page elements through TypeSafe's hosted Jev API, using a small LLM only to type text. +- Content Extraction + - [markdownify](https://github.com/matthewwithanm/python-markdownify) - Convert HTML to Markdown, with customizable tag handling. + - [feedparser](https://github.com/kurtmckee/feedparser) - Universal feed parser. + - [trafilatura](https://github.com/adbar/trafilatura) - A tool for gathering text and metadata from the web, with built-in content filtering. -* [gunicorn](https://github.com/benoitc/gunicorn) - Pre-forked, ported from Ruby's Unicorn project. -* [uwsgi](https://uwsgi-docs.readthedocs.io/en/latest/) - A project aims at developing a full stack for building hosting services, written in C. -* [waitress](https://github.com/Pylons/waitress) - Multi-threaded, powers Pyramid. -* [werkzeug](https://github.com/pallets/werkzeug) - A WSGI utility library for Python that powers Flask and can easily be embedded into your own projects. +### Email -# Resources +_Libraries for sending email._ + +- [aiosmtplib](https://github.com/cole/aiosmtplib) - An asyncio SMTP client. +- [django-anymail](https://github.com/anymail/django-anymail) - Django email backends and webhooks for transactional email services such as Amazon SES, Brevo, Mailgun, Postmark, and Resend. +- [yagmail](https://github.com/kootenpv/yagmail) - Yet another Gmail/SMTP client. + +**Database & Storage** + +### ORM + +_Libraries that implement Object-Relational Mapping or data mapping techniques._ + +- Relational Databases + - [sqlalchemy](https://github.com/sqlalchemy/sqlalchemy) - The Python SQL Toolkit and Object Relational Mapper. + - [awesome-sqlalchemy](https://github.com/dahlia/awesome-sqlalchemy) + - [django.db.models](https://github.com/django/django) - (part of Django) The Django [ORM](https://docs.djangoproject.com/en/stable/topics/db/models/). + - [peewee](https://github.com/coleifer/peewee) - A small, expressive ORM. + - [sqlmodel](https://github.com/fastapi/sqlmodel) - SQLModel is based on Python type annotations, and powered by Pydantic and SQLAlchemy. +- NoSQL Databases + - [pynamodb](https://github.com/pynamodb/PynamoDB) - A Pythonic interface for [Amazon DynamoDB](https://aws.amazon.com/dynamodb/). + - [mongoengine](https://github.com/MongoEngine/mongoengine) - A Python Object-Document-Mapper for working with MongoDB. + - [beanie](https://github.com/BeanieODM/beanie) - An asynchronous Python object-document mapper (ODM) for MongoDB. + - [django-mongodb-backend](https://github.com/mongodb/django-mongodb-backend) - Official MongoDB database backend for Django. + +### Database Drivers + +_Libraries for connecting and operating databases._ + +- MySQL - [awesome-mysql](https://github.com/shlomi-noach/awesome-mysql) + - [pymysql](https://github.com/PyMySQL/PyMySQL) - A pure-Python MySQL and MariaDB client library, based on PEP 249. + - [mysqlclient](https://github.com/PyMySQL/mysqlclient) - MySQL and MariaDB connector ([MySQLdb1](https://github.com/farcepest/MySQLdb1) fork). +- PostgreSQL - [awesome-postgres](https://github.com/dhamaniasad/awesome-postgres) + - [psycopg](https://github.com/psycopg/psycopg) - A PostgreSQL adapter for Python, the successor to psycopg2. + - [asyncpg](https://github.com/MagicStack/asyncpg) - A fast PostgreSQL Database Client Library for Python/asyncio. +- SQLite - [awesome-sqlite](https://github.com/planetopendata/awesome-sqlite) + - [sqlite3](https://docs.python.org/3/library/sqlite3.html) - (Python standard library) SQLite interface compliant with DB-API 2.0. + - [sqlite-utils](https://github.com/simonw/sqlite-utils) - Python CLI utility and library for manipulating SQLite databases. +- ClickHouse + - [clickhouse-connect](https://github.com/ClickHouse/clickhouse-connect) - The official ClickHouse client, with SQLAlchemy and Superset connectors. + - [clickhouse-driver](https://github.com/mymarilyn/clickhouse-driver) - Python driver with native interface for ClickHouse. +- Other Relational Databases + - [pyodbc](https://github.com/mkleehammer/pyodbc) - An ODBC bridge for connecting to SQL Server and any other ODBC-accessible database. + - [oracledb](https://github.com/oracle/python-oracledb) - The official Python driver for Oracle Database, successor to cx_Oracle. + - [mssql-python](https://github.com/microsoft/mssql-python) - Official Microsoft driver for SQL Server and Azure SQL, built on ODBC for high performance. +- NoSQL Databases + - [redis](https://github.com/redis/redis-py) - The Python client for Redis. + - [pymongo](https://github.com/mongodb/mongo-python-driver) - The official Python client for MongoDB. + - [cassandra-driver](https://github.com/apache/cassandra-python-driver) - The Python Driver for Apache Cassandra. + +### Database + +_In-process databases usable directly from Python._ + +- Analytical + - [duckdb](https://github.com/duckdb/duckdb) - An in-process SQL OLAP database management system; optimized for analytics and fast queries, similar to SQLite but for analytical workloads. + - [chdb](https://github.com/chdb-io/chdb) - In-process OLAP SQL engine with the full ClickHouse dialect, zero-copy pandas/Arrow interop, and federation to remote ClickHouse clusters via `remoteSecure()`. +- Vector + - [lancedb](https://github.com/lancedb/lancedb) - A developer-friendly embedded retrieval database for multimodal AI. + - [chromadb](https://github.com/chroma-core/chroma) - An open-source embedding database for building AI applications with embeddings and semantic search. + - [zvec](https://github.com/alibaba/zvec) - A lightweight, in-process vector database that embeds directly into applications. +- Key-Value & Document + - [tinydb](https://github.com/msiemens/tinydb) - A tiny, document-oriented database. + +### Caching + +_Libraries for caching data._ + +- [cachetools](https://github.com/tkem/cachetools) - Extensible memoizing collections and decorators. +- [diskcache](https://github.com/grantjenks/python-diskcache) - SQLite and file backed cache backend, compatible with Django. +- [hishel](https://github.com/karpetrosyan/hishel) - RFC 9111 compliant HTTP caching for clients like httpx and requests and servers like FastAPI, with sync and async support. +- [django-cacheops](https://github.com/Suor/django-cacheops) - A slick ORM cache with automatic granular event-driven invalidation. +- [dogpile.cache](https://github.com/sqlalchemy/dogpile.cache) - dogpile.cache is a next generation replacement for Beaker made by the same authors. + +### Search + +_Libraries and software for indexing and performing search queries on data._ + +- [opensearch-py](https://github.com/opensearch-project/opensearch-py) - The official low-level Python client for [OpenSearch](https://opensearch.org/). +- [elasticsearch](https://github.com/elastic/elasticsearch-py) - The official low-level Python client for [Elasticsearch](https://www.elastic.co/elasticsearch). +- [meilisearch](https://github.com/meilisearch/meilisearch-python) - The official Python client for the [Meilisearch](https://www.meilisearch.com/) search engine. +- [django-haystack](https://github.com/django-haystack/django-haystack) - Modular search for Django. + +### Serialization + +_Libraries for serializing complex data types._ + +- [msgpack](https://github.com/msgpack/msgpack-python) - MessagePack serializer implementation for Python. +- [orjson](https://github.com/ijl/orjson) - Fast, correct JSON library. +- [marshmallow](https://github.com/marshmallow-code/marshmallow) - A lightweight library for converting complex objects to and from simple Python datatypes. +- [msgspec](https://github.com/msgspec/msgspec) - A fast serialization and validation library with built-in support for JSON, MessagePack, YAML, and TOML. + +**Data & Science** + +### Data Analysis + +_Libraries for data analysis._ + +- [pandas](https://github.com/pandas-dev/pandas) - A library providing high-performance, easy-to-use data structures and data analysis tools. +- [polars](https://github.com/pola-rs/polars) - A fast DataFrame library implemented in Rust with a Python API. +- [ibis-framework](https://github.com/ibis-project/ibis) - A portable Python dataframe library with a single API for 20+ backends. + +### Data Ingestion / ETL + +_Libraries for data extraction, transformation, and loading pipelines across multiple sources and destinations._ + +- General + - [awswrangler](https://github.com/aws/aws-sdk-pandas) - Pandas integration with AWS services like Athena, Glue, Redshift, S3, and DynamoDB. + - [dlt](https://github.com/dlt-hub/dlt) - A Python library for building data pipelines with automatic schema inference, incremental loading, and support for multiple sources and destinations. +- Financial Data + - [yfinance](https://github.com/ranaroussi/yfinance) - Easy Pythonic way to download market and financial data from Yahoo Finance. + - [akshare](https://github.com/akfamily/akshare) - A financial data interface library, with data provided for academic research only. + - [edgartools](https://github.com/dgunning/edgartools) - Library for downloading structured data from SEC EDGAR filings and XBRL financial statements. + - [openbb](https://github.com/openbq-org/OpenBB) - A financial data platform for analysts, quants and AI agents. + +### Data Validation + +_Libraries for validating data._ + +- [pydantic](https://github.com/pydantic/pydantic) - Data validation using Python type hints. +- [jsonschema](https://github.com/python-jsonschema/jsonschema) - An implementation of [JSON Schema](https://json-schema.org/) for Python. +- [great-expectations](https://github.com/fivetran/great_expectations) - A data quality framework for validating, documenting, and profiling data with declarative expectations. +- [pandera](https://github.com/unionai-oss/pandera) - A data validation library for dataframes, with support for pandas, polars, PySpark, and more. + +### Data Visualization + +_Libraries for visualizing data. Also see [awesome-javascript](https://github.com/sorrycc/awesome-javascript#data-visualization)._ + +- Plotting + - [matplotlib](https://github.com/matplotlib/matplotlib) - A comprehensive library for creating static, animated, and interactive visualizations. + - [plotly](https://github.com/plotly/plotly.py) - Interactive graphing library for Python. + - [seaborn](https://github.com/mwaskom/seaborn) - Statistical data visualization using Matplotlib. + - [altair](https://github.com/vega/altair) - Declarative statistical visualization library for Python. + - [bokeh](https://github.com/bokeh/bokeh) - Interactive Web Plotting for Python. +- Specialized + - [graphviz](https://github.com/xflr6/graphviz) - Simple Python interface for creating and rendering Graphviz graphs. + - [cartopy](https://github.com/SciTools/cartopy) - A cartographic python library with matplotlib support. + - [graphify](https://github.com/Graphify-Labs/graphify) - Turn any folder of code, SQL schemas, docs, papers, images, or videos into a queryable knowledge graph. +- Dashboards and Apps + - [streamlit](https://github.com/streamlit/streamlit) - A framework which lets you build dashboards, generate reports, or create chat apps in minutes. + - [dash](https://github.com/plotly/dash) - A framework for building data apps and dashboards in pure Python, built on Plotly. + - [gradio](https://github.com/gradio-app/gradio) - Build and share machine learning apps, all in Python. + +### Geolocation + +_Libraries for geocoding addresses and working with latitudes and longitudes._ + +- [geopandas](https://github.com/geopandas/geopandas) - Python tools for geographic data (GeoSeries/GeoDataFrame) built on pandas. +- [geopy](https://github.com/geopy/geopy) - Python Geocoding Toolbox. +- [geojson](https://github.com/jazzband/geojson) - Python bindings and utilities for GeoJSON. +- [geodjango](https://github.com/django/django) - (part of Django) A world-class [geographic web framework](https://docs.djangoproject.com/en/stable/ref/contrib/gis/). + +### Science + +_Libraries for scientific computing. Also see [Python-for-Scientists](https://github.com/TomNicholas/Python-for-Scientists)._ + +- Core + - [numpy](https://github.com/numpy/numpy) - A fundamental package for scientific computing with Python. + - [scipy](https://github.com/scipy/scipy) - Fundamental algorithms for scientific computing in Python. + - [numba](https://github.com/numba/numba) - A NumPy-aware JIT compiler for Python, using LLVM. +- Symbolic Mathematics + - [sympy](https://github.com/sympy/sympy) - A Python library for symbolic mathematics. +- Statistics + - [statsmodels](https://github.com/statsmodels/statsmodels) - Statistical modeling and econometrics in Python. +- Biology and Chemistry + - [rdkit](https://github.com/rdkit/rdkit) - Cheminformatics and Machine Learning Software. + - [biopython](https://github.com/biopython/biopython) - Biopython is a set of freely available tools for biological computation. +- Physics and Engineering + - [pint](https://github.com/hgrecco/pint) - Operate and manipulate physical quantities with units and dimensional analysis. + - [astropy](https://github.com/astropy/astropy) - A community Python library for Astronomy. + - [obspy](https://github.com/obspy/obspy) - A Python toolbox for seismology. +- Simulation and Modeling + - [pymc](https://github.com/pymc-devs/pymc) - Probabilistic programming and Bayesian modeling in Python. + - [simpy](https://gitlab.com/team-simpy/simpy) - A process-based discrete-event simulation framework. + - [mesa](https://github.com/mesa/mesa) - An agent-based modeling framework for building, analyzing, and visualizing complex system simulations. +- Graphs and Networks + - [networkx](https://github.com/networkx/networkx) - A high-productivity software for complex networks. +- Computational Geometry + - [shapely](https://github.com/shapely/shapely) - Manipulation and analysis of geometric objects in the Cartesian plane. +- Other + - [colour-science](https://github.com/colour-science/colour) - Implementing a comprehensive number of colour theory transformations and algorithms. + - [manim](https://github.com/ManimCommunity/manim) - An animation engine for explanatory math videos. + +### Quantum Computing + +_Libraries for quantum computing._ + +- [qiskit](https://github.com/Qiskit/qiskit) - An IBM-backed quantum SDK for building, simulating, and running circuits on real quantum hardware. +- [qutip](https://github.com/qutip/qutip) - Quantum Toolbox in Python. +- [pennylane](https://github.com/PennyLaneAI/pennylane) - A cross-platform library for quantum computing, quantum machine learning, and quantum chemistry. +- [cirq](https://github.com/quantumlib/Cirq) - A Google-developed framework focused on hardware-aware quantum circuit design for NISQ devices. + +**Developer Tools** + +### Algorithms and Design Patterns + +_Python implementation of data structures, algorithms and design patterns. Also see [awesome-algorithms](https://github.com/tayllan/awesome-algorithms)._ + +- Algorithms + - [sortedcontainers](https://github.com/grantjenks/python-sortedcontainers) - Fast and pure-Python implementation of sorted collections. + - [algorithms](https://github.com/keon/algorithms) - Minimal examples of data structures and algorithms. + - [thealgorithms](https://github.com/TheAlgorithms/Python) - All Algorithms implemented in Python. +- Design Patterns + - [python-patterns](https://github.com/faif/python-patterns) - A collection of design patterns and idioms in Python. + - [python-statemachine](https://github.com/fgmacedo/python-statemachine) - Expressive statecharts and finite state machines with a declarative API, in sync and async codebases. + +### Interactive Interpreter + +_Interactive Python interpreters (REPL)._ + +- [ipython](https://github.com/ipython/ipython) - A powerful interactive Python shell, and the kernel behind Jupyter notebooks. +- [notebook](https://github.com/jupyter/notebook) - A web-based notebook environment for interactive computing. + - [awesome-jupyter](https://github.com/markusschanta/awesome-jupyter) +- [marimo](https://github.com/marimo-team/marimo) - A reactive notebook for Python, stored as pure Python and runnable as a script or app. +- [ptpython](https://github.com/prompt-toolkit/ptpython) - Advanced Python REPL built on top of the [python-prompt-toolkit](https://github.com/prompt-toolkit/python-prompt-toolkit). + +### Code Analysis + +_Tools of static analysis, linters and code quality checkers. Also see [awesome-static-analysis](https://github.com/analysis-tools-dev/static-analysis)._ + +- General + - [import-linter](https://github.com/seddonym/import-linter) - A linter that enforces architectural constraints on imports between Python modules. + - [vulture](https://github.com/jendrikseipp/vulture) - A tool for finding and analyzing dead Python code. + - [complexipy](https://github.com/rohaquinlop/complexipy) - Cognitive complexity analysis for Python code, written in Rust. + - [prospector](https://github.com/prospector-dev/prospector) - A tool to analyze Python code. +- Git Hooks + - [pre-commit](https://github.com/pre-commit/pre-commit) - A framework for managing and maintaining multi-language pre-commit hooks. +- Linters and Formatters + - [ruff](https://github.com/astral-sh/ruff) - An extremely fast Python linter and code formatter. + - [black](https://github.com/psf/black) - The uncompromising Python code formatter. + - [isort](https://github.com/PyCQA/isort) - A Python utility / library to sort imports. + - [pylint](https://github.com/pylint-dev/pylint) - A fully customizable source code analyzer. + - [flake8](https://github.com/PyCQA/flake8) - A wrapper around `pycodestyle`, `pyflakes` and McCabe. + - [awesome-flake8-extensions](https://github.com/DmytroLitvinov/awesome-flake8-extensions) + - [bandit](https://github.com/PyCQA/bandit) - A tool designed to find common security issues in Python code. +- Refactoring + - [rope](https://github.com/python-rope/rope) - Rope is a python refactoring library. +- Type Checkers - [awesome-python-typing](https://github.com/typeddjango/awesome-python-typing) + - [mypy](https://github.com/python/mypy) - A static type checker for Python. + - [ty](https://github.com/astral-sh/ty) - An extremely fast Python type checker and language server. + - [pyright](https://github.com/microsoft/pyright) - Full-featured static type checker for Python from Microsoft, the engine behind Pylance. + - [pyrefly](https://github.com/facebook/pyrefly) - A fast type checker and language server for Python. + +### Testing + +_Libraries for testing codebases and generating test data. Also see [awesome-python-testing](https://github.com/cleder/awesome-python-testing)._ + +- Frameworks + - [pytest](https://github.com/pytest-dev/pytest) - A mature full-featured Python testing tool. + - [awesome-pytest](https://github.com/augustogoulart/awesome-pytest) + - [hypothesis](https://github.com/HypothesisWorks/hypothesis) - Hypothesis is an advanced Quickcheck style property based testing library. + - [robotframework](https://github.com/robotframework/robotframework) - A generic test automation framework. +- Test Runners + - [tox](https://github.com/tox-dev/tox) - Auto builds and tests distributions in multiple Python versions. + - [nox](https://github.com/wntrblm/nox) - Flexible test automation for Python. +- Browser Automation + - [playwright-python](https://github.com/microsoft/playwright-python) - Python version of the Playwright testing and automation library. + - [selenium](https://github.com/SeleniumHQ/selenium) - Python bindings for [Selenium](https://selenium.dev/) [WebDriver](https://selenium.dev/documentation/webdriver/). + - [seleniumbase](https://github.com/seleniumbase/SeleniumBase) - Python framework for web automation & testing, with stealth options. +- Load Testing + - [locust](https://github.com/locustio/locust) - Scalable user load testing tool written in Python. +- API Testing + - [schemathesis](https://github.com/schemathesis/schemathesis) - A tool for automatic property-based testing of web APIs from OpenAPI or GraphQL schemas. +- Mock + - [unittest.mock](https://docs.python.org/3/library/unittest.mock.html) - (Python standard library) A mocking and patching library. + - [responses](https://github.com/getsentry/responses) - A utility library for mocking out the requests Python library. + - [vcrpy](https://github.com/kevin1024/vcrpy) - Record and replay HTTP interactions on your tests. + - [respx](https://github.com/lundberg/respx) - Mock HTTPX with awesome request patterns and response side effects. + - [time-machine](https://github.com/adamchainz/time-machine) - Travel through time in your tests by mocking the current time at the C level. +- Object Factories + - [factory-boy](https://github.com/FactoryBoy/factory_boy) - A test fixtures replacement for Python. + - [polyfactory](https://github.com/litestar-org/polyfactory) - A mock data generation library based on type hints (continuation of `pydantic-factories`). +- Code Coverage + - [coverage](https://github.com/coveragepy/coveragepy) - Code coverage measurement. +- Fake Data + - [faker](https://github.com/joke2k/faker) - A Python package that generates fake data. + - [mimesis](https://github.com/lk-geimfari/mimesis) - A Python library for generating fake but realistic data in multiple languages and locales. + +### Debugging Tools + +_Libraries for debugging code._ + +- pdb-like Debugger + - [ipdb](https://github.com/gotcha/ipdb) - IPython-enabled [pdb](https://docs.python.org/3/library/pdb.html). + - [pudb](https://github.com/inducer/pudb) - A full-screen, console-based Python debugger. +- Tracing + - [viztracer](https://github.com/gaogaotiantian/viztracer) - A low-overhead tool that traces and visualizes Python code execution. +- Profiler + - [py-spy](https://github.com/benfred/py-spy) - A sampling profiler for Python programs. Written in Rust. + - [memray](https://github.com/bloomberg/memray) - A memory profiler that tracks allocations in Python code, native extensions, and the interpreter itself. + - [pyinstrument](https://github.com/joerick/pyinstrument) - A statistical wall-clock profiler with low overhead and readable call-tree output. + - [scalene](https://github.com/plasma-umass/scalene) - A high-performance, high-precision CPU, GPU, and memory profiler for Python. +- Others + - [django-debug-toolbar](https://github.com/django-commons/django-debug-toolbar) - Display various debug information for Django. + - [icecream](https://github.com/gruns/icecream) - Inspect variables, expressions, and program execution with a single, simple function call. + - [flask-debugtoolbar](https://github.com/pallets-eco/flask-debugtoolbar) - A port of the django-debug-toolbar to flask. + +### Build Tools + +_Task runners and software build tools. If you're looking for Python packaging/build tools, see [Package Management](#package-management)._ + +- [invoke](https://github.com/pyinvoke/invoke) - A tool for managing shell-oriented subprocesses and organizing executable Python code into CLI-invokable tasks. +- [poethepoet](https://github.com/nat-n/poethepoet) - A task runner that defines tasks in pyproject.toml and works with poetry or uv. +- [scons](https://github.com/SCons/scons) - A software construction tool. +- [doit](https://github.com/pydoit/doit) - A task runner and build tool. + +### Documentation + +_Libraries for generating project documentation._ + +- [sphinx](https://github.com/sphinx-doc/sphinx/) - Python Documentation generator. + - [awesome-sphinxdoc](https://github.com/ygzgxyz/awesome-sphinxdoc) +- [mkdocs-material](https://github.com/squidfunk/mkdocs-material) - A documentation framework and Material Design theme built on MkDocs. +- [diagrams](https://github.com/mingrammer/diagrams) - Diagram as Code. +- [pdoc](https://github.com/mitmproxy/pdoc) - Auto-generates API documentation for Python projects. +- [zensical](https://github.com/zensical/zensical) - A modern static site generator for technical documentation. + +**DevOps** + +### DevOps Tools + +_Software and libraries for DevOps._ + +- Cloud Providers + - [boto3](https://github.com/boto/boto3) - Python interface to Amazon Web Services. + - [awscli](https://github.com/aws/aws-cli) - Universal Command Line Interface for Amazon Web Services; the PyPI package is v1, in maintenance mode, while v2 ships as AWS's bundled installer. + - [azure-sdk-for-python](https://github.com/Azure/azure-sdk-for-python) - Microsoft Azure SDK for Python, published as per-service packages. + - [google-cloud-python](https://github.com/googleapis/google-cloud-python) - Google Cloud client libraries for Python, published as per-service packages. +- Configuration Management + - [ansible](https://github.com/ansible/ansible) - A radically simple IT automation platform. + - [cloud-init](https://github.com/canonical/cloud-init) - A multi-distribution package that handles early initialization of a cloud instance. + - [pyinfra](https://github.com/pyinfra-dev/pyinfra) - Turns Python code into shell commands and runs them on your servers. + - [salt](https://github.com/saltstack/salt) - Infrastructure automation and management system. +- Deployment + - [fabric](https://github.com/fabric/fabric) - A simple, Pythonic tool for remote execution and deployment. + - [chalice](https://github.com/aws/chalice) - A Python serverless microframework for AWS. +- Monitoring and Processes + - [psutil](https://github.com/giampaolo/psutil) - A cross-platform process and system utilities module. + - [sentry-sdk](https://github.com/getsentry/sentry-python) - Sentry SDK for Python. + - [supervisor](https://github.com/Supervisor/supervisor) - Supervisor process control system for UNIX. + - [flower](https://github.com/mher/flower) - A real-time monitor and web admin for Celery task queues. + - [sh](https://github.com/amoffat/sh) - A full-fledged subprocess replacement for Python. +- Other + - [borgbackup](https://github.com/borgbackup/borg) - A deduplicating archiver with compression and encryption. + +### Distributed Computing + +_Frameworks and libraries for Distributed Computing._ + +- [pyspark](https://github.com/apache/spark) - [Apache Spark](https://spark.apache.org/) Python API. +- [dask](https://github.com/dask/dask) - A flexible parallel computing library for analytic computing. +- [ray](https://github.com/ray-project/ray/) - A unified framework for scaling AI and Python applications. +- [joblib](https://github.com/joblib/joblib) - Parallel computing and disk-based caching for Python functions. +- [mpi4py](https://github.com/mpi4py/mpi4py) - Python bindings for MPI. + +### Task Queues + +_Libraries for working with task queues._ + +- [celery](https://github.com/celery/celery) - An asynchronous task queue/job queue based on distributed message passing. +- [rq](https://github.com/rq/rq) - Simple job queues for Python. +- [dramatiq](https://github.com/Bogdanp/dramatiq) - A fast and reliable background task processing library for Python 3. +- [huey](https://github.com/coleifer/huey) - A little task queue with multi-process, multi-thread, or greenlet workers. +- [taskiq](https://github.com/taskiq-python/taskiq) - Distributed task queue with native asyncio support and pluggable brokers. + +### Messaging + +_Libraries for working with message brokers and event streaming._ + +- [confluent-kafka](https://github.com/confluentinc/confluent-kafka-python) - Confluent's Python client for Apache Kafka, built on librdkafka. +- [pika](https://github.com/pika/pika) - Pure-Python RabbitMQ/AMQP 0-9-1 client library. +- [paho-mqtt](https://github.com/eclipse-paho/paho.mqtt.python) - The Eclipse Paho MQTT client for Python. +- [faststream](https://github.com/ag2ai/faststream) - A framework for building asynchronous services over Apache Kafka, RabbitMQ, NATS, MQTT and Redis. + +### Job Schedulers + +_Libraries for scheduling jobs._ + +- Task Scheduling + - [apscheduler](https://github.com/agronholm/apscheduler) - A light but powerful in-process task scheduler that lets you schedule functions. + - [schedule](https://github.com/dbader/schedule) - Python job scheduling for humans. +- Workflow Orchestration + - [dagster](https://github.com/dagster-io/dagster) - An orchestration platform for the development, production, and observation of data assets. + - [apache-airflow](https://github.com/apache/airflow) - Airflow is a platform to programmatically author, schedule and monitor workflows. + - [prefect](https://github.com/PrefectHQ/prefect) - A modern workflow orchestration framework that makes it easy to build, schedule and monitor robust data pipelines. + +### Logging + +_Libraries for generating and working with logs._ + +- [logging](https://docs.python.org/3/library/logging.html) - (Python standard library) Logging facility for Python. +- [structlog](https://github.com/hynek/structlog) - Structured logging made easy. +- [loguru](https://github.com/Delgan/loguru) - Library which aims to bring enjoyable logging in Python. +- [logfire](https://github.com/pydantic/logfire) - The observability platform for Python, from the makers of Pydantic. + +### Network Virtualization + +_Tools and libraries for packet manipulation and network device automation._ + +- [scapy](https://github.com/secdev/scapy) - A brilliant packet manipulation library. +- [netmiko](https://github.com/ktbyers/netmiko) - Multi-vendor library to simplify CLI connections to network devices. +- [napalm](https://github.com/napalm-automation/napalm) - Cross-vendor API to manipulate network devices. + +**CLI & GUI** + +### CLI Development + +_Libraries for building command-line applications._ + +- General + - [argparse](https://docs.python.org/3/library/argparse.html) - (Python standard library) Command-line option and argument parsing. + - [click](https://github.com/pallets/click/) - A package for creating beautiful command line interfaces in a composable way. + - [typer](https://github.com/fastapi/typer) - Modern CLI framework that uses Python type hints. Built on Click. + - [prompt_toolkit](https://github.com/prompt-toolkit/python-prompt-toolkit) - A library for building powerful interactive command lines. + - [fire](https://github.com/google/python-fire) - A library for creating command line interfaces from absolutely any Python object. +- Terminal Rendering + - [rich](https://github.com/Textualize/rich) - Python library for rich text and beautiful formatting in the terminal. Also provides a great `RichHandler` log handler. + - [tqdm](https://github.com/tqdm/tqdm) - Fast, extensible progress bar for loops and CLI. + - [colorama](https://github.com/tartley/colorama) - Cross-platform colored terminal text. + - [alive-progress](https://github.com/rsalmei/alive-progress) - A new kind of Progress Bar, with real-time throughput, eta and very cool animations. +- TUI Frameworks + - [textual](https://github.com/Textualize/textual) - A framework for building interactive user interfaces that run in the terminal and the browser. + - [urwid](https://github.com/urwid/urwid) - A library for creating terminal GUI applications with strong support for widgets, events, rich colors, etc. + - [asciimatics](https://github.com/peterbrittain/asciimatics) - A package to create full-screen text UIs (from interactive forms to ASCII animations). + +### CLI Tools + +_Useful CLI-based tools._ + +- Database CLIs + - [pgcli](https://github.com/dbcli/pgcli) - PostgreSQL CLI with autocompletion and syntax highlighting. + - [mycli](https://github.com/dbcli/mycli) - MySQL CLI with autocompletion and syntax highlighting. + - [litecli](https://github.com/dbcli/litecli) - SQLite CLI with autocompletion and syntax highlighting. + - [iredis](https://github.com/laixintao/iredis) - Redis CLI with autocompletion and syntax highlighting. +- Downloaders + - [yt-dlp](https://github.com/yt-dlp/yt-dlp) - A command-line program to download videos from YouTube and other video sites, a fork of youtube-dl. +- HTTP Clients + - [httpie](https://github.com/httpie/cli) - A command line HTTP client, a user-friendly cURL replacement. +- Project Scaffolding + - [cookiecutter](https://github.com/cookiecutter/cookiecutter) - A command-line utility that creates projects from cookiecutters (project templates). + - [copier](https://github.com/copier-org/copier) - A library and command-line utility for rendering project templates. +- Shells + - [xonsh](https://github.com/xonsh/xonsh/) - A Python-powered shell. Full-featured and cross-platform. +- Terminal Workflow + - [tmuxp](https://github.com/tmux-python/tmuxp) - A [tmux](https://github.com/tmux/tmux) session manager. + +### GUI Development + +_Libraries for working with graphical user interface applications._ + +- Desktop + - [PyGObject](https://github.com/GNOME/pygobject) - Python Bindings for GLib/GObject/GIO/GTK. + - [dearpygui](https://github.com/hoffstadt/DearPyGui) - A simple GPU-accelerated Python GUI framework. + - [Kivy](https://github.com/kivy/kivy) - An open-source framework for cross-platform GUI apps on desktop, mobile, and embedded platforms. + - [wxPython](https://github.com/wxWidgets/Phoenix) - A cross-platform GUI toolkit that wraps the wxWidgets C++ library. + - [toga](https://github.com/beeware/toga) - A Python native, OS native GUI toolkit. +- Qt + - [PySide6](https://github.com/pyside/pyside-setup) - Qt for Python offers the official Python bindings for [Qt](https://www.qt.io/), largely API-compatible with PyQt6 but with different licensing. + - [PyQt6](https://www.riverbankcomputing.com/static/Docs/PyQt6/) - Python bindings for the [Qt](https://www.qt.io/) cross-platform application and UI framework. +- Tkinter + - [tkinter](https://docs.python.org/3/library/tkinter.html) - (Python standard library) The standard Python interface to the Tcl/Tk GUI toolkit. + - [customtkinter](https://github.com/tomschimansky/customtkinter) - A modern and customizable python UI-library based on Tkinter. +- Web-based + - [pywebview](https://github.com/r0x0r/pywebview/) - A lightweight cross-platform native wrapper around a webview component. + - [nicegui](https://github.com/zauberzeug/nicegui) - An easy-to-use, Python-based UI framework, which shows up in your web browser. + - [flet](https://github.com/flet-dev/flet) - Cross-platform GUI framework for building modern apps in pure Python. + +**Text & Documents** + +### Text Processing + +_Libraries for parsing and manipulating plain texts._ + +- Encoding and Unicode + - [charset-normalizer](https://github.com/jawah/charset_normalizer) - Universal character encoding detector, and a dependency of requests. + - [chardet](https://github.com/chardet/chardet) - Python character encoding detector. + - [ftfy](https://github.com/rspeer/python-ftfy) - Makes Unicode text less broken and more consistent automagically. +- Fuzzy Matching + - [rapidfuzz](https://github.com/rapidfuzz/RapidFuzz) - Rapid fuzzy string matching using various string metrics, with a C++ core. +- General + - [difflib](https://docs.python.org/3/library/difflib.html) - (Python standard library) Helpers for computing deltas. + - [pyfiglet](https://github.com/pwaller/pyfiglet) - An implementation of figlet written in Python. +- Internationalization + - [babel](https://github.com/python-babel/babel) - An internationalization library for Python. +- Parser + - [pygments](https://github.com/pygments/pygments) - A generic syntax highlighter. + - [pyparsing](https://github.com/pyparsing/pyparsing) - A Python library for creating PEG parsers. + - [sqlparse](https://github.com/andialbrecht/sqlparse) - A non-validating SQL parser. + - [phonenumbers](https://github.com/daviddrysdale/python-phonenumbers) - Parsing, formatting, storing and validating international phone numbers. + - [parsy](https://github.com/python-parsy/parsy) - Easy, generic parser combinator library for creating parsers. +- Transliteration and Slugs + - [python-slugify](https://github.com/un33k/python-slugify) - A Python slugify library that translates unicode to ASCII. + - [unidecode](https://github.com/avian2/unidecode) - ASCII transliterations of Unicode text. +- Unique identifiers + - [shortuuid](https://github.com/skorokithakis/shortuuid) - A generator library for concise, unambiguous and URL-safe UUIDs. + +### HTML Manipulation + +_Libraries for working with HTML and XML._ + +- [lxml](https://github.com/lxml/lxml) - A very fast, easy-to-use and versatile library for handling HTML and XML. +- [beautifulsoup4](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) - Providing Pythonic idioms for iterating, searching, and modifying HTML or XML. +- [xmltodict](https://github.com/martinblech/xmltodict) - Working with XML feel like you are working with JSON. +- [markupsafe](https://github.com/pallets/markupsafe) - Safely adds untrusted strings to HTML/XML markup. +- [justhtml](https://github.com/EmilStenstrom/justhtml/) - A pure Python HTML5 parser that sanitizes untrusted HTML by default. + +### File Format Processing + +_Libraries for parsing and manipulating specific file formats._ + +- General + - [pyelftools](https://github.com/eliben/pyelftools) - Parsing and analyzing ELF files and DWARF debugging information. + - [tablib](https://github.com/jazzband/tablib) - A module for Tabular Datasets in XLS, CSV, JSON, YAML. +- File Conversion + - [markitdown](https://github.com/microsoft/markitdown) - Python tool for converting files and office documents to Markdown. + - [docling](https://github.com/docling-project/docling) - Library for converting documents into structured data. +- Excel + - [openpyxl](https://openpyxl.readthedocs.io/en/stable/) - A library for reading and writing Excel 2010 xlsx/xlsm/xltx/xltm files. + - [xlsxwriter](https://github.com/jmcnamara/XlsxWriter) - A Python module for creating Excel .xlsx files. +- Word + - [python-docx](https://github.com/python-openxml/python-docx) - Creates, reads, and updates Microsoft Word (.docx) files. +- PowerPoint + - [python-pptx](https://github.com/scanny/python-pptx) - Python library for creating and updating PowerPoint (.pptx) files. +- PDF + - [pypdf](https://github.com/py-pdf/pypdf) - A library capable of splitting, merging, cropping, and transforming PDF pages. + - [pymupdf](https://github.com/pymupdf/PyMuPDF) - A fast library for extracting, rendering, and editing PDF and other document formats, built on MuPDF. + - [reportlab](https://docs.reportlab.com/) - An open-source library for generating PDFs and graphics. + - [pdfminer.six](https://github.com/pdfminer/pdfminer.six) - A community-maintained fork of PDFMiner for extracting information from PDF documents. +- HTML-to-PDF + - [weasyprint](https://github.com/Kozea/WeasyPrint) - A visual rendering engine for HTML and CSS that can export to PDF. +- Markdown + - [markdown-it-py](https://github.com/executablebooks/markdown-it-py) - Markdown parser with 100% CommonMark support, extensions, and syntax plugins. + - [markdown](https://github.com/Python-Markdown/markdown) - A Python implementation of John Gruber’s Markdown. + - [mistune](https://github.com/lepture/mistune) - A fast yet powerful Python Markdown parser with renderers and plugins. +- Data Formats + - [tomllib](https://docs.python.org/3/library/tomllib.html) - (Python standard library) Parse TOML files. + - [pyyaml](https://github.com/yaml/pyyaml) - A full-featured YAML framework for Python. + +### File Manipulation + +_Libraries for file manipulation._ + +- [mimetypes](https://docs.python.org/3/library/mimetypes.html) - (Python standard library) Map filenames to MIME types. +- [pathlib](https://docs.python.org/3/library/pathlib.html) - (Python standard library) A cross-platform, object-oriented path library. +- [watchfiles](https://github.com/samuelcolvin/watchfiles) - Simple, modern and fast file watching and code reload in python. +- [watchdog](https://github.com/gorakhargosh/watchdog) - API and shell utilities to monitor file system events. +- [python-magic](https://github.com/ahupp/python-magic) - A Python interface to the libmagic file type identification library. + +**Media** + +### Image Processing + +_Libraries for manipulating images._ + +- Barcodes and QR Codes + - [qrcode](https://github.com/lincolnloop/python-qrcode) - A pure Python QR Code generator. + - [python-barcode](https://github.com/WhyNotHugo/python-barcode) - Create barcodes in Python with no extra dependencies. +- General + - [pillow](https://github.com/python-pillow/Pillow) - Pillow is the friendly [PIL](https://pillow.readthedocs.io/en/stable/about.html) fork. + - [scikit-image](https://github.com/scikit-image/scikit-image) - A Python library for (scientific) image processing. + - [rembg](https://github.com/danielgatis/rembg) - A tool to remove image backgrounds. + - [wand](https://github.com/emcconville/wand) - Python bindings for [MagickWand](https://imagemagick.org/magick-wand/), C API for ImageMagick. + - [pyvips](https://github.com/libvips/pyvips) - A binding for libvips, a fast image processing library with low memory needs. +- Image Serving + - [thumbor](https://github.com/thumbor/thumbor) - A smart imaging service. It enables on-demand crop, re-sizing and flipping of images. + +### Audio & Video Processing + +_Libraries for manipulating audio, video, and their metadata._ + +- Audio + - [soundfile](https://github.com/bastibe/python-soundfile) - An audio library for reading and writing sound files, based on libsndfile, CFFI, and NumPy. + - [librosa](https://github.com/librosa/librosa) - Python library for audio and music analysis. + - [pydub](https://github.com/jiaaro/pydub) - Manipulate audio with a simple and easy high level interface. +- Video + - [av](https://github.com/PyAV-Org/PyAV) - Pythonic bindings for FFmpeg's libraries. + - [moviepy](https://github.com/Zulko/moviepy) - A module for script-based movie editing with many formats, including animated GIFs. +- Metadata + - [mutagen](https://github.com/quodlibet/mutagen) - A Python module to handle audio metadata. + - [tinytag](https://github.com/tinytag/tinytag) - A library for reading audio file metadata of MP3, MP4, WAV, OGG, FLAC, WMA, and AIFF files. + - [beets](https://github.com/beetbox/beets) - A music library manager and [MusicBrainz](https://musicbrainz.org/) tagger. + +### Game Development + +_Awesome game development libraries._ + +- 3D Engines + - [panda3d](https://github.com/panda3d/panda3d) - 3D game engine developed jointly by Disney and contributors from around the world. +- Game Frameworks + - [pyglet](https://github.com/pyglet/pyglet) - A cross-platform windowing and multimedia library for Python. + - [pygame-ce](https://github.com/pygame-community/pygame-ce) - An actively developed drop-in replacement with new features and performance improvements ([pygame](https://github.com/pygame/pygame) fork). + - [pygame](https://github.com/pygame/pygame) - Pygame is a set of Python modules designed for writing games. + - [arcade](https://github.com/pythonarcade/arcade) - An easy-to-use library for creating 2D arcade games. +- Visual Novels + - [renpy](https://github.com/renpy/renpy) - A Visual Novel engine. + +**Python Language** + +### Implementations + +_Implementations of Python._ + +- [cpython](https://github.com/python/cpython) - Default, most widely used implementation of the Python programming language written in C. +- [micropython](https://github.com/micropython/micropython) - A lean and efficient Python implementation for microcontrollers and constrained systems. +- [pypy](https://github.com/pypy/pypy) - A very fast and compliant implementation of the Python language. +- [Cython](https://github.com/cython/cython) - Optimizing Static Compiler for Python. +- [pyodide](https://github.com/pyodide/pyodide) - Python distribution for the browser and Node.js based on WebAssembly. + +### Built-in Classes Enhancement + +_Libraries for enhancing Python built-in classes._ + +- [attrs](https://github.com/python-attrs/attrs) - Replacement for `__init__`, `__eq__`, `__repr__`, etc. boilerplate in class definitions. +- [bidict](https://github.com/jab/bidict) - Efficient, Pythonic bidirectional map data structures and related functionality. +- [uuid-utils](https://github.com/aminalaee/uuid-utils) - A fast, Rust-backed drop-in replacement for Python's built-in `uuid` module. +- [python-box](https://github.com/cdgriffith/Box) - Python dictionaries with advanced dot notation access. + +### Functional Programming + +_Functional Programming with Python._ + +- [functools](https://docs.python.org/3/library/functools.html) - (Python standard library) Higher-order functions and operations on callable objects. +- [more-itertools](https://github.com/more-itertools/more-itertools) - More routines for operating on iterables, beyond `itertools`. +- [toolz](https://github.com/pytoolz/toolz) - A collection of functional utilities for iterators, functions, and dictionaries. Also available as [cytoolz](https://github.com/pytoolz/cytoolz/) for Cython-accelerated performance. +- [funcy](https://github.com/Suor/funcy) - A fancy and practical functional tools. +- [returns](https://github.com/dry-python/returns) - A set of type-safe monads, transformers, and composition utilities. + +### Asynchronous Programming + +_Libraries for asynchronous, concurrent and parallel execution. Also see [awesome-asyncio](https://github.com/timofurrer/awesome-asyncio)._ + +- Async I/O + - [asyncio](https://docs.python.org/3/library/asyncio.html) - (Python standard library) Asynchronous I/O, event loop, coroutines and tasks. + - [awesome-asyncio](https://github.com/timofurrer/awesome-asyncio) + - [anyio](https://github.com/agronholm/anyio) - A high-level async concurrency and networking framework that works on top of asyncio or trio. + - [uvloop](https://github.com/MagicStack/uvloop) - Ultra fast asyncio event loop. + - [trio](https://github.com/python-trio/trio) - A friendly library for async concurrency and I/O. + - [gevent](https://github.com/gevent/gevent) - A coroutine-based Python networking library that uses [greenlet](https://github.com/python-greenlet/greenlet). + - [Twisted](https://github.com/twisted/twisted) - An event-driven networking engine. +- Parallelism + - [concurrent.futures](https://docs.python.org/3/library/concurrent.futures.html) - (Python standard library) A high-level interface for asynchronously executing callables. + - [multiprocessing](https://docs.python.org/3/library/multiprocessing.html) - (Python standard library) Process-based parallelism. + +### Date and Time + +_Libraries for working with dates and times._ + +- [zoneinfo](https://docs.python.org/3/library/zoneinfo.html) - (Python standard library) IANA time zone support. Brings the [tz database](https://en.wikipedia.org/wiki/Tz_database) into Python. +- [python-dateutil](https://github.com/dateutil/dateutil) - Extensions to the standard Python [datetime](https://docs.python.org/3/library/datetime.html) module. +- [dateparser](https://github.com/scrapinghub/dateparser) - A Python parser for human-readable dates in over 200 language locales. +- [pendulum](https://github.com/python-pendulum/pendulum) - Python datetimes made easy. +- [whenever](https://github.com/ariebovenberg/whenever) - A modern datetime library, type-safe and DST-safe, in Rust or pure Python. + +**Python Toolchain** + +### Environment Management + +_Libraries for Python version and virtual environment management._ + +- [virtualenv](https://github.com/pypa/virtualenv) - A tool to create isolated Python environments. +- [uv](https://github.com/astral-sh/uv) - An extremely fast Python version, package and project manager, written in Rust. +- [pyenv](https://github.com/pyenv/pyenv) - Simple Python version management. + +### Package Management + +_Libraries for package and dependency management._ + +- Package Managers + - [pip](https://github.com/pypa/pip) - The package installer for Python. + - [uv](https://github.com/astral-sh/uv) - An extremely fast Python version, package and project manager, written in Rust. + - [poetry](https://github.com/python-poetry/poetry) - Python dependency management and packaging made easy. + - [hatch](https://github.com/pypa/hatch) - Modern, extensible Python project manager for environments, builds, and publishing. + - [pipx](https://github.com/pypa/pipx) - Install and Run Python Applications in Isolated Environments. Like `npx` in Node.js. + - [conda](https://github.com/conda/conda/) - Cross-platform, Python-agnostic binary package manager. +- Build Backends + - [setuptools](https://github.com/pypa/setuptools) - The historical and still most widely used pyproject build backend. + - [hatchling](https://github.com/pypa/hatch) - Modern, extensible build backend from the hatch project. + - [poetry-core](https://github.com/python-poetry/poetry-core) - Poetry's PEP 517 build backend, usable without Poetry itself. + - [uv-build](https://github.com/astral-sh/uv) - uv's fast, minimal build backend for pure-Python projects. + +### Package Repositories + +_Local PyPI repository servers, proxies, and mirrors._ + +- [pypiserver](https://github.com/pypiserver/pypiserver) - A minimal PyPI server for uploading and installing packages with pip. +- [devpi](https://github.com/devpi/devpi) - PyPI server and packaging/testing/release tool. +- [bandersnatch](https://github.com/pypa/bandersnatch/) - PyPI mirroring tool provided by Python Packaging Authority (PyPA). + +### Distribution + +_Libraries to create packaged executables for release distribution._ + +- Executables + - [pyinstaller](https://github.com/pyinstaller/pyinstaller) - Converts Python programs into stand-alone executables (cross-platform). + - [Nuitka](https://github.com/Nuitka/Nuitka) - Compiles Python programs into high-performance standalone executables (cross-platform). + - [pex](https://github.com/pex-tool/pex) - A tool for building self-contained Python executable environments (PEP 441 zipapps). + - [cx-Freeze](https://github.com/marcelotduarte/cx_Freeze) - Converts Python scripts into standalone executables and installers for Windows, macOS, and Linux. +- Obfuscation + - [pyarmor](https://github.com/dashingsoft/pyarmor) - A tool used to obfuscate python scripts, bind obfuscated scripts to fixed machine or expire obfuscated scripts. + +### Configuration Files + +_Libraries for storing and parsing configuration options._ + +- [configparser](https://docs.python.org/3/library/configparser.html) - (Python standard library) INI file parser. +- [python-dotenv](https://github.com/theskumar/python-dotenv) - Reads key-value pairs from a `.env` file and sets them as environment variables. +- [pydantic-settings](https://github.com/pydantic/pydantic-settings) - Settings management using Pydantic models with validation, loading from environment variables and secrets files. +- [hydra-core](https://github.com/hydra-ecosystem/hydra) - Hydra is a framework for elegantly configuring complex applications. +- [dynaconf](https://github.com/dynaconf/dynaconf) - Dynaconf is a configuration manager with plugins for Django and Flask. + +**Security** + +### Cryptography + +_Libraries for cryptographic primitives and secure protocols._ + +- [cryptography](https://github.com/pyca/cryptography) - A package designed to expose cryptographic primitives and recipes to Python developers. +- [pynacl](https://github.com/pyca/pynacl) - Python binding to libsodium, a fork of the Networking and Cryptography (NaCl) library. +- [paramiko](https://github.com/paramiko/paramiko) - The leading native Python SSHv2 protocol library. +- [itsdangerous](https://github.com/pallets/itsdangerous) - Safely pass trusted data to untrusted environments and back. + +### Penetration Testing + +_Frameworks and tools for penetration testing._ + +- [mitmproxy](https://github.com/mitmproxy/mitmproxy) - An interactive TLS-capable intercepting HTTP proxy for penetration testers and software developers. +- [impacket](https://github.com/fortra/impacket) - A collection of Python classes for working with network protocols, widely used for Windows and Active Directory testing. +- [sqlmap](https://github.com/sqlmapproject/sqlmap) - Automatic SQL injection and database takeover tool. +- [pwntools](https://github.com/Gallopsled/pwntools) - A CTF framework and exploit development library. +- [sherlock-project](https://github.com/sherlock-project/sherlock) - Hunt down social media accounts by username across social networks. + +### Supply Chain Security + +_Tools for auditing dependencies against known vulnerabilities._ + +- [pip-audit](https://github.com/pypa/pip-audit) - Audits Python environments and dependency trees for known vulnerabilities, using the Python Packaging Advisory Database or OSV. +- [uv-audit](https://github.com/astral-sh/uv) - (part of uv) uv's [dependency vulnerability scanning](https://docs.astral.sh/uv/reference/cli/#uv-audit) backed by OSV. + +### Web Security + +_Libraries for application-layer web security._ + +- [nh3](https://github.com/messense/nh3) - Python binding to the ammonia HTML sanitizer, a fast replacement for bleach. +- [secure](https://github.com/TypeError/secure) - HTTP security headers for Python web applications with ASGI and WSGI middleware. + +**Other** + +### Hardware + +_Libraries for programming with hardware._ + +- [pyserial](https://github.com/pyserial/pyserial) - Python serial port access library for Windows, macOS, Linux, and BSD. +- [pynput](https://github.com/moses-palmer/pynput) - A library to control and monitor input devices. +- [bleak](https://github.com/hbldh/bleak) - A cross platform Bluetooth Low Energy Client for Python using asyncio. +- [jumpstarter](https://github.com/jumpstarter-dev/jumpstarter) - A hardware-in-the-loop testing framework with a Python client library for automated testing on real and virtual hardware. + +### Microsoft Windows + +_Python programming on Microsoft Windows._ + +- [pywin32](https://github.com/mhammond/pywin32) - Python Extensions for Windows. +- [pythonnet](https://github.com/pythonnet/pythonnet) - Python Integration with the .NET Common Language Runtime (CLR). +- [pyenv-win](https://github.com/pyenv-win/pyenv-win) - A Python version manager for Windows ([rbenv-win](https://github.com/nak1114/rbenv-win) fork). +- [winpython](https://github.com/winpython/winpython) - Portable Python distribution for Windows. + +### Miscellaneous + +_Useful libraries or tools that don't fit in the categories above._ + +- [blinker](https://github.com/pallets-eco/blinker) - A fast Python in-process signal/event dispatching system. +- [boltons](https://github.com/mahmoud/boltons) - A set of pure-Python utilities. + +## Resources Where to discover learning resources or new Python libraries. -## Newsletters +### Newsletters -* [Awesome Python Newsletter](http://python.libhunt.com/newsletter) -* [Pycoder's Weekly](https://pycoders.com/) -* [Python Tricks](https://realpython.com/python-tricks/) -* [Python Weekly](https://www.pythonweekly.com/) +- [Awesome Python Newsletter](https://python.libhunt.com/newsletter) +- [Pycoder's Weekly](https://pycoders.com/) +- [Python Tricks](https://realpython.com/python-tricks/) +- [Python Weekly](https://www.pythonweekly.com/) -## Podcasts +### Podcasts -* [Django Chat](https://djangochat.com/) -* [Python Bytes](https://pythonbytes.fm) -* [Talk Python To Me](https://talkpython.fm/) -* [Python Test](https://podcast.pythontest.com/) -* [The Real Python Podcast](https://realpython.com/podcasts/rpp/) +- [Django Chat](https://djangochat.com/) +- [PyPodcats](https://pypodcats.live) +- [Python Bytes](https://pythonbytes.fm) +- [Talk Python To Me](https://talkpython.fm/) +- [The Real Python Podcast](https://realpython.com/podcasts/rpp/) -# Contributing +### Websites + +- [Python Developer Tooling Handbook](https://pydevtools.com/) - Comprehensive guide to modern Python developer tools covering package management, linting, type checking, testing, and more. + +## Contributing Your contributions are always welcome! Please take a look at the [contribution guidelines](https://github.com/vinta/awesome-python/blob/master/CONTRIBUTING.md) first. -- - - - -If you have any question about this opinionated list, do not hesitate to contact me [@VintaChen](https://twitter.com/VintaChen) on Twitter or open an issue on GitHub. +--- +If you have any question about this opinionated list, do not hesitate to contact [@vinta](https://x.com/vinta) on X (Twitter). diff --git a/SPONSORSHIP.md b/SPONSORSHIP.md new file mode 100644 index 00000000..1e27bee5 --- /dev/null +++ b/SPONSORSHIP.md @@ -0,0 +1,62 @@ +# Sponsor awesome-python + +**The #10 most-starred repository on all of GitHub.** + +awesome-python is where Python developers go to discover tools. It ranks on the first page of Google for "best Python libraries," is referenced by ChatGPT and other LLMs when recommending Python tools, and is the list developers check when evaluating frameworks. + +Your sponsorship puts your product in front of developers at the exact moment they're choosing what to use. + +## Audience + +| Metric | Value | +| ------------ | ---------------------------------------------------------------------------------------------------- | +| Stars | ![Stars](https://img.shields.io/github/stars/vinta/awesome-python?style=for-the-badge) | +| Forks | ![Forks](https://img.shields.io/github/forks/vinta/awesome-python?style=for-the-badge) | +| Watchers | ![Watchers](https://img.shields.io/github/watchers/vinta/awesome-python?style=for-the-badge) | +| Contributors | ![Contributors](https://img.shields.io/github/contributors/vinta/awesome-python?style=for-the-badge) | + +**Who visits:** Professional Python developers evaluating libraries and tools for production use. Not beginners browsing tutorials. People making adoption decisions. + +**Where they come from:** Google Search, GitHub, Reddit, YouTube, ChatGPT/LLMs, Hacker News. + +## Sponsorship Tiers + +### Headline Sponsor - $500/month + +- Large logo and one-line description (max 120 characters) pinned at the very top of the README, above all project entries +- Logo link in the sponsor section of [awesome-python.com](https://awesome-python.com/) + +### Featured Sponsor - $200/month + +- Text entry (`[Name](URL) - Description.`, max 120 characters) pinned at the top of the README, directly below Headline sponsors +- Text link in the sponsor section of [awesome-python.com](https://awesome-python.com/) + +### Indie Dev Sponsor - $99/month or $249/quarter + +- Lower-cost tier for indie developers, solo founders, and one-person companies only +- Text entry (`[Name](URL) - Description.`, max 120 characters) in the README sponsor section, directly below Featured sponsors +- Text link in the sponsor section of [awesome-python.com](https://awesome-python.com/) + +Indie Dev Sponsor is available only to founder-led (one-person company) products. Funded startups, larger companies, and products with enterprise sales teams should use Featured or Headline Sponsor. + +## Get Started + +Email [sponsorship@awesome-python.com](mailto:sponsorship@awesome-python.com?subject=Awesome%20Python%20Sponsorship) with: + +- **Tier:** Headline Sponsor ($500/mo), Featured Sponsor ($200/mo), or Indie Dev Sponsor ($99/mo or $249/quarter) +- **Content:** Product name, URL, logo, and description (Headline tier) or `[Name](URL) - Description.` entry (Featured or Indie Dev tier) +- **Duration:** 1, 3, 6 months, or longer +- **Payment method:** card via [Open Collective](https://opencollective.com/awesome-python), US bank transfer (ACH/wire), or PayPal + +One upfront payment per term. Setup takes less than 24 hours. + +## Previously Sponsored By + +- [Warp](https://www.warp.dev/) - The terminal for modern developers. +- [pyr](https://pyrun.dev) - Zero-config Python project manager. + +## Editorial Independence + +Sponsorship is logo or link placement in the README header. It does not influence which projects are listed. Listings are curated on merit through the normal [contribution process](CONTRIBUTING.md). + +We reserve the right to request changes to sponsor text, logos, or links that are misleading, off-topic, or incompatible with the README formatting. diff --git a/docs/CNAME b/docs/CNAME deleted file mode 100644 index 0f6ced66..00000000 --- a/docs/CNAME +++ /dev/null @@ -1 +0,0 @@ -awesome-python.com \ No newline at end of file diff --git a/docs/adr/0001-shortlist-not-catalog.md b/docs/adr/0001-shortlist-not-catalog.md new file mode 100644 index 00000000..095fad38 --- /dev/null +++ b/docs/adr/0001-shortlist-not-catalog.md @@ -0,0 +1,26 @@ +--- +status: accepted +--- + +# awesome-python is a shortlist of obvious choices, not a catalog + +By mid-2026 the list held 576 entries across 75 sections, with entry inflow up 24x year over year (96 additions in the trailing 12 months vs 4 the year before), concentrated in sections like AI and Agents (41 entries). The old acceptance model — three lanes (Industry Standard, Rising Star, Hidden Gem), only the first capped — admitted any project good enough in isolation, so categories grew without bound and stopped answering the reader's actual question: "what do I use for X?" We decided to reposition the list as a curated shortlist, per the Zen of Python: there should be one — and preferably only one — obvious way to do it. + +## The decision + +The list's scope test is "serves Python developers", replacing the old "primarily written in Python (>50%)" requirement: implementation language and packaging are irrelevant when Python developers use the thing in their Python work (uv and ty are Rust; agent skill packs are markdown), while a pure-Python project nobody uses in Python work does not belong. Each Use Case (a subcategory, or a flat section) lists at most its Obvious Choices — up to 3, plus up to 2 marked Challengers, hard maximum 5 (numbers provisional, to be reviewed after the prune). Admission is by maintainer editorial judgment, informed primarily by PyPI download counts rather than GitHub stars, and stated as final; judgment overrides the signal's known failure modes (CI/dependency-inflated counts, model releases consumed as weights rather than pip installs, large-but-specific audiences misread as "niche"). Once a Use Case is at cap, the only way in is Displacement: the PR names the entry it replaces and argues the newcomer does that job better. Use Cases are defined by the list's existing structure; an entry PR can never create the subcategory it needs. Standard-library entries hold a slot only where the stdlib module is itself the obvious choice. The existing stock gets the same test retroactively: a staged, worst-first prune (per-section sweep commits), with removed entries deleted outright — git history is the archive. Resources sections (Newsletters, Podcasts, Websites) are out of scope for now. + +## Considered options + +- Keep the three-lane model with caps on every lane: rejected because the lanes answer the wrong question ("is this good enough to enter?") once admission is comparative; rising-star momentum becomes evidence for a Challenger slot or a Displacement rather than an admission ticket of its own, and Hidden Gem is definitionally incompatible with "obvious". +- Rules-only, no retroactive prune: rejected because every rejection would face "but X is listed" precedent arguments, and readers would see no change. +- Archive removed entries in a separate file: rejected because it recreates the catalog one click away and dilutes the identity the change exists to restore. + +## Consequences + +- The list shrinks substantially: the maintainer-reviewed preview of the three largest sections kept 45 of 80 entries. Most future PRs will be rejected for fullness, not badness. +- awesome-python.com loses long-tail search traffic for the hundreds of niche tool names it will no longer carry. Accepted deliberately: reader trust over search surface. +- Fast-moving domains (e.g. AI and Agents) list current leaders by usage and absorb churn through Displacement; an oversized Use Case is either trimmed or Split into finer Use Cases by the maintainer. +- Linked awesome-* lists (e.g. awesome-python-testing) remain as the escape valve for readers who want exhaustive catalogs. + +See CONTEXT.md for the vocabulary (Use Case, Obvious Choice, Challenger, Displacement, Split). diff --git a/docs/audit-logs.md b/docs/audit-logs.md new file mode 100644 index 00000000..b1eae78f --- /dev/null +++ b/docs/audit-logs.md @@ -0,0 +1,67 @@ +# Audit Log + +awesome-python is audited section by section. Every entry gets re-verified against live data, and every removal lands in a commit whose body carries the reason. Git history is the archive. This file is the at-a-glance register of maintainer decisions that a single commit can't show. + +## Overrides + +[CONTRIBUTING.md](../CONTRIBUTING.md) allows the maintainer to exceed any limit for a specific entry or use case. Each override is recorded here. + +### Naming Exceptions + +Display names follow the canonical PyPI package name. These entries keep a different name by maintainer decision: + +- autobahn-python -- `autobahn`. +- django-rest-framework -- `djangorestframework`. +- django-rules -- `rules`. +- fasthtml -- `python-fasthtml`. +- graphify -- `graphifyy`. +- jinja -- `Jinja2`. +- mem0 -- `mem0ai`. +- pangu.py -- `pangu`. +- playwright-python -- `playwright`. +- pytorch -- `torch`. +- strawberry -- `strawberry-graphql`. +- strawberry-django -- `strawberry-graphql-django`. +- thealgorithms -- not on PyPI; its GitHub repo name is `Python`. + +### Stability Exceptions + +Entries admitted despite the Stable quality requirement, by maintainer decision: + +- zensical -- admitted 2026-08-23 at version 0.0.57, PyPI classifier `Development Status :: 3 - Alpha`, with no 1.0 target announced. Admitted as a challenger displacing mkdocs because the Documentation section was carrying a dying upstream: the Material for MkDocs team's announcement of 2025-11-05 put Material into maintenance mode for twelve months and called MkDocs itself unmaintained since 2024-08 and a supply chain risk. Re-check when 1.0 lands. +- openviking -- admitted 2026-08-25 (PR #3302) at version 0.4.16, PyPI classifier `Development Status :: 3 - Alpha`, with a README that said "OpenViking is still in its early stages". Admitted as a Data Layer challenger because the adoption is real: 459,446 downloads/month on pypistats (mirror-excluded) on 2026-08-24, and releases every few days. Re-check when 1.0 lands. +- claude-agent-sdk -- kept 2026-10-01 in the AI and Agents audit at version 0.2.163, PyPI classifier `Development Status :: 3 - Alpha`. Kept as a Vendor Agent SDKs obvious choice because the adoption is real: 30,033,689 downloads/month on ClickPy on 2026-10-01, up from 1.2M in March 2026 on pypistats. Re-check when 1.0 lands. +- uv-audit -- kept 2026-08-16 as a Supply Chain Security entry, recorded 2026-10-02. uv itself prints "`uv audit` is experimental and may change without warning". Kept because it is part of uv, the obvious package manager. Re-check when uv drops the experimental notice. + +### Challenger Exceptions + +Entries admitted as challengers without adoption-trajectory evidence, by maintainer decision: + +- seleniumbase -- admitted 2026-08-16 (PR #3284) as a Browser Automation challenger. It wraps selenium, the successor trajectory in the use case belongs to playwright, and its distinct value is its stealth/UC mode. Admitted because that stealth value counts toward Browser Automation. +- semantica -- kept 2026-10-01 in the AI and Agents audit as a Data Layer challenger at 16,582 downloads/month on ClickPy, with no adoption-trajectory evidence found. Kept by maintainer decision. +- jev-ultrafast -- kept 2026-09-24 as a Web Scraping Frameworks challenger, recorded 2026-10-02. Admitted while the repo was under a month old and not on PyPI, so it has no download signal and no adoption-trajectory evidence; its README calls it an MVP and it needs a hosted TypeSafe API key. Kept by maintainer decision. + +### Cap Exceptions + +Use cases holding more than 5 entries, by maintainer decision: + +- Asynchronous Programming > Async I/O -- 6 entries since the 2026-08-16 split (19875cf), recorded 2026-10-02. +- Code Analysis > Linters and Formatters -- 6 entries since Security Linters merged in on 2026-08-16 (3290ce5), recorded 2026-10-02. +- Package Management > Package Managers -- 6 entries since the subcategory was minted on 2026-08-16 (1b51907), recorded 2026-10-02. + +### Mature-stable Keeps + +These entries sit past the 12-month activity requirement without an override. Each one is kept by editorial judgment: mature, stable, and no successor exists. + +- annoy -- kept 2026-10-02 in the AI & ML audit, last push 2025-10-29. Unlike the others it has a successor, Spotify's own Voyager (2023), but kept by maintainer decision on 911,913 downloads/month on pypistats. +- blinker -- recorded 2026-10-02 ahead of its activity-line crossing on 2026-11-19 (last push 2025-11-19, last release 1.9.0 on 2024-11-08). Flask requires it. +- diskcache -- kept 2026-10-02, last push 2024-08-10, last release 5.6.3 (2023-08-31). Kept despite an unpatched pickle deserialization advisory, CVE-2025-69872 (GHSA-w8v5-vhqr-4h9v), on 25,526,045 downloads/month on ClickPy. +- django-rules -- recorded 2026-10-02 ahead of its activity-line crossing on 2026-10-11 (last push 2025-10-11, last release 3.5 on 2024-09-02). +- ftfy +- httpie -- kept 2026-08-16, recorded 2026-10-02. Last commit 2024-12-17, last release 3.2.4 on 2024-11-01. +- itsdangerous +- jieba +- jinja +- python-pptx -- kept 2026-10-02, last commit 2024-08-06, last release 1.0.2 on 2024-08-07. No successor for generating .pptx files. +- schedule -- kept 2026-08-16, recorded 2026-10-02. Last commit 2024-05-25. +- sortedcontainers diff --git a/docs/css/extra.css b/docs/css/extra.css deleted file mode 100644 index 1230d796..00000000 --- a/docs/css/extra.css +++ /dev/null @@ -1,9 +0,0 @@ -@media (min-width: 960px) { - html { - scroll-behavior: smooth; - } - - .md-content__inner > ul:nth-child(5) { - display: none; - } -} diff --git a/mkdocs.yml b/mkdocs.yml deleted file mode 100644 index 67a0373a..00000000 --- a/mkdocs.yml +++ /dev/null @@ -1,26 +0,0 @@ -site_name: Awesome Python -site_url: https://awesome-python.com -site_description: A curated list of awesome Python frameworks, libraries and software -site_author: Vinta Chen -repo_name: vinta/awesome-python -repo_url: https://github.com/vinta/awesome-python -theme: - name: material - palette: - primary: red - accent: pink -extra: - social: - - type: github - link: https://github.com/vinta - - type: twitter - link: https://twitter.com/vinta - - type: linkedin - link: https://www.linkedin.com/in/vinta -google_analytics: - - UA-510626-7 - - auto -extra_css: - - css/extra.css -nav: - - "Life is short, you need Python.": "index.md" diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 00000000..6f825068 --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,53 @@ +[project] +name = "awesome-python" +version = "0.1.0" +description = "An opinionated guide to the best Python frameworks, libraries, and tools." +authors = [{ name = "Vinta Chen", email = "vinta.chen@gmail.com" }] +readme = "README.md" +license = "MIT" +requires-python = ">=3.14" +dependencies = [] + +[project.urls] +Homepage = "https://awesome-python.com/" +Repository = "https://github.com/vinta/awesome-python" + +[dependency-groups] +build = ["httpx>=0.28.1", "jinja2>=3.1.6", "markdown-it-py>=4.0.0"] +lint = ["ruff>=0.15.6", "ty>=0.0.33"] +test = ["pytest>=9.0.3"] +preview = ["watchfiles>=1.2.0"] +dev = [ + { include-group = "build" }, + { include-group = "lint" }, + { include-group = "test" }, + { include-group = "preview" }, +] + +[tool.uv] +exclude-newer = "3 days" +no-build = true + +[tool.uv.pip] +only-binary = [":all:"] + +[tool.ruff] +line-length = 200 + +[tool.ty.environment] +python-version = "3.14" +root = ["website"] + +[tool.ty.terminal] +error-on-warning = true + +[tool.ty.rules] +division-by-zero = "error" +possibly-missing-attribute = "error" +possibly-missing-import = "error" +possibly-unresolved-reference = "error" +unused-ignore-comment = "error" + +[tool.pytest.ini_options] +testpaths = ["website/tests"] +pythonpath = ["website"] diff --git a/requirements.txt b/requirements.txt deleted file mode 100644 index 89d64c30..00000000 --- a/requirements.txt +++ /dev/null @@ -1,2 +0,0 @@ -mkdocs==1.0.4 -mkdocs-material==4.0.2 diff --git a/sort.py b/sort.py deleted file mode 100755 index 431bd02b..00000000 --- a/sort.py +++ /dev/null @@ -1,83 +0,0 @@ -#!/usr/bin/env python -# coding: utf-8 - -""" - The approach taken is explained below. I decided to do it simply. - Initially I was considering parsing the data into some sort of - structure and then generating an appropriate README. I am still - considering doing it - but for now this should work. The only issue - I see is that it only sorts the entries at the lowest level, and that - the order of the top-level contents do not match the order of the actual - entries. - - This could be extended by having nested blocks, sorting them recursively - and flattening the end structure into a list of lines. Revision 2 maybe ^.^. -""" - -def sort_blocks(): - # First, we load the current README into memory - with open('README.md', 'r') as read_me_file: - read_me = read_me_file.read() - - # Separating the 'table of contents' from the contents (blocks) - table_of_contents = ''.join(read_me.split('- - -')[0]) - blocks = ''.join(read_me.split('- - -')[1]).split('\n# ') - for i in range(len(blocks)): - if i == 0: - blocks[i] = blocks[i] + '\n' - else: - blocks[i] = '# ' + blocks[i] + '\n' - - # Sorting the libraries - inner_blocks = sorted(blocks[0].split('##')) - for i in range(1, len(inner_blocks)): - if inner_blocks[i][0] != '#': - inner_blocks[i] = '##' + inner_blocks[i] - inner_blocks = ''.join(inner_blocks) - - # Replacing the non-sorted libraries by the sorted ones and gathering all at the final_README file - blocks[0] = inner_blocks - final_README = table_of_contents + '- - -' + ''.join(blocks) - - with open('README.md', 'w+') as sorted_file: - sorted_file.write(final_README) - -def main(): - # First, we load the current README into memory as an array of lines - with open('README.md', 'r') as read_me_file: - read_me = read_me_file.readlines() - - # Then we cluster the lines together as blocks - # Each block represents a collection of lines that should be sorted - # This was done by assuming only links ([...](...)) are meant to be sorted - # Clustering is done by indentation - blocks = [] - last_indent = None - for line in read_me: - s_line = line.lstrip() - indent = len(line) - len(s_line) - - if any([s_line.startswith(s) for s in ['* [', '- [']]): - if indent == last_indent: - blocks[-1].append(line) - else: - blocks.append([line]) - last_indent = indent - else: - blocks.append([line]) - last_indent = None - - with open('README.md', 'w+') as sorted_file: - # Then all of the blocks are sorted individually - blocks = [ - ''.join(sorted(block, key=str.lower)) for block in blocks - ] - # And the result is written back to README.md - sorted_file.write(''.join(blocks)) - - # Then we call the sorting method - sort_blocks() - - -if __name__ == "__main__": - main() diff --git a/uv.lock b/uv.lock new file mode 100644 index 00000000..9029df7b --- /dev/null +++ b/uv.lock @@ -0,0 +1,349 @@ +version = 1 +revision = 3 +requires-python = ">=3.14" + +[options] +exclude-newer = "0001-01-01T00:00:00Z" # This has no effect and is included for backwards compatibility when using relative exclude-newer values. +exclude-newer-span = "P3D" + +[[package]] +name = "anyio" +version = "4.14.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "idna" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/61/cc/a381afa6efea9f496eff839d4a6a1aed3bfafc7b3ab4b0d1b243a12573dd/anyio-4.14.2.tar.gz", hash = "sha256:cfa139f3ed1a23ee8f88a145ddb5ac7605b8bbfd8592baacd7ce3d8bb4313c7f", size = 260176, upload-time = "2026-07-12T20:29:07.082Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/da/35/f2287558c17e29fafc8ef3daf819bb9834061cfa43bff8014f7df7f63bdc/anyio-4.14.2-py3-none-any.whl", hash = "sha256:9f505dda5ac9f0c8309b5e8bd445a8c2bf7246f3ce950121e45ea15bc41d1494", size = 125813, upload-time = "2026-07-12T20:29:05.763Z" }, +] + +[[package]] +name = "awesome-python" +version = "0.1.0" +source = { virtual = "." } + +[package.dev-dependencies] +build = [ + { name = "httpx" }, + { name = "jinja2" }, + { name = "markdown-it-py" }, +] +dev = [ + { name = "httpx" }, + { name = "jinja2" }, + { name = "markdown-it-py" }, + { name = "pytest" }, + { name = "ruff" }, + { name = "ty" }, + { name = "watchfiles" }, +] +lint = [ + { name = "ruff" }, + { name = "ty" }, +] +preview = [ + { name = "watchfiles" }, +] +test = [ + { name = "pytest" }, +] + +[package.metadata] + +[package.metadata.requires-dev] +build = [ + { name = "httpx", specifier = ">=0.28.1" }, + { name = "jinja2", specifier = ">=3.1.6" }, + { name = "markdown-it-py", specifier = ">=4.0.0" }, +] +dev = [ + { name = "httpx", specifier = ">=0.28.1" }, + { name = "jinja2", specifier = ">=3.1.6" }, + { name = "markdown-it-py", specifier = ">=4.0.0" }, + { name = "pytest", specifier = ">=9.0.3" }, + { name = "ruff", specifier = ">=0.15.6" }, + { name = "ty", specifier = ">=0.0.33" }, + { name = "watchfiles", specifier = ">=1.2.0" }, +] +lint = [ + { name = "ruff", specifier = ">=0.15.6" }, + { name = "ty", specifier = ">=0.0.33" }, +] +preview = [{ name = "watchfiles", specifier = ">=1.2.0" }] +test = [{ name = "pytest", specifier = ">=9.0.3" }] + +[[package]] +name = "certifi" +version = "2026.2.25" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/af/2d/7bf41579a8986e348fa033a31cdd0e4121114f6bce2457e8876010b092dd/certifi-2026.2.25.tar.gz", hash = "sha256:e887ab5cee78ea814d3472169153c2d12cd43b14bd03329a39a9c6e2e80bfba7", size = 155029, upload-time = "2026-02-25T02:54:17.342Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/9a/3c/c17fb3ca2d9c3acff52e30b309f538586f9f5b9c9cf454f3845fc9af4881/certifi-2026.2.25-py3-none-any.whl", hash = "sha256:027692e4402ad994f1c42e52a4997a9763c646b73e4096e4d5d6db8af1d6f0fa", size = 153684, upload-time = "2026-02-25T02:54:15.766Z" }, +] + +[[package]] +name = "colorama" +version = "0.4.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d8/53/6f443c9a4a8358a93a6792e2acffb9d9d5cb0a5cfd8802644b7b1c9a02e4/colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44", size = 27697, upload-time = "2022-10-25T02:36:22.414Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" }, +] + +[[package]] +name = "h11" +version = "0.16.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/01/ee/02a2c011bdab74c6fb3c75474d40b3052059d95df7e73351460c8588d963/h11-0.16.0.tar.gz", hash = "sha256:4e35b956cf45792e4caa5885e69fba00bdbc6ffafbfa020300e549b208ee5ff1", size = 101250, upload-time = "2025-04-24T03:35:25.427Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/04/4b/29cac41a4d98d144bf5f6d33995617b185d14b22401f75ca86f384e87ff1/h11-0.16.0-py3-none-any.whl", hash = "sha256:63cf8bbe7522de3bf65932fda1d9c2772064ffb3dae62d55932da54b31cb6c86", size = 37515, upload-time = "2025-04-24T03:35:24.344Z" }, +] + +[[package]] +name = "httpcore" +version = "1.0.9" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "certifi" }, + { name = "h11" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/06/94/82699a10bca87a5556c9c59b5963f2d039dbd239f25bc2a63907a05a14cb/httpcore-1.0.9.tar.gz", hash = "sha256:6e34463af53fd2ab5d807f399a9b45ea31c3dfa2276f15a2c3f00afff6e176e8", size = 85484, upload-time = "2025-04-24T22:06:22.219Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7e/f5/f66802a942d491edb555dd61e3a9961140fd64c90bce1eafd741609d334d/httpcore-1.0.9-py3-none-any.whl", hash = "sha256:2d400746a40668fc9dec9810239072b40b4484b640a8c38fd654a024c7a1bf55", size = 78784, upload-time = "2025-04-24T22:06:20.566Z" }, +] + +[[package]] +name = "httpx" +version = "0.28.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, + { name = "certifi" }, + { name = "httpcore" }, + { name = "idna" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b1/df/48c586a5fe32a0f01324ee087459e112ebb7224f646c0b5023f5e79e9956/httpx-0.28.1.tar.gz", hash = "sha256:75e98c5f16b0f35b567856f597f06ff2270a374470a5c2392242528e3e3e42fc", size = 141406, upload-time = "2024-12-06T15:37:23.222Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2a/39/e50c7c3a983047577ee07d2a9e53faf5a69493943ec3f6a384bdc792deb2/httpx-0.28.1-py3-none-any.whl", hash = "sha256:d909fcccc110f8c7faf814ca82a9a4d816bc5a6dbfea25d6591d6985b8ba59ad", size = 73517, upload-time = "2024-12-06T15:37:21.509Z" }, +] + +[[package]] +name = "idna" +version = "3.15" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/82/77/7b3966d0b9d1d31a36ddf1746926a11dface89a83409bf1483f0237aa758/idna-3.15.tar.gz", hash = "sha256:ca962446ea538f7092a95e057da437618e886f4d349216d2b1e294abfdb65fdc", size = 199245, upload-time = "2026-05-12T22:45:57.011Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d2/23/408243171aa9aaba178d3e2559159c24c1171a641aa83b67bdd3394ead8e/idna-3.15-py3-none-any.whl", hash = "sha256:048adeaf8c2d788c40fee287673ccaa74c24ffd8dcf09ffa555a2fbb59f10ac8", size = 72340, upload-time = "2026-05-12T22:45:55.733Z" }, +] + +[[package]] +name = "iniconfig" +version = "2.3.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/72/34/14ca021ce8e5dfedc35312d08ba8bf51fdd999c576889fc2c24cb97f4f10/iniconfig-2.3.0.tar.gz", hash = "sha256:c76315c77db068650d49c5b56314774a7804df16fee4402c1f19d6d15d8c4730", size = 20503, upload-time = "2025-10-18T21:55:43.219Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/cb/b1/3846dd7f199d53cb17f49cba7e651e9ce294d8497c8c150530ed11865bb8/iniconfig-2.3.0-py3-none-any.whl", hash = "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12", size = 7484, upload-time = "2025-10-18T21:55:41.639Z" }, +] + +[[package]] +name = "jinja2" +version = "3.1.6" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "markupsafe" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/df/bf/f7da0350254c0ed7c72f3e33cef02e048281fec7ecec5f032d4aac52226b/jinja2-3.1.6.tar.gz", hash = "sha256:0137fb05990d35f1275a587e9aee6d56da821fc83491a0fb838183be43f66d6d", size = 245115, upload-time = "2025-03-05T20:05:02.478Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/62/a1/3d680cbfd5f4b8f15abc1d571870c5fc3e594bb582bc3b64ea099db13e56/jinja2-3.1.6-py3-none-any.whl", hash = "sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67", size = 134899, upload-time = "2025-03-05T20:05:00.369Z" }, +] + +[[package]] +name = "markdown-it-py" +version = "4.0.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "mdurl" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/5b/f5/4ec618ed16cc4f8fb3b701563655a69816155e79e24a17b651541804721d/markdown_it_py-4.0.0.tar.gz", hash = "sha256:cb0a2b4aa34f932c007117b194e945bd74e0ec24133ceb5bac59009cda1cb9f3", size = 73070, upload-time = "2025-08-11T12:57:52.854Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/94/54/e7d793b573f298e1c9013b8c4dade17d481164aa517d1d7148619c2cedbf/markdown_it_py-4.0.0-py3-none-any.whl", hash = "sha256:87327c59b172c5011896038353a81343b6754500a08cd7a4973bb48c6d578147", size = 87321, upload-time = "2025-08-11T12:57:51.923Z" }, +] + +[[package]] +name = "markupsafe" +version = "3.0.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/7e/99/7690b6d4034fffd95959cbe0c02de8deb3098cc577c67bb6a24fe5d7caa7/markupsafe-3.0.3.tar.gz", hash = "sha256:722695808f4b6457b320fdc131280796bdceb04ab50fe1795cd540799ebe1698", size = 80313, upload-time = "2025-09-27T18:37:40.426Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/33/8a/8e42d4838cd89b7dde187011e97fe6c3af66d8c044997d2183fbd6d31352/markupsafe-3.0.3-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:eaa9599de571d72e2daf60164784109f19978b327a3910d3e9de8c97b5b70cfe", size = 11619, upload-time = "2025-09-27T18:37:06.342Z" }, + { url = "https://files.pythonhosted.org/packages/b5/64/7660f8a4a8e53c924d0fa05dc3a55c9cee10bbd82b11c5afb27d44b096ce/markupsafe-3.0.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:c47a551199eb8eb2121d4f0f15ae0f923d31350ab9280078d1e5f12b249e0026", size = 12029, upload-time = "2025-09-27T18:37:07.213Z" }, + { url = "https://files.pythonhosted.org/packages/da/ef/e648bfd021127bef5fa12e1720ffed0c6cbb8310c8d9bea7266337ff06de/markupsafe-3.0.3-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f34c41761022dd093b4b6896d4810782ffbabe30f2d443ff5f083e0cbbb8c737", size = 24408, upload-time = "2025-09-27T18:37:09.572Z" }, + { url = "https://files.pythonhosted.org/packages/41/3c/a36c2450754618e62008bf7435ccb0f88053e07592e6028a34776213d877/markupsafe-3.0.3-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:457a69a9577064c05a97c41f4e65148652db078a3a509039e64d3467b9e7ef97", size = 23005, upload-time = "2025-09-27T18:37:10.58Z" }, + { url = "https://files.pythonhosted.org/packages/bc/20/b7fdf89a8456b099837cd1dc21974632a02a999ec9bf7ca3e490aacd98e7/markupsafe-3.0.3-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:e8afc3f2ccfa24215f8cb28dcf43f0113ac3c37c2f0f0806d8c70e4228c5cf4d", size = 22048, upload-time = "2025-09-27T18:37:11.547Z" }, + { url = "https://files.pythonhosted.org/packages/9a/a7/591f592afdc734f47db08a75793a55d7fbcc6902a723ae4cfbab61010cc5/markupsafe-3.0.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:ec15a59cf5af7be74194f7ab02d0f59a62bdcf1a537677ce67a2537c9b87fcda", size = 23821, upload-time = "2025-09-27T18:37:12.48Z" }, + { url = "https://files.pythonhosted.org/packages/7d/33/45b24e4f44195b26521bc6f1a82197118f74df348556594bd2262bda1038/markupsafe-3.0.3-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:0eb9ff8191e8498cca014656ae6b8d61f39da5f95b488805da4bb029cccbfbaf", size = 21606, upload-time = "2025-09-27T18:37:13.485Z" }, + { url = "https://files.pythonhosted.org/packages/ff/0e/53dfaca23a69fbfbbf17a4b64072090e70717344c52eaaaa9c5ddff1e5f0/markupsafe-3.0.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:2713baf880df847f2bece4230d4d094280f4e67b1e813eec43b4c0e144a34ffe", size = 23043, upload-time = "2025-09-27T18:37:14.408Z" }, + { url = "https://files.pythonhosted.org/packages/46/11/f333a06fc16236d5238bfe74daccbca41459dcd8d1fa952e8fbd5dccfb70/markupsafe-3.0.3-cp314-cp314-win32.whl", hash = "sha256:729586769a26dbceff69f7a7dbbf59ab6572b99d94576a5592625d5b411576b9", size = 14747, upload-time = "2025-09-27T18:37:15.36Z" }, + { url = "https://files.pythonhosted.org/packages/28/52/182836104b33b444e400b14f797212f720cbc9ed6ba34c800639d154e821/markupsafe-3.0.3-cp314-cp314-win_amd64.whl", hash = "sha256:bdc919ead48f234740ad807933cdf545180bfbe9342c2bb451556db2ed958581", size = 15341, upload-time = "2025-09-27T18:37:16.496Z" }, + { url = "https://files.pythonhosted.org/packages/6f/18/acf23e91bd94fd7b3031558b1f013adfa21a8e407a3fdb32745538730382/markupsafe-3.0.3-cp314-cp314-win_arm64.whl", hash = "sha256:5a7d5dc5140555cf21a6fefbdbf8723f06fcd2f63ef108f2854de715e4422cb4", size = 14073, upload-time = "2025-09-27T18:37:17.476Z" }, + { url = "https://files.pythonhosted.org/packages/3c/f0/57689aa4076e1b43b15fdfa646b04653969d50cf30c32a102762be2485da/markupsafe-3.0.3-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:1353ef0c1b138e1907ae78e2f6c63ff67501122006b0f9abad68fda5f4ffc6ab", size = 11661, upload-time = "2025-09-27T18:37:18.453Z" }, + { url = "https://files.pythonhosted.org/packages/89/c3/2e67a7ca217c6912985ec766c6393b636fb0c2344443ff9d91404dc4c79f/markupsafe-3.0.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:1085e7fbddd3be5f89cc898938f42c0b3c711fdcb37d75221de2666af647c175", size = 12069, upload-time = "2025-09-27T18:37:19.332Z" }, + { url = "https://files.pythonhosted.org/packages/f0/00/be561dce4e6ca66b15276e184ce4b8aec61fe83662cce2f7d72bd3249d28/markupsafe-3.0.3-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1b52b4fb9df4eb9ae465f8d0c228a00624de2334f216f178a995ccdcf82c4634", size = 25670, upload-time = "2025-09-27T18:37:20.245Z" }, + { url = "https://files.pythonhosted.org/packages/50/09/c419f6f5a92e5fadde27efd190eca90f05e1261b10dbd8cbcb39cd8ea1dc/markupsafe-3.0.3-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:fed51ac40f757d41b7c48425901843666a6677e3e8eb0abcff09e4ba6e664f50", size = 23598, upload-time = "2025-09-27T18:37:21.177Z" }, + { url = "https://files.pythonhosted.org/packages/22/44/a0681611106e0b2921b3033fc19bc53323e0b50bc70cffdd19f7d679bb66/markupsafe-3.0.3-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:f190daf01f13c72eac4efd5c430a8de82489d9cff23c364c3ea822545032993e", size = 23261, upload-time = "2025-09-27T18:37:22.167Z" }, + { url = "https://files.pythonhosted.org/packages/5f/57/1b0b3f100259dc9fffe780cfb60d4be71375510e435efec3d116b6436d43/markupsafe-3.0.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:e56b7d45a839a697b5eb268c82a71bd8c7f6c94d6fd50c3d577fa39a9f1409f5", size = 24835, upload-time = "2025-09-27T18:37:23.296Z" }, + { url = "https://files.pythonhosted.org/packages/26/6a/4bf6d0c97c4920f1597cc14dd720705eca0bf7c787aebc6bb4d1bead5388/markupsafe-3.0.3-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:f3e98bb3798ead92273dc0e5fd0f31ade220f59a266ffd8a4f6065e0a3ce0523", size = 22733, upload-time = "2025-09-27T18:37:24.237Z" }, + { url = "https://files.pythonhosted.org/packages/14/c7/ca723101509b518797fedc2fdf79ba57f886b4aca8a7d31857ba3ee8281f/markupsafe-3.0.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:5678211cb9333a6468fb8d8be0305520aa073f50d17f089b5b4b477ea6e67fdc", size = 23672, upload-time = "2025-09-27T18:37:25.271Z" }, + { url = "https://files.pythonhosted.org/packages/fb/df/5bd7a48c256faecd1d36edc13133e51397e41b73bb77e1a69deab746ebac/markupsafe-3.0.3-cp314-cp314t-win32.whl", hash = "sha256:915c04ba3851909ce68ccc2b8e2cd691618c4dc4c4232fb7982bca3f41fd8c3d", size = 14819, upload-time = "2025-09-27T18:37:26.285Z" }, + { url = "https://files.pythonhosted.org/packages/1a/8a/0402ba61a2f16038b48b39bccca271134be00c5c9f0f623208399333c448/markupsafe-3.0.3-cp314-cp314t-win_amd64.whl", hash = "sha256:4faffd047e07c38848ce017e8725090413cd80cbc23d86e55c587bf979e579c9", size = 15426, upload-time = "2025-09-27T18:37:27.316Z" }, + { url = "https://files.pythonhosted.org/packages/70/bc/6f1c2f612465f5fa89b95bead1f44dcb607670fd42891d8fdcd5d039f4f4/markupsafe-3.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:32001d6a8fc98c8cb5c947787c5d08b0a50663d139f1305bac5885d98d9b40fa", size = 14146, upload-time = "2025-09-27T18:37:28.327Z" }, +] + +[[package]] +name = "mdurl" +version = "0.1.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d6/54/cfe61301667036ec958cb99bd3efefba235e65cdeb9c84d24a8293ba1d90/mdurl-0.1.2.tar.gz", hash = "sha256:bb413d29f5eea38f31dd4754dd7377d4465116fb207585f97bf925588687c1ba", size = 8729, upload-time = "2022-08-14T12:40:10.846Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b3/38/89ba8ad64ae25be8de66a6d463314cf1eb366222074cfda9ee839c56a4b4/mdurl-0.1.2-py3-none-any.whl", hash = "sha256:84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8", size = 9979, upload-time = "2022-08-14T12:40:09.779Z" }, +] + +[[package]] +name = "packaging" +version = "26.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/65/ee/299d360cdc32edc7d2cf530f3accf79c4fca01e96ffc950d8a52213bd8e4/packaging-26.0.tar.gz", hash = "sha256:00243ae351a257117b6a241061796684b084ed1c516a08c48a3f7e147a9d80b4", size = 143416, upload-time = "2026-01-21T20:50:39.064Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b7/b9/c538f279a4e237a006a2c98387d081e9eb060d203d8ed34467cc0f0b9b53/packaging-26.0-py3-none-any.whl", hash = "sha256:b36f1fef9334a5588b4166f8bcd26a14e521f2b55e6b9de3aaa80d3ff7a37529", size = 74366, upload-time = "2026-01-21T20:50:37.788Z" }, +] + +[[package]] +name = "pluggy" +version = "1.6.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f9/e2/3e91f31a7d2b083fe6ef3fa267035b518369d9511ffab804f839851d2779/pluggy-1.6.0.tar.gz", hash = "sha256:7dcc130b76258d33b90f61b658791dede3486c3e6bfb003ee5c9bfb396dd22f3", size = 69412, upload-time = "2025-05-15T12:30:07.975Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746", size = 20538, upload-time = "2025-05-15T12:30:06.134Z" }, +] + +[[package]] +name = "pygments" +version = "2.20.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c3/b2/bc9c9196916376152d655522fdcebac55e66de6603a76a02bca1b6414f6c/pygments-2.20.0.tar.gz", hash = "sha256:6757cd03768053ff99f3039c1a36d6c0aa0b263438fcab17520b30a303a82b5f", size = 4955991, upload-time = "2026-03-29T13:29:33.898Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f4/7e/a72dd26f3b0f4f2bf1dd8923c85f7ceb43172af56d63c7383eb62b332364/pygments-2.20.0-py3-none-any.whl", hash = "sha256:81a9e26dd42fd28a23a2d169d86d7ac03b46e2f8b59ed4698fb4785f946d0176", size = 1231151, upload-time = "2026-03-29T13:29:30.038Z" }, +] + +[[package]] +name = "pytest" +version = "9.0.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32'" }, + { name = "iniconfig" }, + { name = "packaging" }, + { name = "pluggy" }, + { name = "pygments" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/7d/0d/549bd94f1a0a402dc8cf64563a117c0f3765662e2e668477624baeec44d5/pytest-9.0.3.tar.gz", hash = "sha256:b86ada508af81d19edeb213c681b1d48246c1a91d304c6c81a427674c17eb91c", size = 1572165, upload-time = "2026-04-07T17:16:18.027Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d4/24/a372aaf5c9b7208e7112038812994107bc65a84cd00e0354a88c2c77a617/pytest-9.0.3-py3-none-any.whl", hash = "sha256:2c5efc453d45394fdd706ade797c0a81091eccd1d6e4bccfcd476e2b8e0ab5d9", size = 375249, upload-time = "2026-04-07T17:16:16.13Z" }, +] + +[[package]] +name = "ruff" +version = "0.15.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/51/df/f8629c19c5318601d3121e230f74cbee7a3732339c52b21daa2b82ef9c7d/ruff-0.15.6.tar.gz", hash = "sha256:8394c7bb153a4e3811a4ecdacd4a8e6a4fa8097028119160dffecdcdf9b56ae4", size = 4597916, upload-time = "2026-03-12T23:05:47.51Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/9e/2f/4e03a7e5ce99b517e98d3b4951f411de2b0fa8348d39cf446671adcce9a2/ruff-0.15.6-py3-none-linux_armv6l.whl", hash = "sha256:7c98c3b16407b2cf3d0f2b80c80187384bc92c6774d85fefa913ecd941256fff", size = 10508953, upload-time = "2026-03-12T23:05:17.246Z" }, + { url = "https://files.pythonhosted.org/packages/70/60/55bcdc3e9f80bcf39edf0cd272da6fa511a3d94d5a0dd9e0adf76ceebdb4/ruff-0.15.6-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:ee7dcfaad8b282a284df4aa6ddc2741b3f4a18b0555d626805555a820ea181c3", size = 10942257, upload-time = "2026-03-12T23:05:23.076Z" }, + { url = "https://files.pythonhosted.org/packages/e7/f9/005c29bd1726c0f492bfa215e95154cf480574140cb5f867c797c18c790b/ruff-0.15.6-py3-none-macosx_11_0_arm64.whl", hash = "sha256:3bd9967851a25f038fc8b9ae88a7fbd1b609f30349231dffaa37b6804923c4bb", size = 10322683, upload-time = "2026-03-12T23:05:33.738Z" }, + { url = "https://files.pythonhosted.org/packages/5f/74/2f861f5fd7cbb2146bddb5501450300ce41562da36d21868c69b7a828169/ruff-0.15.6-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:13f4594b04e42cd24a41da653886b04d2ff87adbf57497ed4f728b0e8a4866f8", size = 10660986, upload-time = "2026-03-12T23:05:53.245Z" }, + { url = "https://files.pythonhosted.org/packages/c1/a1/309f2364a424eccb763cdafc49df843c282609f47fe53aa83f38272389e0/ruff-0.15.6-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:e2ed8aea2f3fe57886d3f00ea5b8aae5bf68d5e195f487f037a955ff9fbaac9e", size = 10332177, upload-time = "2026-03-12T23:05:56.145Z" }, + { url = "https://files.pythonhosted.org/packages/30/41/7ebf1d32658b4bab20f8ac80972fb19cd4e2c6b78552be263a680edc55ac/ruff-0.15.6-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:70789d3e7830b848b548aae96766431c0dc01a6c78c13381f423bf7076c66d15", size = 11170783, upload-time = "2026-03-12T23:06:01.742Z" }, + { url = "https://files.pythonhosted.org/packages/76/be/6d488f6adca047df82cd62c304638bcb00821c36bd4881cfca221561fdfc/ruff-0.15.6-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:542aaf1de3154cea088ced5a819ce872611256ffe2498e750bbae5247a8114e9", size = 12044201, upload-time = "2026-03-12T23:05:28.697Z" }, + { url = "https://files.pythonhosted.org/packages/71/68/e6f125df4af7e6d0b498f8d373274794bc5156b324e8ab4bf5c1b4fc0ec7/ruff-0.15.6-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:1c22e6f02c16cfac3888aa636e9eba857254d15bbacc9906c9689fdecb1953ab", size = 11421561, upload-time = "2026-03-12T23:05:31.236Z" }, + { url = "https://files.pythonhosted.org/packages/f1/9f/f85ef5fd01a52e0b472b26dc1b4bd228b8f6f0435975442ffa4741278703/ruff-0.15.6-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:98893c4c0aadc8e448cfa315bd0cc343a5323d740fe5f28ef8a3f9e21b381f7e", size = 11310928, upload-time = "2026-03-12T23:05:45.288Z" }, + { url = "https://files.pythonhosted.org/packages/8c/26/b75f8c421f5654304b89471ed384ae8c7f42b4dff58fa6ce1626d7f2b59a/ruff-0.15.6-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:70d263770d234912374493e8cc1e7385c5d49376e41dfa51c5c3453169dc581c", size = 11235186, upload-time = "2026-03-12T23:05:50.677Z" }, + { url = "https://files.pythonhosted.org/packages/fc/d4/d5a6d065962ff7a68a86c9b4f5500f7d101a0792078de636526c0edd40da/ruff-0.15.6-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:55a1ad63c5a6e54b1f21b7514dfadc0c7fb40093fa22e95143cf3f64ebdcd512", size = 10635231, upload-time = "2026-03-12T23:05:37.044Z" }, + { url = "https://files.pythonhosted.org/packages/d6/56/7c3acf3d50910375349016cf33de24be021532042afbed87942858992491/ruff-0.15.6-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:8dc473ba093c5ec238bb1e7429ee676dca24643c471e11fbaa8a857925b061c0", size = 10340357, upload-time = "2026-03-12T23:06:04.748Z" }, + { url = "https://files.pythonhosted.org/packages/06/54/6faa39e9c1033ff6a3b6e76b5df536931cd30caf64988e112bbf91ef5ce5/ruff-0.15.6-py3-none-musllinux_1_2_i686.whl", hash = "sha256:85b042377c2a5561131767974617006f99f7e13c63c111b998f29fc1e58a4cfb", size = 10860583, upload-time = "2026-03-12T23:05:58.978Z" }, + { url = "https://files.pythonhosted.org/packages/cb/1e/509a201b843b4dfb0b32acdedf68d951d3377988cae43949ba4c4133a96a/ruff-0.15.6-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:cef49e30bc5a86a6a92098a7fbf6e467a234d90b63305d6f3ec01225a9d092e0", size = 11410976, upload-time = "2026-03-12T23:05:39.955Z" }, + { url = "https://files.pythonhosted.org/packages/6c/25/3fc9114abf979a41673ce877c08016f8e660ad6cf508c3957f537d2e9fa9/ruff-0.15.6-py3-none-win32.whl", hash = "sha256:bbf67d39832404812a2d23020dda68fee7f18ce15654e96fb1d3ad21a5fe436c", size = 10616872, upload-time = "2026-03-12T23:05:42.451Z" }, + { url = "https://files.pythonhosted.org/packages/89/7a/09ece68445ceac348df06e08bf75db72d0e8427765b96c9c0ffabc1be1d9/ruff-0.15.6-py3-none-win_amd64.whl", hash = "sha256:aee25bc84c2f1007ecb5037dff75cef00414fdf17c23f07dc13e577883dca406", size = 11787271, upload-time = "2026-03-12T23:05:20.168Z" }, + { url = "https://files.pythonhosted.org/packages/7f/d0/578c47dd68152ddddddf31cd7fc67dc30b7cdf639a86275fda821b0d9d98/ruff-0.15.6-py3-none-win_arm64.whl", hash = "sha256:c34de3dd0b0ba203be50ae70f5910b17188556630e2178fd7d79fc030eb0d837", size = 11060497, upload-time = "2026-03-12T23:05:25.968Z" }, +] + +[[package]] +name = "ty" +version = "0.0.33" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/84/44/9478c50c266826c1bf30d1692e589755bffa8f1c0a3eb7af8a346c255991/ty-0.0.33.tar.gz", hash = "sha256:46d63bda07403322cb6c28ccfdd5536be916e13df725c29f7ccd0a21f06bd9e8", size = 5559373, upload-time = "2026-04-28T10:45:13.18Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e9/24/e287388c63a19191be26b32ff4dbd06029834068150ebe2532939bc4c851/ty-0.0.33-py3-none-linux_armv6l.whl", hash = "sha256:94d0a9d2234261a8911396d59e506b5923fe0971dbda43b9dcea287936887fcc", size = 11021308, upload-time = "2026-04-28T10:45:43.34Z" }, + { url = "https://files.pythonhosted.org/packages/00/ca/ba1eed819895bd239fba8ee35dfcd5fcb266c203b0914a17a59579096bb5/ty-0.0.33-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:e4a2b5ba078f90de342f56b5f7979bb77c9b9b1d8625a041352ffc6ee93c4073", size = 10777272, upload-time = "2026-04-28T10:45:32.905Z" }, + { url = "https://files.pythonhosted.org/packages/25/a8/c3131d37b44b3fea1d6654a1c929a0cd0873822f77a90482b8ec28f6fbbd/ty-0.0.33-py3-none-macosx_11_0_arm64.whl", hash = "sha256:84ff5707825e9af9668d2bcf66975f93e520a63b524ab494e3a8265735be2563", size = 10201078, upload-time = "2026-04-28T10:45:23.374Z" }, + { url = "https://files.pythonhosted.org/packages/7b/db/d8e37ff0045810cc65e1ff36aa0da0a2253c05659787ac987df8a16c7897/ty-0.0.33-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e375285736f57886868e7af0b11c7b0ec5b6543fa15e7ad2a714fed9f077d4e0", size = 10732347, upload-time = "2026-04-28T10:45:21.444Z" }, + { url = "https://files.pythonhosted.org/packages/e0/1a/20e83a412506a918e4684fc67b567cf7cc13b105470b3428cb23c3d5aa13/ty-0.0.33-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:5680f6350c3b4e46b8bff6d7bb132366ea239463d6cad4892725d06046e65464", size = 10808238, upload-time = "2026-04-28T10:45:38.565Z" }, + { url = "https://files.pythonhosted.org/packages/5d/4b/d0a39f4464dc6cb4cc2c159473ce216bd1846bfb684c0323a3cb36dce5c6/ty-0.0.33-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:c5535538bad8d0f7e62bcdff02197cdb30e41451d80b35d27e17d128f2e1dc5d", size = 11288348, upload-time = "2026-04-28T10:45:08.419Z" }, + { url = "https://files.pythonhosted.org/packages/35/7e/f1745e0f9583363d7a83d9a4990fc244f76ecc30840ddad83dc16a33c52d/ty-0.0.33-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:da196c42bbbc069e1e21e3e52107c061aa9660352dae57a41930690b56e2c02d", size = 11789907, upload-time = "2026-04-28T10:45:19.064Z" }, + { url = "https://files.pythonhosted.org/packages/a5/71/25f39f46a12d662859d45bc648555d0661044eb43db6b5648c9947487da9/ty-0.0.33-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:9281672921ef6d4460e03146b5e6c18cb1a3e3a3b8a1a88f6f33226d05a469b7", size = 11500774, upload-time = "2026-04-28T10:45:48.012Z" }, + { url = "https://files.pythonhosted.org/packages/94/ec/136959ecbb7c71cb90537f5aea441c73f4ab24612868a6ecdc9d7444d32d/ty-0.0.33-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:82c1b8f303f82da64e878108e764be3ecbcd7c9903ac0a7f7031614ed00b97ab", size = 11360314, upload-time = "2026-04-28T10:45:05.402Z" }, + { url = "https://files.pythonhosted.org/packages/cf/95/32809575c222f00beed498cb728e9290a0f5009f930025381bb7253b2206/ty-0.0.33-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:efe3af412c9ff67bce5fa37d0a2b0d8555c24072b145a5bac6c79637f1c83abe", size = 10707785, upload-time = "2026-04-28T10:45:10.836Z" }, + { url = "https://files.pythonhosted.org/packages/13/89/c8e9531f7aa4a093359e15fa32c8e1277fbbe90d16894d7c6032d29f4b34/ty-0.0.33-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:aeec29c91ea768601747da546c3efc20b72c2fb1bd52bcc786a5c6eeff51d27b", size = 10834987, upload-time = "2026-04-28T10:45:40.738Z" }, + { url = "https://files.pythonhosted.org/packages/31/16/9835fbcf5338af1a1917bd28fdb8a7193c210b83f243aa286fa9f79cb3ad/ty-0.0.33-py3-none-musllinux_1_2_i686.whl", hash = "sha256:a535977c52bbb5f7e96b8b70a6ad375ad077f4a9ff2492508ea3816a2b403819", size = 10968968, upload-time = "2026-04-28T10:45:30.26Z" }, + { url = "https://files.pythonhosted.org/packages/36/69/64c76aabc1bc70c7f24b686cd93c3407f8ea430905e395f59bf9603ef571/ty-0.0.33-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:1d732facf39fcb221ba279d469c5040d37883e964f123b1563888efd34818180", size = 11458077, upload-time = "2026-04-28T10:45:45.971Z" }, + { url = "https://files.pythonhosted.org/packages/91/84/fae27b0c4718776a298690d31ca4cc1995f2e3e1c63a7b59e84c41498e9a/ty-0.0.33-py3-none-win32.whl", hash = "sha256:d90960b574428dc252f85e8598ec5fcb7f619794196b2fc95a90da075ed4681c", size = 10345364, upload-time = "2026-04-28T10:45:16.836Z" }, + { url = "https://files.pythonhosted.org/packages/3c/a0/a2938b23ae3e1a09a2d7c189e2ac5f7113676bae4e0e23948b568e18e5f8/ty-0.0.33-py3-none-win_amd64.whl", hash = "sha256:c1c3aec62c44de610c6e95f0a4e97ac3dbc07934bfdbf1fd90d758c9ff72f48e", size = 11342470, upload-time = "2026-04-28T10:45:26.455Z" }, + { url = "https://files.pythonhosted.org/packages/ab/62/7fb948aace38d2f6329261bb33c035a8484549c74f1db28649c7a4c6fed9/ty-0.0.33-py3-none-win_arm64.whl", hash = "sha256:0d44f99ba1b441e55e2aa301b2ac0a21112784931b46a5f66f4ea9efe5620d97", size = 10742673, upload-time = "2026-04-28T10:45:35.555Z" }, +] + +[[package]] +name = "watchfiles" +version = "1.2.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/cd/41/5e1a4bb12aac5f1493fa1bdc11154eca3b258ca4eba65d39c473fe19d8e9/watchfiles-1.2.0.tar.gz", hash = "sha256:c995fba777f1ea992f090f9236e9284cf7a5d1a0130dd5a3d82c598cacd76838", size = 108252, upload-time = "2026-05-18T04:32:04.251Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e7/54/a9c7ea9a82a4ac65e7004c0a03920b5cdd2f9c3b678757d9cd425aa51d53/watchfiles-1.2.0-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:b8c8358484d5fa12ef34f05b7f4168eaf1932f408725ff6d023c33ec17bd79d4", size = 400205, upload-time = "2026-05-18T04:32:05.153Z" }, + { url = "https://files.pythonhosted.org/packages/aa/5d/c9ab3534374a4a67450696905d6ef16a04405448b8dc52bd752ae50423d4/watchfiles-1.2.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:9f04b092229ad2c50126dd3c922c8822e51e605993764a33058d4a791ab42281", size = 392508, upload-time = "2026-05-18T04:30:54.849Z" }, + { url = "https://files.pythonhosted.org/packages/26/ca/1ad30103535cf0cecd7b993e8d50edc5351b1820e38f2d22e3df58962feb/watchfiles-1.2.0-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:7a7ce236284f002a156f70add88efe5c70879cccbb658be0822c54b1306fc09d", size = 452448, upload-time = "2026-05-18T04:30:53.727Z" }, + { url = "https://files.pythonhosted.org/packages/37/a1/ceee2cdf2afbd715fa07758d39c9859513eae411b23196f7fd039e5feedd/watchfiles-1.2.0-cp314-cp314-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:b9909cc2b48468b575eefa944919e1fe8a36c5849d5c7c168f80a8c1db69398e", size = 459605, upload-time = "2026-05-18T04:30:23.312Z" }, + { url = "https://files.pythonhosted.org/packages/e8/f6/421e30fd1cb3907a84ed92ab3f1983e37ba2dca015e9a894a048418417a2/watchfiles-1.2.0-cp314-cp314-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:0a37faaed405c67e28e6be45a1fa4f206ef5a2860f27c237db9fa30704c38242", size = 490757, upload-time = "2026-05-18T04:30:47.358Z" }, + { url = "https://files.pythonhosted.org/packages/41/b0/55ed1b97ed08be7bba6f9a541cac15f2a858e1d74d2b07b6da70a82aab00/watchfiles-1.2.0-cp314-cp314-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:9649193aa27bd9ff2e80ff29bfaa93085496c7a3a377592823cc58b77ee88add", size = 568672, upload-time = "2026-05-18T04:30:38.915Z" }, + { url = "https://files.pythonhosted.org/packages/d1/cf/d8ae8a80dd7bafab395ea7681c10237311bbf34d37704a8c744e7cf31fc7/watchfiles-1.2.0-cp314-cp314-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:4e4ff8e37f99cf1da89e255e07c9c4b37c214038c4283707bdec308cb1b0ea1f", size = 464197, upload-time = "2026-05-18T04:30:09.914Z" }, + { url = "https://files.pythonhosted.org/packages/7c/8a/3076c496ca8dafe0e8cd03fcebdfc47be4b1174b4e5b24ff6e396e6b3af2/watchfiles-1.2.0-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:054dc20fd2e3132b4c3883b4a00d72fd6e1f56fdaf89fccd12e8057d74cd74d7", size = 453181, upload-time = "2026-05-18T04:30:14.829Z" }, + { url = "https://files.pythonhosted.org/packages/e5/10/9745e17c98e7b8a86454df0a3c7b5686bd650383f1e9f26e4ebcbd6cc0c0/watchfiles-1.2.0-cp314-cp314-manylinux_2_31_riscv64.whl", hash = "sha256:e140ed30ebde76796b686e67c182cff10ea2fbab186fafd1560f74bb5a473a6e", size = 465109, upload-time = "2026-05-18T04:30:28.123Z" }, + { url = "https://files.pythonhosted.org/packages/8f/95/8ef4a95481d3e0cb52d62a06fa6e972e81424be2d9698b91a2fecca9904c/watchfiles-1.2.0-cp314-cp314-musllinux_1_1_aarch64.whl", hash = "sha256:bb7e52ecf68ba46d22df23467b87cffeb2146908aa523ebfe803019618cfda06", size = 630653, upload-time = "2026-05-18T04:31:49.304Z" }, + { url = "https://files.pythonhosted.org/packages/fd/e4/3b3bf36b0f829b50c6ebcb8d031583863c59f923d6a6af3d485e470d0fac/watchfiles-1.2.0-cp314-cp314-musllinux_1_1_x86_64.whl", hash = "sha256:23282a321c8baf9b3a3c4afff673f9fe65eb7fdc2338d765ccad9d3d1916a5ba", size = 657838, upload-time = "2026-05-18T04:31:06.497Z" }, + { url = "https://files.pythonhosted.org/packages/21/b1/6cbbb50c1f3002ab568777d44aa21206dfb8807a840990c4037523b51812/watchfiles-1.2.0-cp314-cp314-win32.whl", hash = "sha256:c0db965c5f79aa49fe672d297cf1febc5ad149b658594944f49a54a2b96270a7", size = 275108, upload-time = "2026-05-18T04:30:06.891Z" }, + { url = "https://files.pythonhosted.org/packages/92/45/190ce6db8dcb4536682cf75d3889ff1a27182a58cb519d343cb6d9ea63d8/watchfiles-1.2.0-cp314-cp314-win_amd64.whl", hash = "sha256:71283b39fd17e5408eb123bd37aeecfd9d54c81fc184421943208aadb879d103", size = 288441, upload-time = "2026-05-18T04:32:12.901Z" }, + { url = "https://files.pythonhosted.org/packages/74/0d/3eae1c2313ab08378431d907c3f8095ecca00f3eda33111cf4f0f2591799/watchfiles-1.2.0-cp314-cp314-win_arm64.whl", hash = "sha256:c5c19526f4e54a00f2666a6c0e9e40d582c09e865055ea7378bf0009aab857b3", size = 280684, upload-time = "2026-05-18T04:31:26.902Z" }, + { url = "https://files.pythonhosted.org/packages/b1/75/fb64e6c25d6b5ca636d03df34ffb1c6e9873303e76d27967e045f8df088f/watchfiles-1.2.0-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:d73a585accffa5ae39c17264c36ec3166d2fad7000c780f5ef83b2722afb9dd2", size = 398857, upload-time = "2026-05-18T04:32:17.108Z" }, + { url = "https://files.pythonhosted.org/packages/73/4e/9f7adf01754cbf81843722ccfec169d8f26c69778281a302855cecd2ee08/watchfiles-1.2.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:ae99b14c5f21e026e0e9d96f40e07d8570ebee6cafd9d8fc318354606daa7a28", size = 392413, upload-time = "2026-05-18T04:31:07.911Z" }, + { url = "https://files.pythonhosted.org/packages/47/c8/bec626bcc2d69f44b9acb24ce7d60ed7b16b73628eea747fcbd169d8edda/watchfiles-1.2.0-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4429f3b105524a10b72c3a819b091c495d2811d419c1e1e8df773a5a5974f831", size = 452409, upload-time = "2026-05-18T04:31:20.142Z" }, + { url = "https://files.pythonhosted.org/packages/00/b7/b6362068e81e7c556d155a34c35d40ac3ef42d747b06d7f6e5bf58e359c2/watchfiles-1.2.0-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:43d818978d06062d9b22c4fab2ebe44cf5213d42dc8e62bda8c2760cfa2eeb33", size = 458827, upload-time = "2026-05-18T04:32:06.219Z" }, + { url = "https://files.pythonhosted.org/packages/67/f8/9a813fa42afb1e0b4625e75f0479826644d3ee8dc287e093799bc01f390c/watchfiles-1.2.0-cp314-cp314t-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:b9f732dc58b2dbe69e464ccf8fff7a03b0dd0be439da4c0720d3558527d3d6b4", size = 490104, upload-time = "2026-05-18T04:31:56.034Z" }, + { url = "https://files.pythonhosted.org/packages/2f/bf/27dfb6094ca4c9aad21298b5525b6c53cb36121ee454331d05161e58d130/watchfiles-1.2.0-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:8f200104103feb097de4cab8fe4f5dd18a2026934c7dea98c55a2f5fd6d5a33b", size = 571360, upload-time = "2026-05-18T04:31:57.133Z" }, + { url = "https://files.pythonhosted.org/packages/fb/39/44a096d67270ea93df91d33877dbe91fbda3aa4f8ec2edf799d93eda8736/watchfiles-1.2.0-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:63ac26eefbf4af1741247d6fb68b11c49a25b2f7413fbd318a83a12aaa9cf666", size = 464644, upload-time = "2026-05-18T04:30:57.33Z" }, + { url = "https://files.pythonhosted.org/packages/0e/80/c7472203bad6268e3ef1ad260739704847898938ad7ea8b63a5131f46b50/watchfiles-1.2.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0c4997d4e4a55f0d02b6cde327322daf3a0400e5df6c6b15948994bf72497925", size = 454771, upload-time = "2026-05-18T04:30:48.736Z" }, + { url = "https://files.pythonhosted.org/packages/51/cf/3b10b268b4b7f0fc26e9debb5eef1998b515887840f444cd3ec80c688755/watchfiles-1.2.0-cp314-cp314t-manylinux_2_31_riscv64.whl", hash = "sha256:4c887eba18b7945ac73067a8b4a66f21cd46c2539b2bc68588f7be6c7eb6d26b", size = 463494, upload-time = "2026-05-18T04:31:33.826Z" }, + { url = "https://files.pythonhosted.org/packages/3d/3e/a4302545cd589262a0dc7d140e86f7688eba3f9c72776c27f7e23b8864c4/watchfiles-1.2.0-cp314-cp314t-musllinux_1_1_aarch64.whl", hash = "sha256:3416ff151bb6b5a8d8d11664974fbef4d9305b9b2957839ab5a270468fd8df30", size = 629383, upload-time = "2026-05-18T04:31:15.596Z" }, + { url = "https://files.pythonhosted.org/packages/db/99/d5649df0a9a410d45b7c882304d0b790903ac9b6e8f2cfd12114e0c6b9f2/watchfiles-1.2.0-cp314-cp314t-musllinux_1_1_x86_64.whl", hash = "sha256:0e831a271c035d89789cffc386b6aa1375f39f1cd25eb7ca0997e4970d152fc5", size = 656093, upload-time = "2026-05-18T04:31:58.707Z" }, + { url = "https://files.pythonhosted.org/packages/92/b9/362702539275019a54dd2e94511b31a9b89c5f9e6a21966de7eb692549fc/watchfiles-1.2.0-cp315-cp315-macosx_10_12_x86_64.whl", hash = "sha256:37a6721cdf3f65dbb13aa9503510ccb4451603ac837e44d265d7992a597e1374", size = 400109, upload-time = "2026-05-18T04:31:16.879Z" }, + { url = "https://files.pythonhosted.org/packages/8f/75/71d5ba62db781e5587bded1d944c675374bc4aa37ff33d5018d98e8b6538/watchfiles-1.2.0-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:2b37d10b5a63bd4d87e18472d80fa525bd670586fae62e5dd580452764879b65", size = 392167, upload-time = "2026-05-18T04:31:28.058Z" }, + { url = "https://files.pythonhosted.org/packages/3c/01/c66dd95d0423fe30d31820e2d1d5bda773764131bbb6ac0cb1cf303ac328/watchfiles-1.2.0-cp315-cp315-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0a105bc2283f67e8fbec74253ec2d94925de92ed72c0393f1206bf326b7b7b69", size = 452372, upload-time = "2026-05-18T04:31:00.836Z" }, + { url = "https://files.pythonhosted.org/packages/91/15/2fe99557e72f85627c6a8eed50d889e8d101623e060a22ad75b875cb932d/watchfiles-1.2.0-cp315-cp315-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:5327989a465505f05cfe06f04fa9d0c2fd5432bb243e10e6f012b1bdca3c8579", size = 459596, upload-time = "2026-05-18T04:31:34.96Z" }, + { url = "https://files.pythonhosted.org/packages/ed/23/d4acfa0023367428ed48351b3b9b267893037b6cadae55620c61c24bcfd4/watchfiles-1.2.0-cp315-cp315-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:ecb47f183a8025b2aa18b546725c3657e542112ae9c0613a2af79b4fa8d04ad7", size = 490869, upload-time = "2026-05-18T04:31:59.923Z" }, + { url = "https://files.pythonhosted.org/packages/a4/5f/3164cbdce06c9fb95c4f7b9e2f9760b5e2797af43a9ecc317ef42a23a278/watchfiles-1.2.0-cp315-cp315-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:8520a4ab0e37f770afc34459c4f8f7019e153f9124dc101c15538365875d1ab2", size = 571641, upload-time = "2026-05-18T04:32:00.948Z" }, + { url = "https://files.pythonhosted.org/packages/41/e6/85d3731c55e65cd7690f3f803d24c139588aaf863e4bf2148fe7a7fa1a19/watchfiles-1.2.0-cp315-cp315-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:71cd71740ed2c15211ebb237ced4e39a1cdf6f80566e5fe95428da1626f4fde6", size = 464444, upload-time = "2026-05-18T04:30:34.298Z" }, + { url = "https://files.pythonhosted.org/packages/f4/7d/562641012b8b09872742c3b8adf9629ec479fd78f8d68ae4a0c13da8add6/watchfiles-1.2.0-cp315-cp315-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f88af53d6ddaf72179ef613ddc905e6f4785f712b49b80b3bef9f3525e6194b4", size = 453593, upload-time = "2026-05-18T04:31:23.464Z" }, + { url = "https://files.pythonhosted.org/packages/56/fe/cb8ef3d6f929d14158fdaaad9925985b7310abc9384dcd4d82dd0016fb59/watchfiles-1.2.0-cp315-cp315-manylinux_2_31_riscv64.whl", hash = "sha256:cee9d5efd929efdac5f7e58f72b3376f676b64050a91c5b99a7094c5b2317488", size = 465096, upload-time = "2026-05-18T04:31:30.384Z" }, + { url = "https://files.pythonhosted.org/packages/25/91/80908e835e100527a9267147b08c0eee1fa6ab0ffec15edc04d1d44885f7/watchfiles-1.2.0-cp315-cp315-musllinux_1_1_aarch64.whl", hash = "sha256:b718bf356bbc15e559bd8ef41782b573b8ae0e3f177ab244b440568d7ea02cfb", size = 630638, upload-time = "2026-05-18T04:30:49.89Z" }, + { url = "https://files.pythonhosted.org/packages/46/4b/95ab2f256bb4af3cb2eb23b9317bda984ee6e0f11733a5c004a6c95b06e3/watchfiles-1.2.0-cp315-cp315-musllinux_1_1_x86_64.whl", hash = "sha256:922c0e019fe68b3ae392965a766b02a71ba1168c932cebc3733cd52c5fe5b377", size = 657684, upload-time = "2026-05-18T04:31:32.027Z" }, +] diff --git a/website/build.py b/website/build.py new file mode 100644 index 00000000..e3dd66b4 --- /dev/null +++ b/website/build.py @@ -0,0 +1,945 @@ +#!/usr/bin/env python3 +"""Build a single-page HTML site from README.md for the awesome-python website.""" + +import json +import re +import shutil +import subprocess +import xml.etree.ElementTree as ET +from collections import Counter +from collections.abc import Sequence +from datetime import UTC, datetime +from pathlib import Path +from typing import TypedDict + +from fetch_pypi_downloads_via_clickpy import OVERRIDES_FILE, normalize +from jinja2 import Environment, FileSystemLoader +from markdown_it import MarkdownIt +from markdown_it.tree import SyntaxTreeNode +from readme_parser import AlsoSee, ParsedGroup, ParsedSection, parse_readme, parse_sponsors, render_inline_text, slugify + +GITHUB_REPO_URL_RE = re.compile(r"^https?://github\.com/([^/]+/[^/]+?)(?:\.git)?/?$") +MARKDOWN_LINK_RE = re.compile(r"\[([^\]]+)\]\(([^)\s]+)\)") +BULLET_LINE_RE = re.compile(r"^\s*-\s") +ANCHOR_LINK_ATTRS_RE = re.compile(r'href="(#[^"]*)" target="_blank" rel="noopener"') +SITE_URL = "https://awesome-python.com/" +SITEMAP_URL = f"{SITE_URL}sitemap.xml" +SITEMAP_NS = "http://www.sitemaps.org/schemas/sitemap/0.9" +PLURAL_NOUNS = {"Clients", "Drivers", "Engines", "Files", "Frameworks", "Generators", "Implementations", "Panels", "Queues", "Repositories", "Schedulers", "Servers", "Stdlib", "Tools"} + +BUILTIN_FILTER = "Stdlib" +BUILTIN_SLUG = "built-in" +BUILTIN_PATH = f"/categories/{BUILTIN_SLUG}/" +BUILTIN_PUBLIC_URL = f"{SITE_URL}categories/{BUILTIN_SLUG}/" + +SPONSORSHIP_PATH = "/sponsorship/" +SPONSORSHIP_PUBLIC_URL = f"{SITE_URL}sponsorship/" +SPONSORSHIP_DESCRIPTION = "Sponsorship for awesome-python: tiers, audience, and how to get your product in front of professional Python developers evaluating tools for production use." + +BUNDLED_PREFIX_RE = re.compile(r"^\(part of ") + +SOURCE_TYPE_DOMAINS = { + "docs.python.org": BUILTIN_FILTER, + "gitlab.com": "GitLab", + "bitbucket.org": "Bitbucket", +} + + +class TemplateSubcategory(TypedDict): + name: str + value: str + slug: str + url: str + + +class TemplateEntry(TypedDict): + name: str + url: str + description: str + categories: list[str] + groups: list[str] + subcategories: list[TemplateSubcategory] + stars: int | None + downloads: int | None + owner: str | None + last_commit_at: str | None + source_type: str | None + bundled: bool + badge: str | None + also_see: list[AlsoSee] + + +class EntryGroup(TypedDict): + name: str # empty for a page with a single unnamed group + slug: str + url: str # links the group heading to its own page, empty when it has none + lead: str # the section intro's first paragraph as HTML, shown under the heading on group pages + entries: list[TemplateEntry] + + +class SyntheticCategory(TypedDict): + name: str + slug: str + description: str + description_html: str + + +TemplateCategory = ParsedSection | SyntheticCategory + + +def detect_source_type(url: str) -> str | None: + """Detect source type from URL domain. Returns None for GitHub URLs.""" + if GITHUB_REPO_URL_RE.match(url): + return None + for domain, source_type in SOURCE_TYPE_DOMAINS.items(): + if domain in url: + return source_type + if "github.com" not in url: + return "External" + return None + + +def extract_github_repo(url: str) -> str | None: + """Extract owner/repo from a GitHub repo URL. Returns None for non-GitHub URLs.""" + m = GITHUB_REPO_URL_RE.match(url) + return m.group(1) if m else None + + +def load_stars(path: Path) -> dict[str, dict]: + """Load star data from JSON. Returns empty dict if file doesn't exist or is corrupt.""" + if path.exists(): + try: + return json.loads(path.read_text(encoding="utf-8")) + except json.JSONDecodeError: + return {} + return {} + + +def load_pypi_badges() -> dict[str, str]: + """Load the badge each entry shows in place of a download count. + + Keyed by normalized README name. Only entries whose missing count needs + explaining beyond "Not on PyPI" carry one, e.g. the SDK monorepos that do + ship on PyPI, just as many per-service packages. + """ + raw = json.loads(OVERRIDES_FILE.read_text()) + return {name: entry["badge"] for name, entry in raw.items() if entry.get("badge")} + + +def load_downloads(path: Path) -> dict[str, int]: + """Load last-30-day download counts from the TSV cache, keyed by normalized README name. + + Columns: name, package, downloads, fetched_at. Skips the header and + NOT_FOUND rows. Returns empty dict if the file doesn't exist. + """ + if not path.exists(): + return {} + downloads: dict[str, int] = {} + for line in path.read_text(encoding="utf-8").splitlines()[1:]: + parts = line.split("\t") + if len(parts) >= 3 and parts[2].isdigit(): + downloads[parts[0]] = int(parts[2]) + return downloads + + +def sort_entries(entries: Sequence[TemplateEntry]) -> list[TemplateEntry]: + """Sort entries by downloads descending, then name ascending. + + Four tiers: entries with download counts first, then starred entries by + stars descending, then stdlib, then the rest. + """ + + def sort_key(entry: TemplateEntry) -> tuple[int, int, int, str]: + downloads = entry["downloads"] + stars = entry["stars"] + name = entry["name"].lower() + if downloads is not None: + return (0, -downloads, 0, name) + if stars is not None: + builtin = 1 if entry.get("source_type") == BUILTIN_FILTER else 0 + return (1, -stars, builtin, name) + if entry.get("source_type") == BUILTIN_FILTER: + return (2, 0, 0, name) + return (3, 0, 0, name) + + return sorted(entries, key=sort_key) + + +def build_robots_txt() -> str: + return f"User-agent: *\nContent-Signal: search=yes, ai-input=yes, ai-train=yes\nAllow: /\n\nSitemap: {SITEMAP_URL}\n" + + +WEBSITE_ID = f"{SITE_URL}#website" +ISPARTOF_WEBSITE = {"@type": "WebSite", "@id": WEBSITE_ID} + + +def _website_node() -> dict: + return { + "@type": "WebSite", + "@id": WEBSITE_ID, + "name": "Awesome Python", + "url": SITE_URL, + "inLanguage": "en", + "sameAs": "https://github.com/vinta/awesome-python", + } + + +def _item_list_payload(entries: Sequence[TemplateEntry]) -> dict: + return { + "@type": "ItemList", + "numberOfItems": len(entries), + "itemListElement": [ + { + "@type": "ListItem", + "position": i, + "name": entry["name"], + "url": entry["url"], + } + for i, entry in enumerate(entries, start=1) + ], + } + + +def build_homepage_json_ld(entries: Sequence[TemplateEntry], total_categories: int) -> dict: + description = ( + "An opinionated guide to the best Python frameworks, libraries, and tools. " + f"Explore {len(entries)} curated projects across {total_categories} categories, " + "from AI and agents to data science and web development." + ) + return { + "@context": "https://schema.org", + "@graph": [ + _website_node(), + { + "@type": "CollectionPage", + "@id": SITE_URL, + "name": "Awesome Python", + "url": SITE_URL, + "description": description, + "isPartOf": ISPARTOF_WEBSITE, + "inLanguage": "en", + "mainEntity": _item_list_payload(entries), + }, + ], + } + + +def category_meta_title(name: str, parent_name: str | None = None) -> str: + if parent_name: + # Lead with the section's title so the page matches searches like "python jwt library". + title = f"{category_meta_title(parent_name).removesuffix(' - Awesome Python')}: {name}" + # Google shows the site name above each result, so a long title drops it rather than get cut off. + return f"{title} - Awesome Python" if len(title) <= 43 else title + # Names ending in one of these nouns already say what the entries are. + noun = "" if name.rsplit(" ", 1)[-1] in PLURAL_NOUNS else " Libraries" + prefix = "" if name.startswith("Python ") else "Python " + title = f"{prefix}{name}{noun} - Awesome Python" + if len(title) <= 60: + return title + return f"{name} - Awesome Python" + + +def category_meta_description(name: str, entry_count: int, description: str) -> str: + count_sentence = f"Explore {entry_count} curated Python project{'s' if entry_count != 1 else ''} in {name}." + if description: + lead = description if description.endswith((".", "!", "?")) else f"{description}." + return f"{lead} {count_sentence}" + return f"{count_sentence} Part of the Awesome Python catalog." + + +def subcategory_meta_description(name: str, parent_name: str, entry_names: Sequence[str]) -> str: + names = entry_names[0] if len(entry_names) == 1 else f"{', '.join(entry_names[:-1])}{',' if len(entry_names) > 2 else ''} and {entry_names[-1]}" + return f"The {name} picks in Awesome Python's {parent_name} list: {names}." + + +def load_category_intro(path: Path) -> tuple[str, str, str, str]: + """Render a category intro file to HTML, split at the end of its "How to choose:" list. + + Returns the part shown above the table, the guide shown below it, and the + first paragraph as plain text for the meta description and as HTML for + group pages. A file without the list keeps everything above the table. + Returns empty strings if the category has no intro file. + """ + if not path.exists(): + return "", "", "", "" + md = MarkdownIt("commonmark") + tokens = md.parse(path.read_text(encoding="utf-8")) + for token in tokens: + for child in token.children or []: + if child.type == "link_open": + child.attrSet("target", "_blank") + child.attrSet("rel", "noopener") + lead = next(node for node in SyntaxTreeNode(tokens).children if node.type == "paragraph") + split_at = len(tokens) + for i, token in enumerate(tokens): + if token.type == "inline" and token.level == 1 and token.content == "How to choose:" and i + 2 < len(tokens) and tokens[i + 2].type == "bullet_list_open": + split_at = next(j for j in range(i + 3, len(tokens)) if tokens[j].type == "bullet_list_close" and tokens[j].level == 0) + 1 + break + render = md.renderer.render + return ( + render(tokens[:split_at], md.options, {}), + render(tokens[split_at:], md.options, {}), + render_inline_text(lead.children[0].children), + render(lead.to_tokens(), md.options, {}), + ) + + +def group_section_entries(section: ParsedSection, entries_by_key: dict[tuple[str, str], TemplateEntry]) -> list[EntryGroup]: + """Group a section's entries by use case (subcategory), both in README order.""" + groups: dict[str, EntryGroup] = {} + for parsed in section["entries"]: + name = parsed["subcategory"] + slug = slugify(name) if name else "" + group = groups.setdefault(name, EntryGroup(name=name, slug=slug, url=subcategory_path(section["slug"], slug) if name else "", lead="", entries=[])) + group["entries"].append(entries_by_key[(parsed["url"], parsed["name"])]) + return list(groups.values()) + + +def group_entries_by_section(sections: Sequence[ParsedSection], entries_by_key: dict[tuple[str, str], TemplateEntry], intros_dir: Path) -> list[EntryGroup]: + """Group a thematic group's entries by section, both in README order, listing each entry once.""" + placed: set[tuple[str, str]] = set() + groups: list[EntryGroup] = [] + for section in sections: + entries: list[TemplateEntry] = [] + for parsed in section["entries"]: + key = (parsed["url"], parsed["name"]) + if key not in placed: + placed.add(key) + entries.append(entries_by_key[key]) + groups.append(EntryGroup(name=section["name"], slug=section["slug"], url=category_path(section), lead=load_category_intro(intros_dir / f"{section['slug']}.md")[3], entries=entries)) + return groups + + +def build_breadcrumb_json_ld(items: Sequence[tuple[str, str]]) -> dict: + return { + "@type": "BreadcrumbList", + "itemListElement": [ + { + "@type": "ListItem", + "position": i, + "name": name, + "item": url, + } + for i, (name, url) in enumerate(items, start=1) + ], + } + + +def build_category_json_ld( + name: str, + url: str, + description: str, + entries: Sequence[TemplateEntry], + breadcrumbs: Sequence[tuple[str, str]], +) -> dict: + return { + "@context": "https://schema.org", + "@graph": [ + _website_node(), + { + "@type": "CollectionPage", + "@id": url, + "name": name, + "url": url, + "description": description, + "isPartOf": ISPARTOF_WEBSITE, + "inLanguage": "en", + "mainEntity": _item_list_payload(entries), + }, + build_breadcrumb_json_ld(breadcrumbs), + ], + } + + +def build_sponsorship_json_ld() -> dict: + return { + "@context": "https://schema.org", + "@graph": [ + _website_node(), + { + "@type": "WebPage", + "@id": SPONSORSHIP_PUBLIC_URL, + "name": "Sponsor Awesome Python", + "url": SPONSORSHIP_PUBLIC_URL, + "description": SPONSORSHIP_DESCRIPTION, + "isPartOf": ISPARTOF_WEBSITE, + "inLanguage": "en", + }, + build_breadcrumb_json_ld( + [ + ("Awesome Python", SITE_URL), + ("Sponsorship", SPONSORSHIP_PUBLIC_URL), + ] + ), + ], + } + + +def category_path(category: ParsedSection) -> str: + return f"/categories/{category['slug']}/" + + +def category_public_url(category: ParsedSection) -> str: + return f"{SITE_URL}categories/{category['slug']}/" + + +def group_path(group_slug: str) -> str: + return f"/categories/{group_slug}/" + + +def group_public_url(group_slug: str) -> str: + return f"{SITE_URL}categories/{group_slug}/" + + +def subcategory_path(category_slug: str, subcategory_slug: str) -> str: + return f"/categories/{category_slug}/{subcategory_slug}/" + + +def subcategory_public_url(category_slug: str, subcategory_slug: str) -> str: + return f"{SITE_URL}categories/{category_slug}/{subcategory_slug}/" + + +def synthetic_category(name: str, slug: str) -> SyntheticCategory: + return {"name": name, "slug": slug, "description": "", "description_html": ""} + + +def git_last_change_date(repo_root: Path, *log_args: str) -> str: + """Return the date of the last commit matching `git log` args, or "" without git history.""" + result = subprocess.run(["git", "log", "-1", "--format=%cs", "--no-patch", *log_args], cwd=repo_root, capture_output=True, text=True) + return result.stdout.strip() + + +def section_line_range(readme_text: str, name: str) -> str: + """Return a section's README lines as a `git log -L` range, from its heading to the line before the next heading.""" + lines = readme_text.split("\n") + start = next(i for i, line in enumerate(lines) if re.fullmatch(rf"#+ {re.escape(name)}", line)) + end = next((i for i in range(start + 1, len(lines)) if lines[i].startswith("#")), len(lines)) + return f"{start + 1},{end}:README.md" + + +def write_sitemap_xml(path: Path, urls: Sequence[tuple[str, str]]) -> None: + ET.register_namespace("", SITEMAP_NS) + urlset = ET.Element(f"{{{SITEMAP_NS}}}urlset") + for url, lastmod in urls: + url_el = ET.SubElement(urlset, f"{{{SITEMAP_NS}}}url") + loc_el = ET.SubElement(url_el, f"{{{SITEMAP_NS}}}loc") + loc_el.text = url + lastmod_el = ET.SubElement(url_el, f"{{{SITEMAP_NS}}}lastmod") + lastmod_el.text = lastmod + + tree = ET.ElementTree(urlset) + ET.indent(tree, space=" ") + tree.write(path, encoding="utf-8", xml_declaration=True) + with path.open("ab") as f: + f.write(b"\n") + + +def top_level_heading_text(line: str) -> str | None: + stripped = line.strip() + match = re.match(r"^(#{1,2})\s+(.+)$", stripped) + if match is None: + return None + return match.group(2).strip().strip("#").strip().strip("*").strip() + + +def extract_categories_body(markdown: str) -> str: + """Return content from `Categories` through `Projects`, excluding later sections.""" + lines = markdown.splitlines(keepends=True) + start_idx = None + end_idx = len(lines) + for i, line in enumerate(lines): + heading = top_level_heading_text(line) + if heading is None: + continue + if start_idx is None and heading.lower() == "categories": + start_idx = i + 1 + while start_idx < len(lines) and lines[start_idx].strip() == "": + start_idx += 1 + elif start_idx is not None and heading.lower() in ("resources", "contributing"): + end_idx = i + break + if start_idx is None: + return "" + return "".join(lines[start_idx:end_idx]).rstrip() + "\n" + + +def github_markdown_anchor(text: str) -> str: + anchor = text.strip().lower() + anchor = re.sub(r"[^\w\s-]", "", anchor) + anchor = re.sub(r"\s", "-", anchor) + return f"#{anchor}" + + +def link_llms_category_index_to_canonical_pages(markdown: str, categories: Sequence[ParsedSection]) -> str: + """Point the README-derived category index at canonical category pages.""" + category_urls = {} + for category in categories: + public_url = category_public_url(category) + category_urls[f"#{category['slug']}"] = public_url + category_urls[github_markdown_anchor(category["name"])] = public_url + lines = markdown.splitlines(keepends=True) + out: list[str] = [] + + def replace_link(match: re.Match[str]) -> str: + target = match.group(2) + url = category_urls.get(target) + if url is None: + return match.group(0) + return match.group(0).replace(f"({target})", f"({url})", 1) + + for line in lines: + out.append(MARKDOWN_LINK_RE.sub(replace_link, line)) + + return "".join(out) + + +def link_description_anchors_to_category_pages(categories: Sequence[ParsedSection]) -> None: + """Point README anchor links in section descriptions at category pages, which lack those anchors.""" + category_paths = {} + for category in categories: + category_paths[f"#{category['slug']}"] = category_path(category) + category_paths[github_markdown_anchor(category["name"])] = category_path(category) + + def replace_attrs(match: re.Match[str]) -> str: + path = category_paths.get(match.group(1)) + return f'href="{path}"' if path else match.group(0) + + for category in categories: + category["description_html"] = ANCHOR_LINK_ATTRS_RE.sub(replace_attrs, category["description_html"]) + + +def build_llms_txt( + template_text: str, + *, + readme_text: str, + subtitle: str, + build_date: str, + stars_data: dict[str, dict], + downloads_data: dict[str, int], + categories: Sequence[ParsedSection], + total_entries: int, +) -> str: + """Render the llms.txt entry point with the curated category shortlist.""" + categories_md = annotate_entries_with_stats( + link_llms_category_index_to_canonical_pages( + extract_categories_body(readme_text).rstrip(), + categories, + ), + stars_data, + downloads_data, + ) + text_env = Environment(autoescape=False, trim_blocks=True, lstrip_blocks=True) + rendered = text_env.from_string(template_text).render( + subtitle=subtitle, + build_date=build_date, + site_url=SITE_URL, + github_repo_url="https://github.com/vinta/awesome-python", + contributing_url="https://github.com/vinta/awesome-python/blob/master/CONTRIBUTING.md", + sponsorship_url=SPONSORSHIP_PUBLIC_URL, + sitemap_url=SITEMAP_URL, + categories_md=categories_md, + total_entries=total_entries, + total_categories=len(categories), + ) + return rendered.rstrip() + "\n" + + +def annotate_entries_with_stats( + markdown: str, + stars_data: dict[str, dict], + downloads_data: dict[str, int], +) -> str: + """Append download and star counts to bullet entry lines. + + Downloads are looked up by the first link's display name (the same key the + TSV cache uses); stars by the first GitHub link with known star data. + """ + lines = markdown.splitlines(keepends=True) + out: list[str] = [] + for line in lines: + if not BULLET_LINE_RE.match(line): + out.append(line) + continue + links = MARKDOWN_LINK_RE.findall(line) + parts: list[str] = [] + if links: + name, url = links[0] + # Category-index bullets link into the site itself, and stdlib entries + # would hit same-named PyPI backports (e.g. logging) — no counts for either. + if not url.startswith((SITE_URL, "#")) and detect_source_type(url) != BUILTIN_FILTER: + downloads = downloads_data.get(normalize(name)) + if downloads is not None: + parts.append(f"PyPI downloads/month: {downloads}") + for _, url in links: + repo_key = extract_github_repo(url) + if not repo_key: + continue + entry = stars_data.get(repo_key) + if entry and "stars" in entry: + parts.append(f"GitHub stars: {entry['stars']}") + break + if not parts: + out.append(line) + continue + stripped = line.rstrip("\n") + ending = line[len(stripped) :] + out.append(f"{stripped} ({', '.join(parts)}){ending}") + return "".join(out) + + +def remove_sponsors_section(markdown: str) -> str: + lines = markdown.splitlines(keepends=True) + start_idx = None + for i, line in enumerate(lines): + heading = top_level_heading_text(line) + if heading and heading.lower() == "sponsors": + start_idx = i + break + + if start_idx is None: + return markdown + + end_idx = len(lines) + for i, line in enumerate(lines[start_idx + 1 :], start=start_idx + 1): + if top_level_heading_text(line): + end_idx = i + break + + return "".join(lines[:start_idx] + lines[end_idx:]) + + +def extract_entries( + categories: list[ParsedSection], + groups: list[ParsedGroup], +) -> list[TemplateEntry]: + """Flatten categories into individual library entries for table display. + + Entries appearing in multiple categories are merged into a single entry + with lists of categories and groups. + """ + cat_to_group = {cat["name"]: group["name"] for group in groups for cat in group["categories"]} + + seen: dict[tuple[str, str], TemplateEntry] = {} # (url, name) -> entry + entries: list[TemplateEntry] = [] + for cat in categories: + group_name = cat_to_group.get(cat["name"], "Other") + for entry in cat["entries"]: + key = (entry["url"], entry["name"]) + existing = seen.get(key) + if existing is None: + existing = TemplateEntry( + name=entry["name"], + url=entry["url"], + description=entry["description"], + categories=[], + groups=[], + subcategories=[], + stars=None, + downloads=None, + owner=None, + last_commit_at=None, + source_type=detect_source_type(entry["url"]), + bundled=bool(BUNDLED_PREFIX_RE.match(entry["description"])), + badge=None, + also_see=entry["also_see"], + ) + seen[key] = existing + entries.append(existing) + if cat["name"] not in existing["categories"]: + existing["categories"].append(cat["name"]) + if group_name not in existing["groups"]: + existing["groups"].append(group_name) + subcat = entry["subcategory"] + if subcat: + scoped = f"{cat['name']} > {subcat}" + if not any(s["value"] == scoped for s in existing["subcategories"]): + sub_slug = slugify(subcat) + existing["subcategories"].append( + TemplateSubcategory( + name=subcat, + value=scoped, + slug=sub_slug, + url=f"/categories/{cat['slug']}/{sub_slug}/", + ) + ) + return entries + + +def build(repo_root: Path) -> None: + """Main build: parse README, render single-page HTML via Jinja2 templates.""" + website = repo_root / "website" + readme_text = (repo_root / "README.md").read_text(encoding="utf-8") + + subtitle = "" + for line in readme_text.split("\n"): + stripped = line.strip() + if stripped and not stripped.startswith("#"): + subtitle = stripped + break + + parsed_groups = parse_readme(readme_text) + sponsors = parse_sponsors(readme_text) + + categories = [cat for g in parsed_groups for cat in g["categories"]] + cat_slugs = [cat["slug"] for cat in categories] + group_slugs = [g["slug"] for g in parsed_groups] + all_top_level_slugs = cat_slugs + group_slugs + [BUILTIN_SLUG] + duplicates = {s for s, n in Counter(all_top_level_slugs).items() if n > 1} + if duplicates: + raise ValueError(f"slug collision in /categories/ namespace: {sorted(duplicates)}. Rename a category or group so their slugs differ.") + link_description_anchors_to_category_pages(categories) + total_entries = sum(c["entry_count"] for c in categories) + entries = extract_entries(categories, parsed_groups) + build_date = datetime.now(UTC) + + stars_data = load_stars(website / "data" / "github_stars.json") + downloads_data = load_downloads(website / "data" / "pypi_downloads.tsv") + pypi_badges = load_pypi_badges() + + repo_self = stars_data.get("vinta/awesome-python", {}) + repo_stars = None + if "stars" in repo_self: + stars_val = repo_self["stars"] + repo_stars = f"{stars_val // 1000}k" if stars_val >= 1000 else str(stars_val) + + for entry in entries: + repo_key = extract_github_repo(entry["url"]) + if not repo_key and entry.get("source_type") == BUILTIN_FILTER: + repo_key = "python/cpython" + if repo_key and repo_key in stars_data: + sd = stars_data[repo_key] + entry["stars"] = sd["stars"] + entry["owner"] = sd["owner"] + entry["last_commit_at"] = sd.get("last_commit_at", "") + # Stdlib entries would hit same-named PyPI backports (e.g. asyncio), not the stdlib. + if entry.get("source_type") != BUILTIN_FILTER: + entry["downloads"] = downloads_data.get(normalize(entry["name"])) + entry["badge"] = pypi_badges.get(normalize(entry["name"])) + + entries = sort_entries(entries) + category_urls = {cat["name"]: category_path(cat) for cat in categories} + + filter_urls: dict[str, str] = dict(category_urls) + for group in parsed_groups: + filter_urls[group["name"]] = group_path(group["slug"]) + builtin_entries = [e for e in entries if e.get("source_type") == BUILTIN_FILTER] + + env = Environment( + loader=FileSystemLoader(website / "templates"), + autoescape=True, + trim_blocks=True, + lstrip_blocks=True, + ) + site_dir = website / "output" + if site_dir.exists(): + shutil.rmtree(site_dir) + site_dir.mkdir(parents=True) + + homepage_json_ld = json.dumps( + build_homepage_json_ld(entries, len(categories)), + ensure_ascii=False, + ).replace(" None: + page_dir.mkdir(parents=True, exist_ok=True) + parent_name = parent_category["name"] if parent_category else None + category_title = category_meta_title(category["name"], parent_name) + intro_html, guide_html, intro_lead, _ = load_category_intro(website / "data" / "category_intros" / f"{current_path.removeprefix('/categories/').strip('/')}.md") + if intro_lead: + category_description = intro_lead + elif parent_name: + category_description = subcategory_meta_description(category["name"], parent_name, [e["name"] for e in entries]) + else: + category_description = category_meta_description(category["name"], len(entries), category["description"]) + breadcrumbs = [("Awesome Python", SITE_URL)] + if parent_category: + breadcrumbs.append((parent_category["name"], category_public_url(parent_category))) + breadcrumbs.append((category["name"], category_url)) + page_entries = [entry for group in entry_groups for entry in group["entries"]] if entry_groups else entries + category_json_ld = json.dumps( + build_category_json_ld(category_title.removesuffix(" - Awesome Python"), category_url, category_description, page_entries, breadcrumbs), + ensure_ascii=False, + ).replace(" (cat_slug, sub_slug, sub_name) + cat_slug_by_url_prefix = {f"/categories/{c['slug']}/": c["slug"] for c in categories} + cat_by_slug = {c["slug"]: c for c in categories} + for entry in entries: + for sub in entry.get("subcategories", []): + value = sub["value"] + subcat_to_entries.setdefault(value, []).append(entry) + if value not in subcat_meta: + for prefix, cat_slug in cat_slug_by_url_prefix.items(): + if sub["url"].startswith(prefix): + subcat_meta[value] = (cat_slug, sub["slug"], sub["name"]) + break + + for value, (cat_slug, sub_slug, sub_name) in subcat_meta.items(): + render_category( + synthetic_category(sub_name, sub_slug), + category_url=subcategory_public_url(cat_slug, sub_slug), + entries=subcat_to_entries[value], + current_path=subcategory_path(cat_slug, sub_slug), + page_dir=categories_dir / cat_slug / sub_slug, + parent_category=cat_by_slug[cat_slug], + entry_groups=[EntryGroup(name="", slug="", url="", lead="", entries=group["entries"]) for group in section_groups[cat_by_slug[cat_slug]["name"]] if group["name"] == sub_name], + ) + + redirects_file = website / "data" / "redirects.json" + redirects = json.loads(redirects_file.read_text(encoding="utf-8")) if redirects_file.exists() else {} + for old_path, new_path in redirects.items(): + tpl_redirect = env.get_template("redirect.html") + stub = site_dir / old_path.strip("/") / "index.html" + if stub.exists(): + raise ValueError(f"redirect source {old_path} is a live page; remove it from redirects.json") + if not (site_dir / new_path.strip("/") / "index.html").exists(): + raise ValueError(f"redirect target {new_path} for {old_path} does not exist") + stub.parent.mkdir(parents=True, exist_ok=True) + stub.write_text(tpl_redirect.render(target_url=SITE_URL + new_path.lstrip("/")), encoding="utf-8") + + static_src = website / "static" + static_dst = site_dir / "static" + if static_src.exists(): + shutil.copytree(static_src, static_dst, dirs_exist_ok=True) + + llms_template = (website / "templates" / "llms.txt").read_text(encoding="utf-8") + llms_txt = build_llms_txt( + llms_template, + readme_text=readme_text, + subtitle=subtitle, + build_date=build_date.strftime("%Y/%m/%d"), + stars_data=stars_data, + downloads_data=downloads_data, + categories=categories, + total_entries=total_entries, + ) + (site_dir / "robots.txt").write_text(build_robots_txt(), encoding="utf-8") + # Daily star and download refreshes are not significant changes, so lastmod follows commits to the page's own content. + build_day = build_date.date().isoformat() + readme_date = git_last_change_date(repo_root, "--", "README.md") or build_day + section_dates = { + c["slug"]: max( + git_last_change_date(repo_root, "-L", section_line_range(readme_text, c["name"])), + git_last_change_date(repo_root, "--", f"website/data/category_intros/{c['slug']}.md"), + ) + or build_day + for c in categories + } + sitemap_urls = [(SITE_URL, readme_date)] + sitemap_urls.extend((category_public_url(c), section_dates[c["slug"]]) for c in categories) + sitemap_urls.extend((group_public_url(g["slug"]), max(section_dates[c["slug"]] for c in g["categories"])) for g in parsed_groups) + if builtin_entries: + sitemap_urls.append((BUILTIN_PUBLIC_URL, readme_date)) + for cat_slug, sub_slug, _ in sorted(subcat_meta.values()): + sitemap_urls.append((subcategory_public_url(cat_slug, sub_slug), section_dates[cat_slug])) + sitemap_urls.append((SPONSORSHIP_PUBLIC_URL, git_last_change_date(repo_root, "--", "SPONSORSHIP.md") or build_day)) + write_sitemap_xml(site_dir / "sitemap.xml", sitemap_urls) + (site_dir / "llms.txt").write_text(llms_txt, encoding="utf-8") + + print(f"Built site with {len(parsed_groups)} groups, {len(categories)} categories") + print(f"Total entries: {total_entries}") + print(f"Output: {site_dir}") + + +if __name__ == "__main__": + build(Path(__file__).parent.parent) diff --git a/website/data/category_intros/admin-panels.md b/website/data/category_intros/admin-panels.md new file mode 100644 index 00000000..66ff686f --- /dev/null +++ b/website/data/category_intros/admin-panels.md @@ -0,0 +1,18 @@ +Django saves you from building a Python admin panel: its admin comes built in. Flask-Admin and SQLAdmin add one to Flask and FastAPI apps. + +How to choose: + +- A Flask app: Flask-Admin +- SQLAlchemy models in FastAPI or Starlette: SQLAdmin +- A Django admin grown into dashboards and internal tools: Unfold +- A new look for the Django admin, on your stock ModelAdmin classes: Grappelli + +Flask-Admin [gives each model a `ModelView`](https://flask-admin.readthedocs.io/en/latest/introduction/#adding-model-views): one `admin.add_view(ModelView(User, db.session))` line gets you list, create, and edit pages for it. To fit a model, [subclass `ModelView`](https://flask-admin.readthedocs.io/en/latest/introduction/#customizing-built-in-views) and set class attributes like `can_delete` or `column_list`. Expect fewer defaults than in Django's admin: [it's up to you to tell Flask-Admin](https://flask-admin.readthedocs.io/en/latest/advanced/#design-philosophy) what to display and how. Besides SQLAlchemy, it has [backends](https://flask-admin.readthedocs.io/en/latest/advanced/#using-different-database-backends) for Peewee, pymongo, and MongoEngine. + +SQLAdmin brings Flask-Admin's approach to SQLAlchemy models on FastAPI or Starlette: Flask-Admin [inspired most of its features and configuration](https://smithyhq.github.io/sqladmin/#related-projects-and-inspirations). Create an `Admin` from your app and engine, then [write one `ModelView` subclass per model](https://smithyhq.github.io/sqladmin/#quickstart), declared with `model=User`, and register it with `admin.add_view`. + +Unfold is a theme for Django's own admin, [made for building dashboards, internal tools, and business applications](https://github.com/unfoldadmin/django-unfold). Put `"unfold"` [before `django.contrib.admin`](https://unfoldadmin.com/docs/installation/quickstart/) in `INSTALLED_APPS`, then make your admin classes inherit from `unfold.admin.ModelAdmin`: the stock `ModelAdmin` leaves forms unstyled. + +Grappelli is [a jazzy skin for the Django admin](https://django-grappelli.readthedocs.io/en/latest/) that also adds collapsibles, sortable inlines, and related and autocomplete lookups. Put `'grappelli'` [before `django.contrib.admin`](https://django-grappelli.readthedocs.io/en/latest/quickstart.html) in `INSTALLED_APPS`, and include `grappelli.urls`, which the lookups need. You switch features on with [attributes on your `admin.ModelAdmin` classes](https://django-grappelli.readthedocs.io/en/latest/customization.html#autocomplete-lookups), like `autocomplete_lookup_fields`. + +By default, Django's admin [lets in only users with `is_staff` set](https://docs.djangoproject.com/en/stable/ref/contrib/admin/#overview). Flask-Admin and SQLAdmin leave access control to you: in Flask-Admin, [override `is_accessible`](https://flask-admin.readthedocs.io/en/latest/introduction/#rolling-your-own) on your views, and SQLAdmin [enforces no authentication](https://smithyhq.github.io/sqladmin/authentication/) until you pass it an `AuthenticationBackend`. Django's docs limit its admin to [an internal management tool](https://docs.djangoproject.com/en/stable/ref/contrib/admin/), and once you need a process-centric interface, they say to write your own views. diff --git a/website/data/category_intros/ai-and-agents.md b/website/data/category_intros/ai-and-agents.md new file mode 100644 index 00000000..6c72c3f5 --- /dev/null +++ b/website/data/category_intros/ai-and-agents.md @@ -0,0 +1,55 @@ +Build a typed agent in Pydantic AI. Long-running, stateful agents suit LangGraph. Apps tied to one model vendor can use that vendor's Python AI agent library. + +How to choose: + +- Typed agents: Pydantic AI, or LangGraph for long-running, stateful ones +- Agents on Claude, OpenAI, or Gemini models: Claude Agent SDK, OpenAI Agents SDK, or Google ADK +- Skills for your coding agent: Django AI Skills, Sentry Skills, or Trail of Bits Skills +- MCP servers and clients: MCP Python SDK, or FastMCP to compose and proxy servers +- A personal assistant in your chat apps: Hermes Agent, or AstrBot for QQ, Feishu, and DingTalk +- Prompts tuned against a metric: DSPy +- Agent data: Instructor for structured output, LlamaIndex for RAG, Mem0 for memory +- Open models from the Hugging Face Hub: Transformers +- Serving open models: vLLM or SGLang, or MLX LM on Apple silicon +- One OpenAI-style API for many model providers: LiteLLM +- Image and video generation: Diffusers +- Fine-tuning: TRL with PEFT adapters, Unsloth for less GPU memory, Axolotl from YAML +- Speech: faster-whisper, FunASR for Chinese, gTTS for text to speech + +Pydantic AI comes from the Pydantic team and is [typed end to end](https://pydantic.dev/docs/ai/overview/). Outputs, tools, and dependencies all carry types, so your type checker knows what an agent returns. Switching models takes one string. [Give an agent an output type and tools](https://github.com/pydantic/pydantic-ai), and every run comes back validated. + +LangGraph is a [low-level orchestration framework](https://docs.langchain.com/oss/python/langgraph/overview) for long-running, stateful agents, and it can mix hand-coded steps with LLM-driven ones in one graph. For your first agent, its docs suggest LangChain's agents instead: [a minimal harness](https://docs.langchain.com/oss/python/langchain/overview) built on LangGraph, which you extend through middleware. + +The Claude Agent SDK gives you [the tools and agent loop behind Claude Code](https://code.claude.com/docs/en/agent-sdk/overview), so it fits agents that read files, run commands, and edit code. Call `query()` for a one-off task, and switch to [`ClaudeSDKClient`](https://code.claude.com/docs/en/agent-sdk/python#choosing-between-query-and-claudesdkclient) when the next step depends on Claude's reply, as in a chat. + +The OpenAI Agents SDK keeps to [a small set of primitives](https://openai.github.io/openai-agents-python/): use it when you want the runtime to handle turns, tool calls, handoffs, and guardrails for you. Despite the name, it [isn't limited to OpenAI models](https://github.com/openai/openai-agents-python). + +Google ADK is [optimized for Gemini but model-agnostic](https://github.com/google/adk-python). Start a project with [`adk create`](https://adk.dev/get-started/python/). + +All three skill collections install from a Claude Code plugin marketplace. Django AI Skills and Sentry Skills follow the open [Agent Skills](https://agentskills.io/) format, so other coding agents can load them too. Sentry Skills is [written for Sentry's own engineers](https://github.com/getsentry/skills), so some of its skills follow Sentry's internal standards. + +CrewAI splits an app into Flows, which manage state and control execution, and Crews, teams of agents that work on one task together. For a production app, [start with a Flow](https://docs.crewai.com/en/introduction), and hand a Crew only the steps that need autonomy. + +The MCP Python SDK reads each tool's schema [from its type hints and docstring](https://py.sdk.modelcontextprotocol.io/get-started/first-steps/), so you write no JSON Schema or request handlers. FastMCP wrote the high-level API that the SDK [took in](https://gofastmcp.com/getting-started/welcome). The standalone project is a fuller framework: it can [mount several servers into one](https://gofastmcp.com/servers/composition) and [proxy another MCP server](https://gofastmcp.com/servers/providers/proxy) through your own. + +Hermes Agent talks to you on Telegram, Discord, Slack, WhatsApp, and Signal [from one gateway process](https://github.com/NousResearch/hermes-agent), and writes itself new skills as it works. Since it runs commands for whoever it lets in, allow only your own account on its gateway with an [allowlist](https://hermes-agent.nousresearch.com/docs/user-guide/security#user-authorization-gateway), and leave approval prompts on for dangerous commands. AstrBot covers chat apps such as QQ, Feishu, and DingTalk. It's AGPL, while Hermes Agent is MIT. + +DSPy has you write [structured signatures, not prompts](https://dspy.ai/current/), and its optimizers tune the prompts for you. Before they can, [define a metric and score a baseline](https://dspy.ai/current/getting-started/metrics/) to beat. + +Instructor [turns an LLM's reply into a validated Pydantic model](https://github.com/567-labs/instructor), and when validation fails, it retries with the error message. Its docs draw the line at [extraction versus agents](https://python.useinstructor.com/): Instructor for the first, Pydantic AI for the second. + +LlamaIndex builds [agents over your own data](https://developers.llamaindex.ai/python/framework/). Its high-level API goes from documents to answers in a few lines, and its lower-level API lets you replace any part. + +Mem0 gives an agent [memory of each user](https://github.com/mem0ai/mem0): search their memories before the model answers, then add the new exchange. Run it [as a library in your app](https://docs.mem0.ai/open-source/overview), or as a self-hosted server for a team. + +Transformers is the [model definition that most training frameworks and inference engines share](https://huggingface.co/docs/transformers/index), Axolotl, Unsloth, vLLM, and SGLang among them. Load a model from the Hugging Face Hub with [`from_pretrained()`](https://huggingface.co/docs/transformers/quicktour), then run it through `Pipeline` for inference or `Trainer` for training. For large production deployments, its docs [send you to vLLM or SGLang](https://huggingface.co/docs/transformers/serve-cli/serving). + +vLLM runs [batch inference offline or an OpenAI-compatible server](https://docs.vllm.ai/en/latest/getting_started/quickstart/), so apps written for the OpenAI API work unchanged. SGLang is [optimized for agentic workloads, RL rollouts, and large-scale serving](https://github.com/sgl-project/sglang). On a Mac, MLX LM [runs and fine-tunes models on Apple silicon](https://github.com/ml-explore/mlx-lm). + +LiteLLM [returns every provider's responses in the OpenAI format](https://docs.litellm.ai/docs/) and maps their errors to OpenAI's exception types, so one code path covers them all. Use its Python SDK inside your app, or run its proxy as a self-hosted gateway for your team. + +Diffusers centers on the [`DiffusionPipeline`](https://huggingface.co/docs/diffusers/index): a few lines generate an image, video, or audio clip. + +Start fine-tuning with LoRA or QLoRA rather than training every weight, as both [PEFT](https://huggingface.co/docs/peft/methods/overview) and [Unsloth](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide) advise. TRL trains with methods from SFT to DPO and GRPO, and every TRL trainer takes a [`peft_config`](https://huggingface.co/docs/trl/peft_integration) for PEFT adapters. Axolotl keeps the whole pipeline, from dataset preprocessing to inference, in [one YAML file](https://github.com/axolotl-ai-cloud/axolotl). + +faster-whisper reimplements Whisper on CTranslate2 for [the same accuracy in less time and memory](https://github.com/SYSTRAN/faster-whisper). For Chinese audio, FunASR's `AutoModel` [chains speech recognition with voice activity detection, punctuation, and speaker models](https://github.com/modelscope/FunASR) in one call. gTTS speaks through [Google Translate's undocumented speech feature](https://github.com/pndurette/gTTS), so upstream changes can break it without notice. diff --git a/website/data/category_intros/algorithms-and-design-patterns.md b/website/data/category_intros/algorithms-and-design-patterns.md new file mode 100644 index 00000000..ff8234b1 --- /dev/null +++ b/website/data/category_intros/algorithms-and-design-patterns.md @@ -0,0 +1,21 @@ +If a Python algorithms library has to run in production, Sorted Containers gives you sorted lists, dicts, and sets. To learn, read The Algorithms - Python. + +How to choose: + +- Lists, dicts, and sets that stay sorted as you add and remove: Sorted Containers +- Small, typed implementations you can install and read: algorithms +- Algorithm code to study, browsed by topic: The Algorithms - Python +- How classic design patterns look in Python, and which ones to skip: python-patterns +- Workflows as states and transitions, up to full statecharts: python-statemachine + +Python's standard library [is great until you need a sorted collections type](https://grantjenks.com/docs/sortedcontainers/), and Sorted Containers fills that gap in pure Python, as fast as C extensions. SortedList keeps its values in order as you `add()` them, SortedDict is [a dict that keeps a sorted list of its keys](https://grantjenks.com/docs/sortedcontainers/introduction.html#sorted-dict), and SortedSet does the same for a set. + +algorithms collects [minimal, clean, and well-documented implementations](https://github.com/keon/algorithms) of data structures and algorithms. Each file stands alone, with docstrings, type hints, and complexity notes, and its README says they're designed to be read and learned from. Install it with `pip install algorithms`, then [import an implementation by topic](https://github.com/keon/algorithms#quick-start), like a sort or a graph search. + +The Algorithms - Python implements algorithms [for education](https://github.com/TheAlgorithms/Python), and its README says they may be less efficient than the standard library's own: read them to learn, not to ship. Start from [its directory](https://github.com/TheAlgorithms/Python/blob/master/DIRECTORY.md), which sorts them by topic. Its contributing guide asks each algorithm to [carry type hints and doctests](https://github.com/TheAlgorithms/Python/blob/master/CONTRIBUTING.md#what-is-an-algorithm) that test both valid and erroneous input, so the doctests show you how it behaves. + +python-patterns is [a collection of design patterns and idioms](https://github.com/faif/python-patterns) in Python, one file per pattern, grouped as creational, structural, behavioral, and more. Its README asks you to pay more attention to why you choose a pattern than to how you implement it. It also lists the patterns [not recommended in Python](https://github.com/faif/python-patterns#-anti-patterns): skip a Singleton class, since modules are already singletons, and prefer composition and delegation over deep inheritance. + +python-statemachine builds [flat state machines or full statecharts](https://python-statemachine.readthedocs.io/en/latest/) with compound states, parallel regions, and history, in sync and async code. Declare states as class attributes and transitions with `state.to(target)`, and the [events are the names you assign them to](https://python-statemachine.readthedocs.io/en/latest/tutorial.html#your-first-state-machine). + +Before you hand-write python-patterns' State pattern for a real workflow, read python-statemachine's guide [for coming from the State pattern](https://python-statemachine.readthedocs.io/en/latest/how-to/coming_from_state_pattern.html). The whole workflow sits in one class body, and unreachable states fail when the class is defined, not at runtime. For more places to learn and practice algorithms, see [awesome-algorithms](https://github.com/tayllan/awesome-algorithms). diff --git a/website/data/category_intros/asynchronous-programming.md b/website/data/category_intros/asynchronous-programming.md new file mode 100644 index 00000000..f71a36b9 --- /dev/null +++ b/website/data/category_intros/asynchronous-programming.md @@ -0,0 +1,30 @@ +I/O-bound or CPU-bound? I/O-bound code wants a Python async library, and asyncio comes built in. CPU-bound work goes to a concurrent.futures process pool. + +How to choose: + +- I/O-bound code written with async/await: asyncio +- CPU-bound work, or blocking calls, in a pool of processes or threads: concurrent.futures +- Trio-style task groups and cancel scopes on asyncio, or a library that runs on both asyncio and Trio: AnyIO +- A faster, drop-in event loop for an asyncio app on Linux or macOS: uvloop +- Structured concurrency from the ground up, with its own libraries: Trio +- Existing synchronous code, made concurrent without async/await: gevent +- Network servers and clients with protocols built in, like SSH, mail, and DNS: Twisted +- Processes you manage yourself, talking through queues and pipes: multiprocessing + +asyncio is the standard library's way to [write concurrent code with async/await](https://docs.python.org/3/library/asyncio.html), and many async web servers, database drivers, and task queues build on it. Start your program with `asyncio.run()`, [called once as the main entry point](https://docs.python.org/3/library/asyncio-runner.html#asyncio.run). App code should [rarely need the event loop object](https://docs.python.org/3/library/asyncio-eventloop.html) itself. Run related tasks in an `asyncio.TaskGroup`: when one task fails, it [cancels the rest, which `gather()` doesn't](https://docs.python.org/3/library/asyncio-task.html#asyncio.gather). CPU-heavy code holds up every task on the loop, so [run it in another process](https://docs.python.org/3/library/asyncio-dev.html#asyncio-handle-blocking): hand it to a `ProcessPoolExecutor` with `loop.run_in_executor()`. + +concurrent.futures runs callables on threads or processes behind [the same interface](https://docs.python.org/3/library/concurrent.futures.html). `ThreadPoolExecutor` is [for overlapping I/O](https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor). For CPU-bound work on a multi-core machine, the threading docs [advise processes](https://docs.python.org/3/library/threading.html) instead, and `ProcessPoolExecutor` runs them for you. It [takes only picklable functions and arguments](https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor), though. Use either executor [in a `with` block](https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Executor.shutdown), which shuts it down and waits for its work to finish. + +AnyIO brings [Trio-like structured concurrency to asyncio](https://anyio.readthedocs.io/en/stable/). Code written against its API runs unmodified on asyncio or Trio, so a library built on it doesn't choose for its users. Its docs see [strong merits in its APIs for applications too](https://anyio.readthedocs.io/en/stable/why.html), starting with cancel scopes for [more predictable cancellation](https://anyio.readthedocs.io/en/stable/why.html#design-problems-with-cancellation). Start with `anyio.run(main)`, which [runs on asyncio unless you pass `backend="trio"`](https://anyio.readthedocs.io/en/stable/basics.html#running-async-programs). Spawn tasks in a [task group](https://anyio.readthedocs.io/en/stable/tasks.html): when one child task raises, the rest are cancelled. + +uvloop is [a drop-in replacement for asyncio's event loop](https://github.com/MagicStack/uvloop), built on libuv. Its README prefers `uvloop.run(main())`, which configures `asyncio.run()` to use uvloop, so the rest of your asyncio code stays the same. uvloop runs on Linux and macOS. + +Trio has [an obsessive focus on usability and correctness](https://trio.readthedocs.io/en/stable/). Child tasks run in a nursery, opened with `async with trio.open_nursery()`, and [Trio never discards their exceptions](https://trio.readthedocs.io/en/stable/tutorial.html#okay-let-s-see-something-cool-already). Functions [take no timeout arguments](https://trio.readthedocs.io/en/stable/reference-core.html#blocking-and-non-blocking-methods): you wrap the code in a cancel scope like `trio.move_on_after()`. Trio runs its own event loop, so asyncio functions [don't work inside `trio.run()`](https://trio.readthedocs.io/en/stable/tutorial.html#task-switching-illustrated). Check [the Trio library list](https://trio.readthedocs.io/en/stable/awesome-trio-libraries.html) for what you need first. + +gevent uses greenlets to give you [a synchronous API on top of an event loop](https://www.gevent.org/intro.html). Its monkey patching swaps the standard library's blocking sockets for cooperative ones, so code that knows nothing about gevent runs concurrently. Most programs should [patch everything with `monkey.patch_all()`](https://www.gevent.org/intro.html#beyond-sockets), and the main module should do it [before any other imports](https://www.gevent.org/api/gevent.monkey.html). + +Twisted is [an event-based framework for internet applications](https://github.com/twisted/twisted) that ships clients and servers for HTTP, SSH, IMAP, POP3, SMTP, DNS, IRC, and XMPP. Write new Twisted code as `async def` coroutines, which its docs [prefer over `inlineCallbacks`](https://docs.twisted.org/en/stable/core/howto/defer-intro.html#inline-callbacks-using-yield), and start one with [`Deferred.fromCoroutine()`](https://docs.twisted.org/en/stable/core/howto/defer-intro.html#coroutines-with-async-await). + +multiprocessing runs work in subprocesses, so it can [use every processor on a machine](https://docs.python.org/3/library/multiprocessing.html#introduction). Its own docs point to `ProcessPoolExecutor` as the higher-level interface for pooled tasks. Reach for multiprocessing when you need what it adds, like killing a running process or passing data through queues and pipes. Its guidelines say to [avoid shared state](https://docs.python.org/3/library/multiprocessing.html#all-start-methods) and keep the data moving between processes small. + +On an event loop, a blocking call holds up every other task, so push it to a worker thread: asyncio has [`asyncio.to_thread()`](https://docs.python.org/3/library/asyncio-task.html#asyncio.to_thread), AnyIO [`to_thread.run_sync()`](https://anyio.readthedocs.io/en/stable/threads.html), and Trio [`trio.to_thread.run_sync()`](https://trio.readthedocs.io/en/stable/reference-core.html#threads-if-you-must). When multiprocessing or `ProcessPoolExecutor` runs your code in child processes, define its functions in a module. Guard your entry point with `if __name__ == '__main__':` too, so each new process can [import your main module safely](https://docs.python.org/3/library/multiprocessing.html#multiprocessing-safe-main-import). diff --git a/website/data/category_intros/audio-video-processing.md b/website/data/category_intros/audio-video-processing.md new file mode 100644 index 00000000..feac35ff --- /dev/null +++ b/website/data/category_intros/audio-video-processing.md @@ -0,0 +1,28 @@ +To track tempo and beats with a Python audio processing library, load music into librosa. MoviePy cuts and joins video in code. Mutagen reads and writes tags. + +How to choose: + +- Tempo, beats, and other music analysis: librosa +- Scripted video edits, titles, and compositing: MoviePy +- Reading and writing tags: Mutagen +- Reading and writing sound files as NumPy arrays: soundfile +- Slicing, fading, and joining audio clips: pydub +- Frame- and packet-level access to FFmpeg: PyAV +- Reading tags only, without GPL code: tinytag +- Organizing a music collection with MusicBrainz data: beets + +librosa is a Python library for [audio and music signal processing](https://librosa.org/doc/latest/index.html), with the building blocks for music information retrieval: spectrograms, chroma, onsets, tempo, and beats. Load each file with [`librosa.load()`](https://librosa.org/doc/latest/auto_tutorials/01-intro/01-load.html), which resamples it to a standard rate and mixes it down to mono, the defaults the rest of librosa is designed around. + +soundfile reads and writes [any format libsndfile supports](https://python-soundfile.readthedocs.io/en/latest/#read-write-functions), like WAV, FLAC, and OGG: `sf.read()` returns the samples as a NumPy array along with the sample rate, and `sf.write()` saves them. librosa reads files through soundfile, and its docs [recommend using soundfile directly](https://librosa.org/doc/latest/ioformats.html) to write audio, or when you need more control than loading a whole recording into memory. + +MoviePy is for [automating video edits](https://zulko.github.io/moviepy/getting_started/quick_presentation.html#do-i-need-moviepy): cutting scenes, adding titles and subtitles, or composing many videos into one. A [typical script](https://zulko.github.io/moviepy/getting_started/quick_presentation.html#example-code) loads videos as clips, modifies them, puts them together in a `CompositeVideoClip`, and writes the result with `write_videofile()`. + +PyAV binds FFmpeg's libraries for [direct and precise access](https://pyav.basswood.io/docs/stable/) to your media's containers, streams, packets, codecs, and frames. Open a file with `av.open()` and loop over `container.decode(video=0)` to get frames, which [convert to NumPy arrays](https://pyav.basswood.io/docs/stable/cookbook/numpy.html) with `to_ndarray()` and back with `av.VideoFrame.from_ndarray()`. + +Mutagen reads and writes tags in many audio formats through [one API](https://mutagen.readthedocs.io/en/latest/#why-mutagen) that's roughly the same across them. `mutagen.File()` [guesses the file's type](https://mutagen.readthedocs.io/en/latest/user/gettingstarted.html), you set tags like dict keys, and `save()` writes them. Mutagen is GPL. + +tinytag, which is MIT and pure Python, only reads: `TinyTag.get(path)` returns the [title, artist, album, duration, and more](https://github.com/tinytag/tinytag) as attributes. + +beets runs from the command line: it [catalogs your music collection](https://beets.io/) and improves its metadata from MusicBrainz as it goes. Point [`beet import`](https://beets.readthedocs.io/en/stable/guides/tagger.html) at a folder of albums, and it tags the files as it adds them to your library. The docs [recommend this autotagged import](https://beets.readthedocs.io/en/stable/guides/main.html#importing-your-music): it asks you questions along the way, but gets every song's tags right from the start. + +MoviePy and PyAV both run on FFmpeg. When the `ffmpeg` command [does the job](https://pyav.basswood.io/docs/stable/) on its own, call it directly: MoviePy's docs say that's [faster and more memory-efficient](https://zulko.github.io/moviepy/getting_started/quick_presentation.html#do-i-need-moviepy). diff --git a/website/data/category_intros/authentication.md b/website/data/category_intros/authentication.md new file mode 100644 index 00000000..a02aef1a --- /dev/null +++ b/website/data/category_intros/authentication.md @@ -0,0 +1,27 @@ +Letting users log in with Google takes one Python authentication library: django-allauth on Django, which does password sign-up too, or Authlib elsewhere. + +How to choose: + +- Sign-up, password login, and Google or GitHub login on a Django site: django-allauth +- Google or GitHub login, OAuth API calls, or an OAuth server outside Django: Authlib +- Adding OAuth to a web framework or HTTP library you maintain: oauthlib +- Your own OAuth 2.0 server on Django, or tokens for a Django REST framework API: django-oauth-toolkit +- Signing and verifying JSON Web Tokens: PyJWT +- Permissions you grant per object to users and groups: django-guardian +- Permissions that follow from the object's data, with no extra tables: django-rules + +django-allauth exists to handle [local and social accounts in one app](https://docs.allauth.org/en/latest/introduction/index.html): a user can sign up with a password or log in with Google. Follow the [quickstart](https://docs.allauth.org/en/latest/installation/quickstart.html): add its authentication backend, apps, and middleware, then include `allauth.urls` under `accounts/`. Its login, logout, and password views can replace the ones in `django.contrib.auth.urls`. Give each OAuth provider its client ID and secret in a [`SocialApp` or the `SOCIALACCOUNT_PROVIDERS` setting](https://docs.allauth.org/en/latest/installation/quickstart.html#post-installation). For a single-page or mobile app, add [`allauth.headless`](https://docs.allauth.org/en/latest/headless/introduction.html). + +Authlib covers both sides of OAuth 2.0 and OpenID Connect. As a client, it comes in [two kinds](https://docs.authlib.org/en/latest/oauth2/client/index.html): HTTP clients for scripts and service-to-service calls, and web clients that log users in through Flask, Django, Starlette, or FastAPI. For login, [create an `OAuth` registry and `register()` each provider](https://docs.authlib.org/en/latest/oauth2/client/web/index.html). For an OpenID Connect provider like Google, pass its discovery URL as [`server_metadata_url`](https://docs.authlib.org/en/latest/oauth2/client/web/index.html#parsing-id-token), and Authlib reads the other endpoints from it. To [run your own OAuth 2.0 or OpenID Connect provider](https://docs.authlib.org/en/latest/oauth2/authorization-server/index.html), it has server integrations for Flask and Django. + +PyJWT [encodes and decodes JSON Web Tokens](https://pyjwt.readthedocs.io/en/latest/). Treat every token as untrusted input: [hard-code the `algorithms` you pass to `jwt.decode()`](https://pyjwt.readthedocs.io/en/latest/algorithms.html#specifying-an-algorithm), never read them from the token's own header, and don't mix HS and RS algorithms. For tokens from an OpenID Connect provider, point [`PyJWKClient` at its JWKS endpoint](https://pyjwt.readthedocs.io/en/latest/usage.html#retrieve-rsa-signing-keys-from-a-jwks-endpoint) instead of hard-coding its public keys. + +oauthlib implements OAuth's logic [without assuming a web framework or HTTP request object](https://github.com/oauthlib/oauthlib), so a framework or library maintainer can write a thin layer on top and get OAuth support. Its FAQ says [most people use it indirectly](https://oauthlib.readthedocs.io/en/latest/faq.html#how-do-i-use-oauthlib-with-google-twitter-and-other-providers): django-oauth-toolkit and django-allauth both build on it. To build a provider on it, most of the work is [a `RequestValidator`](https://oauthlib.readthedocs.io/en/latest/oauth2/server.html#implement-a-validator) that maps its checks to your storage. + +django-oauth-toolkit is an [OAuth 2.0 authorization server for Django](https://django-oauth-toolkit.readthedocs.io/en/latest/): it issues and manages tokens from your existing project, and can also protect a Django or Django REST framework API. Django REST framework's docs [recommend it for OAuth 2.0](https://www.django-rest-framework.org/api-guide/authentication/#django-oauth-toolkit). [Install it](https://django-oauth-toolkit.readthedocs.io/en/latest/install.html) by adding `oauth2_provider` to `INSTALLED_APPS`, including its URLs under `o/`, and migrating. For an API, [set `OAuth2Authentication`](https://django-oauth-toolkit.readthedocs.io/en/latest/rest-framework/getting_started.html) as Django REST framework's authentication class and guard views with `TokenHasScope`. + +Django has [a foundation for object permissions but no implementation](https://docs.djangoproject.com/en/stable/topics/auth/customizing/#handling-object-permissions), and django-guardian fills it with an [extra authentication backend](https://django-guardian.readthedocs.io/en/latest/configuration/). Grant a permission on one object with `assign_perm()`, then check it with Django's own [`user.has_perm(perm, obj)`](https://django-guardian.readthedocs.io/en/latest/userguide/checks/). Django REST framework's `DjangoObjectPermissions` [names it as an example backend](https://www.django-rest-framework.org/api-guide/permissions/#djangoobjectpermissions). + +django-rules gives Django object permissions [without a database](https://github.com/dfunckt/django-rules). A permission is a rule built from predicates, plain functions like `is_book_author(user, book)` that you combine with `|`, `&`, and `~`. Add its [backend to `AUTHENTICATION_BACKENDS`](https://github.com/dfunckt/django-rules#configuring-django), register each permission with `rules.add_perm()`, and check it with `user.has_perm()`. Keep predicates and rules in their own modules, and let [`AutodiscoverRulesConfig` import each app's `rules.py`](https://github.com/dfunckt/django-rules#best-practices) at startup. + +On either side of OAuth, use PKCE. django-oauth-toolkit's security guide maps [the OAuth security recommendations of RFC 9700](https://django-oauth-toolkit.readthedocs.io/en/latest/security.html) to its settings: PKCE is one, and the implicit and password grants must not be used. Authlib's client [turns PKCE on with `code_challenge_method`](https://docs.authlib.org/en/latest/oauth2/client/web/index.html#oauth-2-0-code-challenge). Also, don't pick JWTs for sessions just because they sound stateless. django-allauth's docs point out that [if a token must stop working at logout, you need state to revoke it](https://docs.allauth.org/en/latest/headless/token-strategies/jwt-tokens.html). JWTs help most when each service checks tokens without asking a central server. diff --git a/website/data/category_intros/build-tools.md b/website/data/category_intros/build-tools.md new file mode 100644 index 00000000..0e50a00b --- /dev/null +++ b/website/data/category_intros/build-tools.md @@ -0,0 +1,16 @@ +Tasks like running your tests go in pyproject.toml for Poe the Poet to run. C and C++ projects that replace Make with a Python build tool pick SCons. + +How to choose: + +- Project tasks in pyproject.toml, run in your uv or Poetry environment: Poe the Poet +- Compiling C or C++ in place of Make: SCons +- Tasks written as Python functions that run shell commands: Invoke +- Steps that skip when their input files haven't changed: doit + +Poe the Poet reads tasks from a [`[tool.poe.tasks]` table](https://poethepoet.natn.io/tasks/index.html) in pyproject.toml. A string like `test = "pytest"` is a whole task, run as a command without a shell. In a uv or Poetry project, poe [runs tasks in the project's virtualenv](https://poethepoet.natn.io/#quick-start), so you don't type `uv run` or `poetry run` in front. + +Invoke tasks are Python functions in a [`tasks.py` file](https://docs.pyinvoke.org/en/latest/getting-started.html#defining-and-running-task-functions), decorated with `@task` and taking a context as their first argument. Inside, `c.run()` runs a shell command. A function's arguments [become CLI flags](https://docs.pyinvoke.org/en/latest/getting-started.html#task-parameters), and a task can name [pre-tasks](https://docs.pyinvoke.org/en/latest/getting-started.html#declaring-pre-tasks) to always run before it. + +doit reads tasks from a [`dodo.py` file](https://pydoit.org/tutorial-1.html), where each `task_` function returns a dict of actions, file dependencies, and targets instead of running anything. It saves a [signature of each `file_dep`](https://pydoit.org/tasks.html#file-dep) and skips a task whose inputs haven't changed. A task that [declares no input](https://pydoit.org/dependencies.html#doit-up-to-date-definition) never counts as up to date, so list the files each task reads in `file_dep`. + +SCons reads your build from an [`SConstruct` file, which is a Python script](https://scons.org/doc/production/HTML/scons-user.html#sect-sconstruct-python): `Program('hello.c')` there is enough for `scons` to compile a program. Calling a builder like `Program()` [declares what you want built](https://scons.org/doc/production/HTML/scons-user.html#sect-order-independent) instead of building it on that line, and SCons works out when to run each step. diff --git a/website/data/category_intros/built-in-classes-enhancement.md b/website/data/category_intros/built-in-classes-enhancement.md new file mode 100644 index 00000000..dfe74eed --- /dev/null +++ b/website/data/category_intros/built-in-classes-enhancement.md @@ -0,0 +1,16 @@ +Skip hand-written dunder methods, and when you compare Python dataclass alternatives, pick attrs for its validators and converters. Box gives dicts dot access. + +How to choose: + +- Classes with generated dunder methods, validators, and converters: attrs +- Dicts read with dot notation, nested ones included: python-box +- Looking up a key by its value, with both directions in sync: bidict +- A faster drop-in for the `uuid` module: uuid-utils + +attrs [writes the dunder methods](https://www.attrs.org/en/latest/) that implement object protocols, so you don't have to. For new code, its docs [recommend the modern API](https://www.attrs.org/en/latest/names.html#tl-dr): `@define` on the class and `field()` for an attribute, or `@frozen` for immutable instances, with slots on by default. Type annotations stay optional. The standard library's dataclasses [gave up features for simplicity](https://www.attrs.org/en/latest/why.html#data-classes), like validators and converters, so a class that needs them goes to attrs. It [isn't a full serialization library](https://www.attrs.org/en/latest/overview.html#what-attrs-is-not), though: to serialize and validate your attrs classes, its docs point you to its sibling project, cattrs. + +python-box installs Box, a [`dict` subclass](https://github.com/cdgriffith/Box) whose keys you can also read as attributes, even ones like `"imdb stars"`. Nested dicts and lists become Box and BoxList objects, so dot access works all the way down: `movie_box.Robin_Hood_Men_in_Tights.imdb_stars`. It [reads and writes JSON, YAML, TOML, and msgpack](https://github.com/cdgriffith/Box/wiki/Converters) with methods like `from_json()` and `to_json()`, and `to_dict()` gives you plain dicts back. Keep Box for data whose keys change: attrs' docs say a dict with a fixed and known set of keys [is an object, not a hash](https://www.attrs.org/en/latest/why.html#dicts), and belongs in a class. + +bidict gives you [bidirectional mappings](https://bidict.readthedocs.io/intro.html) that work like dicts: look up a value by its key as usual, or a key by its value through `.inverse`, which stays in sync as you update the mapping. A single dict holding both directions mixes keys with values. Modeling the mapping correctly takes two one-way mappings kept in sync, [which is what bidict does](https://bidict.readthedocs.io/intro.html#why-can-t-i-just-use-a-dict) under the hood. + +uuid-utils is a [fast, drop-in replacement for Python's `uuid` module](https://aminalaee.github.io/uuid-utils/latest/), powered by Rust. Import it under the module's name, `import uuid_utils as uuid`, and call `uuid.uuid4()` or `uuid.uuid7()` as usual. Django and some other frameworks require the standard library's own `uuid.UUID` instances. For those, [import `uuid_utils.compat`](https://aminalaee.github.io/uuid-utils/latest/#compatibility-with-python-uuid) instead, which returns them and still outperforms the standard library. diff --git a/website/data/category_intros/caching.md b/website/data/category_intros/caching.md new file mode 100644 index 00000000..5cc943f9 --- /dev/null +++ b/website/data/category_intros/caching.md @@ -0,0 +1,23 @@ +In memory, on disk, or on a cache server: the Python caching library you want is cachetools, DiskCache, or dogpile.cache. For HTTP responses, it's Hishel. + +How to choose: + +- Function results in one process's memory, with a size limit or a time-to-live: cachetools +- A cache on local disk that processes on one machine share, with no server to run: DiskCache +- A Django cache backend on local disk: DiskCache +- One cache on memcached or Redis for many processes or servers: dogpile.cache +- HTTP responses in HTTPX or Requests: Hishel +- `Cache-Control` headers and response caching in FastAPI or another ASGI app: Hishel +- Django querysets, invalidated when a model changes: Cacheops + +cachetools offers [variants of the standard library's `@lru_cache`](https://cachetools.readthedocs.io/en/stable/) with more cache algorithms, including a `TTLCache` whose items [expire after a time-to-live](https://cachetools.readthedocs.io/en/stable/#cachetools.TTLCache). Wrap a function in [`@cached`](https://cachetools.readthedocs.io/en/stable/#cachetools.cached) and pass it the cache to use. The cache classes [aren't thread-safe](https://cachetools.readthedocs.io/en/stable/#cache-implementations), so when threads share a cache, give `@cached` a `threading.Lock`. The cache lives in your process's memory, so each worker process of a web app keeps its own copy. + +DiskCache is a [disk and file backed cache library](https://grantjenks.com/docs/diskcache/) in pure Python, built on SQLite, with no other process to run. Create a `Cache` with a directory path: [two `Cache` objects on the same directory](https://grantjenks.com/docs/diskcache/tutorial.html#cache) can live in separate processes, so the workers on one machine share one cache. Wrap a function in [`@cache.memoize()`](https://grantjenks.com/docs/diskcache/tutorial.html#fanoutcache), which takes arguments like `lru_cache`'s. In Django, set [`diskcache.DjangoCache`](https://grantjenks.com/docs/diskcache/tutorial.html#djangocache) as the cache `BACKEND`. Keep the directory on a local disk, since SQLite [isn't recommended on NFS mounts](https://grantjenks.com/docs/diskcache/tutorial.html#caveats). + +dogpile.cache is [a caching API over backends of any variety](https://dogpilecache.sqlalchemy.org/en/latest/), memcached and Redis among them. You ask it for a value and hand it a function that [creates the value only when needed](https://dogpilecache.sqlalchemy.org/en/latest/usage.html#overview). When the value expires, one worker regenerates it instead of every worker at once. Create a region with `make_region()` at import time, decorate functions with `@region.cache_on_arguments()`, and [call `configure()` later](https://dogpilecache.sqlalchemy.org/en/latest/usage.html#rudimentary-usage) with the backend and expiration time from your config file. When several processes share one Redis or memcached server, turn on the backend's [`distributed_lock`](https://dogpilecache.sqlalchemy.org/en/latest/api.html#dogpile.cache.backends.redis.RedisBackend) so that lock covers them all. + +Hishel caches HTTP responses by [RFC 9111](https://hishel.com/overview.html), the HTTP caching rules browsers follow, in both sync and async code. With HTTPX, [swap your client for Hishel's cache-enabled one](https://hishel.com/httpx.html#quick-start), or [give a client you already have its cache transport](https://hishel.com/httpx.html#cache-transports). With Requests, [mount its cache adapter](https://hishel.com/requests.html#quick-start) on a `Session`. In FastAPI, [a dependency sets `Cache-Control` headers](https://hishel.com/fastapi.html#quick-start) for browsers and CDNs, and its ASGI middleware also caches responses on your server. + +Cacheops caches Django querysets in Redis and [invalidates them](https://github.com/Suor/django-cacheops#user-content-invalidation) on `save()`, `delete()`, and many-to-many changes. [Add it to `INSTALLED_APPS`](https://github.com/Suor/django-cacheops#user-content-setup) and point it at its own Redis database, as its docs highly recommend. Turn caching on per app with `'app_name.*'`, since `'*.*'` can also cache tables you don't mean to, like migrations. Call `.cache()` on a queryset to cache it by hand, and wrap a function in [`@cached_as(Article)`](https://github.com/Suor/django-cacheops#user-content-usage) to drop its result whenever an `Article` changes. + +Plan how stale entries leave the cache, too. cachetools' decorators add a [`cache_clear()`](https://cachetools.readthedocs.io/en/stable/#memoizing-decorators) function, DiskCache [evicts every key with a given tag](https://grantjenks.com/docs/diskcache/tutorial.html#cache), and a function decorated by dogpile.cache takes [`invalidate()`](https://dogpilecache.sqlalchemy.org/en/latest/api.html#dogpile.cache.region.CacheRegion.cache_on_arguments) with the same arguments you'd call it with. diff --git a/website/data/category_intros/cli-development.md b/website/data/category_intros/cli-development.md new file mode 100644 index 00000000..9c2009bc --- /dev/null +++ b/website/data/category_intros/cli-development.md @@ -0,0 +1,38 @@ +Python's own argparse covers basic command-line apps. Beyond that, pick Click or Typer as your Python CLI library, and style the output with Rich. + +How to choose: + +- Basic command-line app with no dependencies: argparse +- Nested commands, or subcommands loaded lazily: Click +- Options declared once as type-hinted function parameters: Typer +- Interactive prompts and REPLs: prompt_toolkit +- CLI from existing code without writing a parser: Python Fire +- Progress bar for a loop: tqdm, or alive-progress for animated bars +- Colored text, tables, and logs: Rich, or colorama for ANSI colors on Windows +- Full-screen app in the terminal or a browser: Textual +- Terminal widgets on an event loop you already run: Urwid +- Full-screen forms and ASCII animations: asciimatics + +argparse is in the standard library. Python's docs call it [the recommended choice](https://docs.python.org/3/library/optparse.html#choosing-an-argument-parser) when you have no more specific needs, since it gives you the most out of the box for the least code. It [writes the help and usage messages](https://docs.python.org/3/library/argparse.html) and reports invalid arguments for you. For commands written as decorated functions, the same docs point to Click, and for a CLI that works with static type checking, to Typer. + +Click builds a CLI from [commands declared with decorators](https://click.palletsprojects.com/en/stable/quickstart/). You can nest them to any depth, and Click can [load subcommands lazily](https://click.palletsprojects.com/en/stable/) at runtime. It also [reads option values from environment variables](https://click.palletsprojects.com/en/stable/why/) and comes with helpers for ANSI colors, terminal size, and launching editors. + +Typer builds a CLI from your function signatures: you [declare each argument and option once](https://typer.tiangolo.com/), as a type-hinted function parameter. Write those hints with `Annotated`, which its tutorial [recommends wherever possible](https://typer.tiangolo.com/tutorial/arguments/optional/). Its docs say you get [the best results by pairing it with Rich](https://typer.tiangolo.com/tutorial/printing/): Typer structures the commands and options, and Rich displays the output. + +prompt_toolkit is for interactive input. It can [replace GNU readline](https://python-prompt-toolkit.readthedocs.io/en/stable/) or build full-screen apps. For a REPL, use a [`PromptSession`](https://python-prompt-toolkit.readthedocs.io/en/stable/pages/asking_for_input.html), which keeps the history for the whole session. + +Python Fire turns any Python object into a CLI: [call `fire.Fire()`](https://github.com/google/python-fire/blob/master/docs/guide.md) at the end of your program. Its README pitches it for [developing and debugging your code](https://github.com/google/python-fire), exploring existing code, and turning other people's code into a CLI. It takes each argument's type from the value you pass, not from the function signature. + +tqdm adds a progress bar to any loop: [wrap the iterable in `tqdm()`](https://github.com/tqdm/tqdm). Import it from `tqdm.auto`, which picks the console bar or the Jupyter widget for you. alive-progress wraps the loop in an [`alive_bar` context manager](https://github.com/rsalmei/alive-progress) instead, with a spinner that speeds up and slows down with your throughput. + +Rich writes colored text, tables, Markdown, and syntax-highlighted code to the terminal. Start with its [drop-in `print`](https://rich.readthedocs.io/en/stable/introduction.html), then create [one `Console` at module level](https://rich.readthedocs.io/en/stable/console.html) for the rest of your app. For colored logs, send the logging module's output through [`RichHandler`](https://rich.readthedocs.io/en/stable/logging.html). + +colorama makes ANSI escape codes work on Windows and does nothing on other platforms. If that's all you need, call [`just_fix_windows_console()`](https://github.com/tartley/colorama). + +Textual apps run in the terminal, and the `textual serve` command from textual-dev [serves them in a browser](https://textual.textualize.io/guide/devtools/). Build one by [subclassing `App`](https://textual.textualize.io/guide/app/), and style its widgets with [CSS](https://textual.textualize.io/guide/CSS/). + +Urwid is a [console widget construction set](https://urwid.org/manual/overview.html) rather than a finished UI library. It runs on [your choice of event loop](https://github.com/urwid/urwid): asyncio, or another one you already use. It's [licensed under the LGPL](https://github.com/urwid/urwid/blob/master/COPYING), while Textual and asciimatics use permissive licenses. + +asciimatics does [full-screen text UIs, from interactive forms to ASCII animations](https://github.com/peterbrittain/asciimatics): create a Screen, build a Scene from Effect objects, and let the Screen play it. + +Ship your CLI as an installable package with a `[project.scripts]` entry point, not a file users run with `python`. [Click recommends it](https://click.palletsprojects.com/en/stable/entry-points/), and shell completion in Click and [Typer](https://typer.tiangolo.com/tutorial/package/) works through that entry point. Test your commands in-process: Click and Typer both provide a [`CliRunner`](https://click.palletsprojects.com/en/stable/testing/), and Textual's [`run_test()`](https://textual.textualize.io/guide/testing/) drives your app as if you were using the keyboard and mouse. diff --git a/website/data/category_intros/cli-tools.md b/website/data/category_intros/cli-tools.md new file mode 100644 index 00000000..e630e5b2 --- /dev/null +++ b/website/data/category_intros/cli-tools.md @@ -0,0 +1,26 @@ +Swap cURL for HTTPie, and your database's own client for a Python CLI tool with autocompletion: pgcli, mycli, litecli, or IRedis. + +How to choose: + +- Calling HTTP APIs from the terminal: HTTPie +- A database shell with autocompletion: pgcli for PostgreSQL, mycli for MySQL, litecli for SQLite, IRedis for Redis +- Downloading video or audio from YouTube and other sites: yt-dlp +- A new project from a template: Cookiecutter, or Copier to pull in later template changes +- A shell you script in Python: xonsh +- A tmux session with its windows and panes, from one YAML file: tmuxp + +HTTPie is a command-line HTTP client for [testing, debugging, and generally interacting with APIs](https://httpie.io/docs/cli) and HTTP servers. It formats and colors the output. A request reads like `http PUT pie.dev/put X-API-Token:123 name=John`: `Header:Value` items set headers, and `field=value` items fill the body, which HTTPie [sends as JSON by default](https://httpie.io/docs/cli/json). + +pgcli, mycli, litecli, and IRedis are terminal clients with autocompletion and syntax highlighting, one per database. Give [pgcli](https://github.com/dbcli/pgcli) a database name or a `postgresql://` URI, and [litecli](https://github.com/dbcli/litecli) the path to a SQLite file. mycli also [works with MariaDB](https://github.com/dbcli/mycli), Percona, TiDB, and Apache Doris. IRedis behaves like Redis's own client in most cases, but it's [safer on production servers](https://github.com/laixintao/iredis): it stops you from accidentally running dangerous commands like `KEYS *`. + +yt-dlp downloads audio and video from [thousands of sites](https://github.com/yt-dlp/yt-dlp). Install ffmpeg too: yt-dlp [needs it to merge separate video and audio files](https://github.com/yt-dlp/yt-dlp#dependencies). + +Cookiecutter creates projects from templates, and its templates [work for any language](https://github.com/cookiecutter/cookiecutter), not only Python. [Run `cookiecutter gh:audreyfeldroy/cookiecutter-pypackage`](https://cookiecutter.readthedocs.io/en/stable/usage.html), answer its prompts, and you get a new project from that GitHub template. + +Copier also generates projects from templates, but it calls itself [a code lifecycle management tool](https://copier.readthedocs.io/en/stable/comparisons/). When the template changes, `copier update` brings the changes into projects you already generated. [It works best](https://copier.readthedocs.io/en/stable/updating/) when both are in Git: the template tagged, your project clean, with the `.copier-answers.yml` file that records your answers. Templates can run code: Cookiecutter's [pre- and post-generate scripts](https://github.com/cookiecutter/cookiecutter), Copier's tasks. [Generate projects only from templates you trust](https://copier.readthedocs.io/en/stable/generating/). + +xonsh is a shell whose language is [a superset of Python](https://github.com/xonsh/xonsh), with shell commands built in. It isn't POSIX-compatible, so don't make it your login shell with `chsh`. Its docs [recommend a xonsh profile in your terminal emulator](https://xon.sh/install.html#before-installing) instead. + +tmuxp launches a whole tmux session [from one YAML or JSON file](https://tmuxp.git-pull.com/quickstart/): its windows, its panes, and the commands in them. Save the file as `.tmuxp.yaml` in a project, and [`tmuxp load path/to/project/`](https://github.com/tmux-python/tmuxp) builds the session. + +Several of these tools' docs have you [install them as CLI tools](https://github.com/cookiecutter/cookiecutter), and add them to a project only to use them from Python. Looking for a library to build your own Python CLI tool? That's [CLI Development](/categories/cli-development/). diff --git a/website/data/category_intros/cms.md b/website/data/category_intros/cms.md new file mode 100644 index 00000000..fa32524f --- /dev/null +++ b/website/data/category_intros/cms.md @@ -0,0 +1,12 @@ +Both Python CMS picks run on Django: Wagtail for page types your developers define, like articles, and django CMS for editors building pages on the live site. + +How to choose: + +- Content types your developers define, like articles and events: Wagtail +- Editors composing pages from reusable components on the live site: django CMS + +Wagtail is [not an instant website in a box](https://docs.wagtail.org/en/stable/getting_started/the_zen_of_wagtail.html#wagtail-is-not-an-instant-website-in-a-box): expect to write code. Start a project with [`wagtail start`](https://docs.wagtail.org/en/stable/getting_started/quick_install.html). Each page type is [a Django model that inherits from `Page`](https://docs.wagtail.org/en/stable/topics/pages.html), so give each kind of content its own type with its own fields. An event page with a date and a location [can show up in a calendar](https://docs.wagtail.org/en/stable/getting_started/the_zen_of_wagtail.html#a-cms-should-get-information-out-of-an-editor-s-head-and-into-a-database-as-efficiently-and-directly-as-possible), and a styled heading on a generic page can't. Use [StreamField](https://docs.wagtail.org/en/stable/topics/streamfield.html) for pages without a fixed structure, like blog posts, and [snippets](https://docs.wagtail.org/en/stable/topics/snippets/index.html) for content that doesn't need its own page, like headers and footers. For a headless site, use its [built-in API](https://docs.wagtail.org/en/stable/advanced_topics/api/index.html). + +In django CMS, [editors compose pages from plugins](https://docs.django-cms.org/en/stable/explanation/philosophy.html#three-disciplines-three-surfaces) in a toolbar on the live site, and developers write standard Django. Each template [declares its placeholders](https://docs.django-cms.org/en/stable/tutorials/02-templates-placeholders.html) with `{% placeholder %}`, and `CMS_TEMPLATES` lists the templates editors can pick from. For a region that is the same on every page, like a footer, use [`{% static_alias %}`](https://docs.django-cms.org/en/stable/tutorials/02-templates-placeholders.html#a-reusable-region-with-static-alias), so the content is stored once. For content from your own app, ask [where it lives](https://docs.django-cms.org/en/stable/explanation/composition.html). If it fits in a placeholder on a page, write a plugin. If it has its own list view, detail view, and URL, mount it as an app with an apphook. The core [publishes as you edit](https://docs.django-cms.org/en/stable/explanation/publishing.html), which is rarely enough for a production site with editors, so add a versioning package. + +Both are built on Django. Wagtail [deploys like a Django site](https://docs.wagtail.org/en/stable/deployment/index.html). You can add either one to an existing Django project: Wagtail [integrates into one](https://docs.wagtail.org/en/stable/getting_started/integrating_into_django.html), and django CMS [doesn't make you rebuild around it](https://docs.django-cms.org/en/stable/explanation/philosophy.html#implications-for-projects). Neither fits a very small or static site. For a two-page brochure, django CMS calls itself [overkill](https://docs.django-cms.org/en/stable/explanation/philosophy.html#when-django-cms-may-not-be-the-right-fit), and a static site generator is lighter. diff --git a/website/data/category_intros/code-analysis.md b/website/data/category_intros/code-analysis.md new file mode 100644 index 00000000..180413fa --- /dev/null +++ b/website/data/category_intros/code-analysis.md @@ -0,0 +1,49 @@ +Lint and format with Ruff. It doesn't check types, so add a type checker, and let pre-commit run each Python code analysis tool on every commit. + +How to choose: + +- Linting, formatting, and import sorting in one tool: Ruff +- Running every check before each commit: pre-commit +- Rules on which of your modules may import which: Import Linter +- Unused code: Vulture +- Functions too complex for people to read: complexipy +- Several analyzers under one command, with ready-made profiles: Prospector +- Black's own formatting, with imports sorted to match: Black and isort +- Checks Ruff can't run: Pylint for deeper inference, Flake8 for your own plugins +- Security issues: Bandit +- Renaming and moving code from your editor or a script: Rope +- Type checking Django or Pydantic code: mypy with its plugins, Pyrefly without +- Adding types to a partly typed codebase: ty +- Type checking in VS Code: Pyright, which Pylance runs on + +Ruff [can replace Flake8, Black, and isort](https://docs.astral.sh/ruff/), plus dozens of Flake8 plugins. Its docs suggest you [start with a small rule set](https://docs.astral.sh/ruff/linter/#rule-selection), add one group at a time, and list them explicitly in `select`. Ruff is [a linter, not a type checker](https://docs.astral.sh/ruff/faq/#how-does-ruff-compare-to-ty-mypy-pyright-or-pyre), so run a type checker alongside it. + +pre-commit installs and runs the hooks you list in `.pre-commit-config.yaml`, [written in any language](https://pre-commit.com/). Run `pre-commit install` [first thing after you clone](https://pre-commit.com/#usage) a repo that uses it, so the hooks run on every `git commit`, and add `pre-commit run --all-files` [as a CI step](https://pre-commit.com/#usage-in-continuous-integration). + +Import Linter checks [contracts on the imports between your modules](https://import-linter.readthedocs.io/en/stable/): `lint-imports` fails when code breaks one. A [layers contract](https://import-linter.readthedocs.io/en/stable/contract_types/layers/) lets higher layers import lower ones but not the other way around, and it counts chains of imports through other modules too. + +Vulture finds unused code, and [running it on your tests too](https://github.com/jendrikseipp/vulture) shows you untested code. + +complexipy measures [cognitive complexity](https://complexipy.com/): how hard code is for people to understand. Unlike cyclomatic complexity, it counts nesting. Its docs recommend [running it alongside Ruff](https://complexipy.com/comparison-with-ruff/#use-both-recommended), whose complexity rule measures something else. + +Prospector runs Pylint and other analyzers together, and aims to be [useful out of the box](https://prospector.landscape.io/en/stable/), with default profiles so you don't have to sort out which warnings matter. When your project uses Django, Flask, or Celery, it [adapts its checks](https://prospector.landscape.io/en/stable/#adapting-to-dependencies) to that framework. + +With Black, you [give up control over hand-formatting](https://black.readthedocs.io/en/stable/), and in return, code looks the same in every project and diffs stay small. Its style options are [deliberately limited](https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html). If you sort imports with isort, set [the black profile](https://isort.readthedocs.io/en/latest/configuration/black_compatibility.html) so the two agree. + +Pylint [infers the actual values](https://pylint.readthedocs.io/en/latest/#what-differentiates-pylint) in your code instead of trusting its type hints. That makes it slower than other linters, but it finds more in code that isn't fully typed, and it's why Ruff [isn't a drop-in replacement](https://docs.astral.sh/ruff/faq/#how-does-ruffs-linter-compare-to-pylint) for Pylint. Because of that speed, run it [in CI or a pre-push hook](https://pylint.readthedocs.io/en/latest/user_guide/installation/pre-commit-integration.html) rather than on every commit. + +Flake8 [wraps pyflakes, pycodestyle, and McCabe](https://github.com/PyCQA/flake8) in one command, and [its plugins](https://flake8.pycqa.org/en/latest/user/using-plugins.html) are much of its appeal. Keep it when you depend on a plugin of your own, since Ruff [doesn't support custom lint rules](https://docs.astral.sh/ruff/faq/#how-does-ruffs-linter-compare-to-flake8). + +Bandit [finds common security issues](https://bandit.readthedocs.io/en/latest/) in Python code. Scan a whole tree with [`bandit -r`](https://bandit.readthedocs.io/en/latest/start.html). + +Rope is a refactoring library that [many IDEs and editors use](https://rope.readthedocs.io/en/latest/overview.html) to rename, extract, and move code, and scripts can run the same refactorings through [its library API](https://rope.readthedocs.io/en/latest/library.html). + +mypy is [designed for gradual typing](https://mypy.readthedocs.io/en/stable/): by default, it [skips functions without annotations](https://mypy.readthedocs.io/en/stable/getting_started.html#dynamic-vs-static-typing), so you can add type hints slowly. Check in one config file, [run mypy in CI](https://mypy.readthedocs.io/en/stable/existing_code.html#run-mypy-consistently-and-prevent-regressions) so new type errors don't land, and aim for `--strict`. Its [plugins](https://mypy.readthedocs.io/en/stable/extending_mypy.html#extending-mypy-using-plugins) teach it libraries the standard type system can't describe. + +Pyrefly is a type checker and language server with [built-in support for Pydantic and Django](https://github.com/facebook/pyrefly), so those work without plugins. It reads an existing mypy or Pyright config, so you can [try it alongside your current checker](https://pyrefly.org/en/docs/compare/#try-it-alongside-your-current-checker) before you switch. + +ty is a type checker and language server [designed for adoption](https://docs.astral.sh/ty/) on partly typed code. Unlike mypy, it [checks the bodies of unannotated functions](https://docs.astral.sh/ty/coming-from-mypy-or-pyright/), and it leaves [missing-annotation checks](https://docs.astral.sh/ty/reference/typing-faq/#why-doesnt-ty-warn-about-missing-type-annotations) to Ruff's `ANN` rules. + +Pyright [checks all code by default](https://github.com/microsoft/pyright/blob/main/docs/mypy-comparison.md#type-checking-unannotated-code), annotated or not, which its language server features rely on. In VS Code, its docs [recommend Pylance](https://github.com/microsoft/pyright/blob/main/docs/installation.md), which includes Pyright. Run it [in CI to keep the project type clean](https://github.com/microsoft/pyright/blob/main/docs/getting-started.md), then turn on `# pyright: strict` file by file. + +Pylint, Prospector, Import Linter, and ty all need to see your project's dependencies, so run them [from the project's own environment](https://pylint.readthedocs.io/en/latest/user_guide/installation/pre-commit-integration.html), not a separate one. For more Python code analysis tools, see [awesome-static-analysis](https://github.com/analysis-tools-dev/static-analysis). diff --git a/website/data/category_intros/computer-vision.md b/website/data/category_intros/computer-vision.md new file mode 100644 index 00000000..1fa53d77 --- /dev/null +++ b/website/data/category_intros/computer-vision.md @@ -0,0 +1,27 @@ +OpenCV processes images and video. Training your own detector needs more than a Python computer vision library scoped to inference, so reach for Ultralytics. + +How to choose: + +- Reading, editing, and analyzing images and video: OpenCV +- Detecting, segmenting, or tracking objects with a trained model: Ultralytics +- Image operations inside a PyTorch model or training loop: Kornia +- Browsing a dataset, finding label mistakes, and seeing where a model fails: FiftyOne +- Printed text in clean scans: pytesseract +- Text in photos or degraded scans: EasyOCR +- Tables and page layouts, turned into Markdown or JSON: PaddleOCR + +OpenCV's Python bindings [convert every OpenCV array to and from a NumPy array](https://github.com/opencv/opencv/blob/5.x/doc/py_tutorials/py_setup/py_intro/py_intro.markdown), so the images you load work with any other library that uses NumPy. + +Ultralytics runs YOLO models for detection, segmentation, pose estimation, and tracking from [one Python package and CLI](https://docs.ultralytics.com/). Load a pretrained model and [call `.train()` on your own dataset](https://docs.ultralytics.com/guides/finetuning-guide/): the weights carry over with no extra code. Ultralytics is AGPL, so [using it without open-sourcing your project](https://www.ultralytics.com/license) takes its Enterprise License. + +Kornia is [a differentiable computer vision library like OpenCV, with strong GPU support](https://kornia.readthedocs.io/en/latest/get-started/introduction.html), built on PyTorch tensors. Its FAQ states the split: Kornia [provides operators to train networks, while OpenCV's scope is inference](https://kornia.readthedocs.io/en/latest/community/faqs.html). + +FiftyOne is for building better datasets. [Load your images and labels as a Dataset](https://docs.voxel51.com/user_guide/basics.html) to search and sort samples and spot label mistakes. Then add your model's predictions and [evaluate them sample by sample](https://docs.voxel51.com/user_guide/evaluation/index.html) in the FiftyOne App to see where it goes wrong. + +pytesseract is [a wrapper for Google's Tesseract engine](https://github.com/madmaze/pytesseract), which you install on its own. `image_to_string()` returns the text, and `image_to_data()` adds boxes and confidences. + +EasyOCR reads [many languages and scripts](https://github.com/JaidedAI/EasyOCR) with deep learning models on PyTorch. [Create one `Reader` and reuse it](https://www.jaided.ai/easyocr/tutorial/): it loads the models into memory once, then reads as many images as you give it. + +PaddleOCR does two jobs: [general OCR](https://www.paddleocr.ai/latest/en/version3.x/pipeline_usage/OCR.html) reads text, and document parsing [turns PDFs and document images into Markdown and JSON](https://www.paddleocr.ai/latest/en/index.html) that keep the original structure. + +Load and crop images with OpenCV, then pass the NumPy arrays on: [EasyOCR](https://www.jaided.ai/easyocr/tutorial/), [PaddleOCR](https://www.paddleocr.ai/latest/en/version3.x/pipeline_usage/OCR.html), and [Ultralytics](https://docs.ultralytics.com/modes/predict/) all take them as input. diff --git a/website/data/category_intros/configuration-files.md b/website/data/category_intros/configuration-files.md new file mode 100644 index 00000000..e6e2921a --- /dev/null +++ b/website/data/category_intros/configuration-files.md @@ -0,0 +1,21 @@ +Settings that change between deployments belong in the environment, whatever Python configuration library you use. pydantic-settings reads them as typed fields. + +How to choose: + +- Typed, validated settings from environment variables and `.env` files: pydantic-settings +- An INI file your users edit, with nothing to install: configparser +- A `.env` file loaded into the environment during development: python-dotenv +- Research code that composes its config and overrides it from the command line: Hydra +- Layered settings per environment, or settings for Django and Flask: Dynaconf + +pydantic-settings loads [a settings class from environment variables or secrets files](https://pydantic.dev/docs/validation/latest/concepts/pydantic_settings/). Subclass `BaseSettings` and declare each setting as a type-hinted field. [Any field you don't pass to the initializer gets its value from the environment](https://pydantic.dev/docs/validation/latest/concepts/pydantic_settings/#usage), or its default when the variable isn't set. To read a `.env` file too, [set `env_file` in `model_config`](https://pydantic.dev/docs/validation/latest/concepts/pydantic_settings/#dotenv-env-support), and its values get validated like the rest. + +configparser comes with Python. It reads [a basic configuration language structured like Windows INI files](https://docs.python.org/3/library/configparser.html), so end users can customize your program by editing a file. Read values with mapping access, `config['section']['option']`, which the docs [prefer for new projects](https://docs.python.org/3/library/configparser.html#legacy-api-examples) over the legacy get and set methods. Values always come back as strings, so convert them with [`getint()`, `getfloat()`, and `getboolean()`](https://docs.python.org/3/library/configparser.html#supported-datatypes). + +python-dotenv is for apps that [take their configuration from environment variables](https://saurabh-kumar.com/python-dotenv/#getting-started). In development, setting each one yourself isn't practical, so call `load_dotenv()` before the rest of your code. It reads a `.env` file when there is one and adds its values to `os.environ`, so your code reads them with `os.getenv()` as if they came from the real environment. + +Hydra is a framework for [research and other complex applications](https://hydra.cc/docs/1.3/intro/). It builds a hierarchical configuration by composition, and you override it from config files and the command line. Decorate your entry point with [`@hydra.main()`](https://hydra.cc/docs/1.3/intro/#basic-example) and point it at a YAML config. Then change a value with an argument like `db.user=root`. For options you switch between, like two databases, [create a config group](https://hydra.cc/docs/1.3/intro/#composition-example) with one file per option, and pick one in the config's `defaults` list. With [`--multirun`](https://hydra.cc/docs/1.3/intro/#multirun), one command runs your function once for each configuration you list. + +Dynaconf reads settings from [files in several formats, environment variables, and Vault or Redis](https://www.dynaconf.com/#features). Start with [`dynaconf init -f toml`](https://www.dynaconf.com/#using-python-only), which creates `config.py`, `settings.toml`, and `.secrets.toml`. Then import `settings` from `config` in your code. TOML is its default and most recommended format. To give development and production their own sections in one file, [set `environments=True`](https://www.dynaconf.com/#layered-environments-on-files). In a Django app, `django.conf.settings` [becomes a Dynaconf settings object](https://www.dynaconf.com/#using-django), and in Flask, `app.config` does. + +Keep settings that change between deployments out of your code. The [twelve-factor app stores config in environment variables](https://12factor.net/config), and python-dotenv and Dynaconf both cite it. python-dotenv, pydantic-settings, and Dynaconf let the real environment win over files. `load_dotenv()` [doesn't override variables already set](https://saurabh-kumar.com/python-dotenv/#getting-started), pydantic-settings [ranks environment variables above `.env` and secrets files](https://pydantic.dev/docs/validation/latest/concepts/pydantic_settings/#field-value-priority), and Dynaconf [prioritizes environment variables over files](https://www.dynaconf.com/envvars/). Keep secrets out of Git, too: add `.env` to your `.gitignore`, and `dynaconf init` adds `.secrets.*` there for you. diff --git a/website/data/category_intros/cryptography.md b/website/data/category_intros/cryptography.md new file mode 100644 index 00000000..21f803b3 --- /dev/null +++ b/website/data/category_intros/cryptography.md @@ -0,0 +1,19 @@ +Don't touch a Python cryptography library's raw ciphers: encrypt with cryptography's Fernet or PyNaCl. Paramiko runs SSH, and ItsDangerous signs tokens. + +How to choose: + +- Encrypting data, X.509 certificates, and other standard formats: cryptography +- Encryption and signatures between your own apps, with algorithms picked for you: PyNaCl +- Hashing passwords for storage: PyNaCl +- SSH clients and servers, and SFTP: Paramiko +- Signed links, tokens, and cookies that users may read but not change: ItsDangerous + +cryptography has 2 layers: recipes that need few decisions, and low-level primitives it calls the "hazmat" layer. Its docs [recommend the recipes layer whenever possible](https://cryptography.io/en/stable/#layout), and hazmat only when necessary. For data you encrypt with a key, the recipe is [Fernet](https://cryptography.io/en/stable/fernet/): a message encrypted with it can't be read or changed without the key. To encrypt with a password, run it through a key derivation function first. The docs [recommend Argon2id](https://cryptography.io/en/stable/fernet/#using-passwords-with-fernet), and you keep the salt to derive the same key again. When Fernet doesn't fit, the docs point you to [authenticated encryption](https://cryptography.io/en/stable/hazmat/primitives/symmetric-encryption/) before any raw cipher, since encryption alone keeps data secret but doesn't stop tampering. The recipes also cover [X.509 certificates](https://cryptography.io/en/stable/x509/tutorial/), from signing requests to self-signed certs. + +PyNaCl is a binding to libsodium, a fork of NaCl. cryptography's FAQ explains the split: cryptography is [general purpose and interoperable with existing systems](https://cryptography.io/en/stable/faq/#how-does-cryptography-compare-to-nacl-networking-and-cryptography-library), while NaCl gives you a set of hand-selected algorithms. If you prefer NaCl's design, the FAQ recommends PyNaCl. With a shared key, encrypt through `SecretBox` or `Aead` and let PyNaCl [generate a random nonce](https://pynacl.readthedocs.io/en/latest/secret/#nacl.secret.Aead.encrypt) for each message, which its docs strongly recommend. Between 2 parties, `Box` [authenticates both sides](https://pynacl.readthedocs.io/en/latest/public/#nacl-public-box), and `SealedBox` sends a message only the recipient can decrypt, without proving who sent it. To store passwords, `nacl.pwhash.str()` [hashes with argon2id by default](https://pynacl.readthedocs.io/en/latest/password_hashing/#password-storage-and-verification), and `nacl.pwhash.verify()` checks a password against the hash. + +Paramiko implements SSHv2 in pure Python, as both client and server. For common client jobs like running remote commands or transferring files, its homepage [recommends a higher-level library built on it](https://www.paramiko.org/). Use Paramiko directly for low-level primitives or to run an SSH server in Python. Its client API [starts with `SSHClient`](https://docs.paramiko.org/en/latest/), and checking the server's host key is your job: call `load_system_host_keys()` before `connect()`. A host key it can't find is [rejected by default](https://docs.paramiko.org/en/latest/api/client.html#paramiko.client.SSHClient.set_missing_host_key_policy) with an `SSHException`. `open_sftp()` opens an SFTP session on the same connection. + +ItsDangerous signs data, so you can send it somewhere untrusted and get it back: the receiver [can see the data but can't modify it](https://itsdangerous.palletsprojects.com/en/stable/) without your key. Flask's default sessions are [cookies signed with it](https://flask.palletsprojects.com/en/stable/api/#flask.sessions.SecureCookieSessionInterface). Signing hides nothing, so when the data must stay secret, encrypt it with Fernet instead. Its docs say you'll [typically want a serializer, not a signer](https://itsdangerous.palletsprojects.com/en/stable/concepts/#serializer-vs-signer). `URLSafeSerializer` turns data into a URL-safe string, and `URLSafeTimedSerializer` rejects tokens older than the `max_age` you pass to `loads()`. Give each use its own [salt](https://itsdangerous.palletsprojects.com/en/stable/concepts/#the-salt), so an activation link's signature won't pass as an upgrade link's. + +Whichever pick you use, keep secret keys out of your code. ItsDangerous's docs say the key [shouldn't be saved in source code or committed to version control](https://itsdangerous.palletsprojects.com/en/stable/concepts/#the-secret-key), and suggest reading it from an environment variable. When you generate a key or token yourself, use `os.urandom()`, [never the `random` module](https://cryptography.io/en/stable/random-numbers/), which isn't cryptographically secure. Plan for rotation, too: cryptography's [`MultiFernet`](https://cryptography.io/en/stable/fernet/#cryptography.fernet.MultiFernet) and ItsDangerous both [take a list of keys](https://itsdangerous.palletsprojects.com/en/stable/concepts/#key-rotation), so old tokens keep working after you add a new key. diff --git a/website/data/category_intros/data-analysis.md b/website/data/category_intros/data-analysis.md new file mode 100644 index 00000000..5d640802 --- /dev/null +++ b/website/data/category_intros/data-analysis.md @@ -0,0 +1,17 @@ +Once your data nears the size of your RAM, switch Python data analysis libraries from pandas to Polars. Once it lives in a database, Ibis runs your code there. + +How to choose: + +- Data that fits in memory: pandas +- Handing DataFrames to libraries that expect pandas: pandas +- Large data on one machine, even bigger than your RAM: Polars +- Data already in a database or warehouse: Ibis +- Code you prototype locally, then run on a warehouse or cluster: Ibis + +pandas aims to be [the fundamental high-level building block](https://pandas.pydata.org/docs/getting_started/overview.html) for practical data analysis in Python. Its DataFrame fits time series and tables with mixed column types, like an SQL table or a spreadsheet. It's built on NumPy to work with the rest of the scientific Python stack. In production code, the docs [recommend `.loc` and `.iloc` over `[]`](https://pandas.pydata.org/docs/user_guide/indexing.html) to select data. Set values in [one `.loc` call](https://pandas.pydata.org/docs/user_guide/copy_on_write.html#chained-assignment), not chained indexing like `df["foo"][mask] = 100`. pandas keeps everything in memory, so a dataset that takes a sizable share of your RAM [gets unwieldy](https://pandas.pydata.org/docs/user_guide/scale.html), and the docs point you to other libraries. + +Polars is [built for multithreaded computing on a single machine](https://docs.pola.rs/user-guide/misc/comparison/#pandas), and its stricter API leads to fewer schema bugs. It has no index: rows are known by their position, so [no index state can change what a query means](https://docs.pola.rs/user-guide/migration/pandas/#polars-does-not-have-a-multi-indexindex). Make the lazy API [your default](https://docs.pola.rs/user-guide/migration/pandas/#be-lazy): start from a function like `scan_csv()` or call `.lazy()`, then `.collect()` the result. That lets the optimizer [filter rows and pick columns while reading the data](https://docs.pola.rs/user-guide/concepts/lazy-api/). Stay eager for exploratory work, when [you don't know yet what your query will look like](https://docs.pola.rs/user-guide/concepts/lazy-api/#when-to-use-which). Write expressions inside `select`, `with_columns`, `filter`, and `group_by`, since Polars code that [looks like pandas code](https://docs.pola.rs/user-guide/migration/pandas/#key-syntax-differences) likely runs slower than it should. For data that doesn't fit in memory, run the query on the [streaming engine](https://docs.pola.rs/user-guide/concepts/streaming/), which works through it in batches. + +Ibis [compiles one Python dataframe API](https://ibis-project.org/why#how-does-ibis-work) into each backend's native language, mostly SQL, so the database or engine does the work. pandas' ecosystem page lists it for [bridging local Python and remote databases](https://pandas.pydata.org/community/ecosystem.html#ibis). Start on the default DuckDB backend, as the tutorial [recommends](https://ibis-project.org/tutorials/basics#install-ibis), then [change the connection string](https://ibis-project.org/why#scaling-up-and-out) to run the same code on PySpark, BigQuery, or Trino. Expressions are lazy: nothing runs until you call a method like `to_pandas()`, and only then does Ibis [send the compiled query](https://ibis-project.org/tutorials/coming-from/pandas) to the backend. + +You can mix all three. Polars converts a DataFrame [with `to_pandas()`](https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.to_pandas.html), Ibis returns results [as pandas or Polars DataFrames](https://ibis-project.org/reference/expression-tables#ibis.expr.types.relations.Table.to_pandas), and pandas [can use PyArrow](https://pandas.pydata.org/docs/user_guide/pyarrow.html) to trade data with Arrow-based libraries like Polars. Do the heavy work in Polars or Ibis, and hand a pandas DataFrame to the libraries that expect one. diff --git a/website/data/category_intros/data-ingestion-etl.md b/website/data/category_intros/data-ingestion-etl.md new file mode 100644 index 00000000..2d488f53 --- /dev/null +++ b/website/data/category_intros/data-ingestion-etl.md @@ -0,0 +1,24 @@ +dlt loads data from APIs, databases, and files into your warehouse. You pip install the Python ETL library into your own code, with no platform to run. + +How to choose: + +- Data from APIs, databases, or files, loaded into a warehouse or DuckDB: dlt +- pandas DataFrames in and out of S3, Athena, Redshift, and other AWS services: awswrangler +- Stock prices, financials, and options from Yahoo Finance, for personal research: yfinance +- Chinese stocks, futures, and funds, for academic research: AKShare +- US company filings and financial statements from SEC EDGAR: EdgarTools +- Many data providers behind one Python API, also served over REST and MCP: OpenBB + +dlt [loads data from messy sources into well-structured datasets](https://dlthub.com/docs/intro): it infers the schema and data types, normalizes the data, and handles nested structures. Start a project with [`dlt init rest_api duckdb`](https://dlthub.com/docs/tutorial/rest-api), which sets up a pipeline script with a REST API source and a DuckDB destination. Build and test on DuckDB, then [switch out the destination](https://dlthub.com/docs/reference/explainers/how-dlt-works) when you deploy. Beyond the built-in sources, any Python iterable can feed it, since [a resource is just a generator](https://github.com/dlt-hub/dlt). dlt [runs anywhere Python runs](https://dlthub.com/docs/walkthroughs/deploy-a-pipeline), so schedule it with the orchestrator you already have. + +AWS SDK for pandas [extends pandas to AWS](https://aws-sdk-pandas.readthedocs.io/en/stable/about.html): import it as `awswrangler`, and it connects DataFrames to S3, Athena, Glue, Redshift, and DynamoDB. Its [quick start](https://github.com/aws/aws-sdk-pandas) stores a DataFrame on S3 as a Parquet dataset with `wr.s3.to_parquet()`, passing a database and table name. Then `wr.athena.read_sql_query()` queries that table with SQL. + +yfinance offers [a Pythonic way to fetch financial and market data](https://github.com/ranaroussi/yfinance) from Yahoo Finance. Its [quick start](https://ranaroussi.github.io/yfinance/) uses `Ticker` for one symbol's prices, financial statements, and options, and `download` for prices across several symbols at once. + +AKShare [aims to simplify fetching financial data](https://github.com/akfamily/akshare), with one function call per dataset, like `ak.stock_zh_a_hist()` for the daily history of a Chinese stock. Its README credits mostly Chinese exchanges and finance sites as its sources. Its [docs are in Chinese](https://akshare.akfamily.xyz/introduction.html), and each data interface comes with an example you can copy and paste. + +EdgarTools [turns any SEC filing into a typed Python object](https://github.com/dgunning/edgartools), so a 10-K's revenue is one line instead of an afternoon of XBRL parsing. EDGAR requires an email with every request, so [set your identity](https://edgartools.readthedocs.io/en/latest/configuration/) with `set_identity()` before anything else. Everything starts with a `Company` or a `Filing`. `Company("AAPL").get_financials().income_statement()` returns a standardized income statement, and `.obj()` turns a filing into an object built for its form, with its data as pandas DataFrames. + +OpenBB is a ["connect once, consume everywhere" layer](https://docs.openbb.co/odp) over financial data sources. [Each provider package](https://docs.openbb.co/odp/python) connects one source and adds its own namespace to the `obb` client. The same commands run from Python, as a REST API, and as MCP tools. A call returns an OBBject, and [`to_dataframe()`](https://docs.openbb.co/odp/python/quickstart/user) turns it into a pandas DataFrame. + +Before you build on financial data, check whose terms you're under. yfinance [isn't affiliated with Yahoo](https://ranaroussi.github.io/yfinance/) and is meant for research and education, and Yahoo's API is for personal use only. AKShare's data is [for academic research](https://github.com/akfamily/akshare#statement) only. OpenBB [hosts no data itself](https://docs.openbb.co/odp/python/extensions/providers): each provider sets its own coverage and terms of use. diff --git a/website/data/category_intros/data-validation.md b/website/data/category_intros/data-validation.md new file mode 100644 index 00000000..5302718c --- /dev/null +++ b/website/data/category_intros/data-validation.md @@ -0,0 +1,18 @@ +Pydantic checks API input against your type hints. Python data validation extends to dataframes in Pandera, where each column gets a type hint. + +How to choose: + +- API payloads and other records, typed with hints: Pydantic +- Dataframes in pandas, Polars, PySpark, and more: Pandera +- Data checked against a JSON Schema document: jsonschema +- Data quality checks on databases and files, with alerts and reports: Great Expectations + +Pydantic [takes its schema from Python type hints](https://pydantic.dev/docs/validation/latest/get-started/why/#type-hints), so your type checker and IDE read the same code it validates. Subclass `BaseModel` and annotate its fields, then pass untrusted data to the model: if no `ValidationError` is raised, [every field matches its declared type](https://pydantic.dev/docs/validation/latest/concepts/models/). Your models can also [generate JSON Schema](https://pydantic.dev/docs/validation/latest/get-started/why/#json-schema) for self-documenting APIs and other tools. + +Pandera [validates pandas, Polars, PySpark, and other dataframes against one schema](https://pandera.readthedocs.io/en/latest/). Write that schema as a `DataFrameModel`, [much the way you'd define a Pydantic model](https://pandera.readthedocs.io/en/latest/dataframe_models.html), with a typed field per column. Then decorate your pipeline functions with `check_types()`, which [validates both inputs and outputs](https://pandera.readthedocs.io/en/latest/decorators.html#check-inputs-and-outputs) from their type annotations. + +jsonschema [implements the JSON Schema specification](https://python-jsonschema.readthedocs.io/en/latest/), so the schema is a JSON document rather than Python code. That suits a schema that has to work outside Python too, since JSON Schema [establishes a common language for data exchange](https://json-schema.org/overview/what-is-jsonschema) across systems. Declare `$schema` in each schema to [identify which version it's written for](https://python-jsonschema.readthedocs.io/en/latest/referencing/). + +Great Expectations is [a framework for describing data with expressive tests](https://docs.greatexpectations.io/docs/core/introduction/gx_overview/#gx-core-components-and-workflows) and validating data against them, from databases to files in cloud storage. Every script [starts by creating a Data Context](https://docs.greatexpectations.io/docs/core/set_up_a_gx_environment/create_a_data_context/). Group your production Expectations into [Expectation Suites](https://docs.greatexpectations.io/docs/core/define_expectations/organize_expectation_suites/), and run those through [a Checkpoint](https://docs.greatexpectations.io/docs/core/introduction/gx_overview/#run-validations), which can send email or Slack alerts and write the results to Data Docs, a human-readable report. + +Records and tables need different validators, but the picks work together. A Pandera `DataFrameModel` [works as a field on a Pydantic model](https://pandera.readthedocs.io/en/latest/pydantic_integration.html), with Pydantic checking the built-in types next to it. Pandera's maintainer suggests [Pandera for in-memory dataframes and Great Expectations for data on disk](https://github.com/unionai-oss/pandera/discussions/598). Pandera needs zero configuration, while Great Expectations takes some setup up front. diff --git a/website/data/category_intros/data-visualization.md b/website/data/category_intros/data-visualization.md new file mode 100644 index 00000000..263d992d --- /dev/null +++ b/website/data/category_intros/data-visualization.md @@ -0,0 +1,39 @@ +Charts for print come from Matplotlib, and interactive web ones from Plotly. Put either Python data visualization library in a Streamlit script for a dashboard. + +How to choose: + +- Publication figures you control down to the last detail: Matplotlib +- Interactive charts with little code: Plotly +- A dashboard or data app from a plain Python script: Streamlit +- Statistical charts straight from a pandas DataFrame: seaborn +- Charts you declare by mapping columns to x, y, and color: Vega-Altair +- Interactive browser plots with full control, or Python code behind them: Bokeh +- Diagrams of nodes and edges, drawn by Graphviz: graphviz +- Maps of large areas, in the projection you choose: Cartopy +- A knowledge graph of your codebase for your AI coding assistant: graphify +- Dashboards built from callbacks and custom layouts: Dash +- A demo UI for an ML model or any Python function: Gradio + +Matplotlib [makes easy things easy and hard things possible](https://matplotlib.org/), and it's built for publication-quality plots. Start with `fig, ax = plt.subplots()` and draw on the `Axes` it returns. The docs [suggest this object-oriented style](https://matplotlib.org/stable/users/explain/quick_start.html#coding-styles) for complicated plots and for code you'll reuse. The pyplot style, like `plt.plot()`, is handy for quick interactive work. + +Plotly draws interactive charts that show up in Jupyter or [run inside Dash apps](https://plotly.com/python/getting-started/). Start with Plotly Express, the [recommended starting point](https://plotly.com/python/plotly-express/) for most common figures. Every Plotly Express function returns a graph objects `Figure`, so you can still fine-tune it with `update_layout()` and `add_trace()`. + +Streamlit turns a Python script into a web app: add a few `st.` calls, then run it with `streamlit run`. Whenever something on the screen must update, Streamlit [reruns your entire script from top to bottom](https://docs.streamlit.io/get-started/fundamentals/main-concepts#data-flow). So wrap slow work in a cached function: [`st.cache_data`](https://docs.streamlit.io/develop/concepts/architecture/caching) for data like a loaded DataFrame or a query result, and `st.cache_resource` for ML models and database connections. + +seaborn builds on Matplotlib and [integrates closely with pandas](https://seaborn.pydata.org/tutorial/introduction.html): you name DataFrame columns, and it draws the statistical chart. Its docs [recommend figure-level functions](https://seaborn.pydata.org/tutorial/function_overview.html#relative-merits-of-figure-level-functions) like `displot()` for most plots. + +Vega-Altair is declarative: you [declare links between data columns and encoding channels](https://altair-viz.github.io/getting_started/overview.html) like the x-axis, y-axis, and color, and it handles the rest. + +Bokeh [creates JavaScript-powered visualizations without writing any JavaScript](https://docs.bokeh.org/en/latest/index.html). Build plots with `figure()` from `bokeh.plotting`, its primary interface. Add the Bokeh server when your plots need to [respond to browser events with Python code](https://docs.bokeh.org/en/latest/docs/user_guide/server/server_introduction.html), like widgets that rerun a query. + +graphviz writes graphs in the DOT language, and [renders them with Graphviz](https://graphviz.readthedocs.io/en/latest/), which you install separately. Build a `Graph` or `Digraph`, add nodes and edges, then call [`render()`](https://graphviz.readthedocs.io/en/latest/manual.html) to save the DOT source and the rendered file. + +Cartopy draws maps on Matplotlib, and its docs say it's [especially useful for large areas](https://cartopy.readthedocs.io/latest/), where flat-map assumptions break down at the poles and the dateline. [Give the axes a map projection](https://cartopy.readthedocs.io/latest/matplotlib/intro.html), like `plt.axes(projection=ccrs.PlateCarree())`, then add coastlines and your data. Your data's coordinate system is separate from that projection, so [always pass `transform=`](https://cartopy.readthedocs.io/latest/tutorials/understanding_transform.html) when you plot. + +graphify maps your project into a knowledge graph that your AI coding assistant can [query instead of grepping](https://github.com/Graphify-Labs/graphify) through files. Code is parsed locally, with no LLM, and you can click through the graph in a browser. + +Dash [ties UI elements like dropdowns, sliders, and graphs](https://github.com/plotly/dash) to your Python code, and draws charts with Plotly through `dcc.Graph`. You add interactivity with [callbacks](https://dash.plotly.com/basic-callbacks): functions Dash calls whenever an input changes, to update another component. + +Gradio [builds a demo or web app](https://www.gradio.app/guides/quickstart) for an ML model, an API, or any Python function, with no JavaScript or CSS. Start with `gr.Interface`, which takes your function plus its inputs and outputs. Use `gr.ChatInterface` for a chatbot, and `gr.Blocks` when you need your own layout and data flow. + +Pick the chart library and the way you share it separately. Plotly, Vega-Altair, and Bokeh all save interactive charts as standalone HTML files: [`write_html()`](https://plotly.com/python/interactive-html-export/), [`chart.save("chart.html")`](https://altair-viz.github.io/user_guide/saving_charts.html), and [`output_file()`](https://docs.bokeh.org/en/latest/docs/user_guide/intro.html). Streamlit has [chart elements](https://docs.streamlit.io/develop/api-reference/charts) for Matplotlib, Vega-Altair, Plotly, and Graphviz, and Gradio's [`gr.Plot`](https://www.gradio.app/docs/gradio/plot) takes Matplotlib, Plotly, Vega-Altair, and Bokeh figures. diff --git a/website/data/category_intros/database-drivers.md b/website/data/category_intros/database-drivers.md new file mode 100644 index 00000000..702cf9fa --- /dev/null +++ b/website/data/category_intros/database-drivers.md @@ -0,0 +1,36 @@ +Each database calls for a Python database driver built for it. Reach PostgreSQL through Psycopg, and MySQL through mysqlclient. sqlite3 comes with Python. + +How to choose: + +- PostgreSQL, from sync or async code: Psycopg +- MySQL or MariaDB, Django projects included: mysqlclient +- SQLite with nothing to install: sqlite3 +- MySQL in pure Python: PyMySQL +- An asyncio-only PostgreSQL service: asyncpg +- Loading JSON or CSV into SQLite, from Python or the shell: sqlite-utils +- ClickHouse, even behind proxies and load balancers: clickhouse-connect +- ClickHouse over its native TCP protocol: clickhouse-driver +- Any database with an ODBC driver: pyodbc +- Oracle Database: python-oracledb +- SQL Server or Azure SQL, with no driver manager to set up: mssql-python +- Redis, MongoDB, or Apache Cassandra: redis-py, PyMongo, or cassandra-driver + +Psycopg is the PostgreSQL driver [Django recommends](https://docs.djangoproject.com/en/stable/ref/databases/#postgresql-notes). One design serves sync and async code: [`AsyncConnection` and `AsyncCursor`](https://www.psycopg.org/psycopg3/docs/advanced/async.html#asynchronous-operations) mirror the sync classes, so you mostly add `await`. Open each connection in a [`with` block](https://www.psycopg.org/psycopg3/docs/basic/usage.html#connection-context). It commits when the block ends, rolls back on an exception, and closes the connection either way. + +asyncpg is built for asyncio only, and [skips DB-API on purpose](https://magicstack.github.io/asyncpg/current/faq.html) to fit PostgreSQL better. In a server, [create a pool](https://magicstack.github.io/asyncpg/current/usage.html#connection-pools) with `asyncpg.create_pool()` at startup, then take a connection per request with `async with pool.acquire()`. + +Django calls mysqlclient [the recommended choice](https://docs.djangoproject.com/en/stable/ref/databases/#mysql-db-api-drivers) for MySQL, and ships its own adapter for it. It's a native driver built against MySQL's client library. Where you can't install that library and its headers, use PyMySQL, which SQLAlchemy calls [a pure Python port](https://docs.sqlalchemy.org/en/latest/dialects/mysql.html#mysql-python-compatibility) of MySQLdb that targets full compatibility. The licenses differ too: mysqlclient is [GPL](https://github.com/PyMySQL/mysqlclient/blob/main/LICENSE), and PyMySQL is MIT. + +sqlite3 comes with Python, and SQLite needs no server. Use the connection as a [context manager](https://docs.python.org/3/library/sqlite3.html#sqlite3-connection-context-manager): it commits when the block succeeds and rolls back when it raises. + +sqlite-utils sits on top of sqlite3 to create and fill databases, not to be a full ORM. Pass [`insert_all()`](https://sqlite-utils.datasette.io/en/latest/python-api.html#bulk-inserts) a list of dicts, and it creates the table and columns for you. Its command-line tool pipes JSON or CSV straight into a new database file. + +clickhouse-connect is ClickHouse's official client and [talks HTTP](https://clickhouse.com/docs/integrations/language-clients/python), so it works through load balancers, proxies, and enterprise network controls. clickhouse-driver uses the [native TCP protocol](https://clickhouse-driver.readthedocs.io/en/latest/) instead, which its docs say is more configurable and sends compact binary data. + +pyodbc reaches any database through that database's ODBC driver, plus one [driver manager](https://github.com/mkleehammer/pyodbc/wiki/Drivers-and-Driver-Managers) for all of them. Windows has one built in; on macOS and Unix, you install one first. For SQL Server and Azure SQL, Microsoft's own mssql-python [connects without an external driver manager](https://learn.microsoft.com/en-us/sql/connect/python/mssql-python/python-sql-driver-mssql-python). + +python-oracledb is Oracle's official driver. In its default [Thin mode](https://python-oracledb.readthedocs.io/en/latest/user_guide/initialization.html), it connects straight to Oracle Database without Oracle Client libraries. + +redis-py is [the Python client for Redis](https://redis.io/docs/latest/develop/clients/redis-py/), and MongoDB's docs call PyMongo [the recommended way](https://www.mongodb.com/docs/languages/python/pymongo-driver/current/) to work with MongoDB from Python. cassandra-driver talks to Apache Cassandra over Cassandra's binary protocol, with sync and async APIs. + +With any SQL database, pass values as query parameters instead of formatting them into the SQL string. The sqlite3 docs say this [avoids SQL injection](https://docs.python.org/3/library/sqlite3.html#sqlite3-placeholders). For more per database, see [awesome-mysql](https://github.com/shlomi-noach/awesome-mysql), [awesome-postgres](https://github.com/dhamaniasad/awesome-postgres), and [awesome-sqlite](https://github.com/planetopendata/awesome-sqlite). diff --git a/website/data/category_intros/database.md b/website/data/category_intros/database.md new file mode 100644 index 00000000..408271eb --- /dev/null +++ b/website/data/category_intros/database.md @@ -0,0 +1,22 @@ +Inside your process, a Python database library runs without a server. DuckDB handles analytical SQL there; to search vectors, start with Chroma. + +How to choose: + +- Analytical SQL over Parquet, CSV, or DataFrames: DuckDB +- Vector search in a prototype or notebook: Chroma +- ClickHouse's SQL dialect, or pandas code that runs on ClickHouse: chDB +- Vectors stored with their raw images and text, on disk: LanceDB +- Vector search embedded in an app or an edge device: Zvec +- A small JSON document store for one script: TinyDB + +Start DuckDB with `duckdb.sql()`, which runs on an [in-memory database kept inside the module](https://duckdb.org/docs/current/clients/python/overview#basic-api-usage). It queries Parquet, CSV, and JSON files by name, and [pandas DataFrames, Polars DataFrames, and Arrow tables](https://duckdb.org/docs/current/clients/python/overview#dataframes) by variable name, with no import step. When the data should stay, [pass a file name to `duckdb.connect()`](https://duckdb.org/docs/current/clients/python/overview#persistent-storage). + +chDB runs ClickHouse inside your process, so you get its SQL dialect without a ClickHouse server. If you come from pandas, [start with the DataStore API](https://clickhouse.com/docs/chdb#for-pandas-users): change the import, and your pandas code works unchanged. For SQL, use the [connection-based API](https://clickhouse.com/docs/chdb/install/python#connection-based-api), and query DataFrames with the [`Python()` table engine](https://clickhouse.com/docs/chdb/install/python#python-table-engine-recommended). + +Chroma's [in-memory client](https://docs.trychroma.com/docs/run-chroma/clients) is the quick way to try it in a notebook, and `PersistentClient` keeps the data on disk. Add text documents, and Chroma [embeds and indexes them for you](https://docs.trychroma.com/docs/overview/getting-started), so you can query with text too. + +LanceDB starts as an [embedded library pointed at a local directory](https://docs.lancedb.com/quickstart). Its indexes live on disk, and a table [stores the raw data, metadata, and embeddings together](https://docs.lancedb.com/faq/faq-oss), which suits images and other multimodal data. Its [embedding function registry](https://docs.lancedb.com/embedding) generates vectors as you insert rows, and Python code can then query with text. + +Zvec runs entirely in-process, with no server or daemon, and its docs pitch it [from prototypes to embedded apps and edge devices](https://zvec.org/en/docs/db/). Its [quickstart](https://zvec.org/en/docs/db/quickstart/) defines a schema of scalar and vector fields first, then combines vector search with filters on those scalar fields. + +TinyDB is pure Python with no dependencies, and stores Python dicts in a JSON file: [open it with `TinyDB('db.json')` and search with `Query()`](https://tinydb.readthedocs.io/en/latest/getting-started.html). Its own docs say to [pick something else](https://tinydb.readthedocs.io/en/latest/intro.html#why-not-use-tinydb) when you need access from several processes or threads, indexes, or ACID guarantees. diff --git a/website/data/category_intros/date-and-time.md b/website/data/category_intros/date-and-time.md new file mode 100644 index 00000000..fcaf5a8c --- /dev/null +++ b/website/data/category_intros/date-and-time.md @@ -0,0 +1,21 @@ +Handle time zones with the built-in zoneinfo. Parsing date strings and adding months is where a Python date library earns its install: python-dateutil. + +How to choose: + +- Time zones by IANA name, like Europe/Paris: zoneinfo +- Parsing date strings, adding months, and recurring dates: python-dateutil +- Dates as people write them, like "3 days ago", in many languages: dateparser +- An easier datetime API whose objects are still datetimes: pendulum +- Exact and local times as separate types, with DST-safe math: whenever + +zoneinfo brings the IANA time zone database to Python, and the datetime docs say [its usage is recommended](https://docs.python.org/3/library/datetime.html#tzinfo-objects). Attach a ZoneInfo to a datetime [through the constructor, `replace()`, or `astimezone()`](https://docs.python.org/3/library/zoneinfo.html#using-zoneinfo). Some systems, Windows among them, have no IANA database, so if your code runs across platforms, [declare a dependency on tzdata](https://docs.python.org/3/library/zoneinfo.html#data-sources). + +python-dateutil adds [extensions to the standard datetime module](https://dateutil.readthedocs.io/en/stable/); install it as `python-dateutil` and import it as `dateutil`. Its parser [reads most known formats](https://dateutil.readthedocs.io/en/stable/parser.html) and returns a datetime even for an ambiguous date. For input like 01/05/09, set `dayfirst` or `yearfirst` to match your data. For calendar math, `relativedelta` takes [plural arguments that add and singular ones that replace](https://dateutil.readthedocs.io/en/stable/relativedelta.html): `months=+1` moves a month ahead, and `day=1` jumps to the first. `rrule` builds [recurring dates from iCalendar rules](https://dateutil.readthedocs.io/en/stable/rrule.html). + +dateparser reads relative dates like "two weeks ago" and absolute ones in more than 200 language locales. Its docs say it [stands out](https://dateparser.readthedocs.io/en/latest/#common-use-cases) for scraped pages, logs, and other data from mixed sources, and for letting users type dates in their own words. Call `dateparser.parse()`, and [pass `languages` when you know them](https://dateparser.readthedocs.io/en/latest/#how-to-use), so it skips language detection. When you parse many dates from one source, [use `DateDataParser`](https://dateparser.readthedocs.io/en/latest/usage.html), which remembers the languages it has found. + +pendulum's classes are [drop-in replacements for the native ones](https://pendulum.eustace.io/docs/#introduction), since they inherit from datetime. Every instance is time zone aware and in UTC by default. Its docs call aware datetimes [the preferred and recommended way](https://pendulum.eustace.io/docs/#instantiation) to use it. For tests, install `pendulum[test]` and [travel in time](https://pendulum.eustace.io/docs/#testing). + +whenever puts exact time and local time in [separate types](https://whenever.readthedocs.io/en/latest/guide/choosing-a-type.html): an instant when only the moment matters, a zoned datetime when the local time matters too. Mixing up naive and aware [becomes a type error](https://whenever.readthedocs.io/en/latest/), and DST is handled in all arithmetic. A standard datetime [does no time zone adjustment](https://docs.python.org/3/library/datetime.html#datetime-objects) when you add a timedelta to it. In production, [turn whenever's DST warnings into errors](https://whenever.readthedocs.io/en/latest/faq.html#why-warnings-instead-of-errors) with Python's standard warnings filter. + +Decide whether you extend datetime or replace it. zoneinfo, python-dateutil, and dateparser all use standard datetime objects, so they work together. pendulum's objects are datetimes too, but code that checks the exact type, like sqlite3 and some database drivers, [needs an adapter registered](https://pendulum.eustace.io/docs/#limitations). whenever [doesn't subclass datetime at all](https://whenever.readthedocs.io/en/latest/faq.html#why-no-drop-in-replacement-for-datetime), so [convert to and from standard datetimes](https://whenever.readthedocs.io/en/latest/guide/stdlib-convert.html) where other code needs one. diff --git a/website/data/category_intros/debugging-tools.md b/website/data/category_intros/debugging-tools.md new file mode 100644 index 00000000..53606b86 --- /dev/null +++ b/website/data/category_intros/debugging-tools.md @@ -0,0 +1,36 @@ +Stepping through code in ipdb gets you pdb with IPython's tab completion. Python debugging tools cover slow code too: py-spy profiles a running Python program. + +How to choose: + +- pdb with tab completion and syntax highlighting: ipdb +- Finding where a running program spends its time, without changing it: py-spy +- A full-screen debugger in your terminal: PuDB +- A timeline of every function call: VizTracer +- Memory use and leaks, native extensions included: Memray +- A readable call tree of where your script spends wall-clock time: pyinstrument +- CPU, GPU, and memory use, line by line: Scalene +- SQL queries and request data on Django pages: Django Debug Toolbar +- Print debugging that labels each value: IceCream +- The same toolbar on Flask pages: Flask-DebugToolbar + +ipdb gives you IPython's debugger, with [tab completion, syntax highlighting, and better tracebacks](https://github.com/gotcha/ipdb), behind the same interface as pdb. Drop into it with `import ipdb; ipdb.set_trace()`, or call `ipdb.pm()` for post-mortem debugging after an exception. + +py-spy is a sampling profiler that shows what a program spends its time on [without restarting it or modifying its code](https://github.com/benfred/py-spy). It runs outside the profiled process, which makes it safe to use on production code. `py-spy record` writes a [flame graph](https://github.com/benfred/py-spy#record) of a process ID or a command you hand it, and `py-spy dump` prints [the call stack where a program is hung](https://github.com/benfred/py-spy#dump). + +PuDB keeps the source, the stack, breakpoints, and variables [all visible at once](https://documen.tician.de/pudb/) in one full-screen terminal view. Start it from your code with `from pudb import set_trace; set_trace()`, or [run a whole script](https://documen.tician.de/pudb/starting.html#starting-the-debugger) with `python -m pudb my-script.py`. + +VizTracer records every function's entry and exit time across a whole run, which [helps catch sporadic performance issues](https://viztracer.readthedocs.io/en/latest/). Run your script [with `viztracer my_script.py`](https://viztracer.readthedocs.io/en/latest/basic_usage.html#command-line) instead of `python`, then open the `result.json` it writes with `vizviewer`. + +Memray tracks allocations [in Python code, native extension modules, and the interpreter itself](https://bloomberg.github.io/memray/overview.html), to find memory leaks and the code behind high memory use. Profiling [takes two steps](https://bloomberg.github.io/memray/getting_started.html): `memray run` saves the allocations to a file, then `memray flamegraph` builds a report from it. + +pyinstrument measures wall-clock time, so time spent [downloading data, reading files, and talking to databases](https://pyinstrument.readthedocs.io/en/latest/how-it-works.html#wall-clock-time-not-cpu-time) counts too. Type [`pyinstrument script.py`](https://pyinstrument.readthedocs.io/en/latest/guide.html#profile-a-python-script) instead of `python script.py`, or wrap a block in `with pyinstrument.profile():`. + +Scalene profiles CPU, GPU, and memory [line by line and per function](https://github.com/plasma-umass/scalene#fast-and-accurate). It [separates time in Python from time in native code](https://github.com/plasma-umass/scalene#cpu-profiling), so you can focus on code you can change. Run [`scalene run your_prog.py`](https://github.com/plasma-umass/scalene#using-scalene), then `scalene view` opens the profile in your browser. + +Django Debug Toolbar adds [panels](https://django-debug-toolbar.readthedocs.io/en/latest/panels.html) to your pages for the current request's SQL queries, templates, cache calls, and more. [Its setup](https://django-debug-toolbar.readthedocs.io/en/latest/installation.html) adds the app, its URLs, and its middleware, and shows the toolbar only to IP addresses in `INTERNAL_IPS`. + +IceCream's `ic()` [prints each expression along with its value](https://github.com/gruns/icecream). With no arguments, it prints [the file, line, and function it runs in](https://github.com/gruns/icecream#inspect-execution). Since `ic()` [returns its arguments](https://github.com/gruns/icecream#return-value), you can wrap an expression in existing code without rewriting the line. + +Flask-DebugToolbar is [a port of Django Debug Toolbar](https://github.com/pallets-eco/flask-debugtoolbar) to Flask. Wrap your app in `DebugToolbarExtension(app)`, and the toolbar [shows up in HTML responses while debug mode is on](https://flask-debugtoolbar.readthedocs.io/en/latest/). + +Both debuggers also open from the built-in `breakpoint()`: set the [`PYTHONBREAKPOINT` environment variable](https://docs.python.org/3/using/cmdline.html#envvar-PYTHONBREAKPOINT) to `ipdb.set_trace` or `pudb.set_trace`. Among profilers, sampling ones like py-spy, pyinstrument, and Scalene [cost much less overhead than tracing ones](https://pyinstrument.readthedocs.io/en/latest/how-it-works.html#statistical-profiling-not-tracing), while a tracer like VizTracer records every call and [gives you more information](https://github.com/gaogaotiantian/viztracer#performance). diff --git a/website/data/category_intros/deep-learning.md b/website/data/category_intros/deep-learning.md new file mode 100644 index 00000000..9637db9f --- /dev/null +++ b/website/data/category_intros/deep-learning.md @@ -0,0 +1,23 @@ +PyTorch suits new models, since it debugs like plain Python. Gymnasium environments pass arrays to whichever Python deep learning framework trains your agent. + +How to choose: + +- Building and training neural networks: PyTorch +- RL environments, built-in or your own: Gymnasium +- A standard training loop scaled across GPUs: PyTorch Lightning +- NumPy-style code compiled for GPUs and TPUs: JAX +- One model API across JAX, TensorFlow, and PyTorch: Keras +- An existing TensorFlow codebase: TensorFlow +- Ready-made RL algorithms to train an agent: Stable Baselines3 + +PyTorch runs each line of code as it's reached, so [a debugger and stack traces read the way you'd expect](https://github.com/pytorch/pytorch#imperative-experiences). Its [quickstart](https://docs.pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html) shows the pieces of a project: a `Dataset` holds your samples and labels, and a `DataLoader` wraps it in an iterable. A model subclasses `nn.Module`, defines its layers in `__init__`, and passes data through them in `forward`. To keep a trained model, [save its `state_dict`](https://docs.pytorch.org/tutorials/beginner/saving_loading_models.html) rather than pickling the whole model, which ties the file to your exact classes and directory layout. + +PyTorch Lightning [organizes PyTorch code](https://github.com/Lightning-AI/pytorch-lightning#why-pytorch-lightning) so you write the model logic and it handles the engineering. A `LightningModule` is still an `nn.Module`, with your training step and `configure_optimizers` added, and the `Trainer` runs the loop around it. Moving to more GPUs, to TPUs, or to 16-bit precision is [a Trainer argument](https://github.com/Lightning-AI/pytorch-lightning#advanced-features), with no change to your model code. + +JAX pairs a NumPy-like API with [composable transformations](https://github.com/jax-ml/jax#transformations): `jax.grad` takes gradients, `jax.jit` compiles, and `jax.vmap` vectorizes. They're designed for [functionally pure functions](https://docs.jax.dev/en/latest/notebooks/Common_Gotchas_in_JAX.html#pure-functions), so pass every input as an argument and return every result. The core holds little that's specific to deep learning. [For neural networks](https://docs.jaxstack.ai/en/latest/), add the JAX AI Stack: Flax for models, Optax for optimizers, and Orbax for checkpoints. + +Gymnasium, a fork of OpenAI's Gym, is [a standard API for reinforcement learning](https://github.com/Farama-Foundation/Gymnasium): one interface between learning algorithms and environments, plus reference environments that follow it. Its [basic loop](https://gymnasium.farama.org/introduction/basic_usage/) comes down to `make()`, `Env.reset()`, `Env.step()`, and `Env.render()`, and wrappers change an environment without touching its code. For your own problem, [subclass `gymnasium.Env`](https://gymnasium.farama.org/introduction/create_custom_env/) with observation and action spaces, and register it so `gymnasium.make()` builds it like a built-in one. + +Stable Baselines3 is a set of [reliable implementations of RL algorithms in PyTorch](https://github.com/DLR-RM/stable-baselines3), and it assumes you already know some RL. It follows [a scikit-learn-like syntax](https://stable-baselines3.readthedocs.io/en/master/guide/quickstart.html): create a model such as `A2C("MlpPolicy", env)`, call `learn()`, then `predict()`. It trains on any environment that [follows the Gymnasium interface](https://stable-baselines3.readthedocs.io/en/master/guide/custom_env.html). Its [RL tips](https://stable-baselines3.readthedocs.io/en/master/guide/rl_tips.html) pick the algorithm by action space, since DQN handles only discrete actions and SAC only continuous ones. + +For more frameworks, papers, and courses, see [awesome-deep-learning](https://github.com/ChristosChristofidis/awesome-deep-learning). diff --git a/website/data/category_intros/devops-tools.md b/website/data/category_intros/devops-tools.md new file mode 100644 index 00000000..bcb51f7b --- /dev/null +++ b/website/data/category_intros/devops-tools.md @@ -0,0 +1,50 @@ +Server setup goes in YAML playbooks that Ansible runs, unless you'd rather write Python with pyinfra. Neither Python DevOps tool needs an agent: SSH is enough. + +How to choose: + +- Server setup over SSH: Ansible in YAML, pyinfra in Python +- Cloud APIs: Boto3, Azure SDK for Python, Google Cloud client libraries, and the AWS CLI from your shell +- First-boot setup of a new cloud instance: cloud-init +- Large fleets, with an agent on every host: Salt +- Shell commands on remote servers, run from Python: Fabric +- Serverless apps on AWS Lambda: Chalice +- CPU, memory, disk, and process stats: psutil +- Errors from your app, sent to Sentry: Sentry SDK +- Processes kept running and restarted after a crash: Supervisor +- Celery workers and tasks in a browser: Flower +- Shell programs called as Python functions: sh +- Deduplicated, encrypted backups: BorgBackup + +Ansible is [agentless](https://github.com/ansible/ansible): it uses the SSH daemon your servers already run. You install it on one control node and [declare the desired state](https://docs.ansible.com/projects/ansible/latest/getting_started/introduction.html) of each machine in a playbook. When a machine already matches, Ansible changes nothing, however many times the playbook runs. + +pyinfra calls itself ["ansible but Python instead of YAML"](https://github.com/pyinfra-dev/pyinfra). Its getting-started guide puts your hosts in an `inventory.py` and your operations in a `deploy.py`, then runs `pyinfra inventory.py deploy.py`. The target hosts [need nothing but an SSH daemon](https://docs.pyinfra.com/en/latest/getting-started.html). Most operations describe an end state: pyinfra [checks the host first](https://docs.pyinfra.com/en/latest/using-operations.html) and runs commands only if reality doesn't match. + +Boto3 is the AWS SDK for Python. Its [clients](https://docs.aws.amazon.com/boto3/latest/guide/clients.html) map close to 1:1 onto AWS service APIs and support every service operation. Resources are a higher-level, object-oriented layer on top. + +The AWS CLI gives you [direct access to AWS's public APIs](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) from your terminal, to explore a service or write shell scripts. + +The Azure SDK for Python is a set of per-service packages, so you install only what you use. [Management libraries](https://learn.microsoft.com/en-us/azure/developer/python/sdk/azure-sdk-overview), named `azure-mgmt-*`, create and configure Azure resources, while client libraries work with services already provisioned. + +The Google Cloud client libraries are [the recommended way](https://docs.cloud.google.com/python/docs/reference) to call Google Cloud APIs from code. + +cloud-init handles a new instance's first boot. It [takes an initial configuration you supply](https://docs.cloud-init.io/en/latest/explanation/introduction.html) and applies it when the instance is created, from the hostname and network to user accounts and scripts. [Most clouds and Linux distributions ship it](https://github.com/canonical/cloud-init), so there's nothing to install. Write that configuration as [cloud-config](https://docs.cloud-init.io/en/latest/explanation/format/cloud-config.html), YAML that describes the state you want. + +Salt has [a master managing minions](https://docs.saltproject.io/salt/user-guide/en/latest/topics/overview.html): an agent runs on each host and listens on an event bus. One command can reach thousands of systems in seconds. + +Fabric runs shell commands on remote servers over SSH and [hands you back Python objects](https://www.fabfile.org/). A [`Connection`](https://docs.fabfile.org/en/latest/getting-started.html#run-commands-via-connections-and-run) is the core of its API: `run()` a command on it, and `put()` or `get()` files. For deploys and sysadmin jobs, write `@task` functions in a `fabfile.py` and [run them with `fab -H web1,web2`](https://docs.fabfile.org/en/latest/getting-started.html#addendum-the-fab-command-line-tool). + +Chalice deploys Python apps to AWS Lambda. You write handlers with [a decorator-based API](https://github.com/aws/chalice) for API Gateway, S3, SQS, and other AWS services. Then `chalice deploy` provisions what your app needs, including its IAM policy. + +psutil reads process and system information, like CPU, memory, disks, and network, across platforms. It covers [what tools like `ps`, `top`, and `netstat` show](https://github.com/giampaolo/psutil). + +The Sentry SDK reports your app's errors to Sentry. [Call `sentry_sdk.init()`](https://docs.sentry.io/platforms/python/#initialize-the-sentry-sdk) with your DSN as early as possible, in your app's entry point. Its [integrations](https://docs.sentry.io/platforms/python/integrations/) for Django, Flask, Celery, and many other libraries turn on by themselves. + +Supervisor runs your programs as its subprocesses on UNIX-like systems and [can restart them after a crash](https://supervisord.org/introduction.html). Add a [`[program:x]` section](https://supervisord.org/running.html#adding-a-program) per program to its INI-style config, start `supervisord`, and control processes with `supervisorctl`. + +Flower is Celery's [recommended monitor](https://docs.celeryq.dev/en/stable/userguide/monitoring.html): a web app that shows workers and tasks in real time, and can restart worker pools or shut workers down. Start it with your app's configuration, [`celery -A tasks.app flower`](https://flower.readthedocs.io/en/latest/install.html). + +sh lets you [call any program as if it were a function](https://sh.readthedocs.io/en/latest/): `from sh import git`, then `git("status", "--short")`. It runs the real binary on your `$PATH`, and a non-zero exit code raises an exception like `sh.ErrorReturnCode_2`. + +BorgBackup is a [deduplicating backup program](https://borgbackup.readthedocs.io/en/stable/). It stores only changes, so it suits daily backups, and its authenticated encryption suits backup targets you don't fully trust. Initialize an encrypted repository, create an archive per backup, and prune old ones on a schedule. The repository is useless without its key, so the quick start says to [back up the key](https://borgbackup.readthedocs.io/en/stable/quickstart.html#repository-encryption) and test your backups. + +Keep secrets out of your code and plain files, whichever tool you pick. Let each cloud's SDK find credentials: Boto3 [searches a chain of locations](https://docs.aws.amazon.com/boto3/latest/guide/credentials.html#configuring-credentials), Azure recommends [token-based authentication through Microsoft Entra ID](https://learn.microsoft.com/en-us/azure/developer/python/sdk/authentication/overview#recommended-approach-for-app-authentication), and Google's client libraries pick up [Application Default Credentials](https://docs.cloud.google.com/docs/authentication/application-default-credentials). In config management, encrypt secrets with [Ansible Vault](https://docs.ansible.com/projects/ansible/latest/tips_tricks/ansible_tips_tricks.html#keep-vaulted-variables-safely-visible) or keep them in [Salt's pillar](https://docs.saltproject.io/en/latest/topics/best_practices.html#storing-secure-data). diff --git a/website/data/category_intros/distributed-computing.md b/website/data/category_intros/distributed-computing.md new file mode 100644 index 00000000..10bbf4f8 --- /dev/null +++ b/website/data/category_intros/distributed-computing.md @@ -0,0 +1,22 @@ +From a laptop to a cluster, Python distributed computing comes down to the workload: joblib runs loops, Dask scales pandas, Ray scales ML, and PySpark runs SQL. + +How to choose: + +- Parallel for loops on one machine: joblib +- Scaling pandas, NumPy, or scikit-learn code: Dask +- Training, tuning, and serving ML models, or GPU jobs: Ray +- Your own Python code with stateful workers on a cluster: Ray Core +- ETL and SQL on structured data, or a JVM shop: PySpark +- MPI programs on HPC clusters and supercomputers: mpi4py + +joblib is a [package for parallel computing and disk-based caching](https://joblib.readthedocs.io/en/stable/) that leaves your code as unmodified as possible. You [write a parallel for loop](https://joblib.readthedocs.io/en/stable/user_guide/parallel.html#common-usage) as a generator expression: `Parallel(n_jobs=2)(delayed(sqrt)(i ** 2) for i in range(10))`. By default, joblib runs the calls in separate worker processes. When one machine isn't enough, [switch its backend](https://joblib.readthedocs.io/en/stable/user_guide/parallel.html#setting-up-joblib-s-backend-with-parallel-config), and the same loop runs on a Dask, Ray, or Spark cluster. scikit-learn's [`n_jobs` runs on joblib](https://scikit-learn.org/stable/computing/parallelism.html) too, so it follows the backend you pick. + +Dask [scales pandas, scikit-learn, and NumPy workflows](https://docs.dask.org/en/stable/why.html) with minimal rewriting. A Dask DataFrame is [a collection of pandas dataframes](https://docs.dask.org/en/stable/) on different computers, and Dask Arrays parallelize NumPy. Dask [runs without any setup](https://docs.dask.org/en/stable/deploying.html#local-machine) on your laptop. Its `LocalCluster` follows the same interface as every other Dask cluster manager, so you swap it out when you're ready to scale up. + +Ray is a [unified framework for scaling AI and Python applications](https://docs.ray.io/en/latest/ray-overview/index.html). Its libraries each distribute one ML task: Ray Data, Train, Tune, Serve, and RLlib. Ray Core runs your own code: [decorate a function with `@ray.remote`](https://docs.ray.io/en/latest/ray-core/walkthrough.html), call it with `.remote()`, and fetch the result with `ray.get()`. For workers that keep state between calls, decorate a class the same way to get an actor. Ray runs on one machine with `ray.init()`, and on several nodes once you [deploy a Ray cluster](https://docs.ray.io/en/latest/cluster/getting-started.html). Ray Data [suits GPU workloads for deep learning inference](https://docs.ray.io/en/latest/data/comparisons.html#how-does-ray-data-compare-to-other-solutions-for-offline-inference) better than Spark does, but unlike Spark, it has no SQL interface. + +PySpark is the Python API for Apache Spark, for large-scale data processing. Dask's own comparison suggests Spark [when you prefer SQL, run mostly JVM infrastructure, or want an all-in-one solution](https://docs.dask.org/en/stable/spark.html#reasons-you-might-choose-spark). PySpark's docs [recommend DataFrames over RDDs](https://spark.apache.org/docs/latest/api/python/index.html), so Spark builds the most efficient query for you. Write it in [SQL or the DataFrame API](https://spark.apache.org/docs/latest/api/python/user_guide/sql.html), whichever you think in, and switch between the two as you go. Installing PySpark with pip is [for local use or as a client](https://spark.apache.org/docs/latest/api/python/getting_started/install.html) that connects to a cluster, not for setting up the cluster itself. The `pyspark` package also needs Java. + +mpi4py provides [Python bindings for MPI](https://mpi4py.readthedocs.io/en/stable/), the Message Passing Interface, so your Python code runs across workstations, clusters, and supercomputers. Lowercase methods like `comm.send` pass any picklable Python object, and uppercase ones like `comm.Send` [pass NumPy arrays the fast way](https://mpi4py.readthedocs.io/en/stable/tutorial.html). Run your script with `mpiexec -n 4 python -m mpi4py script.py`, so an unhandled exception [aborts the whole MPI run instead of deadlocking](https://mpi4py.readthedocs.io/en/stable/mpi4py.run.html#exceptions-and-deadlocks). mpi4py runs on an MPI implementation like MPICH or Open MPI, and in production its docs [recommend a custom-built or system-provided one](https://mpi4py.readthedocs.io/en/stable/install.html). + +Start on one machine: parallelism [brings extra complexity and overhead](https://docs.dask.org/en/stable/best-practices.html#start-small), and often you don't need it. When you do scale out, collect results at the end, since Dask's `compute()` and Ray's `ray.get()` both block until the work finishes. Call [`compute()` once](https://docs.dask.org/en/stable/best-practices.html#avoid-calling-compute-repeatedly) rather than in a loop, and [`ray.get()` as late as possible](https://docs.ray.io/en/latest/ray-core/tips-for-first-time.html#tip-1-delay-ray-get). diff --git a/website/data/category_intros/distribution.md b/website/data/category_intros/distribution.md new file mode 100644 index 00000000..fbf1b7f8 --- /dev/null +++ b/website/data/category_intros/distribution.md @@ -0,0 +1,21 @@ +Users without Python installed can still run your app once PyInstaller takes it from Python script to exe. Pyarmor obfuscates the code you ship. + +How to choose: + +- An executable with the least setup: PyInstaller +- Obfuscated scripts, or ones that expire or run on one machine only: Pyarmor +- Your code compiled to C rather than bundled as bytecode: Nuitka +- One file to copy to your servers and laptops: pex +- A Windows installer or macOS disk image along with the executable: cx_Freeze + +PyInstaller [reads your script, finds every module and library it needs](https://pyinstaller.org/en/stable/operating-mode.html), and collects them with the Python interpreter into one folder, or optionally into one executable file. For most programs that takes one command, `pyinstaller myscript.py`, and your users don't need Python installed at all. It [bundles major packages like NumPy and the Qt bindings out of the box](https://github.com/pyinstaller/pyinstaller). The bundle holds compiled `.pyc` files instead of your source code, but those [could in principle be decompiled](https://pyinstaller.org/en/stable/operating-mode.html#hiding-the-source-code) to reveal your logic. + +Pyarmor is a command-line tool for [obfuscating Python scripts, binding them to specific machines, and setting expiration dates](https://pyarmor.readthedocs.io/en/latest/tutorial/getting-started.html#what-s-pyarmor). The obfuscated scripts stay standard `.py` files: `pyarmor gen foo.py` writes them to `dist/` with a runtime package, and you [copy all of `dist/`](https://pyarmor.readthedocs.io/en/latest/tutorial/getting-started.html#distributing-the-obfuscated-script) to the target. For an executable, Pyarmor [packs the obfuscated scripts with PyInstaller](https://pyarmor.readthedocs.io/en/latest/tutorial/obfuscation.html#packing-obfuscated-scripts). Pyarmor is commercial software: its free version [covers scripts that won't make you much money](https://pyarmor.readthedocs.io/en/latest/licenses.html), and anything else needs a paid license. + +Nuitka [translates your Python modules into a C program](https://github.com/Nuitka/Nuitka) that runs on `libpython` the way CPython does, so building takes [a C compiler](https://nuitka.net/user-documentation/user-manual.html#c-compiler). Its homepage pitches that compile step for [protecting your source code](https://nuitka.net/) and for speed. Run it as [`python -m nuitka`](https://nuitka.net/user-documentation/user-manual.html#recommended-way), so you know which interpreter compiles your program. Its default mode still needs Python on the target. For users without Python, build in standalone or onefile mode. + +pex builds `.pex` files, [self-contained executable Python virtual environments](https://docs.pex-tool.org/whatispex.html) in a single zip file, so deploying your app comes down to `scp`. Pass it your requirements as you would to pip, [`pex -r requirements.txt -o my_application.pex`](https://docs.pex-tool.org/buildingpex.html#specifying-requirements), and give it an entry point to run, or the file starts an interpreter. A plain `.pex` runs on the target's own Python; [build it as a scie](https://docs.pex-tool.org/scie.html) to get a native executable with the interpreter inside. + +cx_Freeze [normally produces a folder](https://cx-freeze.readthedocs.io/en/stable/overview.html) with your executable and the shared libraries it needs, and it can also wrap that in a simple Windows installer or a macOS disk image. Keep its options in a [`[tool.cxfreeze]` table in `pyproject.toml`](https://cx-freeze.readthedocs.io/en/stable/setup_script.html), then run `cxfreeze build`. + +Get the folder build working before you switch to a single file, since both [PyInstaller](https://pyinstaller.org/en/stable/operating-mode.html#bundling-to-one-file) and [Nuitka](https://nuitka.net/user-documentation/use-cases.html#standalone-program-distribution) say problems are much easier to debug that way. Then build on each operating system you ship for, as [PyInstaller](https://pyinstaller.org/en/stable/usage.html#supporting-multiple-operating-systems) and [cx_Freeze](https://cx-freeze.readthedocs.io/en/stable/faq.html#freezing-for-other-platforms) both tell you to. diff --git a/website/data/category_intros/documentation.md b/website/data/category_intros/documentation.md new file mode 100644 index 00000000..c0fad335 --- /dev/null +++ b/website/data/category_intros/documentation.md @@ -0,0 +1,21 @@ +Write only docstrings, and pdoc is all the Python documentation generator you need. Write guides too, and build the site with Sphinx or Material for MkDocs. + +How to choose: + +- API docs straight from docstrings, with no configuration: pdoc +- Handwritten docs plus an API reference, as HTML, PDF, and more: Sphinx +- A searchable Markdown site, with no HTML, CSS, or JavaScript to learn: Material for MkDocs +- Architecture diagrams as code, kept in version control: Diagrams +- A Material for MkDocs site, new or existing, on its team's own generator: Zensical + +pdoc [aims to do one thing and do it well](https://pdoc.dev/docs/pdoc.html#what-is-pdoc): API documentation that follows your module hierarchy, with no configuration. Docstrings are Markdown, and it understands Google and numpydoc styles too. Run `pdoc ./demo.py` or `pdoc my_module_name` for a [preview in your browser that reloads](https://pdoc.dev/docs/pdoc.html#quickstart) when you edit the code, then `pdoc ./demo.py -o ./docs` to export the HTML. Its output is self-contained HTML, and for substantially more complex documentation needs, [pdoc's docs recommend Sphinx](https://pdoc.dev/docs/pdoc.html#limitations). + +Sphinx [focuses on handwritten documentation](https://www.sphinx-doc.org/en/stable/usage/quickstart.html) and turns one set of source files into HTML, a PDF via LaTeX, man pages, and more. Its default markup is reStructuredText, and it can [read Markdown through MyST-Parser](https://www.sphinx-doc.org/en/stable/usage/markdown.html). Run `sphinx-quickstart` to set up a source directory with a `conf.py`, then list your pages in the root document's toctree. `make html` builds the site, and `make latexpdf` the PDF. To document your code, [autodoc](https://www.sphinx-doc.org/en/stable/usage/extensions/autodoc.html) pulls in its docstrings, which you mix with your handwritten pages. + +Material for MkDocs is [a documentation framework on top of MkDocs](https://squidfunk.github.io/mkdocs-material/getting-started/), and `pip install mkdocs-material` installs MkDocs with it. You write Markdown and get a searchable static site, with [no HTML, CSS, or JavaScript to know](https://squidfunk.github.io/mkdocs-material/). Run `mkdocs new .`, then [set `site_name`, `site_url`, and `theme: name: material`](https://squidfunk.github.io/mkdocs-material/creating-your-site/#minimal-configuration) in `mkdocs.yml`, and `mkdocs serve` previews the site as you write. For reference docs from docstrings, try [mkdocstrings](https://squidfunk.github.io/mkdocs-material/alternatives/#sphinx) before switching to Sphinx: Material's alternatives page says it builds on MkDocs and adds Sphinx-like functionality. + +Zensical is [a static site generator from the creators of Material for MkDocs](https://zensical.org/docs/get-started/), with the same batteries-included approach. After `pip install zensical`, [`zensical new .`](https://zensical.org/docs/create-your-site/) creates a `docs/` folder, a `zensical.toml` config, and a GitHub Actions workflow. `zensical serve` previews as you write, and `zensical build` writes the static site. It also [builds existing MkDocs projects without changes](https://zensical.org/docs/compatibility/mkdocs/) from their `mkdocs.yml`, and its classic theme variant keeps the Material for MkDocs look. + +Diagrams [draws cloud system architecture in Python code](https://diagrams.mingrammer.com/), so you can track changes to a diagram in version control. It renders with Graphviz, so [install Graphviz first](https://diagrams.mingrammer.com/docs/getting-started/installation), then `pip install diagrams`. Describe the system in a `with Diagram("Web Service", show=False):` block and chain nodes with `>>`. Running `python diagram.py` saves it as a PNG in your working directory. + +Install [Sphinx](https://www.sphinx-doc.org/en/stable/usage/installation.html), [Material for MkDocs](https://squidfunk.github.io/mkdocs-material/getting-started/#with-pip), or [Zensical](https://zensical.org/docs/get-started/#install-with-pip) into your project's virtual environment, as each one's docs recommend. The documentation generators here all write static HTML, so publish it from CI. All four document a GitHub Actions workflow that deploys to GitHub Pages: [Sphinx](https://www.sphinx-doc.org/en/stable/tutorial/deploying.html#publishing-your-html-documentation), [Material for MkDocs](https://squidfunk.github.io/mkdocs-material/publishing-your-site/), [Zensical](https://zensical.org/docs/publish-your-site/), and [pdoc](https://pdoc.dev/docs/pdoc.html#deploying-to-github-pages). diff --git a/website/data/category_intros/email.md b/website/data/category_intros/email.md new file mode 100644 index 00000000..f5140ea2 --- /dev/null +++ b/website/data/category_intros/email.md @@ -0,0 +1,14 @@ +Sending from Django, you learn no new Python email library API: django-anymail runs send_mail(). Async code sends mail through aiosmtplib. + +How to choose: + +- A Django app sending through Amazon SES, Mailgun, Postmark, or another email service: django-anymail +- A Django app that tracks delivery or receives email through that service: django-anymail +- Async code talking to an SMTP server: aiosmtplib +- A script sending from Gmail, or any SMTP account, with attachments: yagmail + +django-anymail extends Django's `django.core.mail` with features email services add, through [one API that doesn't lock your code to one service](https://anymail.dev/en/latest/). [Change Django's email backend to Anymail's backend for your service](https://anymail.dev/en/latest/installation/), and [`send_mail()`, `EmailMessage`, and even `mail_admins()`](https://anymail.dev/en/latest/sending/django_email/) send through it unchanged. Send from a background task, not the view: Anymail calls that a [best practice for sending email from Django](https://anymail.dev/en/latest/tips/transient_errors/), since your views respond faster. To track delivery or receive email, connect the service's webhooks to [Anymail's Django signals](https://anymail.dev/en/latest/sending/tracking/), and [protect them with https and a shared secret](https://anymail.dev/en/latest/tips/securing_webhooks/). + +aiosmtplib is an [async version of smtplib](https://aiosmtplib.readthedocs.io/en/latest/), with similar APIs. Build each message as the standard library's `EmailMessage`, which makes headers easier to work with. Then pass it to the [`send()` coroutine](https://aiosmtplib.readthedocs.io/en/latest/usage.html), which the docs recommend for most use cases. + +yagmail aims to make sending email as simple as possible, with Gmail as its default server. Its [contents argument is guessed](https://yagmail.readthedocs.io/en/latest/usage.html#magical-contents) item by item: a path to a local file becomes an attachment, valid HTML becomes HTML, and anything else is text. For Gmail, sign in with an [app password or OAuth2](https://github.com/kootenpv/yagmail). diff --git a/website/data/category_intros/environment-management.md b/website/data/category_intros/environment-management.md new file mode 100644 index 00000000..599a379c --- /dev/null +++ b/website/data/category_intros/environment-management.md @@ -0,0 +1,15 @@ +Since uv both downloads Python and creates virtual environments, one Python environment manager does the jobs that pyenv and virtualenv split between them. + +How to choose: + +- Python versions and virtual environments from one tool: uv +- A Python API for creating environments, or plugins: virtualenv +- Switching the `python` command between versions, and nothing more: pyenv + +uv [manages Python versions](https://docs.astral.sh/uv/) as well as packages. When a command needs a Python you don't have, uv [downloads it for you](https://docs.astral.sh/uv/guides/install-python/), so you don't need Python installed to get started. A project's virtual environment lives in a `.venv` folder [inside the project, where editors can find it](https://docs.astral.sh/uv/concepts/projects/layout/#the-project-environment); keep it out of version control. Outside a project, `uv venv --python ` [creates an environment](https://docs.astral.sh/uv/pip/environments/) with that version, and downloads it if needed. + +virtualenv [creates isolated Python environments](https://virtualenv.pypa.io/en/latest/), and a subset of it ships with Python as the venv module. Its docs place it [between venv and uv](https://virtualenv.pypa.io/en/latest/explanation.html#virtualenv-vs-venv-vs-uv): faster and more featureful than venv, and still pure Python. Pick it when you need plugins, or a Python API to create environments. It uses the Python it runs under unless you [pass `-p`](https://virtualenv.pypa.io/en/latest/how-to/usage.html#select-a-python-version) with another installed version. As a command-line tool, it [belongs in an isolated environment](https://virtualenv.pypa.io/en/latest/how-to/install.html), not your system Python. + +pyenv does one job: it [lets you switch between multiple versions of Python](https://github.com/pyenv/pyenv), and leaves virtual environments [to you or its pyenv-virtualenv plugin](https://github.com/pyenv/pyenv#in-contrast-with-pythonbrew-and-pythonz-pyenv-does-not). Install it with the [automatic installer](https://github.com/pyenv/pyenv#1-automatic-installer-recommended) or, on macOS, Homebrew, then [set up your shell](https://github.com/pyenv/pyenv#b-set-up-your-shell-environment-for-pyenv) for it. [Choose the version](https://github.com/pyenv/pyenv#switch-between-python-versions) with `pyenv shell` for the session, `pyenv local` for a directory, or `pyenv global` for your user account. [Most versions are built from source](https://github.com/pyenv/pyenv#install-additional-python-versions) as you install them, so install Python's build dependencies first. pyenv [doesn't work on Windows](https://github.com/pyenv/pyenv#windows) outside the Windows Subsystem for Linux, so on Windows, use uv, which [supports Windows](https://docs.astral.sh/uv/). + +Whichever tools you combine, name each project's Python in a `.python-version` file. `pyenv local` [writes that file](https://github.com/pyenv/pyenv#understanding-python-version-selection), and so does `uv python pin`, whose docs [recommend a plain version number](https://docs.astral.sh/uv/concepts/python-versions/#python-version-files) there so other tools can read it. virtualenv [uses the version pyenv selected](https://virtualenv.pypa.io/en/latest/how-to/usage.html#using-version-managers-pyenv-mise-asdf) with no extra configuration. Don't install packages into the Python that came with your operating system, which [often manages its packages itself](https://docs.astral.sh/uv/pip/environments/); a virtual environment keeps yours apart. diff --git a/website/data/category_intros/erp.md b/website/data/category_intros/erp.md new file mode 100644 index 00000000..390c2921 --- /dev/null +++ b/website/data/category_intros/erp.md @@ -0,0 +1,19 @@ +Out of the box, Odoo's CRM, eCommerce, and accounting apps run alone or combine into a full Python ERP. Anything they lack, you add as a module. + +How to choose: + +- Trying Odoo, or customizing it without code: Odoo Online +- Writing your own modules: a source install +- Hosting your own modules in the cloud: Odoo.sh +- Free and open source: Odoo Community +- More features, with support and upgrades: Odoo Enterprise + +Everything in Odoo starts and ends with modules, and [the main user-facing ones are flagged as Apps](https://www.odoo.com/documentation/latest/developer/tutorials/server_framework_101/01_architecture.html#odoo-modules). Odoo Enterprise is [extra modules installed on top of Community](https://www.odoo.com/documentation/latest/developer/tutorials/server_framework_101/01_architecture.html#odoo-editions). [Community is free and open source under the LGPL](https://www.odoo.com/documentation/latest/administration.html#editions). Enterprise is shared source, and its license [ties running it to an Enterprise subscription](https://www.odoo.com/documentation/latest/legal/licenses.html). + +[Odoo Online](https://www.odoo.com/documentation/latest/administration/odoo_online.html) runs in your browser with nothing to install, and handles customizations that need no code. Your own modules need Odoo.sh or your own server. Odoo.sh is Odoo's official cloud platform, and it [builds your modules from a GitHub repository](https://www.odoo.com/documentation/latest/administration/odoo_sh/create_module.html). + +To develop modules, Odoo's developer docs prefer [a source install](https://www.odoo.com/documentation/latest/developer/tutorials/setup_guide.html), which runs Odoo straight from its code. Odoo's logic is [written in Python, and it stores data only in PostgreSQL](https://www.odoo.com/documentation/latest/developer/tutorials/server_framework_101/01_architecture.html#multitier-application). Keep your modules in a directory of their own, and [start the server with `odoo-bin`](https://www.odoo.com/documentation/latest/administration/on_premise/source.html#running-odoo), adding that directory to `--addons-path`. Then work through [Server framework 101](https://www.odoo.com/documentation/latest/developer/tutorials/server_framework_101.html), which builds one module chapter by chapter. + +A module can [add new business logic or change what's already there](https://www.odoo.com/documentation/latest/developer/tutorials/server_framework_101/01_architecture.html#odoo-modules). To change a standard model, extend it from your own module: [model inheritance](https://www.odoo.com/documentation/latest/developer/tutorials/server_framework_101/12_inheritance.html#model-inheritance) adds fields and overrides methods on a model another module defines. Screens work the same way: [view inheritance](https://www.odoo.com/documentation/latest/developer/tutorials/server_framework_101/12_inheritance.html#view-inheritance) applies your extension views on top of the originals instead of overwriting them. + +Before you write a module, [check whether Odoo already covers the case](https://www.odoo.com/documentation/latest/developer/tutorials/server_framework_101/02_newapp.html). A database with custom modules [can't be upgraded until they're ready for the new version](https://www.odoo.com/documentation/latest/administration/upgrade.html), so [cut what duplicates the standard modules](https://www.odoo.com/documentation/latest/developer/howtos/upgrade_custom_db.html#step-1-stop-the-developments). diff --git a/website/data/category_intros/file-format-processing.md b/website/data/category_intros/file-format-processing.md new file mode 100644 index 00000000..e5c52292 --- /dev/null +++ b/website/data/category_intros/file-format-processing.md @@ -0,0 +1,54 @@ +PDFs you merge, split, or encrypt need only pypdf. Which Python file format library you need beyond that depends on whether you read a file or create it. + +How to choose: + +- Merging, splitting, cropping, or encrypting PDFs: pypdf +- ELF binaries and DWARF debug info: pyelftools +- A table you import or export as CSV, JSON, YAML, or Excel: Tablib +- Documents turned into Markdown for an LLM: MarkItDown, or Docling for complex layouts and scans +- Excel: openpyxl to edit a workbook, XlsxWriter to create one with charts and formatting +- Word documents: python-docx +- PowerPoint decks: python-pptx +- Text and images out of PDFs, or pages rendered as images: PyMuPDF, or pdfminer.six for each character's position and font +- PDFs drawn from Python code: ReportLab +- PDFs from HTML and CSS: WeasyPrint +- Markdown to HTML: markdown-it-py for CommonMark, Python-Markdown for its extensions, Mistune for speed +- Config files: tomllib for TOML, PyYAML for YAML + +pypdf is a [pure-Python PDF library](https://pypdf.readthedocs.io/en/stable/) for splitting, merging, cropping, and transforming pages, and it adds passwords too. To merge files, [`append()` each one to a `PdfWriter`](https://pypdf.readthedocs.io/en/stable/user/merging-pdfs.html) and write it out. With [no C dependency](https://pypdf.readthedocs.io/en/stable/meta/comparisons.html), it's easy to install, but it doesn't generate new PDFs. It reads text with `page.extract_text()`, while a scanned page needs [OCR software](https://pypdf.readthedocs.io/en/stable/user/extract-text.html). + +pyelftools [parses ELF files and DWARF debug info](https://github.com/eliben/pyelftools) in pure Python. Pass an open file to [`ELFFile`](https://github.com/eliben/pyelftools/blob/main/doc/user-guide.md), its main entry point, and stay on the high-level API: its guide says to use the low-level structs at your own peril. + +Tablib is a [format-agnostic tabular dataset library](https://tablib.readthedocs.io/en/stable/). `Dataset().load(fh)` [detects the format of the file](https://tablib.readthedocs.io/en/stable/tutorial.html) you load, and `export()` writes the same data as CSV, JSON, or another format. + +MarkItDown converts PDF, Word, Excel, PowerPoint, HTML, and more into Markdown [for LLMs and text analysis](https://github.com/microsoft/markitdown). Its README says it may not be the best option for high-fidelity conversions that people read. Don't pass untrusted input straight to it, and when you only read local files, [call `convert_local()` instead of `convert()`](https://github.com/microsoft/markitdown#security-considerations), which also fetches URLs. + +Docling is for documents where layout matters: it [understands page layout, reading order, and tables, and runs OCR](https://docling-project.github.io/docling/) on scanned PDFs. It converts each file into a `DoclingDocument`, which you export to Markdown. + +openpyxl reads and writes Excel workbooks, so it's the one to use when the file already exists. Install defusedxml when you open spreadsheets from other people, since openpyxl [doesn't guard against XML attacks](https://openpyxl.readthedocs.io/en/stable/#security) by default. + +XlsxWriter [can't read or modify existing files](https://xlsxwriter.readthedocs.io/introduction.html). In return, it supports more Excel features than the alternatives, and its files are close to what Excel itself writes. + +python-docx works on `.docx` files, not the older `.doc`. [Start from a document that holds your styles](https://python-docx.readthedocs.io/en/latest/user/documents.html), headers, and footers, and open it with `Document('template.docx')`: python-docx can only use a style that [the starting document defines](https://python-docx.readthedocs.io/en/latest/user/styles-understanding.html). + +python-pptx [creates and updates PowerPoint files](https://python-pptx.readthedocs.io/en/latest/user/intro.html). A deck's look comes from its theme, slide master, and layouts, so [open a template deck](https://python-pptx.readthedocs.io/en/latest/user/presentations.html) with `Presentation('template.pptx')`. Add slides from its layouts and fill their [placeholders](https://python-pptx.readthedocs.io/en/latest/user/placeholders-understanding.html), which carry the template's formatting. + +PyMuPDF is a [high-performance library](https://pymupdf.readthedocs.io/en/latest/) for extracting, converting, and editing PDFs and other documents, built on MuPDF, and it also renders pages to images. It's [available under the AGPL or a commercial license](https://pymupdf.readthedocs.io/en/latest/about.html#license-and-copyright). For reading text and editing pages without that condition, use pypdf or pdfminer.six. + +ReportLab [creates PDFs directly from Python](https://docs.reportlab.com/reportlab/userguide/ch1_intro/), like reports generated on a web server, with charts and tables. Lay out documents with Platypus: [create a document from a `DocTemplate` class and pass a list of flowables](https://docs.reportlab.com/reportlab/userguide/ch5_platypus/), such as paragraphs and tables, to its `build()` method. + +pdfminer.six [extracts text from a PDF's source](https://github.com/pdfminer/pdfminer.six), with the exact location, font, and color of each piece, but it can't write PDFs. The simplest start is [`extract_text()`](https://pdfminersix.readthedocs.io/en/latest/tutorial/highlevel.html). + +WeasyPrint turns HTML and CSS into PDFs, like [reports, invoices, and tickets](https://doc.courtbouillon.org/weasyprint/stable/), with a layout engine built for pagination instead of a browser engine. It runs [no JavaScript](https://doc.courtbouillon.org/weasyprint/stable/going_further.html). Call `HTML(...).write_pdf()`. If users supply the HTML or CSS, its docs say you'll need [extra configuration](https://doc.courtbouillon.org/weasyprint/stable/first_steps.html#security) against high memory use, endless renderings, and local file leaks. + +markdown-it-py [follows the CommonMark spec](https://markdown-it-py.readthedocs.io/en/latest/), and plugins add syntax or replace existing rules. + +Python-Markdown implements John Gruber's original Markdown. It's [not a CommonMark implementation](https://python-markdown.github.io/), and its docs say to look elsewhere if you want one. Pick it for its extension API and bundled [extensions](https://python-markdown.github.io/extensions/), like tables, fenced code blocks, and a table of contents. + +Mistune is a [fast parser with renderers and plugins](https://mistune.lepture.com/en/latest/), compatible with "sane CommonMark rules". + +tomllib comes with Python and parses TOML, but doesn't write it. With untrusted input, [limit its size](https://docs.python.org/3/library/tomllib.html), since a malicious TOML string can use a lot of CPU and memory. + +PyYAML parses and emits YAML. Load it with `yaml.safe_load()`, since [`yaml.load` can call any Python function](https://pyyaml.org/wiki/PyYAMLDocumentation) when the data comes from an untrusted source. + +Markdown your users write can carry HTML, so give each parser its safe setting. With markdown-it-py, use the [`js-default` preset](https://markdown-it-py.readthedocs.io/en/latest/security.html), which its docs strongly recommend for user content in web apps. Python-Markdown [doesn't sanitize its output](https://python-markdown.github.io/sanitization/), so run a sanitizer over the HTML. With Mistune, build your parser with [`mistune.create_markdown()`](https://mistune.lepture.com/en/latest/guide.html), which escapes HTML, instead of `mistune.html()`, which doesn't. diff --git a/website/data/category_intros/file-manipulation.md b/website/data/category_intros/file-manipulation.md new file mode 100644 index 00000000..ea7aeb0c --- /dev/null +++ b/website/data/category_intros/file-manipulation.md @@ -0,0 +1,19 @@ +Without installing anything, two Python file manipulation libraries cover paths and file types by name: pathlib and mimetypes. python-magic reads the bytes. + +How to choose: + +- Building and joining paths, and listing directories: pathlib +- A MIME type from a file name or URL: mimetypes +- A file's type from its content, like the Unix `file` command: python-magic +- Rerunning or reloading code when files change: watchfiles +- Your own handler for each created, modified, or moved file: watchdog + +If you aren't sure which pathlib class you need, the docs say [`Path` is most likely it](https://docs.python.org/3/library/pathlib.html): it makes a concrete path for the platform your code runs on. [Join paths with the `/` operator](https://docs.python.org/3/library/pathlib.html#basic-use), and list a directory with `iterdir()` or `glob()`. + +mimetypes [maps a file name's extension to a MIME type](https://docs.python.org/3/library/mimetypes.html), and a MIME type back to extensions. A type guess returns 2 values: a type for the Content-Type header, and an encoding like gzip for the Content-Encoding header. The type is [`None` for a missing or unknown suffix](https://docs.python.org/3/library/mimetypes.html#mimetypes.guess_type). + +python-magic wraps libmagic, which [identifies file types by checking their headers](https://github.com/ahupp/python-magic), the way the Unix `file` command does. It's a thin wrapper, so install libmagic too, with `apt-get install libmagic1` or `brew install libmagic`. Call `magic.from_file(path, mime=True)` for a MIME type, or `magic.from_buffer()` on bytes you already have. + +watchfiles is built for [file watching and code reload](https://watchfiles.helpmanual.io/), and its Rust core groups changes into batches instead of firing once per file. `watch()` is a generator that [yields sets of changes](https://watchfiles.helpmanual.io/api/watch/#watchfiles.watch), and `awatch()` is the async version. To restart code when files change, use [`run_process()`](https://watchfiles.helpmanual.io/api/run_process/#watchfiles.run_process) with a function or a command. From a shell, the [`watchfiles` CLI](https://watchfiles.helpmanual.io/cli/) does the same: `watchfiles --filter python 'pytest --lf' src tests`. + +watchdog gives you [an API and a shell tool](https://python-watchdog.readthedocs.io/en/latest/) for monitoring directories. Its [quickstart](https://python-watchdog.readthedocs.io/en/latest/quickstart.html) has you subclass `FileSystemEventHandler`, override methods like `on_created()` and `on_modified()`, schedule the handler on an `Observer`, and start that thread. An observer skips subdirectories unless you pass `recursive=True` to `schedule()`. diff --git a/website/data/category_intros/functional-programming.md b/website/data/category_intros/functional-programming.md new file mode 100644 index 00000000..6374a009 --- /dev/null +++ b/website/data/category_intros/functional-programming.md @@ -0,0 +1,21 @@ +Beyond functools, install more-itertools, as the itertools docs suggest. toolz is a fuller Python functional programming library, and returns adds typed errors. + +How to choose: + +- Partial application, decorators, and caching: functools +- More iterator tools, the itertools recipes included: more-itertools +- Composing functions into pipelines, currying, and dict helpers: toolz, or cytoolz for speed +- Everyday helpers for collections, decorators, retries, and debugging: funcy +- Errors and missing values as typed containers checked by mypy: returns + +functools is the standard library's module [for higher-order functions](https://docs.python.org/3/library/functools.html), functions that act on or return other functions. Python's Functional Programming HOWTO calls `partial()` [the most useful tool in the module](https://docs.python.org/3/howto/functional.html#the-functools-module): it fills in some of a function's arguments and gives you a new function. When you write a decorator, wrap its inner function with [`wraps`](https://docs.python.org/3/library/functools.html#functools.wraps), so the decorated function keeps its name and docstring. The same HOWTO finds many uses of `reduce()` [clearer as a `for` loop](https://docs.python.org/3/howto/functional.html#small-functions-and-the-lambda-expression). + +The itertools docs point you to more-itertools for [their recipes and many more](https://docs.python.org/3/library/itertools.html#itertools-recipes). It collects [building blocks beyond itertools](https://more-itertools.readthedocs.io/en/stable/), for grouping, windowing, lookahead, and more. The itertools recipes sit in its top-level package, so `from more_itertools import flatten` works. + +toolz [extends itertools and functools](https://toolz.readthedocs.io/en/latest/) with functions that are composable, pure, and lazy, and its API [follows Clojure's standard library](https://toolz.readthedocs.io/en/latest/heritage.html). Each function takes and returns only iterables, dictionaries, and functions, so they [compose to solve your own problems](https://toolz.readthedocs.io/en/latest/composition.html). [`pipe`](https://toolz.readthedocs.io/en/latest/api.html#toolz.functoolz.pipe) runs a value through a sequence of functions, like pipes in Unix. Stick with `partial` at first, and once it shows up several times in your code, [switch to the `toolz.curried` namespace](https://toolz.readthedocs.io/en/latest/curry.html#curry). toolz is a general-purpose library, and for data analytics its docs say [a library built for it](https://toolz.readthedocs.io/en/latest/streaming-analytics.html#disclaimer) may serve you better. [cytoolz](https://github.com/pytoolz/cytoolz) implements the same API in Cython, as a drop-in replacement when you need more speed. + +funcy is a collection of functional tools [focused on practicality](https://github.com/Suor/funcy), inspired by Clojure and underscore. Next to sequence tools, it has [collection functions that keep the type](https://funcy.readthedocs.io/en/stable/overview.html) of a dict or set. It also has control flow helpers, like `@retry` and `silent`, and debugging helpers, like `tap` and `log_calls`. Many of its functions take a regex, a mapping, or a set [where you'd pass a function](https://funcy.readthedocs.io/en/stable/extended_fns.html#extended-function-semantics). + +returns puts results in typed containers: [`Maybe` for None and `Result` for exceptions](https://returns.readthedocs.io/en/latest/pages/quickstart.html#why), plus `IO` for impure code and `Future` for async code. Its docs [really recommend mypy](https://returns.readthedocs.io/en/latest/pages/quickstart.html#typechecking-and-other-integrations), and typing [only works correctly with its mypy plugin](https://returns.readthedocs.io/en/latest/pages/result.html). So it fits projects that check types with mypy. Turn functions that raise into ones that return a `Result` with [`@safe`](https://returns.readthedocs.io/en/latest/pages/result.html#safe), and chain the steps with [`flow`](https://returns.readthedocs.io/en/latest/pages/pipeline.html#flow), which its docs call the recommended way to write code with returns. + +Mix functional style with the rest of your code: Python's HOWTO says functional-style programs usually [give a functional-appearing interface](https://docs.python.org/3/howto/functional.html) and use non-functional features inside. For a plain map or filter, toolz's own docs call comprehensions [more Pythonic](https://toolz.readthedocs.io/en/latest/streaming-analytics.html). Learn a core set of functions; toolz says [about a dozen covers most tasks](https://toolz.readthedocs.io/en/latest/control.html), and the right word only helps when your readers know it too. diff --git a/website/data/category_intros/game-development.md b/website/data/category_intros/game-development.md new file mode 100644 index 00000000..5638cc3e --- /dev/null +++ b/website/data/category_intros/game-development.md @@ -0,0 +1,20 @@ +2D games need a game loop: pygame-ce has you write it; Arcade runs it. Ren'Py is built for visual novels, as Panda3D is for 3D in Python game development. + +How to choose: + +- A game loop you write and control yourself: pygame-ce +- A game loop, physics, and screen switching built in: Arcade +- 3D games, visualizations, and simulations: Panda3D +- Visual novels and other writing-heavy games: Ren'Py +- Windowing, OpenGL graphics, and media with no dependencies: pyglet +- Code already written for pygame: pygame-ce + +pygame-ce installs as `pygame-ce` and still imports as `pygame`. It [gives you full control of program execution](https://pyga.me/docs/): you write the loop that handles events, draws the frame, and flips the display. pygame-ce is LGPL, which [still allows closed-source and commercial games](https://github.com/pygame-community/pygame-ce); only changes to pygame-ce itself must be released under a compatible license. + +Arcade is built on pyglet, and it's [ideal for beginning programmers](https://github.com/pythonarcade/arcade) or anyone who wants 2D games without learning a complex framework. Give each screen its own `arcade.View` subclass that overrides `on_draw()` and `on_update()`, then [hand control to `arcade.run()`](https://api.arcade.academy/en/stable/tutorials/views/index.html). Draw sprites through a `SpriteList`, [the only way to draw a sprite](https://api.arcade.academy/en/stable/programming_guide/performance_tips.html), which batches them for you. For movement and collisions, start with the [built-in physics engines](https://api.arcade.academy/en/stable/api_docs/api/physics_engines.html): a simple one for top-down games, and a platformer one with gravity and moving platforms. + +Panda3D is a C++ engine scripted in Python, and [its manual says you must be a skilled programmer](https://docs.panda3d.org/latest/python/introduction/index) to use it. Subclass `ShowBase`, which opens the 3D window, and [call `run()` once, as the last line](https://docs.panda3d.org/latest/python/introduction/tutorial/starting-panda3d). A model shows up only once it's [attached to the scene graph](https://docs.panda3d.org/latest/python/programming/scene-graph/index), a tree rooted at `render`. + +To use Ren'Py, you [download it and work in its launcher](https://www.renpy.org/doc/html/quickstart.html), which creates, edits, and runs your projects. Write the story in its script language, and reach for [its screen language and Python](https://www.renpy.org/why.html) when you need custom interfaces or game logic. + +pyglet is a windowing and multimedia library [with no external dependencies](https://pyglet.readthedocs.io/en/latest/), for games and other visually rich apps. Attach handlers like `on_draw()` with `@window.event` and call `pyglet.app.run()`, since [writing your own event loop is generally not necessary](https://pyglet.readthedocs.io/en/latest/programming_guide/quickstart.html). Draw many sprites in [a `Batch`, which the docs strongly recommend](https://pyglet.readthedocs.io/en/latest/programming_guide/image.html). diff --git a/website/data/category_intros/geolocation.md b/website/data/category_intros/geolocation.md new file mode 100644 index 00000000..3c5757c3 --- /dev/null +++ b/website/data/category_intros/geolocation.md @@ -0,0 +1,20 @@ +Addresses become coordinates with geopy, whose Python geolocation API covers many geocoding services. GeoPandas analyzes map data, and GeoDjango serves it. + +How to choose: + +- Addresses to coordinates, and coordinates back to addresses: geopy +- The distance between two points: geopy +- Geographic data in tables and files, like shapefiles and GeoJSON: GeoPandas +- Web apps that store and query geographic data: GeoDjango +- A visitor's country or city from their IP address, in a Django project: GeoDjango +- Building, encoding, and validating GeoJSON objects by hand: geojson + +geopy is [a client for geocoding services, not a service itself](https://geopy.readthedocs.io/en/stable/#geopy-is-not-a-service), and each service has its own terms of use, quotas, and pricing. Every geocoder has a `geocode()` method that turns an address into a location, and most also have `reverse()` for the other way around. For OpenStreetMap's Nominatim, [set a `user_agent` that names your app](https://geopy.readthedocs.io/en/stable/#geopy.geocoders.Nominatim) and follow [its usage policy](https://operations.osmfoundation.org/policies/nominatim/). The policy bans heavy use, auto-complete, and systematic queries. It also asks you to cache results and show attribution, and it discourages bulk geocoding. To geocode a DataFrame, [wrap the call in `RateLimiter`](https://geopy.readthedocs.io/en/stable/#usage-with-pandas), which adds delays between requests and retries failed ones. Check first that your service allows bulk requests at all. For distances, `geopy.distance.distance` [computes the geodesic distance](https://geopy.readthedocs.io/en/stable/#module-geopy.distance) between two points. + +GeoPandas adds geometry columns to pandas, so you can [do in Python what would otherwise take a spatial database such as PostGIS](https://geopandas.org/en/stable/). Load a shapefile, GeoJSON, or GeoPackage with `read_file()`, which [detects the file type and returns a GeoDataFrame](https://geopandas.org/en/stable/getting_started/introduction.html), and write it back with `to_file()`. For a table of latitudes and longitudes, [build the points with `points_from_xy()`](https://geopandas.org/en/stable/gallery/create_geopandas_from_pandas.html) and set `crs="EPSG:4326"`. To geocode a column of addresses, GeoPandas [calls geopy for you](https://geopandas.org/en/stable/docs/user_guide/geocoding.html), so the geocoding service's terms still apply. + +GeoDjango is [a contrib module that ships with Django](https://docs.djangoproject.com/en/stable/ref/contrib/gis/tutorial/). It adds model fields for geometries, spatial queries to the ORM, and geometry editing to the admin. Its tutorial assumes you already know Django. Run it on PostGIS, which its docs recommend as [the most mature and feature-rich open source spatial database](https://docs.djangoproject.com/en/stable/ref/contrib/gis/install/#spatial-database). To load a shapefile, [`ogrinspect` writes the model and a `LayerMapping` imports the rows](https://docs.djangoproject.com/en/stable/ref/contrib/gis/tutorial/#importing-spatial-data). For IP geolocation, [GeoIP2](https://docs.djangoproject.com/en/stable/ref/contrib/gis/geoip2/) looks up a country or city in a MaxMind or DB-IP database file you download. + +geojson has [a class for every object in the GeoJSON spec](https://github.com/jazzband/geojson), and `geojson.dumps()` and `geojson.loads()` [wrap the standard json functions](https://github.com/jazzband/geojson#geojson-encodingdecoding) to encode and decode them. Check an object with [its `is_valid` property and `errors()` method](https://github.com/jazzband/geojson#validation). To encode your own classes the same way, [give them a `__geo_interface__`](https://github.com/jazzband/geojson#custom-classes), which GeoPandas' `from_features()` [also accepts](https://geopandas.org/en/stable/docs/reference/api/geopandas.GeoDataFrame.from_features.html). + +Latitude and longitude are angles, so measure distance and area in a projected coordinate system, in meters or feet. In GeoPandas, you [always need one](https://geopandas.org/en/stable/getting_started/introduction.html#Projections) for those operations, so reproject with `to_crs()` first. GeoDjango calls choosing a geometry field's SRID [an important decision](https://docs.djangoproject.com/en/stable/ref/contrib/gis/model-api/#selecting-an-srid), since projected systems ease distance calculations. geopy's `distance` works on an ellipsoidal model of the earth, so it takes latitudes and longitudes as they are. diff --git a/website/data/category_intros/gui-development.md b/website/data/category_intros/gui-development.md new file mode 100644 index 00000000..fcf1dc83 --- /dev/null +++ b/website/data/category_intros/gui-development.md @@ -0,0 +1,40 @@ +Desktop apps with room to grow start on Qt, through PySide6. A small tool needs no third-party Python GUI library, since tkinter is in the standard library. + +How to choose: + +- A desktop app that will grow, even a closed-source one: PySide6 +- A small tool, built with the standard library: tkinter +- GNOME apps, or anything else on GTK: PyGObject +- Fast, GPU-drawn tools and live plots: Dear PyGui +- Touch UIs that also run on Android and iOS: Kivy +- Native widgets on Windows, macOS, and Linux: wxPython +- Native widgets on desktop and mobile: Toga +- A Qt app under the GPL, or with a commercial license: PyQt6 +- A modern look and dark mode for tkinter apps: CustomTkinter +- Your own HTML, CSS, and JavaScript in a desktop window: pywebview +- A pure Python UI in the browser or a desktop window: NiceGUI +- One Python codebase for desktop, mobile, and web: Flet + +PySide6 is [the official Python binding for Qt](https://doc.qt.io/qtforpython-6/), under the LGPL, which [lets your app's code stay closed](https://www.qt.io/licensing/open-source-lgpl-obligations) as long as you meet its terms. Build the UI with Qt Widgets for an app that [looks native](https://doc.qt.io/qtforpython-6/faq/whatisqt.html#widgets), or with QML, which was [first made for mobile apps](https://doc.qt.io/qtforpython-6/faq/whatisqt.html#qml) and runs on desktop too. To lay out forms visually, draw them in Qt Widgets Designer, and pyside6-uic [turns the .ui files into Python](https://doc.qt.io/qtforpython-6/tutorials/basictutorial/uifiles.html). + +PyQt6 binds the same Qt, and PySide6 [aims to be API compatible with PyQt](https://doc.qt.io/qtforpython-6/considerations.html#api-changes), with some exceptions. The license is what splits them: PyQt6 is [GPL or commercial, never LGPL](https://www.riverbankcomputing.com/static/Docs/PyQt6/introduction.html#license), so with the GPL version your own code must use a compatible license. + +tkinter is part of the standard library. Use the themed `tkinter.ttk` widgets, which [adapt to each platform's native theme](https://docs.python.org/3/library/tkinter.ttk.html). Most tutorials online still teach the old API, so [the docs point you to TkDocs](https://docs.python.org/3/library/tkinter.html), which teaches the modern one. + +CustomTkinter widgets [work like normal Tkinter widgets](https://github.com/TomSchimansky/CustomTkinter) and mix with them, and they follow the system's light or dark mode. + +PyGObject is [the way to go](https://pygobject.gnome.org/) for a GNOME app or any GTK app. Start with the GNOME Developer Documentation's [beginner lessons](https://developer.gnome.org/documentation/tutorials/beginners.html). In code, subclass `Gtk.Application` and build your window in `do_activate()`, as [the getting-started example](https://pygobject.gnome.org/getting_started.html) does. + +Dear PyGui is built on Dear ImGui and [draws in immediate mode on your GPU](https://github.com/hoffstadt/DearPyGui), for [scientific, engineering, and data science apps](https://dearpygui.readthedocs.io/en/latest/) that need fast, interactive interfaces. Every main script [runs the same steps](https://dearpygui.readthedocs.io/en/latest/tutorials/first-steps.html#first-run): create the context and the viewport, set up, show the viewport, start, then destroy the context. + +Kivy was built [from scratch for multi-touch](https://kivy.org/doc/stable/philosophy.html), and the same code runs on Windows, macOS, Linux, Android, and iOS. Subclass `App` and [return your root widget from `build()`](https://kivy.org/doc/stable/guide/basic.html#create-an-application). As the UI grows, move it into the KV language, which [separates your app's logic from its interface](https://kivy.org/doc/stable/guide/lang.html#concept-behind-the-language). + +wxPython wraps the wxWidgets C++ library and, in most cases, [uses each platform's native widgets](https://wxpython.org/pages/overview/#what-is-wxpython). Its [Hello World](https://wxpython.org/pages/overview/#hello-world) subclasses `wx.Frame`, puts the widgets on a `wx.Panel`, and lays them out with a sizer. + +Toga [uses native system widgets, not themes](https://toga.beeware.org/en/latest/about/philosophy/#native-widgets-not-themes), on macOS, Windows, Linux, Android, and iOS. It's part of BeeWare, whose [tutorial](https://tutorial.beeware.org/) builds an app with Toga. + +pywebview [shows your HTML, CSS, and JavaScript in a native window](https://pywebview.flowrl.com/). It doesn't bundle a GUI toolkit or web renderer, so a frozen app [stays small](https://pywebview.flowrl.com/guide/). Point the window at a web server you already run, or call Python from JavaScript through its [JS API bridge](https://pywebview.flowrl.com/guide/architecture.html#js-api-with-internal-http-server) and serve static files from the built-in HTTP server. + +NiceGUI keeps [all UI logic in Python](https://github.com/zauberzeug/nicegui#architecture) on a FastAPI backend and handles the web details for you. The UI shows up in your browser, or in a native desktop window. + +Flet makes [web, desktop, and mobile apps](https://flet.dev/docs/) without HTML, CSS, or JavaScript, and [builds them with Flutter](https://flet.dev/docs/publish/). `flet run` [opens your app](https://github.com/flet-dev/flet) in a desktop window, and `flet run --web` in the browser. As the app gets more interactive, its docs [favor the declarative style](https://flet.dev/docs/cookbook/declarative-vs-imperative/), where the UI is derived from your app's state. diff --git a/website/data/category_intros/hardware.md b/website/data/category_intros/hardware.md new file mode 100644 index 00000000..6b1ac391 --- /dev/null +++ b/website/data/category_intros/hardware.md @@ -0,0 +1,19 @@ +Serial ports open with pySerial. Past those, pick a Python hardware library by the device you talk to, like Bleak for Bluetooth LE sensors. + +How to choose: + +- Serial ports on Windows, macOS, Linux, or BSD: pySerial +- Bluetooth Low Energy devices, like sensors: Bleak +- Controlling or monitoring the keyboard and mouse: pynput +- Global hotkeys: pynput +- Automated tests on real or virtual hardware, shared across a team or CI: Jumpstarter + +pySerial is imported as `serial`, and `serial.Serial` opens a port as a context manager. [Set a timeout when you open it](https://pyserial.readthedocs.io/en/latest/shortintro.html#readline), or `readline()` can block forever when no newline arrives. The docs also suggest [`serial_for_url()`](https://pyserial.readthedocs.io/en/latest/pyserial_api.html#serial.serial_for_url) over creating `Serial` directly, so the same code takes a local port, a remote one, or a `loop://` loopback for your tests. + +pynput sends input through a `Controller` and watches it through a `Listener`, for the keyboard and the mouse alike. A listener is a thread that runs your callbacks. On some platforms, notably Windows, a slow callback risks freezing input for every process, so [keep callbacks short](https://pynput.readthedocs.io/en/latest/keyboard.html#the-keyboard-listener-thread): hand events to a queue, and let another thread work through it. + +Bleak is async, so [call `asyncio.run()` once](https://bleak.readthedocs.io/en/latest/troubleshooting.html#calling-asyncio-run-more-than-once) and do all your Bluetooth work in one async main function. Scan with `BleakScanner` and connect with `BleakClient`, each [in an `async with` block](https://bleak.readthedocs.io/en/latest/api/client.html#connecting-and-disconnecting), so the device disconnects when the block exits. + +Jumpstarter puts your device behind an exporter, which manages its interfaces, and your tests reach it through a client. [Local mode](https://jumpstarter.dev/main/introduction/#local-mode) needs no Kubernetes or other infrastructure: run `jmp shell --exporter `, then pytest inside it. When teams share hardware, especially in CI, distributed mode adds a Kubernetes-based controller that leases devices out. Write tests as subclasses of [`JumpstarterTest`](https://jumpstarter.dev/main/getting-started/guides/examples/testing.html), whose `client` fixture handles the connection. + +Jumpstarter builds on the other picks: its serial driver runs on pySerial and takes the port [in pySerial's format](https://jumpstarter.dev/main/reference/package-apis/drivers/pyserial.html), and its Bluetooth LE driver runs on Bleak. diff --git a/website/data/category_intros/html-manipulation.md b/website/data/category_intros/html-manipulation.md new file mode 100644 index 00000000..84819777 --- /dev/null +++ b/website/data/category_intros/html-manipulation.md @@ -0,0 +1,21 @@ +Scraped HTML, however broken, parses in Beautiful Soup on lxml's parser. JustHTML packs a sanitizer into a Python HTML library and runs it by default. + +How to choose: + +- Pulling data out of scraped pages: Beautiful Soup, on lxml's parser +- Speed, XPath, or XML documents: lxml +- Sanitizing HTML your users submit: JustHTML +- XML you'd rather handle like JSON: xmltodict +- Escaping text you put into HTML: MarkupSafe + +Beautiful Soup is for [pulling data out of HTML and XML files](https://www.crummy.com/software/BeautifulSoup/bs4/doc/). It sits on a parser you pick and gives you one way to navigate, search, and change the tree. Its docs [recommend lxml as that parser](https://www.crummy.com/software/BeautifulSoup/bs4/doc/#installing-a-parser) for speed. Name the parser in the constructor, as in `BeautifulSoup(markup, "lxml")`. [Different parsers build different trees](https://www.crummy.com/software/BeautifulSoup/bs4/doc/#differences-between-parsers) from the same broken page, and your script should parse it the same way on every machine. The docs also say when to skip Beautiful Soup: [work directly atop lxml](https://www.crummy.com/software/BeautifulSoup/bs4/doc/#improving-performance) when computer time costs more than programmer time, and [parse with lxml](https://www.crummy.com/software/BeautifulSoup/bs4/doc/#css-selectors-through-the-css-property) when CSS selectors are all you need. + +lxml is [a Pythonic binding for libxml2 and libxslt](https://lxml.de/): the speed and XML features of those C libraries, with an API mostly compatible with ElementTree's. For HTML, use [lxml.html](https://lxml.de/lxmlhtml.html), which adds HTML-specific methods to lxml's elements, and every element takes `.xpath()`. On broken pages, lxml's docs say you often need only [Beautiful Soup's encoding detection](https://lxml.de/lxmlhtml.html#really-broken-pages). Leave the rest to lxml's own parser, which is several times faster. XPath has the same injection problem as SQL. When a value comes from outside, [pass it as an XPath variable](https://lxml.de/FAQ.html#how-do-i-use-lxml-safely-as-a-web-service-endpoint) instead of formatting it into the expression. + +JustHTML [parses HTML like a browser](https://github.com/EmilStenstrom/justhtml), including broken markup. `JustHTML(html)` [sanitizes by default](https://emilstenstrom.github.io/justhtml/sanitization.html) against a strict allowlist. The output is safe in a page body, but [not automatically safe inside a ` + + {% block extra_head %}{% endblock %} + + + + + {% block header %}{% endblock %} +
+ + {% block content %}{% endblock %} +
+ + + + + + diff --git a/website/templates/category.html b/website/templates/category.html new file mode 100644 index 00000000..3d59b1b3 --- /dev/null +++ b/website/templates/category.html @@ -0,0 +1,358 @@ +{% extends "base.html" %} +{% block title %}{{ category_title }}{% endblock %} +{% block description %}{{ category_description }}{% endblock %} +{% block canonical_url %}{{ category_url }}{% endblock %} +{% block alternate_links %}{% endblock %} +{% block extra_head %} + +{% endblock %} +{% block header %} +
+ + + +
+ + +
+ {% if parent_category %} +

+ {{ parent_category.name }} +

+ {% endif %} +

{{ category.name }}

+ {% if category.description_html %} +

{{ category.description_html | safe }}

+ {% endif %} + {% if intro_html %} +
{{ intro_html | safe }}
+ {% endif %} + {% set named_groups = entry_groups | selectattr("name") | list %} + {% if (named_groups and not group_categories) or guide_html %} + + {% endif %} +
+ + {% if group_categories %} + + {% endif %} +
+
+{% endblock %} +{% block content %} +{% macro entry_rows(entry, index) %} + {# On section and subcategory pages the group heading already names the row's use case #} + {% set grouped_page = entry_groups and not group_categories %} + {% set section_path = "/categories/" ~ parent_category.slug ~ "/" if parent_category else current_path %} + + {{ index }} + + {{ entry.name }} + {% if entry.subcategories %}{{ entry.subcategories[0].name }}{% + else %}{{ category.name }}{% endif %} + + + {% if entry.downloads is not none %}{{ + "{:,}".format(entry.downloads) }}{% elif entry.source_type == + 'Stdlib' %}Stdlib{% elif + entry.bundled %}Bundled{% elif + entry.badge %}{{ entry.badge }}{% + else %}Not on PyPI{% endif %} + + + {% if entry.stars is not none %}{{ "{:,}".format(entry.stars) }}{% + elif entry.source_type %}{{ entry.source_type }}{% else %}—{% endif %} + + + {% if entry.last_commit_at %}{% else %}—{% endif %} + + + {% for subcat in entry.subcategories %} + + {{ subcat.name }} + + {% endfor %} + {% for cat in entry.categories %} + {{ cat }} + {% endfor %} + {% if entry.groups %} + {% set group_url = filter_urls[entry.groups[0]] %} + + {{ entry.groups[0] }} + + {% endif %} + {% if entry.source_type == 'Stdlib' %} + + Stdlib + + {% endif %} + + → + + {% if entry.description %} + + + +
{{ entry.description | safe }}
+ + + {% endif %} + + + +
+ {% if entry.also_see %} +
+ Also see: {% for see in entry.also_see %}{{ see.name }}{% if not loop.last %}, {% endif %}{% endfor %} +
+ {% endif %} +
+ {% if entry.owner %}{{ entry.owner }}/{% endif %}{{ entry.url | replace("https://", "") }} + {% if entry.last_commit_at %}/{% endif %} {% if entry.downloads is not none %}/{{ + "{:,}".format(entry.downloads) }} downloads/month{% endif %} +
+
+ + + +{% endmacro %} +
+
+ {% if entry_groups %} + {% set named_groups = entry_groups | selectattr("name") | list %} +

+ {% if group_categories %}Listed by section, in editorial order.{% elif named_groups %}Listed in editorial order, grouped by use case.{% else %}Listed in editorial order.{% endif %} + Click a column to re-sort the whole list. +

+ {% else %} +
+

{{ entries | length }} {{ category.name }} projects

+
+ {% endif %} +

+ Press / to search. Tap a tag to filter. Click any row for + details. +

+
+ +
+

Search and filter

+
+ + + + + +
+
+ +

Results

+
+ + + + + + + + + + + + + + {% if entry_groups %} + {% set row = namespace(index=0) %} + {% for group in entry_groups %} + {% if group.name %} + + + + {% endif %} + {% for entry in group.entries %} + {% set row.index = row.index + 1 %} + {{ entry_rows(entry, row.index) }} + {% endfor %} + {% endfor %} + {% else %} + {% for entry in entries %} + {{ entry_rows(entry, loop.index) }} + {% endfor %} + {% endif %} + +
Row number + + + + + + + + Tags + +
+

+ {% if group.url %}{{ group.name }}{% else %}{{ group.name }}{% endif %} + {{ group.entries | length }} project{{ "s" if group.entries | length != 1 }} +

+ {% if group.lead %} +
{{ group.lead | safe }}
+ {% endif %} +
+
+ + +
+ +{% if guide_html %} +
+
+

{{ category.name }} guide

+
{{ guide_html | safe }}
+
+
+{% endif %} + +
+
+

Contribute

+

Know a project that belongs here?

+

Tell us what it does and why it stands out.

+
+ Submit a project + Star the repository +
+
+
+{% endblock %} diff --git a/website/templates/index.html b/website/templates/index.html new file mode 100644 index 00000000..4cb99d4c --- /dev/null +++ b/website/templates/index.html @@ -0,0 +1,331 @@ +{% extends "base.html" %} +{% block extra_head %} + +{% endblock %} +{% block header %} +
+ + + +
+ + +
+
+

The definitive list that answers "I want to do X in Python, which tool should I use?"

+

Awesome Python

+

{{ subtitle }}

+

+ {% if repo_stars %}{{ repo_stars }}+ stars on GitHub{% endif %} {% if repo_stars and build_date %}/{% endif %} {% if build_date %}Updated {{ build_date }}{% endif %} {% if repo_stars or + build_date %}/{% endif %} Maintained by + @vinta + and + @JinyangWang27 +

+
+ Browse the List + View on GitHub +
+
+
+ + +
+
+{% endblock %} +{% block content %} +{% if sponsors %} + +{% endif %} + +
+
+
+

Search every project in one place

+
+

+ Press / to search. Tap a tag to filter. Click any row for + details. +

+
+ +
+

Search and filter

+
+ + + + + +
+
+ +

Results

+
+ + + + + + + + + + + + + + {% for entry in entries %} + + + + + + + + + + {% if entry.description %} + + + + + {% endif %} + + + + + + {% endfor %} + +
Row number + + + + + + + + Tags + +
+
+ {% if entry.description %} +
{{ entry.description | safe }}
+ {% endif %} {% if entry.also_see %} +
+ Also see: {% for see in entry.also_see %}{{ see.name }}{% if not loop.last %}, {% endif %}{% endfor %} +
+ {% endif %} +
+ {% if entry.owner %}{{ entry.owner }}/{% endif %}{{ entry.url | replace("https://", "") }} + {% if entry.last_commit_at %}/{% endif %} {% if entry.downloads is not none %}/{{ + "{:,}".format(entry.downloads) }} downloads/month{% endif %} +
+
+
+
+ + +
+ +
+
+

Contribute

+

Know a project that belongs here?

+

Tell us what it does and why it stands out.

+
+ Submit a project + Star the repository +
+
+
+{% endblock %} diff --git a/website/templates/llms.txt b/website/templates/llms.txt new file mode 100644 index 00000000..c29782d8 --- /dev/null +++ b/website/templates/llms.txt @@ -0,0 +1,19 @@ +# Awesome Python + +{{ subtitle }} + +{{ total_entries }} projects across {{ total_categories }} {% if total_categories == 1 %}category{% else %}categories{% endif %}, updated on {{ build_date }}. + +Scan the category index, then jump to the matching section for direct project links and short descriptions. Entries end with `PyPI downloads/month: N` (last 30 days, mirrors and CI included) and `GitHub stars: N` notes in parentheses where known; treat both as popularity context, not a quality guarantee. Use the homepage for project context outside the shortlist. + +## Primary Links + +- Homepage: {{ site_url }} +- GitHub repository: {{ github_repo_url }} +- Contributing guide: {{ contributing_url }} +- Sponsorship: {{ sponsorship_url }} +- Sitemap: {{ sitemap_url }} + +## Categories + +{{ categories_md }} diff --git a/website/templates/redirect.html b/website/templates/redirect.html new file mode 100644 index 00000000..d62c587d --- /dev/null +++ b/website/templates/redirect.html @@ -0,0 +1,10 @@ + + + +Redirecting… + + + +

Redirecting…

+Click here if you are not redirected. + diff --git a/website/templates/sponsorship.html b/website/templates/sponsorship.html new file mode 100644 index 00000000..34de3517 --- /dev/null +++ b/website/templates/sponsorship.html @@ -0,0 +1,297 @@ +{% extends "base.html" %} +{% block title %}Sponsor Awesome Python{% endblock %} +{% block description %}{{ sponsorship_description }}{% endblock %} +{% block canonical_url %}https://awesome-python.com/sponsorship/{% endblock %} +{% block alternate_links %}{% endblock %} +{% block extra_head %} + +{% endblock %} +{% block header %} +
+ + + +
+ + +
+

Sponsorship

+

Sponsor Awesome Python

+

+ The #10 most-starred repository on GitHub, and the list Python + developers check when choosing what to use. Your sponsorship puts your + product in front of them at the moment of decision. +

+ + {% if hero_stats %} +

+ {% for stat in hero_stats %}{{ stat }}{% if not loop.last %} + / {% endif %}{% endfor %} +

+ {% endif %} + +
+ Email sponsorship@awesome-python.com + View on GitHub +
+
+
+
+{% endblock %} {% block content %} +
+
+
+

Audience

+
+
+

+ Professional Python developers evaluating libraries and tools for + production use. Not beginners browsing tutorials. People making adoption + decisions. +

+
+
+
Who visits
+
+ Mid to senior Python developers arriving with a specific question: a + maintained ORM, a fast HTTP client, a task queue worth running in + production. +
+
+
+
Where they come from
+
+ Google Search, GitHub, Reddit, YouTube, ChatGPT and other LLMs, + Hacker News. +
+
+
+
Why it works
+
+ Ranks on the first page of Google for "best Python libraries". + ChatGPT and other LLMs cite it when recommending Python tools. + Developers send it to each other. +
+
+
+
+
+
+ +
+
+
+

Tiers

+

+ One upfront payment per term. Setup takes less than 24 hours. +

+
+
+
    +
  1. +

    Headline Sponsor

    +

    + $500 + / month +

    +

    + Logo pinned at the top of the README. Logo on the website. +

    +
      +
    • + Large logo and one-line description (max 120 characters) pinned at + the very top of the README, above all project entries. +
      • +
    • Logo link in the sponsor section of awesome-python.com.
      • +
    + Email about Headline tier +
    2. +
  2. +

    Featured Sponsor

    +

    + $200 + / month +

    +

    + Text link pinned at the top of the README. Text link on the website. +

    +
      +
    • + Text entry ([Name](URL) - Description., max 120 + characters) pinned at the top of the README, directly below + Headline sponsors. +
      • +
    • Text link in the sponsor section of awesome-python.com.
      • +
    + Email about Featured tier +
    3. +
  3. +

    Indie Dev Sponsor

    +

    + $99 + / month or $249 / quarter +

    +

    + Lower-cost text link for eligible indie developers and solo + founders. +

    +
      +
    • + Text entry ([Name](URL) - Description., max 120 + characters) in the README sponsor section, directly below Featured + sponsors. +
      • +
    • Text link in the sponsor section of awesome-python.com.
      • +
    • + Available only to founder-led, one-person-company products. Funded + startups, agencies, recruiters, larger companies, and products + with enterprise sales teams should use Featured or Headline. +
      • +
    + Email about Indie Dev tier +
    4. +
+
+
+
+ +
+
+
+

Previously sponsored by

+
+
+
    +
  • + Warp + The terminal for modern developers. +
    • +
  • + pyr + Zero-config Python project manager. +
    • +
+
+
+
+ +
+
+
+

Get started

+
+
+

+ Email + sponsorship@awesome-python.com + with the four items below. +

+
+
+
Tier
+
+ Headline Sponsor ($500/mo), Featured Sponsor ($200/mo), or Indie Dev + Sponsor ($99/mo or $249/quarter). +
+
+
+
Content
+
+ Product name, URL, logo, and description (Headline tier), or + [Name](URL) - Description. entry (Featured or Indie Dev + tier). +
+
+
+
Duration
+
1, 3, 6 months, or longer.
+
+
+
Payment method
+
+ Card via + Open Collective, + US bank transfer (ACH/wire), or PayPal. +
+
+
+
+ Email sponsorship@awesome-python.com +
+
+
+
+ +
+
+
+

Editorial independence

+
+
+

+ Sponsorship is logo and link placement in the README header. It does not + influence which projects are listed. We curate listings on merit through + the normal + contribution process. +

+

+ We reserve the right to request changes to sponsor text, logos, or links + that are misleading, off-topic, or incompatible with the README + formatting. +

+
+
+
+{% endblock %} diff --git a/website/tests/test_build.py b/website/tests/test_build.py new file mode 100644 index 00000000..16af7aa4 --- /dev/null +++ b/website/tests/test_build.py @@ -0,0 +1,1600 @@ +"""Tests for the build module.""" + +import json +import os +import re +import shutil +import subprocess +import textwrap +import xml.etree.ElementTree as ET +from datetime import UTC, date, datetime +from html.parser import HTMLParser +from pathlib import Path + +import pytest +from build import ( + TemplateEntry, + annotate_entries_with_stats, + build, + category_meta_title, + detect_source_type, + extract_entries, + extract_github_repo, + load_category_intro, + load_downloads, + load_pypi_badges, + load_stars, + sort_entries, + subcategory_path, +) +from readme_parser import parse_readme, slugify + + +class HeadMetadataParser(HTMLParser): + def __init__(self): + super().__init__() + self.title_count = 0 + self.title = "" + self.meta_by_name = {} + self.meta_by_property = {} + self.links_by_rel = {} + self._in_title = False + + def handle_starttag(self, tag, attrs): + attrs = dict(attrs) + if tag == "title": + self.title_count += 1 + self._in_title = True + elif tag == "meta": + if "name" in attrs: + self.meta_by_name[attrs["name"]] = attrs.get("content", "") + if "property" in attrs: + self.meta_by_property[attrs["property"]] = attrs.get("content", "") + elif tag == "link" and attrs.get("rel"): + for rel in attrs["rel"].split(): + self.links_by_rel[rel] = attrs.get("href", "") + + def handle_endtag(self, tag): + if tag == "title": + self._in_title = False + + def handle_data(self, data): + if self._in_title: + self.title += data + + +# --------------------------------------------------------------------------- +# slugify +# --------------------------------------------------------------------------- + + +class TestSlugify: + def test_simple(self): + assert slugify("Admin Panels") == "admin-panels" + + def test_uppercase_acronym(self): + assert slugify("RESTful API") == "restful-api" + + def test_hyphenated_input(self): + assert slugify("Command-line Tools") == "command-line-tools" + + def test_extra_spaces(self): + assert slugify(" Date and Time ") == "date-and-time" + + +class TestSubcategoryPath: + def test_builds_path(self): + assert subcategory_path("web-frameworks", "synchronous") == "/categories/web-frameworks/synchronous/" + + +# --------------------------------------------------------------------------- +# build (integration) +# --------------------------------------------------------------------------- + + +class TestBuild: + @pytest.fixture(autouse=True) + def _make_sponsorship_md(self, tmp_path): + (tmp_path / "SPONSORSHIP.md").write_text("# Sponsorship\n", encoding="utf-8") + + def _make_repo(self, tmp_path, readme): + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + tpl_dir = tmp_path / "website" / "templates" + tpl_dir.mkdir(parents=True) + (tpl_dir / "base.html").write_text( + "{% block title %}{% endblock %}" + "" + "{% block content %}{% endblock %}", + encoding="utf-8", + ) + (tpl_dir / "index.html").write_text( + '{% extends "base.html" %}{% block content %}' + "{% for entry in entries %}" + '
' + "{{ entry.name }}" + "{{ entry.categories | join(', ') }}" + "{{ entry.groups | join(', ') }}" + "
" + "{% endfor %}" + "{% endblock %}", + encoding="utf-8", + ) + (tpl_dir / "category.html").write_text( + '{% extends "base.html" %}{% block content %}

{{ category.name }}

{% for entry in entries %}{{ entry.name }}{% endfor %}{% endblock %}', + encoding="utf-8", + ) + (tpl_dir / "sponsorship.html").write_text( + '{% extends "base.html" %}{% block content %}

Sponsor

{% endblock %}', + encoding="utf-8", + ) + (tpl_dir / "llms.txt").write_text( + "# Awesome Python\n\nHomepage: {{ site_url }}\n\n## Categories\n\n{{ categories_md }}\n", + encoding="utf-8", + ) + + def _copy_real_templates(self, tmp_path): + real_tpl = Path(__file__).parent / ".." / "templates" + tpl_dir = tmp_path / "website" / "templates" + shutil.copytree(real_tpl, tpl_dir) + + def test_build_creates_homepage_and_category_pages(self, tmp_path): + readme = textwrap.dedent("""\ + # Awesome Python + + Intro. + + ## Projects + + **Tools** + + ### Widgets + + _Widget libraries. Also see [awesome-widgets](https://example.com/widgets)._ + + - [w1](https://example.com) - A widget. + + ### Gadgets + + _Gadget tools._ + + - [g1](https://example.com) - A gadget. + + ## Resources + + Info. + + ### Newsletters + + - [NL](https://example.com) + + ## Contributing + + Help! + """) + self._make_repo(tmp_path, readme) + build(tmp_path) + + site = tmp_path / "website" / "output" + assert (site / "index.html").exists() + assert (site / "categories" / "widgets" / "index.html").exists() + assert (site / "categories" / "gadgets" / "index.html").exists() + + def test_build_creates_root_discovery_files(self, tmp_path): + readme = textwrap.dedent("""\ + # Awesome Python + + Intro. + + ## Projects + + **Tools** + + ### Widgets + + - Sync + + - [w1](https://example.com) - A widget. + + ## Contributing + + Help! + """) + self._make_repo(tmp_path, readme) + start_date = datetime.now(UTC).date() + build(tmp_path) + end_date = datetime.now(UTC).date() + + site = tmp_path / "website" / "output" + robots = (site / "robots.txt").read_text(encoding="utf-8") + assert robots == ("User-agent: *\nContent-Signal: search=yes, ai-input=yes, ai-train=yes\nAllow: /\n\nSitemap: https://awesome-python.com/sitemap.xml\n") + + sitemap = ET.parse(site / "sitemap.xml") + root = sitemap.getroot() + ns = {"sitemap": "http://www.sitemaps.org/schemas/sitemap/0.9"} + locs = [loc.text or "" for loc in root.findall("sitemap:url/sitemap:loc", ns)] + lastmods = [lastmod.text or "" for lastmod in root.findall("sitemap:url/sitemap:lastmod", ns)] + + assert root.tag == "{http://www.sitemaps.org/schemas/sitemap/0.9}urlset" + assert locs == [ + "https://awesome-python.com/", + "https://awesome-python.com/categories/widgets/", + "https://awesome-python.com/categories/tools/", + "https://awesome-python.com/categories/widgets/sync/", + "https://awesome-python.com/sponsorship/", + ] + assert len(lastmods) == len(locs) + # Outside a git repository every page falls back to the build date + assert all(start_date <= date.fromisoformat(lastmod) <= end_date for lastmod in lastmods) + assert all(loc.startswith("https://awesome-python.com/") for loc in locs) + assert all("?" not in loc for loc in locs) + + def test_sitemap_lastmod_follows_git_history_of_each_page(self, tmp_path): + readme = textwrap.dedent("""\ + # Awesome Python + + Intro. + + ## Projects + + **Tools** + + ### Widgets + + - Sync + + - [w1](https://example.com/w1) - A widget. + + ### Gadgets + + - [g1](https://example.com/g1) - A gadget. + + ## Contributing + + Help! + """) + self._make_repo(tmp_path, readme) + + def commit(day, message): + env = {**os.environ, "GIT_AUTHOR_DATE": f"{day}T12:00:00Z", "GIT_COMMITTER_DATE": f"{day}T12:00:00Z"} + subprocess.run(["git", "add", "-A"], cwd=tmp_path, check=True) + subprocess.run(["git", "-c", "user.name=t", "-c", "user.email=t@t", "commit", "-qm", message], cwd=tmp_path, env=env, check=True) + + subprocess.run(["git", "init", "-q"], cwd=tmp_path, check=True) + commit("2024-01-01", "initial") + (tmp_path / "README.md").write_text(readme.replace("A gadget.", "A better gadget."), encoding="utf-8") + commit("2024-02-01", "edit gadgets") + intros_dir = tmp_path / "website" / "data" / "category_intros" + intros_dir.mkdir(parents=True) + (intros_dir / "widgets.md").write_text("Use w1.\n", encoding="utf-8") + commit("2024-03-01", "add widgets intro") + build(tmp_path) + + sitemap = (tmp_path / "website" / "output" / "sitemap.xml").read_text(encoding="utf-8") + lastmod_by_loc = dict(re.findall(r"https://awesome-python\.com(\S*?)\s*(\S+)", sitemap)) + assert lastmod_by_loc == { + "/": "2024-02-01", + "/categories/widgets/": "2024-03-01", + "/categories/gadgets/": "2024-02-01", + "/categories/tools/": "2024-03-01", + "/categories/widgets/sync/": "2024-03-01", + "/sponsorship/": "2024-01-01", + } + + def test_build_creates_category_pages_with_metadata_and_links(self, tmp_path): + readme = textwrap.dedent("""\ + # Awesome Python + + Intro. + + ## Projects + + **Tools** + + ## Widgets + + _Widget libraries. Also see [awesome-widgets](https://example.com/widgets)._ + + - [w1](https://example.com/w1) - A widget. + - [w2](https://github.com/owner/w2) - A starred widget. + + ## Gadgets + + _Gadget tools._ + + - [g1](https://example.com/g1) - A gadget. + + # Contributing + + Help! + """) + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + self._copy_real_templates(tmp_path) + + data_dir = tmp_path / "website" / "data" + data_dir.mkdir(parents=True) + stars = { + "owner/w2": { + "stars": 42, + "owner": "owner", + "last_commit_at": "2026-01-01T00:00:00+00:00", + "fetched_at": "2026-01-01T00:00:00+00:00", + }, + } + (data_dir / "github_stars.json").write_text(json.dumps(stars), encoding="utf-8") + + build(tmp_path) + + site = tmp_path / "website" / "output" + index_html = (site / "index.html").read_text(encoding="utf-8") + category_html = (site / "categories" / "widgets" / "index.html").read_text(encoding="utf-8") + parser = HeadMetadataParser() + parser.feed(category_html) + + assert 'href="/categories/widgets/"' in index_html + assert parser.title.strip() == "Python Widgets Libraries - Awesome Python" + assert parser.meta_by_name["description"] == "Widget libraries. Also see awesome-widgets. Explore 2 curated Python projects in Widgets." + assert parser.links_by_rel["canonical"] == "https://awesome-python.com/categories/widgets/" + assert parser.meta_by_property["og:url"] == "https://awesome-python.com/categories/widgets/" + assert '' not in category_html + assert 'Sponsorship' in category_html + assert "

Widgets

" in category_html + assert 'Widget libraries. Also see awesome-widgets.' in category_html + assert 'href="https://example.com/w1"' in category_html + assert "A widget." in category_html + assert 'href="https://github.com/owner/w2"' in category_html + assert '' in category_html + assert "42" in category_html + assert "2026-01-01T00:00:00+00:00" in category_html + + def test_build_links_description_anchors_to_category_pages(self, tmp_path): + readme = textwrap.dedent("""\ + # Awesome Python + + Intro. + + ## Projects + + **Tools** + + ## Audio & Video + + _Media tools._ + + - [a1](https://example.com/a1) - A media tool. + + ## Widgets + + _Widget libraries. Also see [Audio & Video](#audio--video) and [Gadgets](#gadgets)._ + + - [w1](https://example.com/w1) - A widget. + + # Contributing + + Help! + """) + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + self._copy_real_templates(tmp_path) + + build(tmp_path) + + category_html = (tmp_path / "website" / "output" / "categories" / "widgets" / "index.html").read_text(encoding="utf-8") + assert 'Also see Audio & Video' in category_html + assert 'Gadgets' in category_html + + def test_build_creates_llms_text_alternate_without_sponsors(self, tmp_path): + readme = textwrap.dedent("""\ + # Awesome Python + + Intro. + + ## **Sponsors** + + - **[Sponsor](https://sponsor.example.com)**: Sponsored tool. + + > Become a sponsor: [Sponsor us](SPONSORSHIP.md). + + ## Categories + + **Tools** + + - [Widgets](#widgets) + + ## Projects + + **Tools** + + ### Widgets + + - [w1](https://example.com) - A widget. + - [w2](https://github.com/owner/w2) - A starred widget. + + ## Contributing + + Help! + """) + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + self._copy_real_templates(tmp_path) + + data_dir = tmp_path / "website" / "data" + data_dir.mkdir(parents=True) + stars = { + "owner/w2": {"stars": 42, "owner": "owner", "fetched_at": "2026-01-01T00:00:00+00:00"}, + } + (data_dir / "github_stars.json").write_text(json.dumps(stars), encoding="utf-8") + (data_dir / "pypi_downloads.tsv").write_text( + "name\tpackage\tdownloads\tfetched_at\nw1\tw1\t777\t2026-08-16\n", + encoding="utf-8", + ) + + build(tmp_path) + + site = tmp_path / "website" / "output" + index_html = (site / "index.html").read_text(encoding="utf-8") + llms_txt = (site / "llms.txt").read_text(encoding="utf-8") + + assert '' in index_html + + assert llms_txt.startswith("# Awesome Python\n\nIntro.\n") + assert "2 projects across 1 category, updated on " in llms_txt + assert "Scan the category index" in llms_txt + assert "Homepage: https://awesome-python.com/" in llms_txt + assert "Markdown homepage" not in llms_txt + assert "https://awesome-python.com/index.md" not in llms_txt + assert "GitHub repository: https://github.com/vinta/awesome-python" in llms_txt + assert "Contributing guide: https://github.com/vinta/awesome-python/blob/master/CONTRIBUTING.md" in llms_txt + assert "Sponsorship: https://awesome-python.com/sponsorship/" in llms_txt + assert "Sitemap: https://awesome-python.com/sitemap.xml" in llms_txt + assert "## Categories" in llms_txt + assert "**Tools**" in llms_txt + assert "- [Widgets](https://awesome-python.com/categories/widgets/)" in llms_txt + assert "- [Widgets](#widgets)" not in llms_txt + assert "### Widgets" in llms_txt + assert "- [w1](https://example.com) - A widget. (PyPI downloads/month: 777)" in llms_txt + assert "- [w2](https://github.com/owner/w2) - A starred widget. (GitHub stars: 42)" in llms_txt + assert llms_txt != readme + assert "# Contributing" not in llms_txt + + def test_build_cleans_stale_output(self, tmp_path): + readme = textwrap.dedent("""\ + # T + + ## Projects + + ## Only + + - [x](https://x.com) - X. + + # Contributing + + Done. + """) + self._make_repo(tmp_path, readme) + + stale = tmp_path / "website" / "output" / "categories" / "stale" + stale.mkdir(parents=True) + (stale / "index.html").write_text("old", encoding="utf-8") + + build(tmp_path) + + assert not (tmp_path / "website" / "output" / "categories" / "stale").exists() + + def test_build_with_stars_sorts_by_stars(self, tmp_path): + readme = textwrap.dedent("""\ + # T + + ## Projects + + ## Stuff + + - [low-stars](https://github.com/org/low) - Low. + - [high-stars](https://github.com/org/high) - High. + - [no-stars](https://example.com/none) - None. + + # Contributing + + Done. + """) + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + + # Copy real templates + real_tpl = Path(__file__).parent / ".." / "templates" + tpl_dir = tmp_path / "website" / "templates" + shutil.copytree(real_tpl, tpl_dir) + + # Create mock star data + data_dir = tmp_path / "website" / "data" + data_dir.mkdir(parents=True) + stars = { + "org/high": {"stars": 5000, "owner": "org", "fetched_at": "2026-01-01T00:00:00+00:00"}, + "org/low": {"stars": 100, "owner": "org", "fetched_at": "2026-01-01T00:00:00+00:00"}, + } + (data_dir / "github_stars.json").write_text(json.dumps(stars), encoding="utf-8") + + build(tmp_path) + + html = (tmp_path / "website" / "output" / "index.html").read_text(encoding="utf-8") + # Star-sorted: high-stars (5000) before low-stars (100) before no-stars (None) + assert html.index("high-stars") < html.index("low-stars") + assert html.index("low-stars") < html.index("no-stars") + # Formatted star counts + assert "5,000" in html + assert "100" in html + # Expand content present + assert "expand-content" in html + + def test_build_with_downloads_renders_column(self, tmp_path): + readme = textwrap.dedent("""\ + # T + + ## Projects + + ## Stuff + + - [My-Lib](https://github.com/org/mylib) - On PyPI. + - [no-pypi](https://example.com/none) - Not on PyPI. + - [asyncio](https://docs.python.org/3/library/asyncio.html) - Stdlib. + - [my-lib.thing](https://github.com/org/mylib) - (part of My-Lib) A bundled feature. + + # Contributing + + Done. + """) + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + self._copy_real_templates(tmp_path) + + data_dir = tmp_path / "website" / "data" + data_dir.mkdir(parents=True) + # Keyed by normalized README display name, like fetch_pypi_downloads_via_clickpy.py writes it + (data_dir / "pypi_downloads.tsv").write_text( + "name\tpackage\tdownloads\tfetched_at\nasyncio\tasyncio\t26305454\t2026-08-16\nmy-lib\tmy-lib\t1234567\t2026-08-16\n", + encoding="utf-8", + ) + + build(tmp_path) + + html = (tmp_path / "website" / "output" / "index.html").read_text(encoding="utf-8") + assert "1,234,567" in html + # Stdlib entries never show PyPI counts: the asyncio row is the backport package + assert "26,305,454" not in html + # Default sort: entries with download counts come first + assert html.index("My-Lib") < html.index("no-pypi") + # Each no-download entry gets the badge matching why it has no count + assert html.count('Stdlib') == 1 + assert html.count('Bundled') == 1 + assert html.count('Not on PyPI') == 1 + + def test_build_fails_when_group_and_category_slug_collide(self, tmp_path): + readme = textwrap.dedent("""\ + # T + + ## Projects + + **Widgets** + + ## Widgets + + - [w1](https://example.com) - W. + + # Contributing + + Done. + """) + self._make_repo(tmp_path, readme) + with pytest.raises(ValueError, match="slug collision"): + build(tmp_path) + + def test_index_contains_aligned_homepage_metadata(self, tmp_path): + readme = (Path(__file__).parents[2] / "README.md").read_text(encoding="utf-8") + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + self._copy_real_templates(tmp_path) + + build(tmp_path) + + parsed_groups = parse_readme(readme) + categories = [cat for group in parsed_groups for cat in group["categories"]] + entries = extract_entries(categories, parsed_groups) + html = (tmp_path / "website" / "output" / "index.html").read_text(encoding="utf-8") + parser = HeadMetadataParser() + parser.feed(html) + + expected_title = "Awesome Python" + expected_description = f"An opinionated guide to the best Python frameworks, libraries, and tools. Explore {len(entries)} curated projects across {len(categories)} categories, from AI and agents to data science and web development." + expected_url = "https://awesome-python.com/" + expected_image = "https://awesome-python.com/static/og-image.png" + + assert parser.title_count == 1 + assert parser.title.strip() == expected_title + assert parser.meta_by_name["description"] == expected_description + assert parser.links_by_rel["canonical"] == expected_url + assert parser.meta_by_property["og:type"] == "website" + assert parser.meta_by_property["og:title"] == expected_title + assert parser.meta_by_property["og:description"] == expected_description + assert parser.meta_by_property["og:image"] == expected_image + assert parser.meta_by_property["og:url"] == expected_url + assert parser.meta_by_name["twitter:card"] == "summary_large_image" + assert parser.meta_by_name["twitter:title"] == expected_title + assert parser.meta_by_name["twitter:description"] == expected_description + assert parser.meta_by_name["twitter:image"] == expected_image + assert "\n Sponsorship' in html + assert 'id="hero-category-heading">Browse by category' in html + assert 'class="hero-category-link" href="/categories/ai-and-agents/"' in html + + def test_index_contains_homepage_json_ld(self, tmp_path): + readme = (Path(__file__).parents[2] / "README.md").read_text(encoding="utf-8") + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + self._copy_real_templates(tmp_path) + + build(tmp_path) + + parsed_groups = parse_readme(readme) + categories = [cat for group in parsed_groups for cat in group["categories"]] + entries = extract_entries(categories, parsed_groups) + html = (tmp_path / "website" / "output" / "index.html").read_text(encoding="utf-8") + + marker = '", start) + block = html[start:end] + assert "" not in block + data = json.loads(block) + + assert data["@context"] == "https://schema.org" + graph = {node["@type"]: node for node in data["@graph"]} + assert set(graph) == {"WebSite", "CollectionPage"} + assert graph["WebSite"]["url"] == "https://awesome-python.com/" + assert graph["WebSite"]["name"] == "Awesome Python" + assert graph["WebSite"]["@id"] == "https://awesome-python.com/#website" + + collection = graph["CollectionPage"] + assert collection["@id"] == "https://awesome-python.com/" + assert collection["url"] == "https://awesome-python.com/" + assert collection["isPartOf"] == {"@type": "WebSite", "@id": graph["WebSite"]["@id"]} + expected_description = f"An opinionated guide to the best Python frameworks, libraries, and tools. Explore {len(entries)} curated projects across {len(categories)} categories, from AI and agents to data science and web development." + assert collection["description"] == expected_description + + item_list = collection["mainEntity"] + assert item_list["@type"] == "ItemList" + assert item_list["numberOfItems"] == len(entries) + assert len(item_list["itemListElement"]) == len(entries) + + positions = [item["position"] for item in item_list["itemListElement"]] + assert positions == list(range(1, len(entries) + 1)) + assert all(item["@type"] == "ListItem" for item in item_list["itemListElement"]) + assert all(item["url"].startswith(("http://", "https://")) for item in item_list["itemListElement"]) + + rendered_names = {item["name"] for item in item_list["itemListElement"]} + rendered_urls = {item["url"] for item in item_list["itemListElement"]} + assert rendered_names == {e["name"] for e in entries} + assert rendered_urls == {e["url"] for e in entries} + + def test_category_page_contains_json_ld(self, tmp_path): + readme = textwrap.dedent("""\ + # Awesome Python + + Intro. + + ## Projects + + **Tools** + + ## Widgets + + _Widget libraries._ + + - [w1](https://example.com/w1) - A widget. + - [w2](https://github.com/owner/w2) - A starred widget. + + # Contributing + + Help! + """) + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + self._copy_real_templates(tmp_path) + build(tmp_path) + + category_html = (tmp_path / "website" / "output" / "categories" / "widgets" / "index.html").read_text(encoding="utf-8") + marker = '", start) + block = category_html[start:end] + assert "" not in block + data = json.loads(block) + + assert data["@context"] == "https://schema.org" + graph = {node["@type"]: node for node in data["@graph"]} + assert set(graph) == {"WebSite", "CollectionPage", "BreadcrumbList"} + assert graph["WebSite"]["@id"] == "https://awesome-python.com/#website" + collection = graph["CollectionPage"] + assert collection["name"] == "Python Widgets Libraries" + assert collection["@id"] == "https://awesome-python.com/categories/widgets/" + assert collection["url"] == "https://awesome-python.com/categories/widgets/" + assert collection["description"] == "Widget libraries. Explore 2 curated Python projects in Widgets." + assert collection["isPartOf"] == {"@type": "WebSite", "@id": "https://awesome-python.com/#website"} + + item_list = collection["mainEntity"] + assert item_list["@type"] == "ItemList" + assert item_list["numberOfItems"] == 2 + names = {item["name"] for item in item_list["itemListElement"]} + urls = {item["url"] for item in item_list["itemListElement"]} + assert names == {"w1", "w2"} + assert urls == {"https://example.com/w1", "https://github.com/owner/w2"} + positions = sorted(item["position"] for item in item_list["itemListElement"]) + assert positions == [1, 2] + + breadcrumbs = graph["BreadcrumbList"]["itemListElement"] + assert breadcrumbs == [ + {"@type": "ListItem", "position": 1, "name": "Awesome Python", "item": "https://awesome-python.com/"}, + {"@type": "ListItem", "position": 2, "name": "Widgets", "item": "https://awesome-python.com/categories/widgets/"}, + ] + + def test_category_json_ld_follows_page_order(self, tmp_path): + readme = textwrap.dedent("""\ + # Awesome Python + + Intro. + + ## Projects + + **Tools** + + ## Widgets + + _Widget libraries._ + + - [zeta](https://example.com/zeta) - Listed first. + - [alpha](https://example.com/alpha) - Listed second. + + # Contributing + + Help! + """) + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + self._copy_real_templates(tmp_path) + build(tmp_path) + + category_html = (tmp_path / "website" / "output" / "categories" / "widgets" / "index.html").read_text(encoding="utf-8") + marker = '", start)]) + collection = next(node for node in data["@graph"] if node["@type"] == "CollectionPage") + items = collection["mainEntity"]["itemListElement"] + assert [(item["position"], item["name"]) for item in items] == [(1, "zeta"), (2, "alpha")] + + def test_group_page_falls_back_to_default_description_in_json_ld(self, tmp_path): + readme = textwrap.dedent("""\ + # T + + ## Projects + + **AI & ML** + + ## Deep Learning + + - [dl1](https://example.com/dl1) - DL. + + # Contributing + + Done. + """) + self._copy_real_templates(tmp_path) + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + build(tmp_path) + + group_html = (tmp_path / "website" / "output" / "categories" / "ai-ml" / "index.html").read_text(encoding="utf-8") + marker = '", start) + data = json.loads(group_html[start:end]) + + graph = {node["@type"]: node for node in data["@graph"]} + collection = graph["CollectionPage"] + assert collection["name"] == "Python AI & ML Libraries" + assert collection["@id"] == "https://awesome-python.com/categories/ai-ml/" + assert collection["url"] == "https://awesome-python.com/categories/ai-ml/" + assert collection["description"] == "Explore 1 curated Python project in AI & ML. Part of the Awesome Python catalog." + + def test_category_title_skips_libraries_after_plural_noun(self, tmp_path): + readme = textwrap.dedent("""\ + # T + + ## Projects + + **Web Development** + + ## Web Frameworks + + - [wf1](https://example.com/wf1) - WF. + + ## Web APIs + + - [api1](https://example.com/api1) - API. + + # Contributing + + Done. + """) + self._copy_real_templates(tmp_path) + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + build(tmp_path) + + categories_dir = tmp_path / "website" / "output" / "categories" + frameworks_html = (categories_dir / "web-frameworks" / "index.html").read_text(encoding="utf-8") + apis_html = (categories_dir / "web-apis" / "index.html").read_text(encoding="utf-8") + parser = HeadMetadataParser() + parser.feed(frameworks_html) + marker = '", start) + graph = {node["@type"]: node for node in json.loads(frameworks_html[start:end])["@graph"]} + + assert parser.title.strip() == "Python Web Frameworks - Awesome Python" + assert parser.meta_by_property["og:title"] == "Python Web Frameworks - Awesome Python" + assert graph["CollectionPage"]["name"] == "Python Web Frameworks" + assert "Python Web APIs Libraries - Awesome Python" in apis_html + + def test_category_title_does_not_repeat_leading_python(self): + assert category_meta_title("Python Language") == "Python Language Libraries - Awesome Python" + + def test_build_creates_subcategory_pages(self, tmp_path): + readme = textwrap.dedent("""\ + # T + + ## Projects + + **Web** + + ## Web Frameworks + + - Synchronous + + - [django](https://example.com/django) - Sync framework. + + - Asynchronous + + - [fastapi](https://example.com/fastapi) - Async framework. + + # Contributing + + Done. + """) + self._copy_real_templates(tmp_path) + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + build(tmp_path) + + site = tmp_path / "website" / "output" + sync = (site / "categories" / "web-frameworks" / "synchronous" / "index.html").read_text(encoding="utf-8") + async_ = (site / "categories" / "web-frameworks" / "asynchronous" / "index.html").read_text(encoding="utf-8") + + assert "django" in sync + assert "fastapi" not in sync + assert "fastapi" in async_ + assert "django" not in async_ + + def test_subcategory_page_shows_breadcrumb(self, tmp_path): + readme = textwrap.dedent("""\ + # T + + ## Projects + + **Web** + + ## Web Frameworks + + - Synchronous + + - [django](https://example.com/django) - Sync. + + # Contributing + + Done. + """) + self._copy_real_templates(tmp_path) + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + build(tmp_path) + + site = tmp_path / "website" / "output" + sync = (site / "categories" / "web-frameworks" / "synchronous" / "index.html").read_text(encoding="utf-8") + assert 'href="/categories/web-frameworks/"' in sync + assert "Web Frameworks" in sync + assert "

Synchronous

" in sync + assert "category-breadcrumb" in sync + + parser = HeadMetadataParser() + parser.feed(sync) + assert parser.title.strip() == "Python Web Frameworks: Synchronous - Awesome Python" + assert parser.meta_by_name["description"] == "The Synchronous picks in Awesome Python's Web Frameworks list: django." + + marker = '", start) + graph = {node["@type"]: node for node in json.loads(sync[start:end])["@graph"]} + assert graph["CollectionPage"]["name"] == "Python Web Frameworks: Synchronous" + assert graph["BreadcrumbList"]["itemListElement"] == [ + {"@type": "ListItem", "position": 1, "name": "Awesome Python", "item": "https://awesome-python.com/"}, + { + "@type": "ListItem", + "position": 2, + "name": "Web Frameworks", + "item": "https://awesome-python.com/categories/web-frameworks/", + }, + { + "@type": "ListItem", + "position": 3, + "name": "Synchronous", + "item": "https://awesome-python.com/categories/web-frameworks/synchronous/", + }, + ] + + parent = (site / "categories" / "web-frameworks" / "index.html").read_text(encoding="utf-8") + assert "category-breadcrumb" not in parent + + def test_sponsorship_page_contains_json_ld(self, tmp_path): + readme = textwrap.dedent("""\ + # T + + ## Projects + + **Tools** + + ## Widgets + + - [w1](https://example.com/w1) - A widget. + + # Contributing + + Done. + """) + self._copy_real_templates(tmp_path) + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + build(tmp_path) + + site = tmp_path / "website" / "output" + html = (site / "sponsorship" / "index.html").read_text(encoding="utf-8") + parser = HeadMetadataParser() + parser.feed(html) + + assert parser.title.strip() == "Sponsor Awesome Python" + assert parser.meta_by_name["description"] == ( + "Sponsorship for awesome-python: tiers, audience, and how to get your product in front of professional Python developers evaluating tools for production use." + ) + assert parser.links_by_rel["canonical"] == "https://awesome-python.com/sponsorship/" + assert 'Sponsorship' in html + + marker = '", start) + graph = {node["@type"]: node for node in json.loads(html[start:end])["@graph"]} + + assert set(graph) == {"WebSite", "WebPage", "BreadcrumbList"} + assert graph["WebPage"]["@id"] == "https://awesome-python.com/sponsorship/" + assert graph["WebPage"]["url"] == "https://awesome-python.com/sponsorship/" + assert graph["BreadcrumbList"]["itemListElement"] == [ + {"@type": "ListItem", "position": 1, "name": "Awesome Python", "item": "https://awesome-python.com/"}, + {"@type": "ListItem", "position": 2, "name": "Sponsorship", "item": "https://awesome-python.com/sponsorship/"}, + ] + + def test_build_creates_group_pages(self, tmp_path): + readme = textwrap.dedent("""\ + # T + + ## Projects + + **AI & ML** + + ## Deep Learning + + - [dl1](https://example.com/dl1) - DL. + + ## Machine Learning + + - [ml1](https://example.com/ml1) - ML. + + **Web Development** + + ## Web Frameworks + + - [wf1](https://example.com/wf1) - WF. + + # Contributing + + Done. + """) + self._copy_real_templates(tmp_path) + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + build(tmp_path) + + site = tmp_path / "website" / "output" + ai_ml = (site / "categories" / "ai-ml" / "index.html").read_text(encoding="utf-8") + web_dev = (site / "categories" / "web-development" / "index.html").read_text(encoding="utf-8") + + assert "dl1" in ai_ml + assert "ml1" in ai_ml + assert "wf1" not in ai_ml + assert "wf1" in web_dev + assert "dl1" not in web_dev + + def test_tags_link_to_pages_and_subcategory_anchors(self, tmp_path): + readme = textwrap.dedent("""\ + # T + + ## Projects + + **AI & ML** + + ## Deep Learning + + - Vision + + - [v1](https://example.com/v1) - Vision lib. + + # Contributing + + Done. + """) + self._copy_real_templates(tmp_path) + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + build(tmp_path) + + site = tmp_path / "website" / "output" + index_html = (site / "index.html").read_text(encoding="utf-8") + + category_html = (site / "categories" / "deep-learning" / "index.html").read_text(encoding="utf-8") + + for html in (index_html, category_html): + assert 'href="/categories/deep-learning/#vision"' in html + assert 'href="/categories/ai-ml/"' in html + assert "data-url=" not in html + assert 'href="/categories/deep-learning/"' in index_html + assert 'id="vision"' in category_html + + _REDIRECT_README = textwrap.dedent("""\ + # Awesome Python + + Intro. + + ## Projects + + **Tools** + + ### Widgets + + - [w1](https://example.com/w1) - A widget. + + ## Contributing + + Help! + """) + + def _write_redirects(self, tmp_path, redirects): + data_dir = tmp_path / "website" / "data" + data_dir.mkdir(parents=True) + (data_dir / "redirects.json").write_text(json.dumps(redirects), encoding="utf-8") + + def test_build_writes_redirect_stub_outside_sitemap(self, tmp_path): + self._copy_real_templates(tmp_path) + (tmp_path / "README.md").write_text(self._REDIRECT_README, encoding="utf-8") + self._write_redirects(tmp_path, {"/categories/old-widgets/": "/categories/widgets/"}) + build(tmp_path) + + site = tmp_path / "website" / "output" + stub = (site / "categories" / "old-widgets" / "index.html").read_text(encoding="utf-8") + assert '' in stub + assert '' in stub + assert "noindex" not in stub + assert "old-widgets" not in (site / "sitemap.xml").read_text(encoding="utf-8") + + def test_build_renders_category_intro_and_uses_lead_as_meta_description(self, tmp_path): + self._copy_real_templates(tmp_path) + (tmp_path / "README.md").write_text(self._REDIRECT_README, encoding="utf-8") + intros_dir = tmp_path / "website" / "data" / "category_intros" + intros_dir.mkdir(parents=True) + (intros_dir / "widgets.md").write_text("Use `w1` for most apps.\n\nSee [the docs](https://example.com/docs).\n\nHow to choose:\n\n- Small apps: w1\n", encoding="utf-8") + build(tmp_path) + + category_html = (tmp_path / "website" / "output" / "categories" / "widgets" / "index.html").read_text(encoding="utf-8") + parser = HeadMetadataParser() + parser.feed(category_html) + assert parser.meta_by_name["description"] == "Use w1 for most apps." + assert '

Use w1 for most apps.

' in category_html + assert "
  • Small apps: w1
    • " in category_html + assert 'the docs' in category_html + + def test_build_renders_category_guide_below_table(self, tmp_path): + self._copy_real_templates(tmp_path) + (tmp_path / "README.md").write_text(self._REDIRECT_README, encoding="utf-8") + intros_dir = tmp_path / "website" / "data" / "category_intros" + intros_dir.mkdir(parents=True) + (intros_dir / "widgets.md").write_text("Use w1.\n\nHow to choose:\n\n- Small apps: w1\n\nSet up w1 once per process.\n", encoding="utf-8") + build(tmp_path) + + category_html = (tmp_path / "website" / "output" / "categories" / "widgets" / "index.html").read_text(encoding="utf-8") + intro_html = category_html.split('
    ', 1)[1].split("
    ", 1)[0] + assert "
  • Small apps: w1
    • " in intro_html + assert "Set up w1" not in intro_html + guide_html = category_html.split('
    ', 1)[1] + assert "

    Widgets guide

    " in guide_html + assert "

    Set up w1 once per process.

    " in guide_html + assert category_html.index('id="guide"') > category_html.index("
    ") + assert 'Widgets guide' in category_html + + def test_section_page_groups_rows_by_use_case_in_readme_order(self, tmp_path): + readme = textwrap.dedent("""\ + # T + + ## Projects + + **Tools** + + ### Widgets + + - Small + - [w2](https://example.com/w2) - Second. + - [w1](https://example.com/w1) - First. + - Large + - [w3](https://example.com/w3) - Third. + - [sqlite3](https://docs.python.org/3/library/sqlite3.html) - Stdlib. + + # Contributing + + Done. + """) + self._copy_real_templates(tmp_path) + (tmp_path / "README.md").write_text(readme, encoding="utf-8") + build(tmp_path) + + site = tmp_path / "website" / "output" / "categories" + html = (site / "widgets" / "index.html").read_text(encoding="utf-8") + assert 'data-default-sort="editorial"' in html + positions = [html.index(marker) for marker in ('

    ', ">w2w1', ">w3Small' in html + assert 'Small' in html + assert '' in html + assert '' in html + + subcategory_html = (site / "widgets" / "small" / "index.html").read_text(encoding="utf-8") + assert 'data-default-sort="editorial"' in subcategory_html + assert "group-row" not in subcategory_html + assert subcategory_html.index(">w2w1Machine Learning') + dl_heading = html.index('Deep Learning') + assert ml_heading < html.index(">ml1dl1ml1

    Start with ml1.

    ') + assert html.index('Machine Learning') < lead < html.index(">ml1 TemplateEntry: + return TemplateEntry( + name=name, + url="", + description="", + categories=[], + groups=[], + subcategories=[], + stars=stars, + downloads=None, + owner=None, + last_commit_at=None, + source_type=source_type, + bundled=False, + badge=None, + also_see=[], + ) + + +class TestSortEntries: + def test_sorts_by_stars_descending(self): + entries = [ + _template_entry("a", 100), + _template_entry("b", 500), + _template_entry("c", 200), + ] + result = sort_entries(entries) + assert [e["name"] for e in result] == ["b", "c", "a"] + + def test_equal_stars_sorted_alphabetically(self): + entries = [ + _template_entry("beta", 100), + _template_entry("alpha", 100), + ] + result = sort_entries(entries) + assert [e["name"] for e in result] == ["alpha", "beta"] + + def test_no_stars_go_to_bottom(self): + entries = [ + _template_entry("no-stars", None), + _template_entry("has-stars", 50), + ] + result = sort_entries(entries) + assert [e["name"] for e in result] == ["has-stars", "no-stars"] + + def test_no_stars_sorted_alphabetically(self): + entries = [ + _template_entry("zebra", None), + _template_entry("apple", None), + ] + result = sort_entries(entries) + assert [e["name"] for e in result] == ["apple", "zebra"] + + def test_builtin_between_starred_and_unstarred(self): + entries = [ + _template_entry("builtin", None, "Stdlib"), + _template_entry("starred", 100), + _template_entry("unstarred", None), + ] + result = sort_entries(entries) + assert [e["name"] for e in result] == ["starred", "builtin", "unstarred"] + + +# --------------------------------------------------------------------------- +# detect_source_type +# --------------------------------------------------------------------------- + + +class TestDetectSourceType: + def test_github_repo_returns_none(self): + assert detect_source_type("https://github.com/psf/requests") is None + + def test_stdlib_url(self): + assert detect_source_type("https://docs.python.org/3/library/asyncio.html") == "Stdlib" + + def test_gitlab_url(self): + assert detect_source_type("https://gitlab.com/org/repo") == "GitLab" + + def test_bitbucket_url(self): + assert detect_source_type("https://bitbucket.org/org/repo") == "Bitbucket" + + def test_non_github_external(self): + assert detect_source_type("https://example.com/tool") == "External" + + def test_github_non_repo_returns_none(self): + assert detect_source_type("https://github.com/org/repo/wiki") is None + + +# --------------------------------------------------------------------------- +# extract_entries +# --------------------------------------------------------------------------- + + +class TestExtractEntries: + def test_basic_extraction(self): + readme = textwrap.dedent("""\ + # T + + ## Projects + + **Tools** + + ## Widgets + + - [widget](https://example.com) - A widget. + + # Contributing + + Done. + """) + groups = parse_readme(readme) + categories = [c for g in groups for c in g["categories"]] + entries = extract_entries(categories, groups) + assert len(entries) == 1 + assert entries[0]["name"] == "widget" + assert entries[0]["categories"] == ["Widgets"] + assert entries[0]["groups"] == ["Tools"] + + def test_duplicate_entry_merged(self): + readme = textwrap.dedent("""\ + # T + + ## Projects + + **Tools** + + ## Alpha + + - [shared](https://example.com/shared) - Shared lib. + + ## Beta + + - [shared](https://example.com/shared) - Shared lib. + + # Contributing + + Done. + """) + groups = parse_readme(readme) + categories = [c for g in groups for c in g["categories"]] + entries = extract_entries(categories, groups) + shared = [e for e in entries if e["name"] == "shared"] + assert len(shared) == 1 + assert sorted(shared[0]["categories"]) == ["Alpha", "Beta"] + + def test_source_type_detected(self): + readme = textwrap.dedent("""\ + # T + + ## Projects + + ## Stdlib + + - [asyncio](https://docs.python.org/3/library/asyncio.html) - Async I/O. + + # Contributing + + Done. + """) + groups = parse_readme(readme) + categories = [c for g in groups for c in g["categories"]] + entries = extract_entries(categories, groups) + assert entries[0]["source_type"] == "Stdlib" + + def test_subcategory_includes_slug_and_url(self): + readme = textwrap.dedent("""\ + # T + + ## Projects + + **Tools** + + ## Web Frameworks + + - Synchronous + + - [django](https://example.com/django) - A framework. + + # Contributing + + Done. + """) + groups = parse_readme(readme) + categories = [c for g in groups for c in g["categories"]] + entries = extract_entries(categories, groups) + assert entries[0]["subcategories"] == [ + { + "name": "Synchronous", + "value": "Web Frameworks > Synchronous", + "slug": "synchronous", + "url": "/categories/web-frameworks/synchronous/", + }, + ] + + +# --------------------------------------------------------------------------- +# annotate_entries_with_stats +# --------------------------------------------------------------------------- + + +class TestAnnotateEntriesWithStats: + def test_appends_star_count_to_bullet(self): + markdown = "- [foo](https://github.com/owner/foo) - A foo.\n" + stars = {"owner/foo": {"stars": 123, "owner": "owner"}} + assert annotate_entries_with_stats(markdown, stars, {}) == ("- [foo](https://github.com/owner/foo) - A foo. (GitHub stars: 123)\n") + + def test_appends_downloads_by_display_name(self): + markdown = "- [Foo.py](https://example.com) - A foo.\n" + assert annotate_entries_with_stats(markdown, {}, {"foo-py": 777}) == ("- [Foo.py](https://example.com) - A foo. (PyPI downloads/month: 777)\n") + + def test_appends_downloads_and_stars_together(self): + markdown = "- [foo](https://github.com/owner/foo) - A foo.\n" + stars = {"owner/foo": {"stars": 123, "owner": "owner"}} + assert annotate_entries_with_stats(markdown, stars, {"foo": 777}) == ("- [foo](https://github.com/owner/foo) - A foo. (PyPI downloads/month: 777, GitHub stars: 123)\n") + + def test_uses_first_github_link(self): + markdown = "- [foo](https://github.com/owner/foo) - A foo. Also [bar](https://github.com/owner/bar).\n" + stars = { + "owner/foo": {"stars": 10, "owner": "owner"}, + "owner/bar": {"stars": 99, "owner": "owner"}, + } + assert annotate_entries_with_stats(markdown, stars, {}) == ("- [foo](https://github.com/owner/foo) - A foo. Also [bar](https://github.com/owner/bar). (GitHub stars: 10)\n") + + def test_skips_entries_without_data(self): + markdown = "- [foo](https://github.com/owner/foo) - A foo.\n" + assert annotate_entries_with_stats(markdown, {}, {}) == markdown + + def test_skips_non_github_links_for_stars(self): + markdown = "- [foo](https://example.com) - A foo.\n" + stars = {"owner/foo": {"stars": 1, "owner": "owner"}} + assert annotate_entries_with_stats(markdown, stars, {}) == markdown + + def test_skips_non_bullet_lines(self): + markdown = "See [foo](https://github.com/owner/foo) for details.\n" + stars = {"owner/foo": {"stars": 1, "owner": "owner"}} + assert annotate_entries_with_stats(markdown, stars, {"foo": 5}) == markdown + + def test_handles_indented_bullets(self): + markdown = " - [foo](https://github.com/owner/foo)\n" + stars = {"owner/foo": {"stars": 7, "owner": "owner"}} + assert annotate_entries_with_stats(markdown, stars, {}) == (" - [foo](https://github.com/owner/foo) (GitHub stars: 7)\n") + + def test_preserves_lines_without_trailing_newline(self): + markdown = "- [foo](https://github.com/owner/foo) - A foo." + stars = {"owner/foo": {"stars": 5, "owner": "owner"}} + assert annotate_entries_with_stats(markdown, stars, {}) == ("- [foo](https://github.com/owner/foo) - A foo. (GitHub stars: 5)") + + +class TestLoadDownloads: + def test_parses_tsv_and_skips_not_found(self, tmp_path): + tsv = tmp_path / "pypi_downloads.tsv" + tsv.write_text( + "name\tpackage\tdownloads\tfetched_at\naiohttp\taiohttp\t649105404\t2026-08-16\npytorch\ttorch\t50000000\t2026-08-16\ndead-pkg\t-\tNOT_FOUND\t2026-08-16\n", + encoding="utf-8", + ) + assert load_downloads(tsv) == {"aiohttp": 649105404, "pytorch": 50000000} + + def test_missing_file_returns_empty(self, tmp_path): + assert load_downloads(tmp_path / "nope.tsv") == {} + + +# --------------------------------------------------------------------------- +# load_category_intro +# --------------------------------------------------------------------------- + + +class TestLoadCategoryIntro: + def test_splits_after_how_to_choose_list(self, tmp_path): + path = tmp_path / "widgets.md" + path.write_text("Use `w1` for most apps.\n\nHow to choose:\n\n- Small apps: w1\n- Big apps: w2\n\nConfigure w1 once.\n\nPin w2.\n", encoding="utf-8") + intro_html, guide_html, lead, lead_html = load_category_intro(path) + assert intro_html == "

    Use w1 for most apps.

    \n

    How to choose:

    \n
      \n
    • Small apps: w1
      • \n
    • Big apps: w2
      • \n
    \n" + assert guide_html == "

    Configure w1 once.

    \n

    Pin w2.

    \n" + assert lead == "Use w1 for most apps." + assert lead_html == "

    Use w1 for most apps.

    \n" + + def test_keeps_everything_above_table_without_how_to_choose_list(self, tmp_path): + path = tmp_path / "widgets.md" + path.write_text("Use w1.\n\n- Small apps: w1\n\nConfigure w1 once.\n", encoding="utf-8") + intro_html, guide_html, _, _ = load_category_intro(path) + assert "Configure w1 once." in intro_html + assert guide_html == "" + + def test_returns_empty_strings_without_intro_file(self, tmp_path): + assert load_category_intro(tmp_path / "missing.md") == ("", "", "", "") diff --git a/website/tests/test_fetch_github_stars.py b/website/tests/test_fetch_github_stars.py new file mode 100644 index 00000000..6f0297b6 --- /dev/null +++ b/website/tests/test_fetch_github_stars.py @@ -0,0 +1,266 @@ +"""Tests for fetch_github_stars module.""" + +import json + +from fetch_github_stars import ( + build_graphql_query, + extract_github_repos, + parse_graphql_response, + save_cache, +) + + +class TestExtractGithubRepos: + def test_extracts_owner_repo_from_github_url(self): + readme = "* [requests](https://github.com/psf/requests) - HTTP lib." + result = extract_github_repos(readme) + assert result == {"psf/requests"} + + def test_multiple_repos(self): + readme = "* [requests](https://github.com/psf/requests) - HTTP.\n* [flask](https://github.com/pallets/flask) - Micro." + result = extract_github_repos(readme) + assert result == {"psf/requests", "pallets/flask"} + + def test_deduplicates(self): + readme = "* [a](https://github.com/org/repo) - A.\n* [b](https://github.com/org/repo) - B." + result = extract_github_repos(readme) + assert result == {"org/repo"} + + def test_strips_fragment(self): + readme = "* [lib](https://github.com/org/repo#section) - Lib." + result = extract_github_repos(readme) + assert result == {"org/repo"} + + +class TestSaveCache: + def test_creates_directory_and_writes_json(self, tmp_path, monkeypatch): + data_dir = tmp_path / "data" + cache_file = data_dir / "stars.json" + monkeypatch.setattr("fetch_github_stars.DATA_DIR", data_dir) + monkeypatch.setattr("fetch_github_stars.CACHE_FILE", cache_file) + save_cache({"a/b": {"stars": 1}}) + assert cache_file.exists() + assert json.loads(cache_file.read_text(encoding="utf-8")) == {"a/b": {"stars": 1}} + + +class TestBuildGraphqlQuery: + def test_single_repo(self): + query = build_graphql_query(["psf/requests"]) + assert "repository" in query + assert 'owner: "psf"' in query + assert 'name: "requests"' in query + assert "stargazerCount" in query + + def test_multiple_repos_use_aliases(self): + query = build_graphql_query(["psf/requests", "pallets/flask"]) + assert "repo_0:" in query + assert "repo_1:" in query + + def test_empty_list(self): + query = build_graphql_query([]) + assert query == "" + + def test_skips_repos_with_quotes_in_name(self): + query = build_graphql_query(['org/"bad"']) + assert query == "" + + def test_skips_only_bad_repos(self): + query = build_graphql_query(["good/repo", 'bad/"repo"']) + assert "good" in query + assert "bad" not in query + + def test_skips_graphql_injection_in_owner(self): + query = build_graphql_query(['org"){evil}/repo']) + assert query == "" + + def test_skips_graphql_injection_in_name(self): + query = build_graphql_query(['org/repo"){evil}']) + assert query == "" + + def test_skips_owner_starting_with_hyphen(self): + query = build_graphql_query(["-bad/repo"]) + assert query == "" + + def test_skips_owner_starting_with_dot(self): + query = build_graphql_query([".bad/repo"]) + assert query == "" + + def test_skips_repo_starting_with_dot(self): + query = build_graphql_query(["org/.hidden"]) + assert query == "" + + def test_allows_repo_with_dots_and_underscores(self): + query = build_graphql_query(["org/my_repo.py"]) + assert 'name: "my_repo.py"' in query + + def test_allows_hyphenated_owner(self): + query = build_graphql_query(["my-org/repo"]) + assert 'owner: "my-org"' in query + + def test_skips_owner_with_underscore(self): + query = build_graphql_query(["bad_owner/repo"]) + assert query == "" + + +class TestParseGraphqlResponse: + def test_parses_star_count_and_owner(self): + data = { + "repo_0": { + "stargazerCount": 52467, + "owner": {"login": "psf"}, + } + } + repos = ["psf/requests"] + result = parse_graphql_response(data, repos) + assert result["psf/requests"]["stars"] == 52467 + assert result["psf/requests"]["owner"] == "psf" + + def test_skips_null_repos(self): + data = {"repo_0": None} + repos = ["deleted/repo"] + result = parse_graphql_response(data, repos) + assert result == {} + + def test_handles_missing_owner(self): + data = {"repo_0": {"stargazerCount": 100}} + repos = ["org/repo"] + result = parse_graphql_response(data, repos) + assert result["org/repo"]["owner"] == "" + + def test_multiple_repos(self): + data = { + "repo_0": {"stargazerCount": 100, "owner": {"login": "a"}}, + "repo_1": {"stargazerCount": 200, "owner": {"login": "b"}}, + } + repos = ["a/x", "b/y"] + result = parse_graphql_response(data, repos) + assert len(result) == 2 + assert result["a/x"]["stars"] == 100 + assert result["b/y"]["stars"] == 200 + + def test_extracts_last_commit_at(self): + data = { + "repo_0": { + "stargazerCount": 100, + "owner": {"login": "org"}, + "defaultBranchRef": {"target": {"committedDate": "2025-06-01T00:00:00Z"}}, + } + } + repos = ["org/repo"] + result = parse_graphql_response(data, repos) + assert result["org/repo"]["last_commit_at"] == "2025-06-01T00:00:00Z" + + def test_missing_default_branch_ref(self): + data = {"repo_0": {"stargazerCount": 50, "owner": {"login": "org"}}} + repos = ["org/repo"] + result = parse_graphql_response(data, repos) + assert result["org/repo"]["last_commit_at"] == "" + + +class TestMainSkipsFreshCache: + """Verify that main() skips fetching when all cache entries are fresh.""" + + def test_skips_fetch_when_cache_is_fresh(self, tmp_path, monkeypatch, capsys): + from datetime import datetime, timedelta, timezone + + from fetch_github_stars import main + + # Set up a minimal README with one repo + readme = tmp_path / "README.md" + readme.write_text("* [req](https://github.com/psf/requests) - HTTP.\n") + monkeypatch.setattr("fetch_github_stars.README_PATH", readme) + + # Pre-populate cache with a fresh entry (1 hour ago) + data_dir = tmp_path / "data" + data_dir.mkdir() + cache_file = data_dir / "github_stars.json" + now = datetime.now(timezone.utc) + fresh_cache = { + "psf/requests": { + "stars": 52000, + "owner": "psf", + "last_commit_at": "2025-01-01T00:00:00+00:00", + "fetched_at": (now - timedelta(hours=1)).isoformat(), + }, + "vinta/awesome-python": { + "stars": 230000, + "owner": "vinta", + "last_commit_at": "2025-01-01T00:00:00+00:00", + "fetched_at": (now - timedelta(hours=1)).isoformat(), + }, + } + cache_file.write_text(json.dumps(fresh_cache), encoding="utf-8") + monkeypatch.setattr("fetch_github_stars.CACHE_FILE", cache_file) + monkeypatch.setattr("fetch_github_stars.DATA_DIR", data_dir) + monkeypatch.setenv("GITHUB_TOKEN", "fake-token") + + main() + + output = capsys.readouterr().out + assert "0 repos to fetch" in output + assert "Cache is up to date" in output + + def test_fetches_when_cache_is_stale(self, tmp_path, monkeypatch, capsys): + from datetime import datetime, timedelta, timezone + from unittest.mock import MagicMock + + from fetch_github_stars import main + + # Set up a minimal README with one repo + readme = tmp_path / "README.md" + readme.write_text("* [req](https://github.com/psf/requests) - HTTP.\n") + monkeypatch.setattr("fetch_github_stars.README_PATH", readme) + + # Pre-populate cache with a stale entry (24 hours ago) + data_dir = tmp_path / "data" + data_dir.mkdir() + cache_file = data_dir / "github_stars.json" + now = datetime.now(timezone.utc) + stale_cache = { + "psf/requests": { + "stars": 52000, + "owner": "psf", + "last_commit_at": "2025-01-01T00:00:00+00:00", + "fetched_at": (now - timedelta(hours=24)).isoformat(), + }, + "vinta/awesome-python": { + "stars": 230000, + "owner": "vinta", + "last_commit_at": "2025-01-01T00:00:00+00:00", + "fetched_at": (now - timedelta(hours=24)).isoformat(), + }, + } + cache_file.write_text(json.dumps(stale_cache), encoding="utf-8") + monkeypatch.setattr("fetch_github_stars.CACHE_FILE", cache_file) + monkeypatch.setattr("fetch_github_stars.DATA_DIR", data_dir) + monkeypatch.setenv("GITHUB_TOKEN", "fake-token") + + # Mock httpx.Client to avoid real API calls + mock_response = MagicMock() + mock_response.json.return_value = { + "data": { + "repo_0": { + "stargazerCount": 53000, + "owner": {"login": "psf"}, + "defaultBranchRef": {"target": {"committedDate": "2025-06-01T00:00:00Z"}}, + }, + "repo_1": { + "stargazerCount": 231000, + "owner": {"login": "vinta"}, + "defaultBranchRef": {"target": {"committedDate": "2025-06-01T00:00:00Z"}}, + }, + } + } + mock_response.raise_for_status = MagicMock() + mock_client = MagicMock() + mock_client.__enter__ = MagicMock(return_value=mock_client) + mock_client.__exit__ = MagicMock(return_value=False) + mock_client.post.return_value = mock_response + monkeypatch.setattr("fetch_github_stars.httpx.Client", lambda **kwargs: mock_client) + + main() + + output = capsys.readouterr().out + assert "2 repos to fetch" in output + assert "Done. Fetched 2 repos" in output + mock_client.post.assert_called_once() diff --git a/website/tests/test_readme_parser.py b/website/tests/test_readme_parser.py new file mode 100644 index 00000000..707134ed --- /dev/null +++ b/website/tests/test_readme_parser.py @@ -0,0 +1,542 @@ +"""Tests for the readme_parser module.""" + +import re +import textwrap +from pathlib import Path + +import pytest + +from fetch_pypi_downloads_via_clickpy import PYPI_NAME_RE, load_overrides, normalize +from readme_parser import ( + _find_inline, + _parse_section_entries, + parse_readme, + render_inline_html, + render_inline_text, +) + +from markdown_it import MarkdownIt +from markdown_it.tree import SyntaxTreeNode + + +def _parse_inline(md_text: str) -> list[SyntaxTreeNode]: + """Helper: parse a single paragraph and return its inline children.""" + md = MarkdownIt("commonmark") + root = SyntaxTreeNode(md.parse(md_text)) + # root > paragraph > inline > children + return root.children[0].children[0].children + + +class TestRenderInlineHtml: + def test_plain_text_escapes_html(self): + children = _parse_inline("Hello & friends") + assert render_inline_html(children) == "Hello <world> & friends" + + def test_link_with_target(self): + children = _parse_inline("[name](https://example.com)") + html = render_inline_html(children) + assert 'href="https://example.com"' in html + assert 'target="_blank"' in html + assert 'rel="noopener"' in html + assert ">name" in html + + def test_emphasis(self): + children = _parse_inline("*italic* text") + assert "italic" in render_inline_html(children) + + def test_strong(self): + children = _parse_inline("**bold** text") + assert "bold" in render_inline_html(children) + + def test_code_inline(self): + children = _parse_inline("`some code`") + assert "some code" in render_inline_html(children) + + def test_mixed_link_and_text(self): + children = _parse_inline("See [foo](https://x.com) for details.") + html = render_inline_html(children) + assert "See " in html + assert ">foo" in html + assert " for details." in html + + +class TestRenderInlineText: + def test_plain_text(self): + children = _parse_inline("Hello world") + assert render_inline_text(children) == "Hello world" + + def test_link_becomes_text(self): + children = _parse_inline("See [awesome-algos](https://github.com/x/y).") + assert render_inline_text(children) == "See awesome-algos." + + def test_emphasis_stripped(self): + children = _parse_inline("*italic* text") + assert render_inline_text(children) == "italic text" + + def test_code_inline_kept(self): + children = _parse_inline("`code` here") + assert render_inline_text(children) == "code here" + + +MINIMAL_README = textwrap.dedent("""\ + # Awesome Python + + Some intro text. + + ## Projects + + ### Alpha + + _Libraries for alpha stuff._ + + - [lib-a](https://example.com/a) - Does A. + - [lib-b](https://example.com/b) - Does B. + + ### Beta + + _Tools for beta._ + + - [lib-c](https://example.com/c) - Does C. + + ## Resources + + Where to discover resources. + + ### Newsletters + + - [News One](https://example.com/n1) + - [News Two](https://example.com/n2) + + ### Podcasts + + - [Pod One](https://example.com/p1) + + ## Contributing + + Please contribute! +""") + + +GROUPED_README = textwrap.dedent("""\ + # Awesome Python + + Some intro text. + + ## Projects + + **Group One** + + ### Alpha + + _Libraries for alpha stuff._ + + - [lib-a](https://example.com/a) - Does A. + - [lib-b](https://example.com/b) - Does B. + + **Group Two** + + ### Beta + + _Tools for beta._ + + - [lib-c](https://example.com/c) - Does C. + + ### Gamma + + - [lib-d](https://example.com/d) - Does D. + + ## Resources + + Where to discover resources. + + ### Newsletters + + - [News One](https://example.com/n1) + + ## Contributing + + Please contribute! +""") + + +class TestParseReadmeSections: + def test_ungrouped_categories_go_to_other(self): + groups = parse_readme(MINIMAL_README) + assert len(groups) == 1 + assert groups[0]["name"] == "Other" + assert len(groups[0]["categories"]) == 2 + + def test_ungrouped_category_names(self): + groups = parse_readme(MINIMAL_README) + cats = groups[0]["categories"] + assert cats[0]["name"] == "Alpha" + assert cats[1]["name"] == "Beta" + + def test_category_slugs(self): + groups = parse_readme(MINIMAL_README) + cats = groups[0]["categories"] + assert cats[0]["slug"] == "alpha" + assert cats[1]["slug"] == "beta" + + def test_category_description(self): + groups = parse_readme(MINIMAL_README) + cats = groups[0]["categories"] + assert cats[0]["description"] == "Libraries for alpha stuff." + assert cats[0]["description_html"] == "Libraries for alpha stuff." + assert cats[1]["description"] == "Tools for beta." + assert cats[1]["description_html"] == "Tools for beta." + + def test_contributing_skipped(self): + groups = parse_readme(MINIMAL_README) + all_names = [] + for g in groups: + all_names.extend(c["name"] for c in g["categories"]) + assert "Contributing" not in all_names + + def test_no_projects_heading(self): + groups = parse_readme("# Just a heading\n\nSome text.\n") + assert groups == [] + + def test_no_description(self): + readme = textwrap.dedent("""\ + # Title + + ## Projects + + ### NullDesc + + - [item](https://x.com) - Thing. + + ## Resources + + ### Tips + + - [tip](https://x.com) + + ## Contributing + + Done. + """) + groups = parse_readme(readme) + cats = groups[0]["categories"] + assert cats[0]["description"] == "" + assert cats[0]["description_html"] == "" + assert cats[0]["entries"][0]["name"] == "item" + + def test_description_with_link_stripped(self): + readme = textwrap.dedent("""\ + # T + + ## Projects + + ### Algos + + _Algorithms. Also see [awesome-algos](https://example.com)._ + + - [lib](https://x.com) - Lib. + + ## Contributing + + Done. + """) + groups = parse_readme(readme) + cats = groups[0]["categories"] + assert cats[0]["description"] == "Algorithms. Also see awesome-algos." + assert cats[0]["description_html"] == 'Algorithms. Also see awesome-algos.' + + +class TestParseGroupedReadme: + def test_group_count(self): + groups = parse_readme(GROUPED_README) + assert len(groups) == 2 + + def test_group_names(self): + groups = parse_readme(GROUPED_README) + assert groups[0]["name"] == "Group One" + assert groups[1]["name"] == "Group Two" + + def test_group_slugs(self): + groups = parse_readme(GROUPED_README) + assert groups[0]["slug"] == "group-one" + assert groups[1]["slug"] == "group-two" + + def test_group_one_has_one_category(self): + groups = parse_readme(GROUPED_README) + assert len(groups[0]["categories"]) == 1 + assert groups[0]["categories"][0]["name"] == "Alpha" + + def test_group_two_has_two_categories(self): + groups = parse_readme(GROUPED_README) + assert len(groups[1]["categories"]) == 2 + assert groups[1]["categories"][0]["name"] == "Beta" + assert groups[1]["categories"][1]["name"] == "Gamma" + + def test_empty_group_skipped(self): + readme = textwrap.dedent("""\ + # T + + ## Projects + + **Empty** + + **HasCats** + + ### Cat + + - [x](https://x.com) - X. + + ## Contributing + + Done. + """) + groups = parse_readme(readme) + assert len(groups) == 1 + assert groups[0]["name"] == "HasCats" + + def test_bold_with_extra_text_not_group_marker(self): + readme = textwrap.dedent("""\ + # T + + ## Projects + + **Note:** This is not a group marker. + + ### Cat + + - [x](https://x.com) - X. + + ## Contributing + + Done. + """) + groups = parse_readme(readme) + # "Note:" has text after the strong node, so it's not a group marker + # Category goes into "Other" + assert len(groups) == 1 + assert groups[0]["name"] == "Other" + + def test_categories_before_any_group_marker(self): + readme = textwrap.dedent("""\ + # T + + ## Projects + + ### Orphan + + - [x](https://x.com) - X. + + **A Group** + + ### Grouped + + - [y](https://x.com) - Y. + + ## Contributing + + Done. + """) + groups = parse_readme(readme) + assert len(groups) == 2 + assert groups[0]["name"] == "Other" + assert groups[0]["categories"][0]["name"] == "Orphan" + assert groups[1]["name"] == "A Group" + assert groups[1]["categories"][0]["name"] == "Grouped" + + +def _content_nodes(md_text: str) -> list[SyntaxTreeNode]: + """Helper: parse markdown and return all block nodes.""" + md = MarkdownIt("commonmark") + root = SyntaxTreeNode(md.parse(md_text)) + return root.children + + +class TestParseSectionEntries: + def test_flat_entries(self): + nodes = _content_nodes("- [django](https://example.com/d) - A web framework.\n- [flask](https://example.com/f) - A micro framework.\n") + entries = _parse_section_entries(nodes) + assert len(entries) == 2 + assert entries[0]["name"] == "django" + assert entries[0]["url"] == "https://example.com/d" + assert "web framework" in entries[0]["description"] + assert entries[0]["also_see"] == [] + assert entries[1]["name"] == "flask" + + def test_link_only_entry(self): + nodes = _content_nodes("- [tool](https://x.com)\n") + entries = _parse_section_entries(nodes) + assert len(entries) == 1 + assert entries[0]["name"] == "tool" + assert entries[0]["description"] == "" + + def test_subcategorized_entries(self): + nodes = _content_nodes("- Algorithms\n - [algos](https://x.com/a) - Algo lib.\n - [sorts](https://x.com/s) - Sort lib.\n- Design Patterns\n - [patterns](https://x.com/p) - Pattern lib.\n") + entries = _parse_section_entries(nodes) + assert len(entries) == 3 + assert entries[0]["name"] == "algos" + assert entries[2]["name"] == "patterns" + + def test_text_before_link_is_subcategory(self): + nodes = _content_nodes( + "- MySQL - [awesome-mysql](http://example.com/awesome-mysql/)\n" + " - [mysqlclient](https://example.com/mysqlclient) - MySQL connector.\n" + " - [pymysql](https://example.com/pymysql) - Pure Python MySQL driver.\n" + ) + entries = _parse_section_entries(nodes) + # awesome-mysql is a subcategory label, not an entry + assert len(entries) == 2 + names = [e["name"] for e in entries] + assert "awesome-mysql" not in names + assert "mysqlclient" in names + assert "pymysql" in names + + def test_also_see_sub_entries(self): + nodes = _content_nodes( + "- [asyncio](https://docs.python.org/3/library/asyncio.html) - Async I/O.\n" + " - [awesome-asyncio](https://github.com/timofurrer/awesome-asyncio)\n" + "- [trio](https://github.com/python-trio/trio) - Friendly async.\n" + ) + entries = _parse_section_entries(nodes) + assert len(entries) == 2 + assert entries[0]["name"] == "asyncio" + assert len(entries[0]["also_see"]) == 1 + assert entries[0]["also_see"][0]["name"] == "awesome-asyncio" + assert entries[1]["name"] == "trio" + assert entries[1]["also_see"] == [] + + def test_entry_count_includes_also_see(self): + readme = textwrap.dedent("""\ + # T + + ## Projects + + ### Async + + - [asyncio](https://x.com) - Async I/O. + - [awesome-asyncio](https://y.com) + - [trio](https://z.com) - Friendly async. + + ## Contributing + + Done. + """) + groups = parse_readme(readme) + cats = groups[0]["categories"] + # 2 main entries + 1 also_see = 3 + assert cats[0]["entry_count"] == 3 + + def test_description_html_escapes_xss(self): + nodes = _content_nodes("- [lib](https://x.com) - A lib.\n") + entries = _parse_section_entries(nodes) + assert "