diff --git a/README.md b/README.md index 73a63a73..ee4bd8ff 100644 --- a/README.md +++ b/README.md @@ -168,10 +168,13 @@ _Libraries for building AI applications, LLM integrations, and autonomous agents - Pre-trained Models - [transformers](https://github.com/huggingface/transformers) - The model-definition framework for pretrained models in text, computer vision, audio, video, and multimodal tasks, for inference and training. - [sentence-transformers](https://github.com/huggingface/sentence-transformers) - Compute embeddings, run retrieval, and rerank results with pretrained and fine-tuned models. -- LLM Inference and Serving +- Local LLM Inference + - [ollama](https://github.com/ollama/ollama-python) - Python client for Ollama, which downloads and runs open-weight LLMs locally. + - [llama-cpp-python](https://github.com/abetlen/llama-cpp-python) - Python bindings for llama.cpp, for running GGUF models locally. + - [mlx-lm](https://github.com/ml-explore/mlx-lm) - Run and fine-tune large language models on Apple Silicon with MLX. +- LLM Serving - [vllm](https://github.com/vllm-project/vllm) - A high-throughput and memory-efficient inference and serving engine for LLMs. - [sglang](https://github.com/sgl-project/sglang) - A high-performance serving framework for large language models and multimodal models. - - [mlx-lm](https://github.com/ml-explore/mlx-lm) - Run and fine-tune large language models on Apple Silicon with MLX. - LLM Gateways - [litellm](https://github.com/BerriAI/litellm) - Call 100+ LLMs using OpenAI format. - Image and Video Generation @@ -207,6 +210,7 @@ _Libraries for Machine Learning. Also see [awesome-machine-learning](https://git - General - [scikit-learn](https://github.com/scikit-learn/scikit-learn) - The most popular Python library for Machine Learning with extensive documentation and community support. + - [imbalanced-learn](https://github.com/scikit-learn-contrib/imbalanced-learn) - Resampling techniques for datasets with imbalanced classes, compatible with scikit-learn. - [pgmpy](https://github.com/pgmpy/pgmpy) - A Python library for causal and probabilistic reasoning with graphical models. - [feature-engine](https://github.com/feature-engine/feature_engine) - sklearn compatible API with the widest toolset for feature engineering and selection. - Gradient Boosting @@ -218,6 +222,9 @@ _Libraries for Machine Learning. Also see [awesome-machine-learning](https://git - [statsforecast](https://github.com/Nixtla/statsforecast) - Fast statistical forecasting models such as ARIMA, ETS, and Theta, compiled with numba. - [sktime](https://github.com/sktime/sktime) - A unified scikit-learn-style framework for forecasting and other time-series learning tasks. - [timesfm](https://github.com/google-research/timesfm) - A pretrained foundation model from Google Research for time-series forecasting, with non-commercial default weights. +- Hyperparameter Optimization + - [optuna](https://github.com/optuna/optuna) - A hyperparameter optimization framework with define-by-run search spaces and pruning of unpromising trials. + - [hyperopt](https://github.com/hyperopt/hyperopt) - Distributed asynchronous hyperparameter optimization with Tree-structured Parzen Estimators. ### Natural Language Processing @@ -231,6 +238,7 @@ _Libraries for working with human languages._ - Chinese - [jieba](https://github.com/fxsjy/jieba) - The most popular Chinese text segmentation library. - [pypinyin](https://github.com/mozillazg/python-pinyin) - Convert Chinese hanzi (漢字) to pinyin (拼音). + - [OpenCC](https://github.com/BYVoid/OpenCC) - Conversion between Traditional and Simplified Chinese, with regional variants and phrase-level rules. - [pangu.py](https://github.com/vinta/pangu.py) - Paranoid text spacing. ### Computer Vision @@ -399,7 +407,7 @@ _Libraries to automate web scraping and extract web content._ - [scrapy](https://github.com/scrapy/scrapy) - A fast high-level web crawling and scraping framework. - [crawl4ai](https://github.com/unclecode/crawl4ai) - An open-source, LLM-friendly web crawler that provides lightning-fast, structured data extraction specifically designed for AI agents. - [stagehand](https://github.com/browserbase/stagehand) - A fast and token-efficient browser automation SDK to extract data and perform self-healing actions on web pages. - - [jev-ultrafast](https://github.com/browser-use/jev-ultrafast) - A fast browser agent that picks actions from an indexed table of page elements through TypeSafe's hosted Jev API, using a small LLM only to type text. + - [scrapling](https://github.com/D4Vinci/Scrapling) - An adaptive scraping framework whose selectors relocate elements after page changes, with stealth fetchers and spiders. - Content Extraction - [markdownify](https://github.com/matthewwithanm/python-markdownify) - Convert HTML to Markdown, with customizable tag handling. - [feedparser](https://github.com/kurtmckee/feedparser) - Universal feed parser. @@ -441,8 +449,10 @@ _Libraries for connecting and operating databases._ - PostgreSQL - [awesome-postgres](https://github.com/dhamaniasad/awesome-postgres) - [psycopg](https://github.com/psycopg/psycopg) - A PostgreSQL adapter for Python, the successor to psycopg2. - [asyncpg](https://github.com/MagicStack/asyncpg) - A fast PostgreSQL Database Client Library for Python/asyncio. + - [pgvector](https://github.com/pgvector/pgvector-python) - pgvector support for Python, adding vector types and similarity search to psycopg, asyncpg, SQLAlchemy, and Django. - SQLite - [awesome-sqlite](https://github.com/planetopendata/awesome-sqlite) - [sqlite3](https://docs.python.org/3/library/sqlite3.html) - (Python standard library) SQLite interface compliant with DB-API 2.0. + - [apsw](https://github.com/rogerbinns/apsw) - Another Python SQLite Wrapper, exposing the complete SQLite C API. - [sqlite-utils](https://github.com/simonw/sqlite-utils) - Python CLI utility and library for manipulating SQLite databases. - ClickHouse - [clickhouse-connect](https://github.com/ClickHouse/clickhouse-connect) - The official ClickHouse client, with SQLAlchemy and Superset connectors. @@ -464,6 +474,7 @@ _In-process databases usable directly from Python._ - [duckdb](https://github.com/duckdb/duckdb) - An in-process SQL OLAP database management system; optimized for analytics and fast queries, similar to SQLite but for analytical workloads. - [chdb](https://github.com/chdb-io/chdb) - In-process OLAP SQL engine with the full ClickHouse dialect, zero-copy pandas/Arrow interop, and federation to remote ClickHouse clusters via `remoteSecure()`. - Vector + - [faiss-cpu](https://github.com/facebookresearch/faiss) - A library for efficient similarity search and clustering of dense vectors. - [lancedb](https://github.com/lancedb/lancedb) - A developer-friendly embedded retrieval database for multimodal AI. - [chromadb](https://github.com/chroma-core/chroma) - An open-source embedding database for building AI applications with embeddings and semantic search. - [zvec](https://github.com/alibaba/zvec) - A lightweight, in-process vector database that embeds directly into applications. @@ -476,9 +487,9 @@ _Libraries for caching data._ - [cachetools](https://github.com/tkem/cachetools) - Extensible memoizing collections and decorators. - [diskcache](https://github.com/grantjenks/python-diskcache) - SQLite and file backed cache backend, compatible with Django. +- [django-redis](https://github.com/jazzband/django-redis) - Full-featured Redis cache backend for Django. - [hishel](https://github.com/karpetrosyan/hishel) - RFC 9111 compliant HTTP caching for clients like httpx and requests and servers like FastAPI, with sync and async support. - [django-cacheops](https://github.com/Suor/django-cacheops) - A slick ORM cache with automatic granular event-driven invalidation. -- [dogpile.cache](https://github.com/sqlalchemy/dogpile.cache) - dogpile.cache is a next generation replacement for Beaker made by the same authors. ### Search @@ -493,10 +504,14 @@ _Libraries and software for indexing and performing search queries on data._ _Libraries for serializing complex data types._ -- [msgpack](https://github.com/msgpack/msgpack-python) - MessagePack serializer implementation for Python. -- [orjson](https://github.com/ijl/orjson) - Fast, correct JSON library. -- [marshmallow](https://github.com/marshmallow-code/marshmallow) - A lightweight library for converting complex objects to and from simple Python datatypes. -- [msgspec](https://github.com/msgspec/msgspec) - A fast serialization and validation library with built-in support for JSON, MessagePack, YAML, and TOML. +- Data Formats + - [msgpack](https://github.com/msgpack/msgpack-python) - MessagePack serializer implementation for Python. + - [orjson](https://github.com/ijl/orjson) - Fast, correct JSON library. + - [marshmallow](https://github.com/marshmallow-code/marshmallow) - A lightweight library for converting complex objects to and from simple Python datatypes. + - [msgspec](https://github.com/msgspec/msgspec) - A fast serialization and validation library with built-in support for JSON, MessagePack, YAML, and TOML. +- Object Pickling + - [cloudpickle](https://github.com/cloudpipe/cloudpickle) - Extends pickle to functions, lambdas, and interactively defined classes, for shipping code to remote workers. + - [dill](https://github.com/uqfoundation/dill) - Extends pickle to serialize almost any Python object, including closures and whole interpreter sessions. **Data & Science** @@ -643,6 +658,7 @@ _Tools of static analysis, linters and code quality checkers. Also see [awesome- - [bandit](https://github.com/PyCQA/bandit) - A tool designed to find common security issues in Python code. - Refactoring - [rope](https://github.com/python-rope/rope) - Rope is a python refactoring library. + - [ast-grep-cli](https://github.com/ast-grep/ast-grep) - A CLI tool for structural code search, linting, and rewriting based on syntax trees. - Type Checkers - [awesome-python-typing](https://github.com/typeddjango/awesome-python-typing) - [mypy](https://github.com/python/mypy) - A static type checker for Python. - [ty](https://github.com/astral-sh/ty) - An extremely fast Python type checker and language server. @@ -700,6 +716,7 @@ _Libraries for debugging code._ - [scalene](https://github.com/plasma-umass/scalene) - A high-performance, high-precision CPU, GPU, and memory profiler for Python. - Others - [django-debug-toolbar](https://github.com/django-commons/django-debug-toolbar) - Display various debug information for Django. + - [django-silk](https://github.com/jazzband/django-silk) - Live profiling and inspection of HTTP requests and database queries for Django. - [icecream](https://github.com/gruns/icecream) - Inspect variables, expressions, and program execution with a single, simple function call. - [flask-debugtoolbar](https://github.com/pallets-eco/flask-debugtoolbar) - A port of the django-debug-toolbar to flask. @@ -922,6 +939,7 @@ _Libraries for parsing and manipulating specific file formats._ - [tablib](https://github.com/jazzband/tablib) - A module for Tabular Datasets in XLS, CSV, JSON, YAML. - File Conversion - [markitdown](https://github.com/microsoft/markitdown) - Python tool for converting files and office documents to Markdown. + - [pypandoc](https://github.com/JessicaTegner/pypandoc) - A thin wrapper for pandoc, the universal document converter. - [docling](https://github.com/docling-project/docling) - Library for converting documents into structured data. - Excel - [openpyxl](https://openpyxl.readthedocs.io/en/stable/) - A library for reading and writing Excel 2010 xlsx/xlsm/xltx/xltm files. @@ -949,11 +967,14 @@ _Libraries for parsing and manipulating specific file formats._ _Libraries for file manipulation._ -- [mimetypes](https://docs.python.org/3/library/mimetypes.html) - (Python standard library) Map filenames to MIME types. -- [pathlib](https://docs.python.org/3/library/pathlib.html) - (Python standard library) A cross-platform, object-oriented path library. -- [watchfiles](https://github.com/samuelcolvin/watchfiles) - Simple, modern and fast file watching and code reload in python. -- [watchdog](https://github.com/gorakhargosh/watchdog) - API and shell utilities to monitor file system events. -- [python-magic](https://github.com/ahupp/python-magic) - A Python interface to the libmagic file type identification library. +- Paths and File Watching + - [pathlib](https://docs.python.org/3/library/pathlib.html) - (Python standard library) A cross-platform, object-oriented path library. + - [watchfiles](https://github.com/samuelcolvin/watchfiles) - Simple, modern and fast file watching and code reload in python. + - [watchdog](https://github.com/gorakhargosh/watchdog) - API and shell utilities to monitor file system events. +- File Type Detection + - [mimetypes](https://docs.python.org/3/library/mimetypes.html) - (Python standard library) Map filenames to MIME types. + - [python-magic](https://github.com/ahupp/python-magic) - A Python interface to the libmagic file type identification library. + - [magika](https://github.com/google/magika) - Detects file content types with a deep-learning model. **Media** @@ -1069,6 +1090,7 @@ _Libraries for Python version and virtual environment management._ - [virtualenv](https://github.com/pypa/virtualenv) - A tool to create isolated Python environments. - [uv](https://github.com/astral-sh/uv) - An extremely fast Python version, package and project manager, written in Rust. - [pyenv](https://github.com/pyenv/pyenv) - Simple Python version management. +- [mise](https://github.com/jdx/mise) - A polyglot tool version manager that installs Python versions and reads .python-version files. ### Package Management @@ -1141,10 +1163,13 @@ _Frameworks and tools for penetration testing._ ### Supply Chain Security -_Tools for auditing dependencies against known vulnerabilities._ +_Tools for auditing dependencies against known vulnerabilities and scanning for leaked secrets._ -- [pip-audit](https://github.com/pypa/pip-audit) - Audits Python environments and dependency trees for known vulnerabilities, using the Python Packaging Advisory Database or OSV. -- [uv-audit](https://github.com/astral-sh/uv) - (part of uv) uv's [dependency vulnerability scanning](https://docs.astral.sh/uv/reference/cli/#uv-audit) backed by OSV. +- Dependency Audits + - [pip-audit](https://github.com/pypa/pip-audit) - Audits Python environments and dependency trees for known vulnerabilities, using the Python Packaging Advisory Database or OSV. + - [uv-audit](https://github.com/astral-sh/uv) - (part of uv) uv's [dependency vulnerability scanning](https://docs.astral.sh/uv/reference/cli/#uv-audit) backed by OSV. +- Secret Scanning + - [betterleaks](https://github.com/betterleaks/betterleaks) - Finds leaked secrets in git history, directories, and files, from the original author of gitleaks. ### Web Security diff --git a/docs/audit-logs.md b/docs/audit-logs.md index 94d295c3..14c6713e 100644 --- a/docs/audit-logs.md +++ b/docs/audit-logs.md @@ -39,7 +39,8 @@ Entries admitted as challengers without adoption-trajectory evidence, by maintai - seleniumbase -- admitted 2026-08-16 (PR #3284) as a Browser Automation challenger. It wraps selenium, the successor trajectory in the use case belongs to playwright, and its distinct value is its stealth/UC mode. Admitted because that stealth value counts toward Browser Automation. - semantica -- kept 2026-10-01 in the AI and Agents audit as a Data Layer challenger at 16,582 downloads/month on ClickPy, with no adoption-trajectory evidence found. Kept by maintainer decision. -- jev-ultrafast -- kept 2026-09-24 as a Web Scraping Frameworks challenger, recorded 2026-10-02. Admitted while the repo was under a month old and not on PyPI, so it has no download signal and no adoption-trajectory evidence; its README calls it an MVP and it needs a hosted TypeSafe API key. Kept by maintainer decision. +- hyperopt -- admitted 2026-10-16 as a Hyperparameter Optimization challenger at 1,793,872 downloads/month on ClickPy. Its 0.3.0 release came after a 4.5-year gap and it has no adoption-trajectory evidence. Admitted by maintainer decision. +- magika -- admitted 2026-10-16 as a File Type Detection challenger at 16,388,380 downloads/month on ClickPy. markitdown requires it, so most of that count is transitive, and no adoption-trajectory evidence was found. Admitted by maintainer decision. ### Cap Exceptions diff --git a/website/data/pypi_name_overrides.json b/website/data/pypi_name_overrides.json index daf57097..0ee6dfa6 100644 --- a/website/data/pypi_name_overrides.json +++ b/website/data/pypi_name_overrides.json @@ -1,6 +1,7 @@ { "autobahn-python": { "package": "autobahn", "reason": null }, "azure-sdk-for-python": { "package": null, "reason": "monorepo umbrella; ships as many azure-* packages, no single package represents it", "badge": "Multiple on PyPI" }, + "betterleaks": { "package": null, "reason": "not published on PyPI" }, "cloud-init": { "package": null, "reason": "installed via distro images, not pip" }, "concurrent-futures": { "package": null, "reason": "stdlib module" }, "conda": { "package": null, "reason": "installed via Miniforge or Anaconda installers; PyPI conda is a stale 2017 upload" }, @@ -16,11 +17,11 @@ "google-cloud-python": { "package": null, "reason": "monorepo umbrella; ships as many google-cloud-* packages", "badge": "Multiple on PyPI" }, "graphify": { "package": "graphifyy", "reason": null }, "httpx-url": { "package": null, "reason": "class in the httpx package, not a package" }, - "jev-ultrafast": { "package": null, "reason": "not published on PyPI" }, "jinja": { "package": "jinja2", "reason": "PyPI jinja is Jinja1; current releases publish as jinja2" }, "mem0": { "package": "mem0ai", "reason": null }, "micropython": { "package": null, "reason": "interpreter distributed as firmware builds, not pip" }, "mimetypes": { "package": null, "reason": "stdlib module" }, + "mise": { "package": null, "reason": "not published on PyPI; PyPI mise is an unrelated package" }, "odoo": { "package": null, "reason": "self-hosted suite installed from odoo.com, not pip" }, "pangu-py": { "package": "pangu", "reason": null }, "playwright-python": { "package": "playwright", "reason": null }, diff --git a/website/data/redirects.json b/website/data/redirects.json index 51e5593b..d5dc47fd 100644 --- a/website/data/redirects.json +++ b/website/data/redirects.json @@ -1,5 +1,6 @@ { "/categories/": "/", + "/categories/ai-and-agents/llm-inference-and-serving/": "/categories/ai-and-agents/llm-serving/", "/categories/ai-and-agents/pre-trained-models-and-inference/": "/categories/ai-and-agents/pre-trained-models/", "/categories/cli-development/cli-development/": "/categories/cli-development/", "/categories/cli-tools/productivity-tools/": "/categories/cli-tools/",