From edee0f545d042d4aa59e3e8055b170d948e0243d Mon Sep 17 00:00:00 2001 From: Vinta Chen Date: Sun, 15 Nov 2026 00:44:08 +0800 Subject: [PATCH 1/7] feat: add compression.zstd to File Manipulation Mints the Compression use case. zstd joined the standard library in Python 3.14, and urllib3 and aiohttp now depend on its backport for older Pythons. The null override keeps the downloads sweep from querying a same-named PyPI package. Co-Authored-By: Claude --- README.md | 2 ++ website/data/pypi_name_overrides.json | 1 + 2 files changed, 3 insertions(+) diff --git a/README.md b/README.md index ed9177a3..17a31514 100644 --- a/README.md +++ b/README.md @@ -999,6 +999,8 @@ _Libraries for file manipulation._ - Remote Files - [fsspec](https://github.com/fsspec/filesystem_spec) - A unified filesystem interface for local, cloud, and remote storage. - [smart-open](https://github.com/piskvorky/smart_open) - Stream files from S3, GCS, Azure, HDFS, and HTTP as a drop-in replacement for open(). +- Compression + - [compression.zstd](https://docs.python.org/3/library/compression.zstd.html) - (Python standard library) Compress and decompress data in the Zstandard format. **Media** diff --git a/website/data/pypi_name_overrides.json b/website/data/pypi_name_overrides.json index ee534c13..4fa0ed6e 100644 --- a/website/data/pypi_name_overrides.json +++ b/website/data/pypi_name_overrides.json @@ -3,6 +3,7 @@ "azure-sdk-for-python": { "package": null, "reason": "monorepo umbrella; ships as many azure-* packages, no single package represents it", "badge": "Multiple on PyPI" }, "betterleaks": { "package": null, "reason": "not published on PyPI" }, "cloud-init": { "package": null, "reason": "installed via distro images, not pip" }, + "compression-zstd": { "package": null, "reason": "stdlib module" }, "concurrent-futures": { "package": null, "reason": "stdlib module" }, "conda": { "package": null, "reason": "installed via Miniforge or Anaconda installers; PyPI conda is a stale 2017 upload" }, "cpython": { "package": null, "reason": "not pip-installable; the cpython PyPI name was squatted" }, From ebaa07e8cd61a640235f2c94fe3aa8fa1b0c9ac4 Mon Sep 17 00:00:00 2001 From: Vinta Chen Date: Sun, 15 Nov 2026 00:44:20 +0800 Subject: [PATCH 2/7] feat: add gzip to File Manipulation The stdlib is the obvious choice for the most common compression format. Co-Authored-By: Claude --- README.md | 1 + website/data/pypi_name_overrides.json | 1 + 2 files changed, 2 insertions(+) diff --git a/README.md b/README.md index 17a31514..2807c84e 100644 --- a/README.md +++ b/README.md @@ -1001,6 +1001,7 @@ _Libraries for file manipulation._ - [smart-open](https://github.com/piskvorky/smart_open) - Stream files from S3, GCS, Azure, HDFS, and HTTP as a drop-in replacement for open(). - Compression - [compression.zstd](https://docs.python.org/3/library/compression.zstd.html) - (Python standard library) Compress and decompress data in the Zstandard format. + - [gzip](https://docs.python.org/3/library/gzip.html) - (Python standard library) Read and write gzip-compressed files and data. **Media** diff --git a/website/data/pypi_name_overrides.json b/website/data/pypi_name_overrides.json index 4fa0ed6e..4366e5ec 100644 --- a/website/data/pypi_name_overrides.json +++ b/website/data/pypi_name_overrides.json @@ -18,6 +18,7 @@ "geodjango": { "package": null, "reason": "part of Django (django.contrib.gis), not a package" }, "google-cloud-python": { "package": null, "reason": "monorepo umbrella; ships as many google-cloud-* packages", "badge": "Multiple on PyPI" }, "graphify": { "package": "graphifyy", "reason": null }, + "gzip": { "package": null, "reason": "stdlib module" }, "httpx-url": { "package": null, "reason": "class in the httpx package, not a package" }, "jinja": { "package": "jinja2", "reason": "PyPI jinja is Jinja1; current releases publish as jinja2" }, "mem0": { "package": "mem0ai", "reason": null }, From d5011bb39d2698cd8be459ba3b8fd2a909e6ca55 Mon Sep 17 00:00:00 2001 From: Vinta Chen Date: Sun, 15 Nov 2026 00:44:27 +0800 Subject: [PATCH 3/7] feat: add zstandard to File Manipulation The third-party zstd binding with the widest API and Python 3.9+ support, 189.8M downloads/month (pypistats, 2026-11-15), part of it via extras such as httpx[zstd]. Co-Authored-By: Claude --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 2807c84e..1a06830a 100644 --- a/README.md +++ b/README.md @@ -1002,6 +1002,7 @@ _Libraries for file manipulation._ - Compression - [compression.zstd](https://docs.python.org/3/library/compression.zstd.html) - (Python standard library) Compress and decompress data in the Zstandard format. - [gzip](https://docs.python.org/3/library/gzip.html) - (Python standard library) Read and write gzip-compressed files and data. + - [zstandard](https://github.com/indygreg/python-zstandard) - Python bindings to the Zstandard (zstd) compression library. **Media** From 7440501bbbbd6cdf7a8176455e6129286a9c71ec Mon Sep 17 00:00:00 2001 From: Vinta Chen Date: Sun, 15 Nov 2026 00:44:35 +0800 Subject: [PATCH 4/7] feat: add brotli to File Manipulation Google's official Brotli binding, 118.1M downloads/month (pypistats, 2026-11-15), most of it via the brotli extras of urllib3, httpx, aiohttp and fonttools. Co-Authored-By: Claude --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 1a06830a..a31c7e77 100644 --- a/README.md +++ b/README.md @@ -1003,6 +1003,7 @@ _Libraries for file manipulation._ - [compression.zstd](https://docs.python.org/3/library/compression.zstd.html) - (Python standard library) Compress and decompress data in the Zstandard format. - [gzip](https://docs.python.org/3/library/gzip.html) - (Python standard library) Read and write gzip-compressed files and data. - [zstandard](https://github.com/indygreg/python-zstandard) - Python bindings to the Zstandard (zstd) compression library. + - [brotli](https://github.com/google/brotli) - Python bindings for Google's Brotli compression library. **Media** From 26ec7d5129d2e42dfcabc9a03b187379bf87322d Mon Sep 17 00:00:00 2001 From: Vinta Chen Date: Sun, 15 Nov 2026 00:44:35 +0800 Subject: [PATCH 5/7] feat: add lz4 to File Manipulation The standard LZ4 binding, 86.1M downloads/month (pypistats, 2026-11-15), part of it required by clickhouse-connect below Python 3.14. Co-Authored-By: Claude --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index a31c7e77..ebfaf9cc 100644 --- a/README.md +++ b/README.md @@ -1004,6 +1004,7 @@ _Libraries for file manipulation._ - [gzip](https://docs.python.org/3/library/gzip.html) - (Python standard library) Read and write gzip-compressed files and data. - [zstandard](https://github.com/indygreg/python-zstandard) - Python bindings to the Zstandard (zstd) compression library. - [brotli](https://github.com/google/brotli) - Python bindings for Google's Brotli compression library. + - [lz4](https://github.com/python-lz4/python-lz4) - Python bindings for the LZ4 fast compression library. **Media** From bc2339d45493510360ecc88619fd7b40d353e1f1 Mon Sep 17 00:00:00 2001 From: Vinta Chen Date: Sun, 15 Nov 2026 00:44:35 +0800 Subject: [PATCH 6/7] docs: log rejected compression candidates Co-Authored-By: Claude --- docs/audit-logs.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/audit-logs.md b/docs/audit-logs.md index 17b71e6e..df29e68e 100644 --- a/docs/audit-logs.md +++ b/docs/audit-logs.md @@ -104,6 +104,15 @@ Projects the maintainer decided not to list, with the evidence at decision time, - m-bain/whisperX -- 2026-11-08, 342,919 downloads/month (pypistats, package whisperx): A wrapper over faster-whisper for word timestamps and diarization, not a successor to any Speech entry. Its monthly downloads dipped instead of rising, and its release line pins torch~=2.8.0 with requires_python <3.14. - openai/codex -- 2026-11-08, no PyPI download signal: A language-agnostic terminal coding agent written in Rust. Not Python work. - openai/openai-python -- 2026-11-08, 295,856,631 downloads/month (pypistats, package openai): A vendor API client, declined with Third-party APIs (2d98426); OpenAI is already listed through openai-agents. Much of its count is pulled in by other libraries. +- milesgranger/cramjam -- 2026-11-15, 31,936,738 downloads/month (pypistats): fastparquet and python-snappy require it, and few people pick it directly. Compression is at cap. +- intake/python-snappy -- 2026-11-15, 11,089,388 downloads/month (pypistats): Now a thin wrapper over cramjam. Snappy is mostly used inside Parquet and Kafka libraries, not called directly. +- Rogdham/backports.zstd -- 2026-11-15, 57,800,006 downloads/month (pypistats, package backports-zstd): A backport of the listed compression.zstd for Python 3.10–3.13 that won't install on 3.14. The stdlib entry covers it. +- Rogdham/pyzstd -- 2026-11-15, 6,838,415 downloads/month (pypistats): compression.zstd was built from it, and its own docs guide users to the stdlib. Its only remaining edge is the seekable format. +- pycompression/python-zlib-ng -- 2026-11-15, 3,387,853 downloads/month (pypistats, package zlib-ng): A faster drop-in for zlib and gzip with a niche audience (16 stars). +- pycompression/python-isal -- 2026-11-15, 2,960,582 downloads/month (pypistats, package isal): A faster drop-in for zlib and gzip using Intel ISA-L, with a niche audience (61 stars). +- python-hyper/brotlicffi -- 2026-11-15, 1,946,691 downloads/month (pypistats): The PyPy version of the listed brotli. HTTP clients pick it automatically on non-CPython. +- uktrade/stream-zip -- 2026-11-15, 980,636 downloads/month (pypistats): Builds ZIP files as a stream. Niche (21 stars), and no archives use case exists. +- Changaco/python-libarchive-c -- 2026-11-15, 824,581 downloads/month (pypistats, package libarchive-c): A libarchive binding with low demand (85 stars) that needs the system library. No archives use case exists. ## Last Audited From a23795f16ac32fd620e6a350b1e7c08f0b13b7d1 Mon Sep 17 00:00:00 2001 From: Vinta Chen Date: Sun, 15 Nov 2026 00:47:41 +0800 Subject: [PATCH 7/7] docs: add compression libraries to File Manipulation intro Co-Authored-By: Claude --- website/data/category_intros/file-manipulation.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/website/data/category_intros/file-manipulation.md b/website/data/category_intros/file-manipulation.md index 616ea657..510264cd 100644 --- a/website/data/category_intros/file-manipulation.md +++ b/website/data/category_intros/file-manipulation.md @@ -10,6 +10,10 @@ How to choose: - Telling text file types apart, like source code from config files: Magika - Listing, globbing, and copying files on cloud or remote storage: fsspec - Streaming one large remote file through a drop-in `open()`: smart_open +- Zstandard files and data: compression.zstd, or zstandard for multi-threaded compression +- Reading and writing .gz files: gzip +- Denser compression than gzip at a similar speed: brotli +- Compressing and decompressing as fast as possible: lz4 If you aren't sure which pathlib class you need, the docs say [`Path` is most likely it](https://docs.python.org/3/library/pathlib.html): it makes a concrete path for the platform your code runs on. [Join paths with the `/` operator](https://docs.python.org/3/library/pathlib.html#basic-use), and list a directory with `iterdir()` or `glob()`. @@ -26,3 +30,13 @@ Magika runs [a compact deep-learning model](https://securityresearch.google/magi fsspec gives you [one API for local, remote, and embedded file systems](https://filesystem-spec.readthedocs.io/en/latest/), and it works the same whatever the storage backend. Get a file system by protocol name, like `fsspec.filesystem("s3")`, then call methods that [follow posix naming and the `os` module](https://filesystem-spec.readthedocs.io/en/latest/usage.html#use-a-file-system): `ls()`, `glob()`, `cp()`, and `rm()`. For one file, [`fsspec.open()` picks the backend from the URL](https://filesystem-spec.readthedocs.io/en/latest/usage.html#higher-level), and opens the file only when you enter its `with` block. smart_open is [a drop-in replacement for Python's built-in `open()`](https://github.com/piskvorky/smart_open#what) that streams very large files from and to S3, GCS, Azure Blob Storage, HDFS, HTTP, and SFTP. Import its `open` in place of the built-in, and pass it a URL like `s3://bucket/key`. A file name ending in `.gz`, `.bz2`, or `.zst` gets [compressed or decompressed on the fly](https://github.com/piskvorky/smart_open#compression-handling). To use your own credentials or settings, [pass the storage service's client object in `transport_params`](https://github.com/piskvorky/smart_open#transport-specific-options). + +compression.zstd covers files, bytes, and streams. [`compression.zstd.open()`](https://docs.python.org/3/library/compression.zstd.html#compression.zstd.open) reads and writes .zst files as file objects, `compress()` and `decompress()` handle bytes in one call, and `ZstdCompressor` and `ZstdDecompressor` work through data a chunk at a time. + +gzip [opens .gz files like regular files](https://docs.python.org/3/library/gzip.html#gzip.open): `gzip.open()` takes `'rb'` or `'wb'` for bytes, and `'rt'` or `'wt'` for text. For data already in memory, call `gzip.compress()` and `gzip.decompress()`. + +zstandard is [the most feature-rich zstd binding](https://python-zstandard.readthedocs.io/en/latest/projectinfo.html#comparison-to-other-python-bindings). Still, its own docs say the standard library is probably the easiest way to use zstd, if it works for you. Use its one-shot APIs for data that fits in memory, and [its streaming APIs for large inputs](https://python-zstandard.readthedocs.io/en/latest/api_usage.html#choosing-an-api). Pass `threads` to `ZstdCompressor` to [compress on several cores](https://python-zstandard.readthedocs.io/en/latest/multithreaded.html), in C threads that run outside the GIL. + +brotli is the Python module of the Brotli project, whose format is [similar in speed to deflate but denser](https://github.com/google/brotli#introduction). Call `brotli.compress()` and `brotli.decompress()` on bytes, or [feed a `brotli.Compressor` piece by piece](https://github.com/google/brotli/blob/master/python/brotli.py) with `process()`, then call `finish()`. + +lz4 binds the LZ4 library, built for [extremely fast compression and an extremely fast decoder](https://github.com/lz4/lz4). Its docs recommend [the `lz4.frame` bindings](https://python-lz4.readthedocs.io/en/stable/quickstart.html#simple-usage), since the frame format works with other LZ4 implementations and language bindings. `lz4.frame.open()` is [the most convenient way to work with compressed files](https://python-lz4.readthedocs.io/en/stable/quickstart.html#working-with-compressed-files), and it works like `gzip.open()`.