Merge branch 'audit/file-manipulation'

README.md

@@ -999,6 +999,12 @@ _Libraries for file manipulation._
 - Remote Files
   - [fsspec](https://github.com/fsspec/filesystem_spec) - A unified filesystem interface for local, cloud, and remote storage.
   - [smart-open](https://github.com/piskvorky/smart_open) - Stream files from S3, GCS, Azure, HDFS, and HTTP as a drop-in replacement for open().
+- Compression
+  - [compression.zstd](https://docs.python.org/3/library/compression.zstd.html) - (Python standard library) Compress and decompress data in the Zstandard format.
+  - [gzip](https://docs.python.org/3/library/gzip.html) - (Python standard library) Read and write gzip-compressed files and data.
+  - [zstandard](https://github.com/indygreg/python-zstandard) - Python bindings to the Zstandard (zstd) compression library.
+  - [brotli](https://github.com/google/brotli) - Python bindings for Google's Brotli compression library.
+  - [lz4](https://github.com/python-lz4/python-lz4) - Python bindings for the LZ4 fast compression library.
 
 **Media**
 

docs/audit-logs.md

@@ -104,6 +104,15 @@ Projects the maintainer decided not to list, with the evidence at decision time,
 - m-bain/whisperX -- 2026-11-08, 342,919 downloads/month (pypistats, package whisperx): A wrapper over faster-whisper for word timestamps and diarization, not a successor to any Speech entry. Its monthly downloads dipped instead of rising, and its release line pins torch~=2.8.0 with requires_python <3.14.
 - openai/codex -- 2026-11-08, no PyPI download signal: A language-agnostic terminal coding agent written in Rust. Not Python work.
 - openai/openai-python -- 2026-11-08, 295,856,631 downloads/month (pypistats, package openai): A vendor API client, declined with Third-party APIs (2d98426); OpenAI is already listed through openai-agents. Much of its count is pulled in by other libraries.
+- milesgranger/cramjam -- 2026-11-15, 31,936,738 downloads/month (pypistats): fastparquet and python-snappy require it, and few people pick it directly. Compression is at cap.
+- intake/python-snappy -- 2026-11-15, 11,089,388 downloads/month (pypistats): Now a thin wrapper over cramjam. Snappy is mostly used inside Parquet and Kafka libraries, not called directly.
+- Rogdham/backports.zstd -- 2026-11-15, 57,800,006 downloads/month (pypistats, package backports-zstd): A backport of the listed compression.zstd for Python 3.10–3.13 that won't install on 3.14. The stdlib entry covers it.
+- Rogdham/pyzstd -- 2026-11-15, 6,838,415 downloads/month (pypistats): compression.zstd was built from it, and its own docs guide users to the stdlib. Its only remaining edge is the seekable format.
+- pycompression/python-zlib-ng -- 2026-11-15, 3,387,853 downloads/month (pypistats, package zlib-ng): A faster drop-in for zlib and gzip with a niche audience (16 stars).
+- pycompression/python-isal -- 2026-11-15, 2,960,582 downloads/month (pypistats, package isal): A faster drop-in for zlib and gzip using Intel ISA-L, with a niche audience (61 stars).
+- python-hyper/brotlicffi -- 2026-11-15, 1,946,691 downloads/month (pypistats): The PyPy version of the listed brotli. HTTP clients pick it automatically on non-CPython.
+- uktrade/stream-zip -- 2026-11-15, 980,636 downloads/month (pypistats): Builds ZIP files as a stream. Niche (21 stars), and no archives use case exists.
+- Changaco/python-libarchive-c -- 2026-11-15, 824,581 downloads/month (pypistats, package libarchive-c): A libarchive binding with low demand (85 stars) that needs the system library. No archives use case exists.
 
 ## Last Audited
 

website/data/category_intros/file-manipulation.md

@@ -10,6 +10,10 @@ How to choose:
 - Telling text file types apart, like source code from config files: Magika
 - Listing, globbing, and copying files on cloud or remote storage: fsspec
 - Streaming one large remote file through a drop-in `open()`: smart_open
+- Zstandard files and data: compression.zstd, or zstandard for multi-threaded compression
+- Reading and writing .gz files: gzip
+- Denser compression than gzip at a similar speed: brotli
+- Compressing and decompressing as fast as possible: lz4
 
 If you aren't sure which pathlib class you need, the docs say [`Path` is most likely it](https://docs.python.org/3/library/pathlib.html): it makes a concrete path for the platform your code runs on. [Join paths with the `/` operator](https://docs.python.org/3/library/pathlib.html#basic-use), and list a directory with `iterdir()` or `glob()`.
 
@@ -26,3 +30,13 @@ Magika runs [a compact deep-learning model](https://securityresearch.google/magi
 fsspec gives you [one API for local, remote, and embedded file systems](https://filesystem-spec.readthedocs.io/en/latest/), and it works the same whatever the storage backend. Get a file system by protocol name, like `fsspec.filesystem("s3")`, then call methods that [follow posix naming and the `os` module](https://filesystem-spec.readthedocs.io/en/latest/usage.html#use-a-file-system): `ls()`, `glob()`, `cp()`, and `rm()`. For one file, [`fsspec.open()` picks the backend from the URL](https://filesystem-spec.readthedocs.io/en/latest/usage.html#higher-level), and opens the file only when you enter its `with` block.
 
 smart_open is [a drop-in replacement for Python's built-in `open()`](https://github.com/piskvorky/smart_open#what) that streams very large files from and to S3, GCS, Azure Blob Storage, HDFS, HTTP, and SFTP. Import its `open` in place of the built-in, and pass it a URL like `s3://bucket/key`. A file name ending in `.gz`, `.bz2`, or `.zst` gets [compressed or decompressed on the fly](https://github.com/piskvorky/smart_open#compression-handling). To use your own credentials or settings, [pass the storage service's client object in `transport_params`](https://github.com/piskvorky/smart_open#transport-specific-options).
+
+compression.zstd covers files, bytes, and streams. [`compression.zstd.open()`](https://docs.python.org/3/library/compression.zstd.html#compression.zstd.open) reads and writes .zst files as file objects, `compress()` and `decompress()` handle bytes in one call, and `ZstdCompressor` and `ZstdDecompressor` work through data a chunk at a time.
+
+gzip [opens .gz files like regular files](https://docs.python.org/3/library/gzip.html#gzip.open): `gzip.open()` takes `'rb'` or `'wb'` for bytes, and `'rt'` or `'wt'` for text. For data already in memory, call `gzip.compress()` and `gzip.decompress()`.
+
+zstandard is [the most feature-rich zstd binding](https://python-zstandard.readthedocs.io/en/latest/projectinfo.html#comparison-to-other-python-bindings). Still, its own docs say the standard library is probably the easiest way to use zstd, if it works for you. Use its one-shot APIs for data that fits in memory, and [its streaming APIs for large inputs](https://python-zstandard.readthedocs.io/en/latest/api_usage.html#choosing-an-api). Pass `threads` to `ZstdCompressor` to [compress on several cores](https://python-zstandard.readthedocs.io/en/latest/multithreaded.html), in C threads that run outside the GIL.
+
+brotli is the Python module of the Brotli project, whose format is [similar in speed to deflate but denser](https://github.com/google/brotli#introduction). Call `brotli.compress()` and `brotli.decompress()` on bytes, or [feed a `brotli.Compressor` piece by piece](https://github.com/google/brotli/blob/master/python/brotli.py) with `process()`, then call `finish()`.
+
+lz4 binds the LZ4 library, built for [extremely fast compression and an extremely fast decoder](https://github.com/lz4/lz4). Its docs recommend [the `lz4.frame` bindings](https://python-lz4.readthedocs.io/en/stable/quickstart.html#simple-usage), since the frame format works with other LZ4 implementations and language bindings. `lz4.frame.open()` is [the most convenient way to work with compressed files](https://python-lz4.readthedocs.io/en/stable/quickstart.html#working-with-compressed-files), and it works like `gzip.open()`.

website/data/pypi_name_overrides.json

@@ -3,6 +3,7 @@
   "azure-sdk-for-python": { "package": null, "reason": "monorepo umbrella; ships as many azure-* packages, no single package represents it", "badge": "Multiple on PyPI" },
   "betterleaks": { "package": null, "reason": "not published on PyPI" },
   "cloud-init": { "package": null, "reason": "installed via distro images, not pip" },
+  "compression-zstd": { "package": null, "reason": "stdlib module" },
   "concurrent-futures": { "package": null, "reason": "stdlib module" },
   "conda": { "package": null, "reason": "installed via Miniforge or Anaconda installers; PyPI conda is a stale 2017 upload" },
   "cpython": { "package": null, "reason": "not pip-installable; the cpython PyPI name was squatted" },
@@ -17,6 +18,7 @@
   "geodjango": { "package": null, "reason": "part of Django (django.contrib.gis), not a package" },
   "google-cloud-python": { "package": null, "reason": "monorepo umbrella; ships as many google-cloud-* packages", "badge": "Multiple on PyPI" },
   "graphify": { "package": "graphifyy", "reason": null },
+  "gzip": { "package": null, "reason": "stdlib module" },
   "httpx-url": { "package": null, "reason": "class in the httpx package, not a package" },
   "jinja": { "package": "jinja2", "reason": "PyPI jinja is Jinja1; current releases publish as jinja2" },
   "mem0": { "package": "mem0ai", "reason": null },