diff --git a/README.md b/README.md index ed9177a3..ebfaf9cc 100644 --- a/README.md +++ b/README.md @@ -999,6 +999,12 @@ _Libraries for file manipulation._ - Remote Files - [fsspec](https://github.com/fsspec/filesystem_spec) - A unified filesystem interface for local, cloud, and remote storage. - [smart-open](https://github.com/piskvorky/smart_open) - Stream files from S3, GCS, Azure, HDFS, and HTTP as a drop-in replacement for open(). +- Compression + - [compression.zstd](https://docs.python.org/3/library/compression.zstd.html) - (Python standard library) Compress and decompress data in the Zstandard format. + - [gzip](https://docs.python.org/3/library/gzip.html) - (Python standard library) Read and write gzip-compressed files and data. + - [zstandard](https://github.com/indygreg/python-zstandard) - Python bindings to the Zstandard (zstd) compression library. + - [brotli](https://github.com/google/brotli) - Python bindings for Google's Brotli compression library. + - [lz4](https://github.com/python-lz4/python-lz4) - Python bindings for the LZ4 fast compression library. **Media** diff --git a/docs/audit-logs.md b/docs/audit-logs.md index 17b71e6e..df29e68e 100644 --- a/docs/audit-logs.md +++ b/docs/audit-logs.md @@ -104,6 +104,15 @@ Projects the maintainer decided not to list, with the evidence at decision time, - m-bain/whisperX -- 2026-11-08, 342,919 downloads/month (pypistats, package whisperx): A wrapper over faster-whisper for word timestamps and diarization, not a successor to any Speech entry. Its monthly downloads dipped instead of rising, and its release line pins torch~=2.8.0 with requires_python <3.14. - openai/codex -- 2026-11-08, no PyPI download signal: A language-agnostic terminal coding agent written in Rust. Not Python work. - openai/openai-python -- 2026-11-08, 295,856,631 downloads/month (pypistats, package openai): A vendor API client, declined with Third-party APIs (2d98426); OpenAI is already listed through openai-agents. Much of its count is pulled in by other libraries. +- milesgranger/cramjam -- 2026-11-15, 31,936,738 downloads/month (pypistats): fastparquet and python-snappy require it, and few people pick it directly. Compression is at cap. +- intake/python-snappy -- 2026-11-15, 11,089,388 downloads/month (pypistats): Now a thin wrapper over cramjam. Snappy is mostly used inside Parquet and Kafka libraries, not called directly. +- Rogdham/backports.zstd -- 2026-11-15, 57,800,006 downloads/month (pypistats, package backports-zstd): A backport of the listed compression.zstd for Python 3.10–3.13 that won't install on 3.14. The stdlib entry covers it. +- Rogdham/pyzstd -- 2026-11-15, 6,838,415 downloads/month (pypistats): compression.zstd was built from it, and its own docs guide users to the stdlib. Its only remaining edge is the seekable format. +- pycompression/python-zlib-ng -- 2026-11-15, 3,387,853 downloads/month (pypistats, package zlib-ng): A faster drop-in for zlib and gzip with a niche audience (16 stars). +- pycompression/python-isal -- 2026-11-15, 2,960,582 downloads/month (pypistats, package isal): A faster drop-in for zlib and gzip using Intel ISA-L, with a niche audience (61 stars). +- python-hyper/brotlicffi -- 2026-11-15, 1,946,691 downloads/month (pypistats): The PyPy version of the listed brotli. HTTP clients pick it automatically on non-CPython. +- uktrade/stream-zip -- 2026-11-15, 980,636 downloads/month (pypistats): Builds ZIP files as a stream. Niche (21 stars), and no archives use case exists. +- Changaco/python-libarchive-c -- 2026-11-15, 824,581 downloads/month (pypistats, package libarchive-c): A libarchive binding with low demand (85 stars) that needs the system library. No archives use case exists. ## Last Audited diff --git a/website/data/category_intros/file-manipulation.md b/website/data/category_intros/file-manipulation.md index 616ea657..510264cd 100644 --- a/website/data/category_intros/file-manipulation.md +++ b/website/data/category_intros/file-manipulation.md @@ -10,6 +10,10 @@ How to choose: - Telling text file types apart, like source code from config files: Magika - Listing, globbing, and copying files on cloud or remote storage: fsspec - Streaming one large remote file through a drop-in `open()`: smart_open +- Zstandard files and data: compression.zstd, or zstandard for multi-threaded compression +- Reading and writing .gz files: gzip +- Denser compression than gzip at a similar speed: brotli +- Compressing and decompressing as fast as possible: lz4 If you aren't sure which pathlib class you need, the docs say [`Path` is most likely it](https://docs.python.org/3/library/pathlib.html): it makes a concrete path for the platform your code runs on. [Join paths with the `/` operator](https://docs.python.org/3/library/pathlib.html#basic-use), and list a directory with `iterdir()` or `glob()`. @@ -26,3 +30,13 @@ Magika runs [a compact deep-learning model](https://securityresearch.google/magi fsspec gives you [one API for local, remote, and embedded file systems](https://filesystem-spec.readthedocs.io/en/latest/), and it works the same whatever the storage backend. Get a file system by protocol name, like `fsspec.filesystem("s3")`, then call methods that [follow posix naming and the `os` module](https://filesystem-spec.readthedocs.io/en/latest/usage.html#use-a-file-system): `ls()`, `glob()`, `cp()`, and `rm()`. For one file, [`fsspec.open()` picks the backend from the URL](https://filesystem-spec.readthedocs.io/en/latest/usage.html#higher-level), and opens the file only when you enter its `with` block. smart_open is [a drop-in replacement for Python's built-in `open()`](https://github.com/piskvorky/smart_open#what) that streams very large files from and to S3, GCS, Azure Blob Storage, HDFS, HTTP, and SFTP. Import its `open` in place of the built-in, and pass it a URL like `s3://bucket/key`. A file name ending in `.gz`, `.bz2`, or `.zst` gets [compressed or decompressed on the fly](https://github.com/piskvorky/smart_open#compression-handling). To use your own credentials or settings, [pass the storage service's client object in `transport_params`](https://github.com/piskvorky/smart_open#transport-specific-options). + +compression.zstd covers files, bytes, and streams. [`compression.zstd.open()`](https://docs.python.org/3/library/compression.zstd.html#compression.zstd.open) reads and writes .zst files as file objects, `compress()` and `decompress()` handle bytes in one call, and `ZstdCompressor` and `ZstdDecompressor` work through data a chunk at a time. + +gzip [opens .gz files like regular files](https://docs.python.org/3/library/gzip.html#gzip.open): `gzip.open()` takes `'rb'` or `'wb'` for bytes, and `'rt'` or `'wt'` for text. For data already in memory, call `gzip.compress()` and `gzip.decompress()`. + +zstandard is [the most feature-rich zstd binding](https://python-zstandard.readthedocs.io/en/latest/projectinfo.html#comparison-to-other-python-bindings). Still, its own docs say the standard library is probably the easiest way to use zstd, if it works for you. Use its one-shot APIs for data that fits in memory, and [its streaming APIs for large inputs](https://python-zstandard.readthedocs.io/en/latest/api_usage.html#choosing-an-api). Pass `threads` to `ZstdCompressor` to [compress on several cores](https://python-zstandard.readthedocs.io/en/latest/multithreaded.html), in C threads that run outside the GIL. + +brotli is the Python module of the Brotli project, whose format is [similar in speed to deflate but denser](https://github.com/google/brotli#introduction). Call `brotli.compress()` and `brotli.decompress()` on bytes, or [feed a `brotli.Compressor` piece by piece](https://github.com/google/brotli/blob/master/python/brotli.py) with `process()`, then call `finish()`. + +lz4 binds the LZ4 library, built for [extremely fast compression and an extremely fast decoder](https://github.com/lz4/lz4). Its docs recommend [the `lz4.frame` bindings](https://python-lz4.readthedocs.io/en/stable/quickstart.html#simple-usage), since the frame format works with other LZ4 implementations and language bindings. `lz4.frame.open()` is [the most convenient way to work with compressed files](https://python-lz4.readthedocs.io/en/stable/quickstart.html#working-with-compressed-files), and it works like `gzip.open()`. diff --git a/website/data/pypi_name_overrides.json b/website/data/pypi_name_overrides.json index ee534c13..4366e5ec 100644 --- a/website/data/pypi_name_overrides.json +++ b/website/data/pypi_name_overrides.json @@ -3,6 +3,7 @@ "azure-sdk-for-python": { "package": null, "reason": "monorepo umbrella; ships as many azure-* packages, no single package represents it", "badge": "Multiple on PyPI" }, "betterleaks": { "package": null, "reason": "not published on PyPI" }, "cloud-init": { "package": null, "reason": "installed via distro images, not pip" }, + "compression-zstd": { "package": null, "reason": "stdlib module" }, "concurrent-futures": { "package": null, "reason": "stdlib module" }, "conda": { "package": null, "reason": "installed via Miniforge or Anaconda installers; PyPI conda is a stale 2017 upload" }, "cpython": { "package": null, "reason": "not pip-installable; the cpython PyPI name was squatted" }, @@ -17,6 +18,7 @@ "geodjango": { "package": null, "reason": "part of Django (django.contrib.gis), not a package" }, "google-cloud-python": { "package": null, "reason": "monorepo umbrella; ships as many google-cloud-* packages", "badge": "Multiple on PyPI" }, "graphify": { "package": "graphifyy", "reason": null }, + "gzip": { "package": null, "reason": "stdlib module" }, "httpx-url": { "package": null, "reason": "class in the httpx package, not a package" }, "jinja": { "package": "jinja2", "reason": "PyPI jinja is Jinja1; current releases publish as jinja2" }, "mem0": { "package": "mem0ai", "reason": null },