[GH-ISSUE #18] [Feature]: Xet protocol support (chunk-level dedup) #18

New Issue

GiteaMirror · 2026-05-22T03:10:11-05:00

GiteaMirror commented

2026-05-22 03:10:11 -05:00

Originally created by @Dynoooooo on GitHub (May 21, 2026).
Original GitHub issue: https://github.com/KohakuBlueleaf/KohakuHub/issues/18

What feature do you want?

Add support for HF's Xet content-addressable storage protocol alongside (or eventually replacing) Git LFS for large file handling. That means content-defined chunking, xorbs as upload containers, and shards for file reconstruction — see the upload protocol spec.

Reference implementation is open source: xet-core (Rust client + CAS logic). The huggingface_hub Python client already speaks it via hf_xet, so a Xet-compatible KohakuHub endpoint would Just Work with existing tooling.

Why do you need it?

Two reasons:

Compatibility drift. HF made Xet the default for all new accounts/orgs as of May 23, 2025 (Hub docs). Any KohakuHub instance mirroring or falling back to huggingface.co will increasingly hit Xet-backed repos. LFS-only means relying on HF's legacy LFS bridge indefinitely.
Chunk-level vs file-level dedup. LFS dedupes whole files; Xet dedupes bytes. When a checkpoint or dataset changes incrementally, only modified chunks move. HF reports 2-3x faster transfers in production (From Chunks to Blocks). Big deal for anyone iterating on models, and for storage costs on the host side.

Additional context

The protocol is being standardized at the IETF via draft-denis-xet-03, so this isn't an HF-exclusive thing going forward.

Good news on the KohakuHub side: a lot of the pieces are already in place — S3 backend, LakeFS versioning, SHA256 content-addressing. Adding Xet would mostly mean layering CDC chunking + xorb assembly on top of the existing storage layer rather than a rewrite.

Totally get it if this is out of scope for alpha. Just wanted to log it. Happy to help test once anything lands. Thanks for building this!

Checklist

I searched existing issues
I checked the roadmap in CONTRIBUTING.md

Originally created by @Dynoooooo on GitHub (May 21, 2026). Original GitHub issue: https://github.com/KohakuBlueleaf/KohakuHub/issues/18 ### What feature do you want? Add support for HF's Xet content-addressable storage protocol alongside (or eventually replacing) Git LFS for large file handling. That means content-defined chunking, xorbs as upload containers, and shards for file reconstruction — see the [upload protocol spec](https://huggingface.co/docs/xet/upload-protocol). Reference implementation is open source: [xet-core](https://github.com/huggingface/xet-core) (Rust client + CAS logic). The huggingface_hub Python client already speaks it via hf_xet, so a Xet-compatible KohakuHub endpoint would Just Work with existing tooling. ### Why do you need it? Two reasons: 1. **Compatibility drift.** HF made Xet the default for all new accounts/orgs as of May 23, 2025 ([Hub docs](https://huggingface.co/docs/hub/xet)). Any KohakuHub instance mirroring or falling back to huggingface.co will increasingly hit Xet-backed repos. LFS-only means relying on HF's legacy LFS bridge indefinitely. 2. **Chunk-level vs file-level dedup.** LFS dedupes whole files; Xet dedupes bytes. When a checkpoint or dataset changes incrementally, only modified chunks move. HF reports 2-3x faster transfers in production ([From Chunks to Blocks](https://huggingface.co/blog/from-chunks-to-blocks)). Big deal for anyone iterating on models, and for storage costs on the host side. ### Additional context The protocol is being standardized at the IETF via [draft-denis-xet-03](https://datatracker.ietf.org/doc/draft-denis-xet/03/), so this isn't an HF-exclusive thing going forward. Good news on the KohakuHub side: a lot of the pieces are already in place — S3 backend, LakeFS versioning, SHA256 content-addressing. Adding Xet would mostly mean layering CDC chunking + xorb assembly on top of the existing storage layer rather than a rewrite. Totally get it if this is out of scope for alpha. Just wanted to log it. Happy to help test once anything lands. Thanks for building this! ### Checklist - [x] I searched existing issues - [x] I checked the roadmap in CONTRIBUTING.md

GiteaMirror added the enhancement label 2026-05-22 03:10:11 -05:00

GiteaMirror commented

2026-07-11 11:04:23 -05:00

@normanheckscher commented on GitHub (Jul 11, 2026):

Can confirm this gap from the client side, for what it's worth.

I've been running a mirror pipeline that pulls from huggingface.co and pushes into a self-hosted KohakuHub instance using huggingface_hub's standard upload_folder()/snapshot_download(). A few things I hit that might be useful data points:

The xet-write-token endpoint isn't implemented. Uploads to KohakuHub 404 on /api/{repo_type}s/{repo_id}/xet-write-token/{revision} — confirms there's currently no server-side path for Xet-aware clients to negotiate an upload at all, not just a slower/fallback one.
HF_HUB_DISABLE_XET doesn't reliably re-decide mid-process. Once huggingface_hub is imported, toggling that env var in the same process and retrying doesn't change xet-on-vs-off for subsequent calls — I had to shell out to a fresh subprocess with the var set before huggingface_hub gets imported to reliably force LFS. Worth knowing if anyone else is debugging "I set the env var and it's still trying Xet" — it's not a bug in your app code, it's an import-time decision on the client side.
Download-from-HF side works fine with hf_xet installed (as expected, since huggingface.co speaks Xet) — the gap is specifically upload-to-KohakuHub.

Not offering a fix, just a workaround (force-disable Xet before the upload leg, always fall back to LFS) plus these repro details in case they save someone time. Repo's here if useful: https://github.com/normanheckscher/hf2kh. Thanks for building this either way!

@normanheckscher commented on GitHub (Jul 11, 2026): Can confirm this gap from the client side, for what it's worth. I've been running a mirror pipeline that pulls from huggingface.co and pushes into a self-hosted KohakuHub instance using `huggingface_hub`'s standard `upload_folder()`/`snapshot_download()`. A few things I hit that might be useful data points: - **The xet-write-token endpoint isn't implemented.** Uploads to KohakuHub 404 on `/api/{repo_type}s/{repo_id}/xet-write-token/{revision}` — confirms there's currently no server-side path for Xet-aware clients to negotiate an upload at all, not just a slower/fallback one. - **`HF_HUB_DISABLE_XET` doesn't reliably re-decide mid-process.** Once `huggingface_hub` is imported, toggling that env var in the same process and retrying doesn't change xet-on-vs-off for subsequent calls — I had to shell out to a fresh subprocess with the var set *before* `huggingface_hub` gets imported to reliably force LFS. Worth knowing if anyone else is debugging "I set the env var and it's still trying Xet" — it's not a bug in your app code, it's an import-time decision on the client side. - Download-from-HF side works fine with `hf_xet` installed (as expected, since huggingface.co speaks Xet) — the gap is specifically upload-to-KohakuHub. Not offering a fix, just a workaround (force-disable Xet before the upload leg, always fall back to LFS) plus these repro details in case they save someone time. Repo's here if useful: https://github.com/normanheckscher/hf2kh. Thanks for building this either way!

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/KohakuHub#18