github-starred/KohakuHub
2
0
Fork 0
mirror of https://github.com/KohakuBlueleaf/KohakuHub.git synced 2026-07-10 19:57:19 -05:00
Code Issues 6 Packages Projects Releases Wiki Activity

[PR #17] [MERGED] sync deepghs:dev/narugo1992 → main: 22 PRs across fallback / errors / cache / perf / UI (Apr–May 2026) #17

New Issue
Closed
opened 2026-05-06 19:48:03 -05:00 by GiteaMirror · 0 comments
GiteaMirror commented 2026-05-06 19:48:03 -05:00
Owner
Copy Link

📋 Pull Request Information

Original PR: https://github.com/KohakuBlueleaf/KohakuHub/pull/17
Author: @narugo1992
Created: 5/5/2026
Status: ✅ Merged
Merged: 5/8/2026
Merged by: @KohakuBlueleaf

Base: main ← Head: dev/narugo1992

📝 Commits (10+)

  • 1397332 feat(dev): unify make ui (admin under /admin) and stop spurious vite reloads
  • d039f12 Merge pull request #34 from deepghs/feat/dev-server-quality-of-life
  • b482171 feat(admin): dependency health dashboard for Postgres / MinIO / LakeFS / SMTP
  • 9eeb5e7 test(admin/health): cover failure paths so the patch matches project coverage
  • b9a286f fix(admin/health): UI parity with sibling pages + real MinIO server version
  • ad0f5bf Merge pull request #48 from deepghs/feat/admin-health-dashboard
  • 275266f feat(admin): global credentials view (sessions / API tokens / SSH keys) with revoke
  • c53c080 feat(seed): plant deterministic API tokens and SSH keys with usable keypairs
  • 2a4246b Merge pull request #49 from deepghs/feat/admin-credentials-view
  • f9bf4b0 feat(ui): read-only browser for hfutils.index TAR + sidecar pairs

📊 Changes

164 files changed (+42341 additions, -1453 deletions)

View changed files

📝 .env.dev.example (+6 -0)
📝 .github/workflows/fullstack-tests.yml (+151 -6)
➕ CHANGELOG.md (+74 -0)
📝 CONTRIBUTING.md (+3 -5)
📝 Makefile (+21 -9)
📝 README.md (+24 -8)
📝 codecov.yml (+27 -0)
📝 config-example.toml (+1 -1)
📝 docker-compose.example.yml (+40 -0)
📝 docs/deployment.md (+15 -5)
📝 docs/deployment/docker.md (+7 -5)
📝 docs/deployment/production.md (+34 -0)
➕ docs/development/cache.md (+174 -0)
📝 docs/development/local-dev.md (+3 -4)
📝 docs/setup.md (+13 -4)
➕ package.json (+18 -0)
➕ pnpm-workspace.yaml (+3 -0)
📝 pyproject.toml (+1 -0)
📝 scripts/README.md (+2 -1)
📝 scripts/deploy.py (+10 -8)

...and 80 more files

📄 Description

Summary

Up-streaming 94 commits / 22 PRs / 162 files (+41,351 / −1,450 lines) that landed on deepghs/KohakuHub:dev/narugo1992 between 2026-04-27 and 2026-05-05. The work clusters into four large feature threads (fallback chain repo-binding, HF error-contract alignment, an L2 cache infrastructure, and a list/info perf pass), plus a set of UI / admin / indexed-tar additions, a dev-server quality-of-life rework, and small bug fixes.

This PR is the scheduled dev/narugo1992 → upstream-main ship. Every URL below points at the corresponding deepghs PR / issue so cross-repo references survive verbatim.

Source: deepghs/KohakuHub:dev/narugo1992 @ f8773c4.
Target: KohakuBlueleaf/KohakuHub:main (parent fork).

1 — Fallback chain: repo-grain binding (largest single thread)

deepghs/KohakuHub#77 — closes deepghs/KohakuHub#75

The fallback chain previously made per-operation, per-file decisions when picking a source. For one repo_id, different reads in the same SPA session could land on different upstream sources — info from source A, tree from source B, resolve from source C, all naming a "bert-base-uncased" but three different repos. This PR enforces a repo_id-scoped binding contract: once any operation confirms a repo lives on a given source, all subsequent reads against that repo_id go to that source for the cache TTL window.

The four try_fallback_* loops now share a 3-state classifier (BIND_AND_RESPOND / BIND_AND_PROPAGATE / TRY_NEXT_SOURCE) whose priority mirrors huggingface_hub.utils._http.hf_raise_for_status. The matrix is empirically anchored to live HF responses (re-probed multiple times during the branch's lifetime, most recently 2026-05-05).

Several follow-up fixes/features landed on the same feat/fallback-repo-binding branch before merge:

  • deepghs/KohakuHub#84 — closes deepghs/KohakuHub#78. TTL default lowered from 300s to 30s; chain-probe decoupling so the simulate path doesn't mutate live cache; new admin chain-tester UI for operator debugging.
  • deepghs/KohakuHub#81 — closes deepghs/KohakuHub#79. Strict-freshness cache contract: per-(user, tokens_hash, repo_id) keying + three-dimensional generation counters (global / per-user / per-repo) + invalidation hooks on every mutation event (admin source mutations, user external-token rotations, local repo CRUD).
  • deepghs/KohakuHub#86 — closes deepghs/KohakuHub#85. Lock liveness regression — relax the binding lock (release before cache-hit I/O) + asyncio.wait_for supervisor so a stuck attempt_fn cannot freeze same-repo callers indefinitely. Bundled with a Plan A redirect-passthrough on resolve GET so multi-GB blobs never traverse the backend.
  • deepghs/KohakuHub#88 — TypeError: 'NoneType' object does not support item assignment regression caught in pre-merge testing. dict.pop(key, default) returns None when the key is present with a None value (default doesn't apply); combined with client_headers or None collapsing empty dicts to None, every token-bearing source 502'd on small-text-file resolves with "category":"network". Fixed at the FallbackClient contract boundary.

The strict-consistency contract is documented in src/kohakuhub/api/fallback/README.md (rewritten as part of this thread) and deliberately narrowed in deepghs/KohakuHub#85 to "safety under stable config, scoped per-process, not a liveness gate."

2 — HuggingFace error-contract alignment for permission denials

deepghs/KohakuHub#89 — closes deepghs/KohakuHub#76

check_repo_read_permission previously raised HTTPException(401) for anonymous-on-private and HTTPException(403) for authed-no-access — both produced FastAPI's default {"detail": "..."} envelope with no X-Error-Code. A huggingface_hub client hitting a private hkub repo got a generic HfHubHTTPError instead of the named RepositoryNotFoundError, so transformers / datasets / etc. lost actionable error semantics.

This PR introduces RepoReadDeniedError (a non-HTTPException exception class) raised by check_repo_read_permission for both anon-on-private and authed-no-access. A global FastAPI exception handler in main.py converts it into 404 + X-Error-Code: RepoNotFound with an empty body — matching what HF returns to authenticated callers. hf_raise_for_status then dispatches RepositoryNotFoundError end-to-end (verified via live round-trip tests).

Privacy-preserving Option A from the issue: anon and authed-no-access produce the same wire shape (no enumeration leak between "missing" and "private"); same client exception class; simpler than mirroring HF's anon-vs-authed split.

A hf_disabled_repo() helper was also added — emits HF's exact X-Error-Message: "Access to this resource is disabled." string with no X-Error-Code (DisabledRepoError dispatch is keyed off the message verbatim). No call site yet; reserved for the future moderation feature. The HFErrorCode docstring labels its KohakuHub-only extensions distinctly from the four HF-recognized codes.

Self-review during this PR caught one critical wrapper-interaction regression: with_repo_fallback has a generic except Exception catch (api/fallback/decorators.py:363) that absorbed the new exception and re-raised it as a 500. The original test suite missed it because the harness sets KOHAKU_HUB_FALLBACK_ENABLED=false and every wire-shape test pre-emptively passes ?fallback=false. Fixed by inserting an explicit except RepoReadDeniedError: raise between the cancellation handler and the generic catch, plus a dedicated regression test that forces cfg.fallback.enabled=True.

3 — Cache & connection-pool infrastructure

  • deepghs/KohakuHub#74 — Valkey-based shared L2 cache backend, opt-in via KOHAKU_HUB_CACHE_URL. Includes a new src/kohakuhub/cache.py module with the cache primitive + boot-time flush coordinator (Mode-B namespaces wiped whenever Valkey's run_id changes since the last seen value, i.e. Valkey was restarted even if the API wasn't). Admin cache monitoring page + Redis probe added to /admin/api/health. 100% test coverage on cache.py, the admin cache router, and config validation.
  • deepghs/KohakuHub#61 — pool the httpx.AsyncClient in LakeFSRestClient instead of constructing one per request. Substantial p99 reduction on list/info paths that previously paid TLS handshake per LakeFS round-trip.

4 — List / info / tree perf pass

  • deepghs/KohakuHub#70 + #72 — replace per-row LakeFS.get_branch + get_commit calls in list endpoints (/api/models, /api/datasets, /api/spaces) with a single SQL aggregate (commit table). #72 is the dev/narugo1992 sync of #70. Coverage includes new upstream-compat tests against huggingface_hub.HfApi.list_models / list_datasets / list_spaces that pin response shape across the matrix (0.20.x → latest).
  • deepghs/KohakuHub#71 — /org/{name} inlines repo_count directly from the DB instead of dispatching three frontend listRepos fan-outs (which themselves N+1'd into LakeFS).
  • deepghs/KohakuHub#58 — drop the per-object DB query in calculate_repository_storage. Quota calculations on large repos no longer scale O(files).
  • deepghs/KohakuHub#60 — resolve lastCommit per page via LakeFS path-filtered logCommits instead of walking commits client-side. Closes deepghs/KohakuHub#59 (tree-expand 30s timeout on WAN-deployed instances).

5 — Indexed-tar feature suite

A coordinated three-PR thread for surfacing hfutils.index TAR-format datasets natively:

  • deepghs/KohakuHub#50 — read-only browser for hfutils.index TAR + sidecar pairs (.tar + .json with member offsets). Detects sibling-tar pairs server-side; backend exposes a TAR-member listing endpoint; SPA renders a tree-style member browser inside the regular file list.
  • deepghs/KohakuHub#51 — lazy in-listing thumbnails for image members of indexed tars. The SPA performs a Range-read request per visible image member; thumbnail extraction is fully client-side, no precomputation.
  • deepghs/KohakuHub#53 (also covers list pagination — see §6) — adds the HEAD-probe that detects whether a .tar has an indexed sibling, gating the indexed-tar UI.

6 — Repo-list / tree UI features (HF-compatible)

7 — Admin tooling

  • deepghs/KohakuHub#48 — dependency health dashboard for Postgres / MinIO / LakeFS / SMTP. Each has a probe with status + latency + last-error message in the admin SPA.
  • deepghs/KohakuHub#49 — global credentials view (active sessions, API tokens, SSH keys) with revoke. Closes deepghs/KohakuHub#36.

8 — Dev-server / monorepo

  • deepghs/KohakuHub#34 — unify make ui (admin SPA mounted under /admin); stop spurious Vite reloads triggered by Python file changes in shared dirs. Establishes the pnpm monorepo layout that subsequent PRs build on (src/kohaku-hub-ui + src/kohaku-hub-admin workspaces).

9 — Bug fixes

  • deepghs/KohakuHub#52 — forward same-origin session cookie on /resolve/ Range probes from the SPA. Without it, private-repo Range reads from the parquet/safetensors preview path silently anonymized themselves and 401'd.
  • deepghs/KohakuHub#88 — covered in §1; client-layer headers=None TypeError regression caught in pre-merge testing of #77.

Open follow-ups (intentionally not blocking this merge)

These were either explicitly scoped out of the PRs above, or surfaced post-merge during pre-dev → main review. All tracked in the deepghs issue tracker; nothing here is new debt introduced by this PR thread without a corresponding tracker entry:

  • deepghs/KohakuHub#82 — header-token cache pollution (availability / LRU eviction). Privacy threat closed by #79; residual availability vector remains.
  • deepghs/KohakuHub#83 — migrate the fallback cache to the shared kohakuhub.cache backend so multi-worker deployments honour the strict-freshness contract across processes (rather than the current per-process scope).
  • deepghs/KohakuHub#87 — inline-200 GET path still buffers the full upstream body for non-HF mirrors. The 30x branch is fully redirect-passthrough after #86; the unbounded-buffer edge applies only to misconfigured non-HF mirrors that 200 a multi-GB blob.
  • deepghs/KohakuHub#90 — git-over-HTTP regression for anonymous-on-private clones surfaced by #89. git/routers/http.py is not wrapped by with_repo_fallback, so the privacy translation hits git transport too — git clone of a private repo now returns 404 + empty body instead of 401 + WWW-Authenticate: Basic. Fix sketched in the issue.
  • deepghs/KohakuHub#66 — with_list_aggregation defaults fallback=true; every list call hits external sources. Pre-existing.
  • deepghs/KohakuHub#26 — broad except Exception antipatterns (168 sites). Pre-existing.

Test posture

CI matrix on each landed PR covers backend tests across Python 3.10 / 3.11 / 3.12 × huggingface_hub versions {0.20.3, 0.30.2, 0.36.2, 1.0.1, 1.6.0, latest} (18 lanes), plus a cache disabled lane, plus codecov + admin frontend + frontend unit tests. Every PR landed in this thread shipped CI-green. The most recent ones (#89, #88, #86) include hf_hub interop tests that drive a real client against a live test server.

Significant new test artefacts:

  • test/kohakuhub/api/fallback/test_strict_consistency.py — invariants pinning the post-#77 binding contract.
  • test/kohakuhub/api/fallback/test_strict_freshness.py — generation-counter race protection (#79).
  • test/kohakuhub/api/fallback/test_chain_enumeration.py + test_e2e_matrix.py — exhaustive matrix covering every wire shape × every operation type × hf_hub_download / model_info / list_repo_tree / get_paths_info dispatch behaviour.
  • test/kohakuhub/api/test_repo_read_denial_hf_alignment.py — wire-shape + interop tests for #76 (#89) including the production-shape regression guard with fallback enabled.
  • test/kohakuhub/test_cache.py — 100% coverage of the L2 cache primitive (#74).
  • test/kohakuhub/test_lakefs_rest_client.py + test_lakefs_rest_client_live.py — pooling / lifecycle coverage for #61.

Migration notes for operators

  • KOHAKU_HUB_FALLBACK_CACHE_TTL_SECONDS default lowered from 300 to 30 (#84). Existing deployments unaffected unless they explicitly relied on the longer window.
  • KOHAKU_HUB_CACHE_URL is the new opt-in for the L2 cache (#74). Defaults to disabled; auto-enabled when set. Single-worker deployments without Redis continue to work as before.
  • Permission-denial wire shape changes from 401 / 403 + JSON to 404 + X-Error-Code: RepoNotFound + empty body for the HF API surfaces (#89). The SPA already maps RepoNotFound and bare 401 to NOT_FOUND, so user-facing error UX shifts from "Forbidden" to "Not found" for outsiders on private repos — privacy-preserving by design.
  • Git-over-HTTP for anonymous-on-private has the same wire-shape change with a known UX regression for git clone (no WWW-Authenticate: Basic challenge); see #90 for the planned fix.
  • pnpm monorepo layout — src/kohaku-hub-ui/ + src/kohaku-hub-admin/ are workspaces; make ui runs both. Anyone vendoring this fork's frontend should update build scripts to point at pnpm workspaces (#34).

Per-PR commit anchor table (ordered by merge time)

For full traceability:

Merge SHA PR Title (one-liner)
d039f12 #34 Unify make ui (admin under /admin) and stop spurious Vite reloads
ad0f5bf #48 feat(admin): dependency health dashboard for Postgres / MinIO / LakeFS / SMTP
2a4246b #49 feat(admin): global credentials view (sessions / tokens / SSH keys) with revoke
b98b482 #50 Read-only browser for hfutils.index TAR + sidecar pairs
1e3d010 #51 Lazy in-listing thumbnails for indexed-tar image members
c064357 #52 fix(ui/range-reads): forward same-origin session on /resolve/ Range probes
174554a #53 feat(ui/repo-list): paginate file list (50/100/200) + HEAD-probe indexed-tar siblings
d68f2ba #55 feat(tree): same-level name_prefix filter on /tree
b0d1279 #57 feat(ui/repo-file-list): swap numbered pager for "Load More" cursor pagination
ad357ee #58 perf(quota): drop the per-object DB query in calculate_repository_storage
eb71342 #60 perf(tree): resolve lastCommit per page via LakeFS path-filtered logCommits
4947f93 #61 perf(lakefs-rest): pool the httpx.AsyncClient in LakeFSRestClient
ca573ea #70 perf(repo/info): replace per-row LakeFS get_branch+get_commit with SQL aggregate
a2ba6c3 #71 perf(org): inline repo_count, drop frontend listRepos fan-out
c8f3757 #72 perf(repo/info): list endpoints SQL aggregate fix (sync of #70 onto dev)
1993885 #74 perf(cache): Valkey-based L2 cache infrastructure
9e7254c #81 feat(fallback): strict-freshness cache contract (closes #79)
4467133 #84 feat(fallback): TTL default 30s + chain-probe decoupling + admin chain-tester UI (closes #78)
4af6419 #86 fix(fallback): relax binding lock + asyncio.wait_for supervisor (closes #85)
a53acdd #77 feat(fallback): enforce repo-grain binding (closes #75)
e3089e6 #88 fix(fallback/client): treat caller-supplied headers=None as empty
f8773c4 #89 feat(auth/permissions): privacy-preserving RepoNotFound for read denials (closes #76)

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/KohakuBlueleaf/KohakuHub/pull/17 **Author:** [@narugo1992](https://github.com/narugo1992) **Created:** 5/5/2026 **Status:** ✅ Merged **Merged:** 5/8/2026 **Merged by:** [@KohakuBlueleaf](https://github.com/KohakuBlueleaf) **Base:** `main` ← **Head:** `dev/narugo1992` --- ### 📝 Commits (10+) - [`1397332`](https://github.com/KohakuBlueleaf/KohakuHub/commit/13973327da11255d08950f35211c8907fdb3856f) feat(dev): unify make ui (admin under /admin) and stop spurious vite reloads - [`d039f12`](https://github.com/KohakuBlueleaf/KohakuHub/commit/d039f12621174a7138abe982eba51eb816be26c5) Merge pull request #34 from deepghs/feat/dev-server-quality-of-life - [`b482171`](https://github.com/KohakuBlueleaf/KohakuHub/commit/b482171e1b2d0412086ae74c2d331ea1650a9d1d) feat(admin): dependency health dashboard for Postgres / MinIO / LakeFS / SMTP - [`9eeb5e7`](https://github.com/KohakuBlueleaf/KohakuHub/commit/9eeb5e7ceb85fa67ddc356f0bb84c5ab15716632) test(admin/health): cover failure paths so the patch matches project coverage - [`b9a286f`](https://github.com/KohakuBlueleaf/KohakuHub/commit/b9a286f57c1438d4b74352a12e08c2644067ef20) fix(admin/health): UI parity with sibling pages + real MinIO server version - [`ad0f5bf`](https://github.com/KohakuBlueleaf/KohakuHub/commit/ad0f5bf5c98b0ad34c15aa8dcad19e5ee77afc37) Merge pull request #48 from deepghs/feat/admin-health-dashboard - [`275266f`](https://github.com/KohakuBlueleaf/KohakuHub/commit/275266f8efd175d0c8dee855bc1e547029028355) feat(admin): global credentials view (sessions / API tokens / SSH keys) with revoke - [`c53c080`](https://github.com/KohakuBlueleaf/KohakuHub/commit/c53c08058d0070c8e0429717f9f60ebc0a5e075d) feat(seed): plant deterministic API tokens and SSH keys with usable keypairs - [`2a4246b`](https://github.com/KohakuBlueleaf/KohakuHub/commit/2a4246bdaa3a48f0888e2c1fc6d68f1168d43d3b) Merge pull request #49 from deepghs/feat/admin-credentials-view - [`f9bf4b0`](https://github.com/KohakuBlueleaf/KohakuHub/commit/f9bf4b0c8fc7d8c355758184e189873fbd238b2e) feat(ui): read-only browser for hfutils.index TAR + sidecar pairs ### 📊 Changes **164 files changed** (+42341 additions, -1453 deletions) <details> <summary>View changed files</summary> 📝 `.env.dev.example` (+6 -0) 📝 `.github/workflows/fullstack-tests.yml` (+151 -6) ➕ `CHANGELOG.md` (+74 -0) 📝 `CONTRIBUTING.md` (+3 -5) 📝 `Makefile` (+21 -9) 📝 `README.md` (+24 -8) 📝 `codecov.yml` (+27 -0) 📝 `config-example.toml` (+1 -1) 📝 `docker-compose.example.yml` (+40 -0) 📝 `docs/deployment.md` (+15 -5) 📝 `docs/deployment/docker.md` (+7 -5) 📝 `docs/deployment/production.md` (+34 -0) ➕ `docs/development/cache.md` (+174 -0) 📝 `docs/development/local-dev.md` (+3 -4) 📝 `docs/setup.md` (+13 -4) ➕ `package.json` (+18 -0) ➕ `pnpm-workspace.yaml` (+3 -0) 📝 `pyproject.toml` (+1 -0) 📝 `scripts/README.md` (+2 -1) 📝 `scripts/deploy.py` (+10 -8) _...and 80 more files_ </details> ### 📄 Description ## Summary Up-streaming **94 commits / 22 PRs / 162 files (+41,351 / −1,450 lines)** that landed on `deepghs/KohakuHub:dev/narugo1992` between 2026-04-27 and 2026-05-05. The work clusters into four large feature threads (fallback chain repo-binding, HF error-contract alignment, an L2 cache infrastructure, and a list/info perf pass), plus a set of UI / admin / indexed-tar additions, a dev-server quality-of-life rework, and small bug fixes. This PR is the scheduled `dev/narugo1992` → upstream-`main` ship. Every URL below points at the corresponding deepghs PR / issue so cross-repo references survive verbatim. Source: `deepghs/KohakuHub:dev/narugo1992` @ `f8773c4`. Target: `KohakuBlueleaf/KohakuHub:main` (parent fork). --- ## 1 — Fallback chain: repo-grain binding (largest single thread) **[deepghs/KohakuHub#77](https://github.com/deepghs/KohakuHub/pull/77)** — closes [deepghs/KohakuHub#75](https://github.com/deepghs/KohakuHub/issues/75) The fallback chain previously made per-operation, per-file decisions when picking a source. For one `repo_id`, different reads in the same SPA session could land on different upstream sources — `info` from source A, `tree` from source B, `resolve` from source C, all naming a "bert-base-uncased" but three different repos. This PR enforces a `repo_id`-scoped binding contract: once any operation confirms a repo lives on a given source, all subsequent reads against that `repo_id` go to that source for the cache TTL window. The four `try_fallback_*` loops now share a 3-state classifier (`BIND_AND_RESPOND` / `BIND_AND_PROPAGATE` / `TRY_NEXT_SOURCE`) whose priority mirrors `huggingface_hub.utils._http.hf_raise_for_status`. The matrix is empirically anchored to live HF responses (re-probed multiple times during the branch's lifetime, most recently 2026-05-05). Several follow-up fixes/features landed on the same `feat/fallback-repo-binding` branch before merge: - **[deepghs/KohakuHub#84](https://github.com/deepghs/KohakuHub/pull/84)** — closes [deepghs/KohakuHub#78](https://github.com/deepghs/KohakuHub/issues/78). TTL default lowered from 300s to 30s; chain-probe decoupling so the simulate path doesn't mutate live cache; new admin chain-tester UI for operator debugging. - **[deepghs/KohakuHub#81](https://github.com/deepghs/KohakuHub/pull/81)** — closes [deepghs/KohakuHub#79](https://github.com/deepghs/KohakuHub/issues/79). Strict-freshness cache contract: per-`(user, tokens_hash, repo_id)` keying + three-dimensional generation counters (global / per-user / per-repo) + invalidation hooks on every mutation event (admin source mutations, user external-token rotations, local repo CRUD). - **[deepghs/KohakuHub#86](https://github.com/deepghs/KohakuHub/pull/86)** — closes [deepghs/KohakuHub#85](https://github.com/deepghs/KohakuHub/issues/85). Lock liveness regression — relax the binding lock (release before cache-hit I/O) + `asyncio.wait_for` supervisor so a stuck `attempt_fn` cannot freeze same-repo callers indefinitely. Bundled with a Plan A redirect-passthrough on resolve GET so multi-GB blobs never traverse the backend. - **[deepghs/KohakuHub#88](https://github.com/deepghs/KohakuHub/pull/88)** — `TypeError: 'NoneType' object does not support item assignment` regression caught in pre-merge testing. `dict.pop(key, default)` returns `None` when the key is present with a `None` value (default doesn't apply); combined with `client_headers or None` collapsing empty dicts to `None`, every token-bearing source 502'd on small-text-file resolves with `"category":"network"`. Fixed at the `FallbackClient` contract boundary. The strict-consistency contract is documented in `src/kohakuhub/api/fallback/README.md` (rewritten as part of this thread) and deliberately narrowed in [deepghs/KohakuHub#85](https://github.com/deepghs/KohakuHub/issues/85) to "safety under stable config, scoped per-process, not a liveness gate." --- ## 2 — HuggingFace error-contract alignment for permission denials **[deepghs/KohakuHub#89](https://github.com/deepghs/KohakuHub/pull/89)** — closes [deepghs/KohakuHub#76](https://github.com/deepghs/KohakuHub/issues/76) `check_repo_read_permission` previously raised `HTTPException(401)` for anonymous-on-private and `HTTPException(403)` for authed-no-access — both produced FastAPI's default `{"detail": "..."}` envelope with no `X-Error-Code`. A `huggingface_hub` client hitting a private hkub repo got a generic `HfHubHTTPError` instead of the named `RepositoryNotFoundError`, so `transformers` / `datasets` / etc. lost actionable error semantics. This PR introduces `RepoReadDeniedError` (a non-`HTTPException` exception class) raised by `check_repo_read_permission` for both anon-on-private and authed-no-access. A global FastAPI exception handler in `main.py` converts it into `404 + X-Error-Code: RepoNotFound` with an empty body — matching what HF returns to authenticated callers. `hf_raise_for_status` then dispatches `RepositoryNotFoundError` end-to-end (verified via live round-trip tests). Privacy-preserving Option A from the issue: anon and authed-no-access produce the same wire shape (no enumeration leak between "missing" and "private"); same client exception class; simpler than mirroring HF's anon-vs-authed split. A `hf_disabled_repo()` helper was also added — emits HF's exact `X-Error-Message: "Access to this resource is disabled."` string with no `X-Error-Code` (DisabledRepoError dispatch is keyed off the message verbatim). No call site yet; reserved for the future moderation feature. The `HFErrorCode` docstring labels its KohakuHub-only extensions distinctly from the four HF-recognized codes. Self-review during this PR caught one critical wrapper-interaction regression: `with_repo_fallback` has a generic `except Exception` catch (`api/fallback/decorators.py:363`) that absorbed the new exception and re-raised it as a 500. The original test suite missed it because the harness sets `KOHAKU_HUB_FALLBACK_ENABLED=false` *and* every wire-shape test pre-emptively passes `?fallback=false`. Fixed by inserting an explicit `except RepoReadDeniedError: raise` between the cancellation handler and the generic catch, plus a dedicated regression test that forces `cfg.fallback.enabled=True`. --- ## 3 — Cache & connection-pool infrastructure - **[deepghs/KohakuHub#74](https://github.com/deepghs/KohakuHub/pull/74)** — Valkey-based shared L2 cache backend, opt-in via `KOHAKU_HUB_CACHE_URL`. Includes a new `src/kohakuhub/cache.py` module with the cache primitive + boot-time flush coordinator (Mode-B namespaces wiped whenever Valkey's `run_id` changes since the last seen value, i.e. Valkey was restarted even if the API wasn't). Admin cache monitoring page + Redis probe added to `/admin/api/health`. 100% test coverage on `cache.py`, the admin cache router, and config validation. - **[deepghs/KohakuHub#61](https://github.com/deepghs/KohakuHub/pull/61)** — pool the `httpx.AsyncClient` in `LakeFSRestClient` instead of constructing one per request. Substantial p99 reduction on list/info paths that previously paid TLS handshake per LakeFS round-trip. --- ## 4 — List / info / tree perf pass - **[deepghs/KohakuHub#70](https://github.com/deepghs/KohakuHub/pull/70)** + **[#72](https://github.com/deepghs/KohakuHub/pull/72)** — replace per-row `LakeFS.get_branch + get_commit` calls in list endpoints (`/api/models`, `/api/datasets`, `/api/spaces`) with a single SQL aggregate (commit table). `#72` is the `dev/narugo1992` sync of `#70`. Coverage includes new upstream-compat tests against `huggingface_hub.HfApi.list_models / list_datasets / list_spaces` that pin response shape across the matrix (0.20.x → latest). - **[deepghs/KohakuHub#71](https://github.com/deepghs/KohakuHub/pull/71)** — `/org/{name}` inlines `repo_count` directly from the DB instead of dispatching three frontend `listRepos` fan-outs (which themselves N+1'd into LakeFS). - **[deepghs/KohakuHub#58](https://github.com/deepghs/KohakuHub/pull/58)** — drop the per-object DB query in `calculate_repository_storage`. Quota calculations on large repos no longer scale O(files). - **[deepghs/KohakuHub#60](https://github.com/deepghs/KohakuHub/pull/60)** — resolve `lastCommit` per page via LakeFS path-filtered `logCommits` instead of walking commits client-side. Closes [deepghs/KohakuHub#59](https://github.com/deepghs/KohakuHub/issues/59) (tree-expand 30s timeout on WAN-deployed instances). --- ## 5 — Indexed-tar feature suite A coordinated three-PR thread for surfacing `hfutils.index` TAR-format datasets natively: - **[deepghs/KohakuHub#50](https://github.com/deepghs/KohakuHub/pull/50)** — read-only browser for `hfutils.index` TAR + sidecar pairs (`.tar` + `.json` with member offsets). Detects sibling-tar pairs server-side; backend exposes a TAR-member listing endpoint; SPA renders a tree-style member browser inside the regular file list. - **[deepghs/KohakuHub#51](https://github.com/deepghs/KohakuHub/pull/51)** — lazy in-listing thumbnails for image members of indexed tars. The SPA performs a Range-read request per visible image member; thumbnail extraction is fully client-side, no precomputation. - **[deepghs/KohakuHub#53](https://github.com/deepghs/KohakuHub/pull/53)** (also covers list pagination — see §6) — adds the HEAD-probe that detects whether a `.tar` has an indexed sibling, gating the indexed-tar UI. --- ## 6 — Repo-list / tree UI features (HF-compatible) - **[deepghs/KohakuHub#53](https://github.com/deepghs/KohakuHub/pull/53)** — paginate the file list at 50 / 100 / 200 per page (same default boundaries HF uses). - **[deepghs/KohakuHub#55](https://github.com/deepghs/KohakuHub/pull/55)** — same-level `name_prefix` filter on `/tree` (closes [deepghs/KohakuHub#54](https://github.com/deepghs/KohakuHub/issues/54)). Matches HF's additive query parameter. - **[deepghs/KohakuHub#57](https://github.com/deepghs/KohakuHub/pull/57)** — swap the numbered pager for HuggingFace-style "Load more" cursor pagination (closes [deepghs/KohakuHub#56](https://github.com/deepghs/KohakuHub/issues/56)). --- ## 7 — Admin tooling - **[deepghs/KohakuHub#48](https://github.com/deepghs/KohakuHub/pull/48)** — dependency health dashboard for Postgres / MinIO / LakeFS / SMTP. Each has a probe with status + latency + last-error message in the admin SPA. - **[deepghs/KohakuHub#49](https://github.com/deepghs/KohakuHub/pull/49)** — global credentials view (active sessions, API tokens, SSH keys) with revoke. Closes [deepghs/KohakuHub#36](https://github.com/deepghs/KohakuHub/issues/36). --- ## 8 — Dev-server / monorepo - **[deepghs/KohakuHub#34](https://github.com/deepghs/KohakuHub/pull/34)** — unify `make ui` (admin SPA mounted under `/admin`); stop spurious Vite reloads triggered by Python file changes in shared dirs. Establishes the pnpm monorepo layout that subsequent PRs build on (`src/kohaku-hub-ui` + `src/kohaku-hub-admin` workspaces). --- ## 9 — Bug fixes - **[deepghs/KohakuHub#52](https://github.com/deepghs/KohakuHub/pull/52)** — forward same-origin session cookie on `/resolve/` Range probes from the SPA. Without it, private-repo Range reads from the parquet/safetensors preview path silently anonymized themselves and 401'd. - **[deepghs/KohakuHub#88](https://github.com/deepghs/KohakuHub/pull/88)** — covered in §1; client-layer `headers=None` `TypeError` regression caught in pre-merge testing of #77. --- ## Open follow-ups (intentionally not blocking this merge) These were either explicitly scoped out of the PRs above, or surfaced post-merge during pre-`dev → main` review. All tracked in the deepghs issue tracker; nothing here is new debt introduced by this PR thread without a corresponding tracker entry: - **[deepghs/KohakuHub#82](https://github.com/deepghs/KohakuHub/issues/82)** — header-token cache pollution (availability / LRU eviction). Privacy threat closed by [#79](https://github.com/deepghs/KohakuHub/issues/79); residual availability vector remains. - **[deepghs/KohakuHub#83](https://github.com/deepghs/KohakuHub/issues/83)** — migrate the fallback cache to the shared `kohakuhub.cache` backend so multi-worker deployments honour the strict-freshness contract across processes (rather than the current per-process scope). - **[deepghs/KohakuHub#87](https://github.com/deepghs/KohakuHub/issues/87)** — inline-200 GET path still buffers the full upstream body for non-HF mirrors. The 30x branch is fully redirect-passthrough after [#86](https://github.com/deepghs/KohakuHub/pull/86); the unbounded-buffer edge applies only to misconfigured non-HF mirrors that 200 a multi-GB blob. - **[deepghs/KohakuHub#90](https://github.com/deepghs/KohakuHub/issues/90)** — git-over-HTTP regression for anonymous-on-private clones surfaced by [#89](https://github.com/deepghs/KohakuHub/pull/89). `git/routers/http.py` is not wrapped by `with_repo_fallback`, so the privacy translation hits git transport too — `git clone` of a private repo now returns `404 + empty body` instead of `401 + WWW-Authenticate: Basic`. Fix sketched in the issue. - **[deepghs/KohakuHub#66](https://github.com/deepghs/KohakuHub/issues/66)** — `with_list_aggregation` defaults `fallback=true`; every list call hits external sources. Pre-existing. - **[deepghs/KohakuHub#26](https://github.com/deepghs/KohakuHub/issues/26)** — broad `except Exception` antipatterns (168 sites). Pre-existing. --- ## Test posture CI matrix on each landed PR covers backend tests across **Python 3.10 / 3.11 / 3.12 × `huggingface_hub` versions {0.20.3, 0.30.2, 0.36.2, 1.0.1, 1.6.0, latest}** (18 lanes), plus a `cache disabled` lane, plus codecov + admin frontend + frontend unit tests. Every PR landed in this thread shipped CI-green. The most recent ones ([#89](https://github.com/deepghs/KohakuHub/pull/89), [#88](https://github.com/deepghs/KohakuHub/pull/88), [#86](https://github.com/deepghs/KohakuHub/pull/86)) include hf_hub interop tests that drive a real client against a live test server. Significant new test artefacts: - `test/kohakuhub/api/fallback/test_strict_consistency.py` — invariants pinning the post-[#77](https://github.com/deepghs/KohakuHub/pull/77) binding contract. - `test/kohakuhub/api/fallback/test_strict_freshness.py` — generation-counter race protection ([#79](https://github.com/deepghs/KohakuHub/issues/79)). - `test/kohakuhub/api/fallback/test_chain_enumeration.py` + `test_e2e_matrix.py` — exhaustive matrix covering every wire shape × every operation type × `hf_hub_download` / `model_info` / `list_repo_tree` / `get_paths_info` dispatch behaviour. - `test/kohakuhub/api/test_repo_read_denial_hf_alignment.py` — wire-shape + interop tests for [#76](https://github.com/deepghs/KohakuHub/issues/76) ([#89](https://github.com/deepghs/KohakuHub/pull/89)) including the production-shape regression guard with fallback enabled. - `test/kohakuhub/test_cache.py` — 100% coverage of the L2 cache primitive ([#74](https://github.com/deepghs/KohakuHub/pull/74)). - `test/kohakuhub/test_lakefs_rest_client.py` + `test_lakefs_rest_client_live.py` — pooling / lifecycle coverage for [#61](https://github.com/deepghs/KohakuHub/pull/61). --- ## Migration notes for operators - **`KOHAKU_HUB_FALLBACK_CACHE_TTL_SECONDS`** default lowered from `300` to `30` ([#84](https://github.com/deepghs/KohakuHub/pull/84)). Existing deployments unaffected unless they explicitly relied on the longer window. - **`KOHAKU_HUB_CACHE_URL`** is the new opt-in for the L2 cache ([#74](https://github.com/deepghs/KohakuHub/pull/74)). Defaults to disabled; auto-enabled when set. Single-worker deployments without Redis continue to work as before. - **Permission-denial wire shape** changes from `401 / 403 + JSON` to `404 + X-Error-Code: RepoNotFound + empty body` for the HF API surfaces ([#89](https://github.com/deepghs/KohakuHub/pull/89)). The SPA already maps `RepoNotFound` and bare 401 to `NOT_FOUND`, so user-facing error UX shifts from "Forbidden" to "Not found" for outsiders on private repos — privacy-preserving by design. - **Git-over-HTTP for anonymous-on-private** has the same wire-shape change with a known UX regression for `git clone` (no `WWW-Authenticate: Basic` challenge); see [#90](https://github.com/deepghs/KohakuHub/issues/90) for the planned fix. - **pnpm monorepo layout** — `src/kohaku-hub-ui/` + `src/kohaku-hub-admin/` are workspaces; `make ui` runs both. Anyone vendoring this fork's frontend should update build scripts to point at `pnpm` workspaces ([#34](https://github.com/deepghs/KohakuHub/pull/34)). --- ## Per-PR commit anchor table (ordered by merge time) For full traceability: | Merge SHA | PR | Title (one-liner) | |---|---|---| | `d039f12` | [#34](https://github.com/deepghs/KohakuHub/pull/34) | Unify `make ui` (admin under `/admin`) and stop spurious Vite reloads | | `ad0f5bf` | [#48](https://github.com/deepghs/KohakuHub/pull/48) | feat(admin): dependency health dashboard for Postgres / MinIO / LakeFS / SMTP | | `2a4246b` | [#49](https://github.com/deepghs/KohakuHub/pull/49) | feat(admin): global credentials view (sessions / tokens / SSH keys) with revoke | | `b98b482` | [#50](https://github.com/deepghs/KohakuHub/pull/50) | Read-only browser for `hfutils.index` TAR + sidecar pairs | | `1e3d010` | [#51](https://github.com/deepghs/KohakuHub/pull/51) | Lazy in-listing thumbnails for indexed-tar image members | | `c064357` | [#52](https://github.com/deepghs/KohakuHub/pull/52) | fix(ui/range-reads): forward same-origin session on `/resolve/` Range probes | | `174554a` | [#53](https://github.com/deepghs/KohakuHub/pull/53) | feat(ui/repo-list): paginate file list (50/100/200) + HEAD-probe indexed-tar siblings | | `d68f2ba` | [#55](https://github.com/deepghs/KohakuHub/pull/55) | feat(tree): same-level `name_prefix` filter on `/tree` | | `b0d1279` | [#57](https://github.com/deepghs/KohakuHub/pull/57) | feat(ui/repo-file-list): swap numbered pager for "Load More" cursor pagination | | `ad357ee` | [#58](https://github.com/deepghs/KohakuHub/pull/58) | perf(quota): drop the per-object DB query in `calculate_repository_storage` | | `eb71342` | [#60](https://github.com/deepghs/KohakuHub/pull/60) | perf(tree): resolve lastCommit per page via LakeFS path-filtered logCommits | | `4947f93` | [#61](https://github.com/deepghs/KohakuHub/pull/61) | perf(lakefs-rest): pool the `httpx.AsyncClient` in `LakeFSRestClient` | | `ca573ea` | [#70](https://github.com/deepghs/KohakuHub/pull/70) | perf(repo/info): replace per-row LakeFS get_branch+get_commit with SQL aggregate | | `a2ba6c3` | [#71](https://github.com/deepghs/KohakuHub/pull/71) | perf(org): inline `repo_count`, drop frontend `listRepos` fan-out | | `c8f3757` | [#72](https://github.com/deepghs/KohakuHub/pull/72) | perf(repo/info): list endpoints SQL aggregate fix (sync of [#70](https://github.com/deepghs/KohakuHub/pull/70) onto dev) | | `1993885` | [#74](https://github.com/deepghs/KohakuHub/pull/74) | perf(cache): Valkey-based L2 cache infrastructure | | `9e7254c` | [#81](https://github.com/deepghs/KohakuHub/pull/81) | feat(fallback): strict-freshness cache contract (closes [#79](https://github.com/deepghs/KohakuHub/issues/79)) | | `4467133` | [#84](https://github.com/deepghs/KohakuHub/pull/84) | feat(fallback): TTL default 30s + chain-probe decoupling + admin chain-tester UI (closes [#78](https://github.com/deepghs/KohakuHub/issues/78)) | | `4af6419` | [#86](https://github.com/deepghs/KohakuHub/pull/86) | fix(fallback): relax binding lock + `asyncio.wait_for` supervisor (closes [#85](https://github.com/deepghs/KohakuHub/issues/85)) | | `a53acdd` | [#77](https://github.com/deepghs/KohakuHub/pull/77) | feat(fallback): enforce repo-grain binding (closes [#75](https://github.com/deepghs/KohakuHub/issues/75)) | | `e3089e6` | [#88](https://github.com/deepghs/KohakuHub/pull/88) | fix(fallback/client): treat caller-supplied `headers=None` as empty | | `f8773c4` | [#89](https://github.com/deepghs/KohakuHub/pull/89) | feat(auth/permissions): privacy-preserving RepoNotFound for read denials (closes [#76](https://github.com/deepghs/KohakuHub/issues/76)) | --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
GiteaMirror added the pull-request label 2026-05-06 19:48:03 -05:00
GiteaMirror changed title from [PR #17] sync deepghs:dev/narugo1992 → main: 22 PRs across fallback / errors / cache / perf / UI (Apr–May 2026) to [PR #17] [MERGED] sync deepghs:dev/narugo1992 → main: 22 PRs across fallback / errors / cache / perf / UI (Apr–May 2026) 2026-05-10 15:44:49 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
enhancement
pull-request
Mirrored from GitHub Pull Request
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/KohakuHub#17
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?