Restructures the PR #77 demo work for the right separation of
concerns:
1. **Primary seed creates the demo repos.** Adds 4 OrganizationSeed
+ 4 RepoSeed entries to ``scripts/dev/seed_demo_data.py``, so a
single ``make reset-and-seed`` produces:
- openai-community/gpt2 (README only; collides with HF GPT-2)
- bigscience/bloom (README + small text note)
- meta-llama/Llama-2-7b (README only; HF version is gated)
- narugo1992-pr77-demo/guide (README; pure-fallback walk-through)
mai_lin owns each org as super-admin. The repos are public
(private=False) so anyone can browse them in the SPA at
``/{namespace}/{name}`` without logging in.
2. **Standalone verification command stays separate.**
``scripts/dev/seed_pr77_demo.py`` is now a thin one-shot script
that drives every demo's contract via the real ``huggingface_hub``
library and asserts each step:
PYTHONPATH=src python scripts/dev/seed_pr77_demo.py
No seeding side-effects — just contract assertions. Exits 0 if
every step holds end-to-end, non-zero on the first failure.
3. **READMEs are full verification guides.** Each demo's README has:
- one-shot verification command (the python script above)
- copy-pasteable curl commands with expected response bodies
- python equivalents using huggingface_hub
- the pre-fix counterfactual where it shines a light on
cross-source mixing
So a reviewer can either run one command, or hand-curl a single
URL, or open the README in the SPA — all three lead to the same
contract verification.
Tested end-to-end: ``make reset-and-seed`` creates everything; the
standalone verify script runs through 11 assertions across the 4
demos and reports "every PR #77 demo contract holds end-to-end".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a fourth demo repo at ``meta-llama/Llama-2-7b`` — same path as
the real (gated) HuggingFace repo, locally a README only. Documents
the strict-consistency contract against a gated upstream:
local-namespace ownership applies even when the upstream variant is
gated. Reading a non-local file under this local repo surfaces
``EntryNotFoundError`` from the local handler — never HF's
``GatedRepo`` 401, never bytes from HF.
Verified end-to-end on the dev backend:
$ curl -i .../meta-llama/Llama-2-7b/resolve/main/config.json
→ 404 + X-Error-Code: EntryNotFound, no X-Source-Count
(local short-circuit; HF's gated response never reached client)
For comparison, request a gated HF repo with no local collision:
$ curl -i .../meta-llama/Llama-2-70b/resolve/main/config.json
→ 401 + X-Error-Code: GatedRepo, x-source-count: 1
(chain ran; aggregate carries GatedRepo)
Same upstream gating, different outcomes based on whether the local
namespace owns the path. The narugo1992-pr77-demo/guide README is
extended with the pure-fallback gated case so reviewers can hit both
sides without leaving the guide.
Drive-by: switched the seed's idempotency check from path-presence to
content-size equality. Re-running the seed after a script-side
README rewrite now correctly re-uploads (was: silently skipped on
size mismatch).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds ``scripts/dev/seed_pr77_demo.py`` — an idempotent secondary
seed run after ``make seed-demo`` that creates a small set of
local repos whose ``(namespace, name)`` collides with real
HuggingFace repos but whose content is intentionally structurally
different. Together they let any reviewer hit a few
``huggingface_hub`` calls and watch the strict-consistency
contract behave as expected on a single-HF-fallback dev backend.
Three repos:
- ``openai-community/gpt2`` — README.md only.
- ``bigscience/bloom`` — README.md + a small text note.
- ``narugo1992-pr77-demo/guide`` — README.md only; documents
the genuine-pure-fallback path (no local collision).
Each repo's ``README.md`` is the verification guide for that
contract: it explains what the repo is for, lists the exact
``HfApi`` / ``hf_hub_download`` calls to drive against the dev
backend, and states what the response should look like on this PR
versus the pre-fix counterfactual. Where applicable, also includes
a wire-level ``curl -i`` against the resolve URL to inspect the
response headers (X-Error-Code / X-Source / X-Source-Count) — the
absence of X-Source* headers is the proof that the fallback chain
did not run when it shouldn't have.
The script is idempotent against itself: re-running it logs
``[=]`` for the entries already present and exits without touching
anything. Default credentials are the seed's ``mai_lin`` account.
Targets ``http://127.0.0.1:48888`` and the dev admin token by
default; override the constants at the top of the script for
non-dev backends.
Why no Makefile target: the existing ``make seed-demo`` shouldn't
be cluttered with PR-specific demos; running this script
explicitly keeps the seed surface minimal. The script's docstring
documents the one-line invocation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two issues surfaced during a make reset-and-seed walk-through against an
existing dev checkout:
1) Backend reported configured_enabled=false even with Valkey running.
Root cause: the contributor's pre-existing .env.dev predates the cache
feature and didn't pick up the new KOHAKU_HUB_CACHE_ENABLED line from
.env.dev.example. ``make init-env`` is intentionally a no-op when
.env.dev exists (per AGENTS.md), so env files drift forever.
Fix is two-layered:
- ``scripts/dev/run_backend.sh`` now defaults
KOHAKU_HUB_CACHE_URL=redis://127.0.0.1:26379/0 if not provided. The
up_infra.sh ensure_valkey step always runs Valkey on host port
26379, so this is correct for any standard local-dev setup. Explicit
env / .env.dev values still win.
- ``config.py`` cache loader now treats the presence of CACHE_URL in
env as implicit-enable when CACHE_ENABLED is unset. Explicit
CACHE_ENABLED=false still wins, so the silent-degradation regression
job in CI keeps validating the off path.
With both layers, ``make reset-and-seed && make backend`` from a stale
.env.dev now boots with the cache live and connected — verified
end-to-end against a clean dev valkey (configured_enabled=true,
client_initialized=true, bootstrap flush completing on first connect).
2) Admin /health page didn't include the cache. Operationally important:
when "low cache hit rate" turns out to be "Valkey unreachable", the
admin UI must surface that signal in the same place the rest of the
dependencies live.
``probe_redis`` added to admin/utils/health.py with the same shape as
the existing probes (PING + INFO server, tight timeout, never raises).
It honors the cache.enabled toggle: configured-off shows up as
``disabled`` (consistent with the smtp probe pattern), configured-on
but unreachable shows up as ``down`` with the underlying connect
error in detail, configured-on and reachable shows up as ``ok`` with
the version field disambiguating Valkey from Redis ("Valkey 7.2.4"
vs "Redis 7.4.0"). The admin UI label is "Redis" for everything else
— protocol identity is what matters operationally.
Verified all three states by toggling KOHAKU_HUB_CACHE_ENABLED and
stopping/starting kohakuhub-dev-valkey while polling the endpoint.
Test updates:
- ``test_health_page.test.js`` SAMPLE_PAYLOAD gains a ``redis`` entry
(status=ok, version="Valkey 8.0.1") so the page test exercises the
new label mapping. Existing assertions use SAMPLE_PAYLOAD.dependencies.length
so they stay correct without mechanical updates.
- All existing cache, config, and admin UI tests still pass (37 backend +
47 admin UI).
Refs: #73, #74
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Brings the L2 cache fully into the operational UI surface and tightens the
local-reset semantics around the cache.
Admin UI surface:
- New Cache page at /cache, modeled on the existing Health page: shows
configured/initialized state, namespace, Valkey memory + maxmemory_policy
+ eviction counter, the bootstrap-flush metadata (last seen run_id,
timestamp, keys flushed), and per-namespace hit/miss/error/set/invalidate
counters with computed hit rate. Auto-refresh selector with 0/5s/15s/60s
options. "Reset counters" button is gated to the operational state and
disabled when the cache is configured-off.
- Three operational states surfaced as a single tag:
* enabled — configured on AND client initialized
* degraded — configured on but client not initialized (silent
fallback to source — operationally important to
surface; without this, "low hit rate" looks like a
miss-heavy workload when actually Valkey is unreachable)
* disabled — configured off
- New AdminLayout nav entry pointing at /cache.
- New API client wrappers getCacheStats / resetCacheMetrics in
src/kohaku-hub-admin/src/utils/api.js, mirroring the read/probe split
used elsewhere.
Tests (test/kohaku-hub-admin/pages/test_cache_page.test.js, 11 cases):
mount + render assertions for each operational state; refresh button
re-fetches; reset confirm dialog flow (both confirmed and cancelled
paths) verified through ElMessageBox.confirm spying; 401/403 force
re-login; missing token redirects to /login; auto-refresh interval
toggles + unmount cleanup. ElMessageBox.confirm is hijacked via
``vi.spyOn`` against the live Element Plus module rather than
``vi.mock("element-plus", ...)``: Element Plus's resolved module identity
inside Vite differs from a test-file ``vi.mock`` interceptor, so spying
on the actual module is the only path that lets the SFC's
``await ElMessageBox.confirm(...)`` land in our test-controllable spy.
Reset semantics:
- ``scripts/dev/reset_local_data.sh`` now follows FLUSHALL with an
explicit synchronous SAVE on the dev Valkey container. Without this
the on-disk RDB still holds entries from before the reset; a Valkey
crash before the next BGSAVE would resurrect cache entries pointing
at repos / commits that no longer exist — observable as ghost cache
hits after ``make reset`` "succeeded".
Refs: #73, #74
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Introduces the prerequisite cache layer tracked in #73. No business code
yet consumes the helpers — this is plumbing only, gated by
KOHAKU_HUB_CACHE_ENABLED (default: false). Subsequent issues will adopt
specific cache patterns on top of this foundation.
Why now: every hot read endpoint currently makes 1–3 LakeFS REST calls
plus several Postgres queries per request, with the only existing cache
being a per-process cachetools.TTLCache in fallback/cache.py — useless
across the default 4-worker uvicorn deployment.
Design highlights (full design in docs/development/cache.md):
- Pure cache, no source-of-truth state. Silent-degradation contract:
every cache call is wrapped in try/except and falls back to L3 when
Valkey is unreachable. CI runs a dedicated cache-disabled job to
regression-guard this.
- L1 (per-worker cachetools) restricted to immutable / content-addressed
data only — multi-worker uvicorn has no portable cross-worker
invalidation channel, and constraining L1 to "key contains its own
version" sidesteps that entirely.
- L2 (Valkey) holds everything. Helpers ship with TTL jitter (±15%
default), two-level singleflight (asyncio.Lock + Valkey SET NX EX),
negative cache, generation counters, and per-namespace metrics.
- Persistence: RDB on, AOF off, persistent volume. Mode-A
(lakefs:commit, lakefs:stat, lakefs:list — commit_id-keyed) survives
restart safely. Mode-B (mutable) namespaces are flushed on every
Valkey restart by a run_id-based bootstrap coordinator that
serializes the flush across workers.
Includes:
- src/kohakuhub/cache.py — the helper module (319 stmts, 83% coverage
via the new test module).
- src/kohakuhub/api/admin/routers/cache.py — admin endpoints exposing
hit/miss/error counters, Valkey memory state, and bootstrap-flush
metadata.
- test/kohakuhub/test_cache.py — 34 tests against a real Valkey,
covering: round-trips, TTL jitter spread, SCAN-based prefix delete
over >SCAN_BATCH_SIZE keys, two-level singleflight (100 concurrent
calls fold to 1 fetch), bootstrap flush selectivity (Mode-A survives,
Mode-B is wiped, exactly), two-worker bootstrap coordination, silent
degradation when Valkey is disabled OR unreachable, generation
counters, negative cache, the Mode-B prefix list shape contract.
- docker-compose.example.yml — adds the valkey service with RDB +
LFU + bind-mounted hub-meta/valkey-data, mirroring the persistence
pattern of the other stateful services.
- scripts/dev/up_infra.sh / down_infra.sh / reset_local_data.sh —
Valkey container plumbing for local dev (host port 26379).
- .github/workflows/fullstack-tests.yml — adds valkey to the existing
matrix services and adds a separate single-Python job
(backend-tests-cache-disabled) running with KOHAKU_HUB_CACHE_ENABLED=false
as the silent-degradation contract regression guard.
- docs/development/cache.md — the design doc referenced by the cache
module's docstrings.
Refs: #73
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The tree endpoint with `expand=true` was reproducing `git log --follow` for
each path on the page by walking the LakeFS commit graph manually: one
unfiltered `log_commits` call followed by per-commit `diff_refs` calls,
client-side matching diff entries against unresolved targets. Latency was
O(commits-walked-from-HEAD-until-the-deepest-target-resolved). On a
100-commit / 50-path page this scaled to ~7 s on loopback and ~25 s on
WAN-deployed instances; on the new 280-commit churn fixture it hit ~20 s
locally and matched the user-reported 30 s+ stalls on hub.deepghs.org
(see issue #59).
The replacement (issue #59 Plan E):
* Per file target → `logCommits(objects=[path], amount=1, limit=true)`.
* Per directory target → `logCommits(prefixes=[path/], amount=1, limit=true)`.
* Both fanned out under `Semaphore(LAST_COMMIT_LOOKUP_CONCURRENCY=16)`.
* Drops `_apply_changed_path`, `TREE_DIFF_PAGE_SIZE`, and
`TREE_COMMIT_SCAN_PAGE_SIZE` — no client-side commit walk anymore.
LakeFS implements the path filter via its content-addressed metarange tree
(`pkg/catalog/catalog.go:checkPathListInCommit`): when a key's containing
range hash matches between two commits, the key didn't change — no diff
fetch, no value comparison, just two range-ID equality checks. Each call is
single-digit milliseconds regardless of how deep the path sits in history.
Behaviour preserved (live-checked on the new stress fixture):
* `lastCommit` payload shape unchanged: `{id, title, date}`.
* Identical `id` values across all entries vs. the old algorithm on a
48-entry recursive page (every (path, commit_id) tuple matches).
* Per-target failures stay non-fatal — the affected entry resolves to
`null` and the rest of the page still surfaces, matching the previous
diff-walk's log-and-continue behaviour.
Measured on the planted `tree-expand-stress-bench` (280 chaotic commits,
~94 surviving paths through add / modify / delete / restore / folder-delete):
| page | before | after | speedup |
|-------------------------------|-----------|-----------|---------|
| root (2 entries: README+dir) | 19.76 s | 0.40 s | 49× |
| /shard/group_00 (10 files) | 1.73 s | 0.72 s | 2.4× |
| /shard recursive (48 files) | 17.66 s | 3.55 s | 5× |
Connection pooling for `LakeFSRestClient` is intentionally NOT bundled here
(also called out as a follow-up in #59); WAN benefit will widen further
once it lands.
LakeFS version requirement: `objects` / `prefixes` / `limit` parameters on
`logCommits` were introduced in LakeFS v0.54.0 (2021-11-08). Anything from
v0.54 onward works; pre-v0.54 servers ignore the params and return the
unfiltered log. KohakuHub's docker bundle pins `treeverse/lakefs:latest` so
default deployments are always compatible. Documented in
`lakefs_rest_client.log_commits` and `tree.resolve_last_commits_for_paths`
docstrings.
Tests:
* `test_tree_unit.py` — three new tests cover the call shape (objects=
vs prefixes= dispatch, amount=1+limit=true contract, fan-out result
map), per-target failure isolation, and the
LAST_COMMIT_LOOKUP_CONCURRENCY semaphore cap. Old diff-walk tests for
`_apply_changed_path` / paginated diff are removed.
* `test_lakefs_rest_client.py` — new test pins the params shape
(list-of-tuples for repeated `objects` / `prefixes`, `"true"`/`"false"`
serialisation for `limit` / `first_parent`). One existing assertion
updated for the params-list shape.
* Full backend suite green: 626 passed in 542 s (was 624 before; +2 new
tests).
Acceptance fixture: a new local-only seed `mai_lin/tree-expand-stress-bench`
(280 commits / ~94 surviving files) exercises the chaotic-history pattern
behind issue #59 — heavy modify / delete / restore cycles biased to a hot
12-path tier, plus periodic `deletedFolder` ops. The seed is byte-deterministic
(hash-based byte stream, no `random` module) per AGENTS §2. Bumps SEED_VERSION
to `local-dev-demo-v8`. `commit_files` extended to dispatch the new
`DeletedFileSeed` / `DeletedFolderSeed` / `CopyFileSeed` op shapes.
(`CopyFileSeed` is wired through but unused for now — KohakuHub's
`process_copy_file` re-links the source's internal LakeFS physical address,
which LakeFS 1.80 rejects with "address is not signed: link address invalid"
for non-LFS sources. Filed as separate issue.)
Refs: #59 (the original perf report and updated Plan E follow-up).
Generating + uploading 1000 tar/json pairs took long enough on a
fresh `make reset-and-seed` that the user noticed it as a stall.
Drop to 250 — 10 pages at the default 50/page, 3 pages at 200/page,
still demonstrably "needs paging" without the wait.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two changes:
1. New `mai_lin/big-indexed-tar-bench` dataset seed: 1000
hfutils.index-compatible tar/json pairs sit under `archives/`
so the new file-list pager can be exercised against a directory
that genuinely needs paging (40 pages at 50/page; 10 at 200/page).
Every bundle clones the same minimal one-member tar — LakeFS
dedupes by content hash so the underlying object storage stays
one bundle, not 1000, and the seed runs in seconds instead of
minutes.
2. RepoViewer pagination patch coverage: add 4 more cases that
close the gap around the new code paths
(`goToFirstFileListPage`, the paginated empty-search placeholder,
and `findReadmeViaPathsInfo`'s success + failure modes). Patch
coverage on the new code now lands at ~94% — the only remaining
uncovered lines are the probeId staleness early-return inside
the indexed-tar HEAD-probe pass, which is a race-condition path
that does not have a deterministic test surface.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The safetensors / parquet / indexed-tar / tar-thumbnail readers all hit
`/resolve/...` directly with `fetch()` and pinned `credentials: "omit"`,
which dropped the SPA session cookie. On a private repo the backend's
HF-compat anti-enumeration path then returned 404 even though the user
was logged in — visible in production where the click-through download
worked but the inline preview's Range probe 404'd. Switch every
range-read entry point to `credentials: "same-origin"`: cookies travel
on the same-origin /resolve/ hop and are dropped automatically on the
cross-origin redirect to S3/MinIO, so presigned URLs answered with
`Access-Control-Allow-Origin: *` still work.
New regression test pins the contract on every entry point and
includes an explicit "404 if credentials would be omitted" case so a
future regression toward `omit` re-trips the test. Demo seed adds two
owner-only repos (`mai_lin/private-range-preview-bench` and
`mai_lin/private-vision-checkpoint`) so the parquet / indexed-tar /
safetensors preview paths can be exercised against a private repo
locally.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Visual feedback after wiring up the lazy thumbnails:
- The container used `object-cover`, which center-cropped the
preview. Switched to `object-contain` so the full image fits
inside the square area (letterboxed when not square). Drops
the "interesting bit got cut off" surprise on tall illustrations.
- The grid-card thumbnail used a fixed 56-px wrapper, so the image
was always smaller than the card. Now the wrapper auto-stretches
to the card's inner width via `aspect-square w-full`, and the
image inside uses `w-full h-full object-contain` so the bitmap
scales up to the card edges with proper letterboxing.
- Bumped THUMB_MAX_DIM from 128 → 256 px (and quality 0.7 → 0.78).
At 128 px the rendered <img> still showed an empty halo inside
the larger grid card (a 128-px source can't fill a 200+ px slot
without being upscaled). 256 px keeps caches small (~25 KB per
thumbnail) while feeding the larger grid layout.
- List-view fixed-size mode also bumped from 28 → 48 px so the
thumbnail is actually identifiable at a glance.
- Component now exposes a no-`size`-prop "auto" mode that the
panel uses for grid view (parent picks the width); the explicit
pixel size mode is retained for list rows.
Seed:
- New `archives/gallery/flat-images.tar` + sidecar carrying all 24
Danbooru showcase images flat at the tar root. Lives next to
`bundle.tar` so a single folder visit produces a long
scroll-driven listing — the right shape for hand-checking the
IntersectionObserver lazy-load + concurrency-pool drain
behaviour. SEED_VERSION → v7.
Visual verification: drove Chromium via Playwright through the
flat-images.tar listing (list mode + grid mode + toggle off);
24/24 thumbnails load, fill the squares without cropping, and the
toggle correctly reverts every row to the placeholder icon.
UI suite remains 366/40 green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Image rows inside an indexed-tar listing now lazily decode a small
preview in place of the static placeholder icon. The work is gated
on visibility (IntersectionObserver), bandwidth-aware (per-format
strategy chain), and globally togglable (localStorage-persisted).
Strategy chain — first non-null wins:
1. jpeg-exif Range-read first 64 KB; parse JPEG markers;
extract APP1 / IFD1 thumbnail JPEG. Saves
up to 99% of bandwidth on EXIF-bearing JPEGs.
2. small-image ≤ 256 KB images: full read + canvas resize.
3. medium-image 256 KB – 5 MB images: full read + canvas
resize (head bytes from #1 are reused via
a shared ExtractionContext).
> 5 MB no strategy matches → placeholder stays.
The chain is registered in `_STRATEGIES`, so future formats
(video first-frame, etc.) plug in by appending another entry.
Pure-client end to end: no backend thumbnail service, no extra API.
Concurrency capped at 4 in flight; an LRU of 100 thumbnail blob URLs
caches across folder navigation and revokes the oldest when evicted.
A toolbar switch in the panel persists the user's choice via
localStorage (key: kohaku-tar-thumbnail-enabled). Default ON; OFF
short-circuits everything — no IO subscription, no Range read, no
cache lookup.
Failure semantics: every error path (network 5xx, decode failure,
abort) silently falls back to the placeholder. Never a thrown
render, never a toast. The user just sees the same icon they would
have seen pre-feature.
Test fixtures generated via scripts/dev/generate_preview_test_fixtures.py:
- with_exif_thumb.jpg — JPEG carrying an embedded thumbnail in
APP1 / IFD1, exercises the EXIF probe success path.
- no_exif_thumb.jpg — same visual, EXIF stripped, exercises
the parser's null-return + caller's strategy fall-through.
Coverage on the new modules:
TarMemberThumbnail.vue 100% / 100%
TarBrowserPanel.vue 99.42% / 84.72% (was 95.98%)
tar-thumbnail.js 78.98% / 75.75% (canvas-decode path
is jsdom-untestable)
Total UI suite: 363 passing across 40 files.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three real-world bugs surfaced once the modal was actually used:
1. Member download returned the entire tar instead of the slice.
The memberView wrapper used by the Download button copied
{path, name, size, sha256} but dropped .offset; downloadMember
then re-called extractMemberBytes with offset === undefined,
the resulting "bytes=undefined-..." Range header was silently
ignored by MinIO, and the response was the full archive. Fix:
include offset in memberView, cache the already-extracted
bytes so the download reuses them with no second round-trip,
and have extractMemberBytes throw a clear TypeError on shape
misuse instead of delegating to the network.
2. Parquet / safetensors preview inside an archive failed with a
CORS-shaped "Browser blocked the request" error. The cause was
handing FilePreviewDialog a `blob:` URL — hyparquet's
asyncBufferFromUrl issues HEAD + Range against the source, and
`blob:` URLs do not honour those reliably. The safetensors
path showed the same failure as a "header too large" garbage
value because the intercepted response carried wrong bytes at
offset 0. Fix: add parseSafetensorsMetadataFromBuffer +
parseParquetMetadataFromBuffer that work on the in-memory
bytes the modal already has; FilePreviewDialog now accepts a
`bytes` prop and routes to the from-buffer parsers.
3. Inner FilePreviewDialog auto-opened on prop change and stacked
over the "Open metadata preview" button, intercepting clicks.
Fix: only open it on explicit button click; clear it on member
close.
README files inside the archive (with or without an extension) now
classify as markdown / text by basename instead of falling through
to "binary". Fixes the missing icon report.
Real-browser verification with Playwright drove the actual flow:
parquet + safetensors metadata dialogs render with the seeded
columns, tensors, parameter counts and __metadata__ — covered by
new from-buffer unit tests plus the screenshots collected during
the verification run.
Danbooru picks regenerated with order:random per the user request:
24 posts spanning post-id 3.6M – 11.2M (no longer adjacent IDs),
same 16-arknights-by-rating + 8-mixed-IP layout.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Plants 24 real Danbooru posts on top of the existing remote-asset
fetch path (hashes verified at seed time, content-addressed CDN URLs):
- 16 Arknights posts, four per Danbooru rating (g/s/q/e per the
howto:rate wiki), grouped under images/arknights/<rating>/<id>.<ext>
- 8 mixed-IP posts at the same rating spread (Genshin Impact / Blue
Archive / Hololive / original) under images/misc/<ip>/<rating>/...
The stale and no-hash demo archives now also reuse a real image so
the showcase no longer carries any synthetic colour rectangles.
The make_solid_color_image_bytes helper is removed accordingly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a third pure-client preview kind alongside safetensors and
parquet. A `.tar` file lights up an archive icon only when the same
listing also carries a `<basename>.json` sidecar — bare tars stay
untreated so the icon is not noise.
Clicking the icon opens TarBrowserDialog, which fetches the sidecar
JSON via /resolve/ and walks the in-tar directory tree client-side.
Member preview is routed to the same renderer the standalone blob
page uses (image / video / audio / pdf / text / markdown), and a
member is materialized via a single Range read against the .tar URL,
mirroring the algorithm in hfutils.index.local_fetch.
Hash banner:
- mismatch (sha256 in sidecar disagrees with the .tar): warning
- empty hash + hash_lfs: info notice (no verification possible)
- otherwise: silent
Member download builds an object URL anchor with a `download`
attribute equal to the member basename so the browser's native
save flow shows the in-tar filename, not the .tar filename.
Seed adds open-media-lab/indexed-tar-showcase with five tar+sidecar
pairs covering nested navigation, ~600-entry pagination, hash
mismatch, missing-hash, and inner safetensors / parquet metadata
preview. SEED_VERSION is bumped to v6 so existing local envs are
told to `make reset-and-seed`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The dev seed (``make reset-and-seed``) and the test baseline now plant a
known set of API tokens and SSH keys per-user. Three real ed25519 keypairs
ship in source, private half included, so local SSH smoke tests against a
freshly seeded backend can sign with the matching key without generating
one.
Dev seed (``scripts/dev/seed_demo_data.py``)
- 6 named tokens spread across mai_lin / leo_park / sara_chen / ivy_ops,
with last_used set to a mix of "recent", "180+ days stale" and
"never used" so admin Credentials filters have something to bite on.
- 3 SSH keys: mai_lin holds two distinct keypairs (Workstation +
Archived MBP), leo_park holds the secondary keypair. The schema's
global UNIQUE on ``fingerprint`` is what forces three keypairs for two
users, hence the third generated keypair.
- Helpers are idempotent: re-running the seed without a reset is a
no-op for already-planted tokens / keys.
- ``SEED_VERSION`` bumped from ``v4`` → ``v5`` so existing dev databases
are flagged as ``incomplete`` and the operator is told to run
``make reset-local-data``.
- Manifest at ``hub-meta/dev/demo-seed-manifest.json`` now lists every
planted token plaintext and SSH private key — those are explicitly
test fixtures, never production credentials.
Test baseline (``test/kohakuhub/support/seed.py``)
- Mirrors the same plants for the test users (owner, member, outsider)
via the same keypair PEMs, so anything written against
``SEED_TOKENS`` / ``SEED_SSH_KEYS`` carries over to dev manually.
- Tokens go in via direct DB insert (the public API never re-emits a
fixed plaintext); SSH keys go through ``POST /api/user/keys`` so the
fingerprint computed by the production code is the canonical one.
Tests (``test/kohakuhub/api/admin/routers/test_credentials.py``)
- 8 new cases assert against the seeded baseline:
- every seed plant surfaces in the admin list,
- planted plaintexts authenticate as real Bearer tokens,
- SSH fingerprints match the constants in ``seed_credentials``,
- per-user isolation holds,
- ``unused_for_days`` filters actually pick stale rows,
- revoking a seeded token immediately kills its Bearer auth.
- ``test_ssh_keys.py``'s "no keys yet" assumption is replaced with a
baseline-aware count check so it coexists with the planted keys.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
verify_seed_data.py hardcoded EXPECTED_SEED_VERSION = "local-dev-demo-v3"
but seed_demo_data.py was bumped to v4 in the preview PR, so the
post-seed verifier would falsely fail with a version mismatch. Extract
the constant to scripts/dev/seed_shared.py and import it from both
sides so the two scripts always agree.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Implements issue #27 v4: file-level HF-compatible metadata preview
computed entirely in the browser via HTTP Range reads against the
existing /resolve/ 302 → presigned S3/MinIO URL. Zero new backend
preview code, zero LRU, zero precomputation, zero new DB state.
Backend (minimal CORS plumbing only):
- main.py CORSMiddleware: add `expose_headers` so browsers can read
Content-Range / X-Linked-* / X-Repo-Commit / ETag / Location off
the final 206 response that follows the /resolve/ 302.
- docker-compose.example.yml + scripts/dev/up_infra.sh: wire
`MINIO_API_CORS_ALLOW_ORIGIN` so the SPA can cross-origin Range-read
presigned targets. Configurable via `DEV_MINIO_CORS_ALLOW_ORIGIN`.
- docs/development/local-dev.md: MinIO CORS section explaining the
hard prerequisite + smoke-test probe + how to recreate the container.
Frontend:
- utils/safetensors.js (~190 LOC): pure-JS parser mirroring
huggingface_hub.parse_safetensors_file_metadata byte-for-byte
(speculative 100 KB first read, two-read fallback for fat headers,
SAFETENSORS_MAX_HEADER_LENGTH guard). Exposes parseSafetensorsMetadata
+ summarizeSafetensors.
- utils/parquet.js: thin wrapper over hyparquet's asyncBufferFromUrl +
parquetMetadataAsync with mode:"cors" + credentials:"omit" so cookies
never leak onto presigned URLs. Normalizes BigInt row counts.
- components/repo/preview/FilePreviewDialog.vue: ElDialog with
per-phase spinner text (range-head → parsing → done for safetensors,
head → footer → parsing → done for parquet), dtype/row-group tables,
and an explicit "CORS likely misconfigured" placeholder on failure.
- RepoViewer.vue: HF-style chart-line-data icon next to .safetensors
and .parquet rows; click opens the modal with the resolved /resolve/
URL for the current branch.
Tests + fixtures:
- test_files.py::test_resolve_get_302_exposes_cors_headers_for_browser_preview
pins the `Access-Control-Expose-Headers` list against regressions.
- test/kohaku-hub-ui/utils/test_safetensors.test.js: 6 cases covering
the real-HF-format fixture, dtype summary, progress phases, fat-header
fallback, oversized-header guard, and non-206 error paths.
- test/kohaku-hub-ui/utils/test_parquet.test.js: footer parse +
progress phase assertions.
- test/kohaku-hub-ui/fixtures/previews/{tiny.safetensors,tiny.parquet}:
byte-identical-to-HF fixtures produced by the real safetensors /
pyarrow libs via scripts/dev/generate_preview_test_fixtures.py
(committed so tests stay offline per AGENTS.md §5.2).
Seed:
- seed_demo_data.py: add two RemoteAsset entries for real HF-hosted
small fixtures pinned by sha256, and wire them into visible paths
(open-media-lab/vision-language-assistant-3b/fixtures/hf-tiny-random-bert.safetensors,
open-media-lab/multimodal-benchmark-suite/fixtures/hf-no-robots-test.parquet)
so the preview can be exercised against files that actually came off
huggingface.co rather than purely local pyarrow/safetensors output.
SEED_VERSION bumped to local-dev-demo-v4.
Verified end-to-end against the dev stack: safetensors parser output
on the seeded fixtures matches huggingface_hub.parse_safetensors_file_metadata
byte-for-byte on the same file (100 tensors, 126,851 params, I64=512
/ F32=126,339, metadata `{format: pt, ...}`). Browser preview modal
renders both file kinds correctly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The local demo seed now installs https://huggingface.co as a low-priority
(priority=1000) global fallback source via the admin API, so a fresh
`make seed-demo` can resolve public HF repos out-of-the-box. Bumps the
seed version to local-dev-demo-v3 and updates verify_seed_data.py to
assert the seeded source is advertised via /api/fallback-sources/available.
The creation step is idempotent: it lists global sources first and skips
the insert when a matching URL already exists.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
