Adds a third pure-client preview kind alongside safetensors and
parquet. A `.tar` file lights up an archive icon only when the same
listing also carries a `<basename>.json` sidecar — bare tars stay
untreated so the icon is not noise.
Clicking the icon opens TarBrowserDialog, which fetches the sidecar
JSON via /resolve/ and walks the in-tar directory tree client-side.
Member preview is routed to the same renderer the standalone blob
page uses (image / video / audio / pdf / text / markdown), and a
member is materialized via a single Range read against the .tar URL,
mirroring the algorithm in hfutils.index.local_fetch.
Hash banner:
- mismatch (sha256 in sidecar disagrees with the .tar): warning
- empty hash + hash_lfs: info notice (no verification possible)
- otherwise: silent
Member download builds an object URL anchor with a `download`
attribute equal to the member basename so the browser's native
save flow shows the in-tar filename, not the .tar filename.
Seed adds open-media-lab/indexed-tar-showcase with five tar+sidecar
pairs covering nested navigation, ~600-entry pagination, hash
mismatch, missing-hash, and inner safetensors / parquet metadata
preview. SEED_VERSION is bumped to v6 so existing local envs are
told to `make reset-and-seed`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The dev seed (``make reset-and-seed``) and the test baseline now plant a
known set of API tokens and SSH keys per-user. Three real ed25519 keypairs
ship in source, private half included, so local SSH smoke tests against a
freshly seeded backend can sign with the matching key without generating
one.
Dev seed (``scripts/dev/seed_demo_data.py``)
- 6 named tokens spread across mai_lin / leo_park / sara_chen / ivy_ops,
with last_used set to a mix of "recent", "180+ days stale" and
"never used" so admin Credentials filters have something to bite on.
- 3 SSH keys: mai_lin holds two distinct keypairs (Workstation +
Archived MBP), leo_park holds the secondary keypair. The schema's
global UNIQUE on ``fingerprint`` is what forces three keypairs for two
users, hence the third generated keypair.
- Helpers are idempotent: re-running the seed without a reset is a
no-op for already-planted tokens / keys.
- ``SEED_VERSION`` bumped from ``v4`` → ``v5`` so existing dev databases
are flagged as ``incomplete`` and the operator is told to run
``make reset-local-data``.
- Manifest at ``hub-meta/dev/demo-seed-manifest.json`` now lists every
planted token plaintext and SSH private key — those are explicitly
test fixtures, never production credentials.
Test baseline (``test/kohakuhub/support/seed.py``)
- Mirrors the same plants for the test users (owner, member, outsider)
via the same keypair PEMs, so anything written against
``SEED_TOKENS`` / ``SEED_SSH_KEYS`` carries over to dev manually.
- Tokens go in via direct DB insert (the public API never re-emits a
fixed plaintext); SSH keys go through ``POST /api/user/keys`` so the
fingerprint computed by the production code is the canonical one.
Tests (``test/kohakuhub/api/admin/routers/test_credentials.py``)
- 8 new cases assert against the seeded baseline:
- every seed plant surfaces in the admin list,
- planted plaintexts authenticate as real Bearer tokens,
- SSH fingerprints match the constants in ``seed_credentials``,
- per-user isolation holds,
- ``unused_for_days`` filters actually pick stale rows,
- revoking a seeded token immediately kills its Bearer auth.
- ``test_ssh_keys.py``'s "no keys yet" assumption is replaced with a
baseline-aware count check so it coexists with the planted keys.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
verify_seed_data.py hardcoded EXPECTED_SEED_VERSION = "local-dev-demo-v3"
but seed_demo_data.py was bumped to v4 in the preview PR, so the
post-seed verifier would falsely fail with a version mismatch. Extract
the constant to scripts/dev/seed_shared.py and import it from both
sides so the two scripts always agree.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Implements issue #27 v4: file-level HF-compatible metadata preview
computed entirely in the browser via HTTP Range reads against the
existing /resolve/ 302 → presigned S3/MinIO URL. Zero new backend
preview code, zero LRU, zero precomputation, zero new DB state.
Backend (minimal CORS plumbing only):
- main.py CORSMiddleware: add `expose_headers` so browsers can read
Content-Range / X-Linked-* / X-Repo-Commit / ETag / Location off
the final 206 response that follows the /resolve/ 302.
- docker-compose.example.yml + scripts/dev/up_infra.sh: wire
`MINIO_API_CORS_ALLOW_ORIGIN` so the SPA can cross-origin Range-read
presigned targets. Configurable via `DEV_MINIO_CORS_ALLOW_ORIGIN`.
- docs/development/local-dev.md: MinIO CORS section explaining the
hard prerequisite + smoke-test probe + how to recreate the container.
Frontend:
- utils/safetensors.js (~190 LOC): pure-JS parser mirroring
huggingface_hub.parse_safetensors_file_metadata byte-for-byte
(speculative 100 KB first read, two-read fallback for fat headers,
SAFETENSORS_MAX_HEADER_LENGTH guard). Exposes parseSafetensorsMetadata
+ summarizeSafetensors.
- utils/parquet.js: thin wrapper over hyparquet's asyncBufferFromUrl +
parquetMetadataAsync with mode:"cors" + credentials:"omit" so cookies
never leak onto presigned URLs. Normalizes BigInt row counts.
- components/repo/preview/FilePreviewDialog.vue: ElDialog with
per-phase spinner text (range-head → parsing → done for safetensors,
head → footer → parsing → done for parquet), dtype/row-group tables,
and an explicit "CORS likely misconfigured" placeholder on failure.
- RepoViewer.vue: HF-style chart-line-data icon next to .safetensors
and .parquet rows; click opens the modal with the resolved /resolve/
URL for the current branch.
Tests + fixtures:
- test_files.py::test_resolve_get_302_exposes_cors_headers_for_browser_preview
pins the `Access-Control-Expose-Headers` list against regressions.
- test/kohaku-hub-ui/utils/test_safetensors.test.js: 6 cases covering
the real-HF-format fixture, dtype summary, progress phases, fat-header
fallback, oversized-header guard, and non-206 error paths.
- test/kohaku-hub-ui/utils/test_parquet.test.js: footer parse +
progress phase assertions.
- test/kohaku-hub-ui/fixtures/previews/{tiny.safetensors,tiny.parquet}:
byte-identical-to-HF fixtures produced by the real safetensors /
pyarrow libs via scripts/dev/generate_preview_test_fixtures.py
(committed so tests stay offline per AGENTS.md §5.2).
Seed:
- seed_demo_data.py: add two RemoteAsset entries for real HF-hosted
small fixtures pinned by sha256, and wire them into visible paths
(open-media-lab/vision-language-assistant-3b/fixtures/hf-tiny-random-bert.safetensors,
open-media-lab/multimodal-benchmark-suite/fixtures/hf-no-robots-test.parquet)
so the preview can be exercised against files that actually came off
huggingface.co rather than purely local pyarrow/safetensors output.
SEED_VERSION bumped to local-dev-demo-v4.
Verified end-to-end against the dev stack: safetensors parser output
on the seeded fixtures matches huggingface_hub.parse_safetensors_file_metadata
byte-for-byte on the same file (100 tensors, 126,851 params, I64=512
/ F32=126,339, metadata `{format: pt, ...}`). Browser preview modal
renders both file kinds correctly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The local demo seed now installs https://huggingface.co as a low-priority
(priority=1000) global fallback source via the admin API, so a fresh
`make seed-demo` can resolve public HF repos out-of-the-box. Bumps the
seed version to local-dev-demo-v3 and updates verify_seed_data.py to
assert the seeded source is advertised via /api/fallback-sources/available.
The creation step is idempotent: it lists global sources first and skips
the insert when a matching URL already exists.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
