Three real-world bugs surfaced once the modal was actually used:
1. Member download returned the entire tar instead of the slice.
The memberView wrapper used by the Download button copied
{path, name, size, sha256} but dropped .offset; downloadMember
then re-called extractMemberBytes with offset === undefined,
the resulting "bytes=undefined-..." Range header was silently
ignored by MinIO, and the response was the full archive. Fix:
include offset in memberView, cache the already-extracted
bytes so the download reuses them with no second round-trip,
and have extractMemberBytes throw a clear TypeError on shape
misuse instead of delegating to the network.
2. Parquet / safetensors preview inside an archive failed with a
CORS-shaped "Browser blocked the request" error. The cause was
handing FilePreviewDialog a `blob:` URL — hyparquet's
asyncBufferFromUrl issues HEAD + Range against the source, and
`blob:` URLs do not honour those reliably. The safetensors
path showed the same failure as a "header too large" garbage
value because the intercepted response carried wrong bytes at
offset 0. Fix: add parseSafetensorsMetadataFromBuffer +
parseParquetMetadataFromBuffer that work on the in-memory
bytes the modal already has; FilePreviewDialog now accepts a
`bytes` prop and routes to the from-buffer parsers.
3. Inner FilePreviewDialog auto-opened on prop change and stacked
over the "Open metadata preview" button, intercepting clicks.
Fix: only open it on explicit button click; clear it on member
close.
README files inside the archive (with or without an extension) now
classify as markdown / text by basename instead of falling through
to "binary". Fixes the missing icon report.
Real-browser verification with Playwright drove the actual flow:
parquet + safetensors metadata dialogs render with the seeded
columns, tensors, parameter counts and __metadata__ — covered by
new from-buffer unit tests plus the screenshots collected during
the verification run.
Danbooru picks regenerated with order:random per the user request:
24 posts spanning post-id 3.6M – 11.2M (no longer adjacent IDs),
same 16-arknights-by-rating + 8-mixed-IP layout.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Plants 24 real Danbooru posts on top of the existing remote-asset
fetch path (hashes verified at seed time, content-addressed CDN URLs):
- 16 Arknights posts, four per Danbooru rating (g/s/q/e per the
howto:rate wiki), grouped under images/arknights/<rating>/<id>.<ext>
- 8 mixed-IP posts at the same rating spread (Genshin Impact / Blue
Archive / Hololive / original) under images/misc/<ip>/<rating>/...
The stale and no-hash demo archives now also reuse a real image so
the showcase no longer carries any synthetic colour rectangles.
The make_solid_color_image_bytes helper is removed accordingly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a third pure-client preview kind alongside safetensors and
parquet. A `.tar` file lights up an archive icon only when the same
listing also carries a `<basename>.json` sidecar — bare tars stay
untreated so the icon is not noise.
Clicking the icon opens TarBrowserDialog, which fetches the sidecar
JSON via /resolve/ and walks the in-tar directory tree client-side.
Member preview is routed to the same renderer the standalone blob
page uses (image / video / audio / pdf / text / markdown), and a
member is materialized via a single Range read against the .tar URL,
mirroring the algorithm in hfutils.index.local_fetch.
Hash banner:
- mismatch (sha256 in sidecar disagrees with the .tar): warning
- empty hash + hash_lfs: info notice (no verification possible)
- otherwise: silent
Member download builds an object URL anchor with a `download`
attribute equal to the member basename so the browser's native
save flow shows the in-tar filename, not the .tar filename.
Seed adds open-media-lab/indexed-tar-showcase with five tar+sidecar
pairs covering nested navigation, ~600-entry pagination, hash
mismatch, missing-hash, and inner safetensors / parquet metadata
preview. SEED_VERSION is bumped to v6 so existing local envs are
told to `make reset-and-seed`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The dev seed (``make reset-and-seed``) and the test baseline now plant a
known set of API tokens and SSH keys per-user. Three real ed25519 keypairs
ship in source, private half included, so local SSH smoke tests against a
freshly seeded backend can sign with the matching key without generating
one.
Dev seed (``scripts/dev/seed_demo_data.py``)
- 6 named tokens spread across mai_lin / leo_park / sara_chen / ivy_ops,
with last_used set to a mix of "recent", "180+ days stale" and
"never used" so admin Credentials filters have something to bite on.
- 3 SSH keys: mai_lin holds two distinct keypairs (Workstation +
Archived MBP), leo_park holds the secondary keypair. The schema's
global UNIQUE on ``fingerprint`` is what forces three keypairs for two
users, hence the third generated keypair.
- Helpers are idempotent: re-running the seed without a reset is a
no-op for already-planted tokens / keys.
- ``SEED_VERSION`` bumped from ``v4`` → ``v5`` so existing dev databases
are flagged as ``incomplete`` and the operator is told to run
``make reset-local-data``.
- Manifest at ``hub-meta/dev/demo-seed-manifest.json`` now lists every
planted token plaintext and SSH private key — those are explicitly
test fixtures, never production credentials.
Test baseline (``test/kohakuhub/support/seed.py``)
- Mirrors the same plants for the test users (owner, member, outsider)
via the same keypair PEMs, so anything written against
``SEED_TOKENS`` / ``SEED_SSH_KEYS`` carries over to dev manually.
- Tokens go in via direct DB insert (the public API never re-emits a
fixed plaintext); SSH keys go through ``POST /api/user/keys`` so the
fingerprint computed by the production code is the canonical one.
Tests (``test/kohakuhub/api/admin/routers/test_credentials.py``)
- 8 new cases assert against the seeded baseline:
- every seed plant surfaces in the admin list,
- planted plaintexts authenticate as real Bearer tokens,
- SSH fingerprints match the constants in ``seed_credentials``,
- per-user isolation holds,
- ``unused_for_days`` filters actually pick stale rows,
- revoking a seeded token immediately kills its Bearer auth.
- ``test_ssh_keys.py``'s "no keys yet" assumption is replaced with a
baseline-aware count check so it coexists with the planted keys.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
verify_seed_data.py hardcoded EXPECTED_SEED_VERSION = "local-dev-demo-v3"
but seed_demo_data.py was bumped to v4 in the preview PR, so the
post-seed verifier would falsely fail with a version mismatch. Extract
the constant to scripts/dev/seed_shared.py and import it from both
sides so the two scripts always agree.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Implements issue #27 v4: file-level HF-compatible metadata preview
computed entirely in the browser via HTTP Range reads against the
existing /resolve/ 302 → presigned S3/MinIO URL. Zero new backend
preview code, zero LRU, zero precomputation, zero new DB state.
Backend (minimal CORS plumbing only):
- main.py CORSMiddleware: add `expose_headers` so browsers can read
Content-Range / X-Linked-* / X-Repo-Commit / ETag / Location off
the final 206 response that follows the /resolve/ 302.
- docker-compose.example.yml + scripts/dev/up_infra.sh: wire
`MINIO_API_CORS_ALLOW_ORIGIN` so the SPA can cross-origin Range-read
presigned targets. Configurable via `DEV_MINIO_CORS_ALLOW_ORIGIN`.
- docs/development/local-dev.md: MinIO CORS section explaining the
hard prerequisite + smoke-test probe + how to recreate the container.
Frontend:
- utils/safetensors.js (~190 LOC): pure-JS parser mirroring
huggingface_hub.parse_safetensors_file_metadata byte-for-byte
(speculative 100 KB first read, two-read fallback for fat headers,
SAFETENSORS_MAX_HEADER_LENGTH guard). Exposes parseSafetensorsMetadata
+ summarizeSafetensors.
- utils/parquet.js: thin wrapper over hyparquet's asyncBufferFromUrl +
parquetMetadataAsync with mode:"cors" + credentials:"omit" so cookies
never leak onto presigned URLs. Normalizes BigInt row counts.
- components/repo/preview/FilePreviewDialog.vue: ElDialog with
per-phase spinner text (range-head → parsing → done for safetensors,
head → footer → parsing → done for parquet), dtype/row-group tables,
and an explicit "CORS likely misconfigured" placeholder on failure.
- RepoViewer.vue: HF-style chart-line-data icon next to .safetensors
and .parquet rows; click opens the modal with the resolved /resolve/
URL for the current branch.
Tests + fixtures:
- test_files.py::test_resolve_get_302_exposes_cors_headers_for_browser_preview
pins the `Access-Control-Expose-Headers` list against regressions.
- test/kohaku-hub-ui/utils/test_safetensors.test.js: 6 cases covering
the real-HF-format fixture, dtype summary, progress phases, fat-header
fallback, oversized-header guard, and non-206 error paths.
- test/kohaku-hub-ui/utils/test_parquet.test.js: footer parse +
progress phase assertions.
- test/kohaku-hub-ui/fixtures/previews/{tiny.safetensors,tiny.parquet}:
byte-identical-to-HF fixtures produced by the real safetensors /
pyarrow libs via scripts/dev/generate_preview_test_fixtures.py
(committed so tests stay offline per AGENTS.md §5.2).
Seed:
- seed_demo_data.py: add two RemoteAsset entries for real HF-hosted
small fixtures pinned by sha256, and wire them into visible paths
(open-media-lab/vision-language-assistant-3b/fixtures/hf-tiny-random-bert.safetensors,
open-media-lab/multimodal-benchmark-suite/fixtures/hf-no-robots-test.parquet)
so the preview can be exercised against files that actually came off
huggingface.co rather than purely local pyarrow/safetensors output.
SEED_VERSION bumped to local-dev-demo-v4.
Verified end-to-end against the dev stack: safetensors parser output
on the seeded fixtures matches huggingface_hub.parse_safetensors_file_metadata
byte-for-byte on the same file (100 tensors, 126,851 params, I64=512
/ F32=126,339, metadata `{format: pt, ...}`). Browser preview modal
renders both file kinds correctly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The local demo seed now installs https://huggingface.co as a low-priority
(priority=1000) global fallback source via the admin API, so a fresh
`make seed-demo` can resolve public HF repos out-of-the-box. Bumps the
seed version to local-dev-demo-v3 and updates verify_seed_data.py to
assert the seeded source is advertised via /api/fallback-sources/available.
The creation step is idempotent: it lists global sources first and skips
the insert when a matching URL already exists.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
