## Summary
Closes#78. Stacks on PR #77 (which carries #79's strict-freshness cache architecture); base is `feat/fallback-repo-binding` so #78's cache-architecture deps are already merged in.
Three pieces:
1. **TTL default 300 → 30s** — `cfg.fallback.cache_ttl_seconds` and `RepoSourceCache.__init__(ttl_seconds=30)`. `config-example.toml` updated. Self-heal window for transient bound-source failures shrinks 10×; the 30s bound is still long enough for hf_hub session consistency.
2. **Chain-probe decoupling — `core.probe_chain` pure function**. New `src/kohakuhub/api/fallback/core.py` with:
- `ProbeAttempt` / `ProbeReport` dataclasses
- `probe_chain(op, repo_type, namespace, name, sources, ...)` — pure async, no cache writes, no binding lock, no FastAPI request context
- Per-op HTTP method dispatch (HEAD for resolve, POST for paths_info, GET for info/tree)
- Reuses `utils.classify_upstream` so verdicts match production
- Captures **body preview** (UTF-8 decoded, 4 KB cap, binary-safe fallback) and curated **response headers** per attempt
- Sets `final_response` to the bound attempt's full shape so the UI can show what a production caller would receive
- 100% coverage in `test_core.py` (31 tests)
3. **Admin Chain Tester UI** — single-page panel on `fallback-sources.vue`:
- **System state — draft editor**. Independent of live config; ``Load from System`` deep-copies live sources into the draft; every edit toggles a "Draft modified" tag; ``Push to System`` runs strong-confirm + atomic `bulk-replace`; ``Discard Draft`` resets.
- **User state — simulation**. Identity radio (anonymous / username / user_id) + free-form Authorization-header-style per-URL token overrides (the `Bearer xxx|url,token|...` shape, pre-decoded).
- **Run + timeline**. Two run buttons: ``Simulate`` against the draft, ``Real`` against live config + impersonated identity. Result renders as final-outcome tag → bound source → per-attempt rows (status, X-Error-Code, curated headers, expandable body preview) → final response section.
- Op selector uses HF-API-equivalent labels (`Repo info` / `List files` / `Resolve file` / `paths_info`).
## New backend endpoints
| Method | Path | Purpose |
|---|---|---|
| `PUT` | `/admin/api/fallback/sources-bulk-replace` | Atomic transactional replace of every `FallbackSource` row. Triggers `cache.clear()`. |
| `POST` | `/admin/api/fallback/test/simulate` | Run `probe_chain` with operator-supplied source list + per-URL token overlay. No cache writes. |
| `POST` | `/admin/api/fallback/test/real` | Run `probe_chain` against live `get_enabled_sources(...)`, optionally impersonating a user (DB tokens layered with header-style overrides per the production `get_merged_external_tokens` precedence). |
## Files touched
| File | Change |
|---|---|
| `src/kohakuhub/api/fallback/cache.py` | Default TTL 300 → 30. Updated docstring. |
| `src/kohakuhub/api/fallback/core.py` | **NEW** — pure `probe_chain` + `ProbeAttempt` / `ProbeReport`. |
| `src/kohakuhub/api/admin/routers/fallback.py` | Three new endpoints (bulk-replace + test/simulate + test/real). |
| `src/kohakuhub/config.py` | Default 300 → 30. |
| `config-example.toml` | Default + comment. |
| `src/kohaku-hub-admin/src/utils/api.js` | Three wrappers (`bulkReplaceFallbackSources`, `testFallbackChainSimulate`, `testFallbackChainReal`). |
| `src/kohaku-hub-admin/src/pages/fallback-sources.vue` | Chain Tester card (~600 added lines). |
| `test/kohakuhub/api/fallback/test_core.py` | **NEW** — 31 tests at 100% line coverage. |
| `test/kohakuhub/api/admin/routers/test_fallback_debug.py` | **NEW** — 21 tests for the three admin endpoints. |
| `test/kohaku-hub-admin/pages/test_fallback_sources_page.test.js` | Extended from 60 to 91 tests covering every tester flow. |
| `test/kohaku-hub-admin/utils/test_api.test.js` | Invocation + URL asserts for the three new wrappers. |
## Test plan
- [x] `pytest test/kohakuhub/api/fallback/test_core.py` — 31/31 pass; `core.py` 100% line coverage
- [x] `pytest test/kohakuhub/api/admin/routers/test_fallback_debug.py` — 21/21 pass
- [x] Combined fallback + admin coverage — `core.py` 100%, admin router 91% (only pre-existing CRUD exception handlers uncovered)
- [x] `npm test` — 91/91 pass; `fallback-sources.vue` 100% lines + statements, 96.47% branches
- [x] Backend regression suite (changed-area scope): **477 passed, 1 skipped** in 239s
- [x] `pnpm build` — admin SPA bundles cleanly
- [ ] CI matrix (Python 3.10/3.11/3.12 × hf_hub versions)
## Notes for reviewers
- The chain tester's probe is intentionally simpler than the production binding path (no HEAD-then-GET commit for resolve; single per-source call). The verdict (BIND_AND_RESPOND / BIND_AND_PROPAGATE / TRY_NEXT_SOURCE) matches production via the shared `classify_upstream`, so a tester says-it-binds source will bind in production too.
- `final_response` is `None` for `CHAIN_EXHAUSTED` — the production aggregate is built from `utils.build_aggregate_failure_response` which depends on op-specific scope semantics the tester deliberately doesn't impose. The per-attempt list is sufficient to diagnose what each source said.
- "Push to system" replaces every row atomically. There's no per-row Save anymore for draft edits — the user explicitly asked for batched-edit + final-save semantics in lieu of inline auto-save (matches the workflow in their direction).
- Live-server browser verification deferred to the user's own `make reset-and-seed backend` per their stated workflow; the SPA infrastructure is identical to PR #81's admin frontend (already Playwright-verified) and the page builds cleanly.
Refs: #78, #77, #79
Introduces the prerequisite cache layer tracked in #73. No business code
yet consumes the helpers — this is plumbing only, gated by
KOHAKU_HUB_CACHE_ENABLED (default: false). Subsequent issues will adopt
specific cache patterns on top of this foundation.
Why now: every hot read endpoint currently makes 1–3 LakeFS REST calls
plus several Postgres queries per request, with the only existing cache
being a per-process cachetools.TTLCache in fallback/cache.py — useless
across the default 4-worker uvicorn deployment.
Design highlights (full design in docs/development/cache.md):
- Pure cache, no source-of-truth state. Silent-degradation contract:
every cache call is wrapped in try/except and falls back to L3 when
Valkey is unreachable. CI runs a dedicated cache-disabled job to
regression-guard this.
- L1 (per-worker cachetools) restricted to immutable / content-addressed
data only — multi-worker uvicorn has no portable cross-worker
invalidation channel, and constraining L1 to "key contains its own
version" sidesteps that entirely.
- L2 (Valkey) holds everything. Helpers ship with TTL jitter (±15%
default), two-level singleflight (asyncio.Lock + Valkey SET NX EX),
negative cache, generation counters, and per-namespace metrics.
- Persistence: RDB on, AOF off, persistent volume. Mode-A
(lakefs:commit, lakefs:stat, lakefs:list — commit_id-keyed) survives
restart safely. Mode-B (mutable) namespaces are flushed on every
Valkey restart by a run_id-based bootstrap coordinator that
serializes the flush across workers.
Includes:
- src/kohakuhub/cache.py — the helper module (319 stmts, 83% coverage
via the new test module).
- src/kohakuhub/api/admin/routers/cache.py — admin endpoints exposing
hit/miss/error counters, Valkey memory state, and bootstrap-flush
metadata.
- test/kohakuhub/test_cache.py — 34 tests against a real Valkey,
covering: round-trips, TTL jitter spread, SCAN-based prefix delete
over >SCAN_BATCH_SIZE keys, two-level singleflight (100 concurrent
calls fold to 1 fetch), bootstrap flush selectivity (Mode-A survives,
Mode-B is wiped, exactly), two-worker bootstrap coordination, silent
degradation when Valkey is disabled OR unreachable, generation
counters, negative cache, the Mode-B prefix list shape contract.
- docker-compose.example.yml — adds the valkey service with RDB +
LFU + bind-mounted hub-meta/valkey-data, mirroring the persistence
pattern of the other stateful services.
- scripts/dev/up_infra.sh / down_infra.sh / reset_local_data.sh —
Valkey container plumbing for local dev (host port 26379).
- .github/workflows/fullstack-tests.yml — adds valkey to the existing
matrix services and adds a separate single-Python job
(backend-tests-cache-disabled) running with KOHAKU_HUB_CACHE_ENABLED=false
as the silent-degradation contract regression guard.
- docs/development/cache.md — the design doc referenced by the cache
module's docstrings.
Refs: #73
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous commit (perf(tree): use LakeFS path-filtered logCommits ...)
relies on the ``objects=`` / ``prefixes=`` / ``limit=`` query parameters on
``logCommits``, introduced in LakeFS v0.54.0 (released 2021-11-08).
Pre-v0.54 servers silently ignore those parameters and would surface
incorrect ``lastCommit`` metadata on the file-list ``expand=true`` page.
The shipped docker bundle pins ``treeverse/lakefs:latest`` so default
deployments are always compatible. The constraint only matters for
self-deployments that pin an older LakeFS image.
Document the requirement in:
* README — Architecture/Stack line and the Quick Start "Prereq" callout.
* docs/setup.md — new "Prerequisites" section.
* docs/deployment.md — new "Component Version Requirements" section.
* docs/deployment/production.md — new "Component Versions" section.
* docs/deployment/docker.md — extended ``lakefs`` service description.
Code-side docstrings on ``LakeFSRestClient.log_commits`` and
``tree.resolve_last_commits_for_paths`` already carry the same note from
the original perf commit.
Implements issue #27 v4: file-level HF-compatible metadata preview
computed entirely in the browser via HTTP Range reads against the
existing /resolve/ 302 → presigned S3/MinIO URL. Zero new backend
preview code, zero LRU, zero precomputation, zero new DB state.
Backend (minimal CORS plumbing only):
- main.py CORSMiddleware: add `expose_headers` so browsers can read
Content-Range / X-Linked-* / X-Repo-Commit / ETag / Location off
the final 206 response that follows the /resolve/ 302.
- docker-compose.example.yml + scripts/dev/up_infra.sh: wire
`MINIO_API_CORS_ALLOW_ORIGIN` so the SPA can cross-origin Range-read
presigned targets. Configurable via `DEV_MINIO_CORS_ALLOW_ORIGIN`.
- docs/development/local-dev.md: MinIO CORS section explaining the
hard prerequisite + smoke-test probe + how to recreate the container.
Frontend:
- utils/safetensors.js (~190 LOC): pure-JS parser mirroring
huggingface_hub.parse_safetensors_file_metadata byte-for-byte
(speculative 100 KB first read, two-read fallback for fat headers,
SAFETENSORS_MAX_HEADER_LENGTH guard). Exposes parseSafetensorsMetadata
+ summarizeSafetensors.
- utils/parquet.js: thin wrapper over hyparquet's asyncBufferFromUrl +
parquetMetadataAsync with mode:"cors" + credentials:"omit" so cookies
never leak onto presigned URLs. Normalizes BigInt row counts.
- components/repo/preview/FilePreviewDialog.vue: ElDialog with
per-phase spinner text (range-head → parsing → done for safetensors,
head → footer → parsing → done for parquet), dtype/row-group tables,
and an explicit "CORS likely misconfigured" placeholder on failure.
- RepoViewer.vue: HF-style chart-line-data icon next to .safetensors
and .parquet rows; click opens the modal with the resolved /resolve/
URL for the current branch.
Tests + fixtures:
- test_files.py::test_resolve_get_302_exposes_cors_headers_for_browser_preview
pins the `Access-Control-Expose-Headers` list against regressions.
- test/kohaku-hub-ui/utils/test_safetensors.test.js: 6 cases covering
the real-HF-format fixture, dtype summary, progress phases, fat-header
fallback, oversized-header guard, and non-206 error paths.
- test/kohaku-hub-ui/utils/test_parquet.test.js: footer parse +
progress phase assertions.
- test/kohaku-hub-ui/fixtures/previews/{tiny.safetensors,tiny.parquet}:
byte-identical-to-HF fixtures produced by the real safetensors /
pyarrow libs via scripts/dev/generate_preview_test_fixtures.py
(committed so tests stay offline per AGENTS.md §5.2).
Seed:
- seed_demo_data.py: add two RemoteAsset entries for real HF-hosted
small fixtures pinned by sha256, and wire them into visible paths
(open-media-lab/vision-language-assistant-3b/fixtures/hf-tiny-random-bert.safetensors,
open-media-lab/multimodal-benchmark-suite/fixtures/hf-no-robots-test.parquet)
so the preview can be exercised against files that actually came off
huggingface.co rather than purely local pyarrow/safetensors output.
SEED_VERSION bumped to local-dev-demo-v4.
Verified end-to-end against the dev stack: safetensors parser output
on the seeded fixtures matches huggingface_hub.parse_safetensors_file_metadata
byte-for-byte on the same file (100 tensors, 126,851 params, I64=512
/ F32=126,339, metadata `{format: pt, ...}`). Browser preview modal
renders both file kinds correctly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
