## Summary
Closes#78. Stacks on PR #77 (which carries #79's strict-freshness cache architecture); base is `feat/fallback-repo-binding` so #78's cache-architecture deps are already merged in.
Three pieces:
1. **TTL default 300 → 30s** — `cfg.fallback.cache_ttl_seconds` and `RepoSourceCache.__init__(ttl_seconds=30)`. `config-example.toml` updated. Self-heal window for transient bound-source failures shrinks 10×; the 30s bound is still long enough for hf_hub session consistency.
2. **Chain-probe decoupling — `core.probe_chain` pure function**. New `src/kohakuhub/api/fallback/core.py` with:
- `ProbeAttempt` / `ProbeReport` dataclasses
- `probe_chain(op, repo_type, namespace, name, sources, ...)` — pure async, no cache writes, no binding lock, no FastAPI request context
- Per-op HTTP method dispatch (HEAD for resolve, POST for paths_info, GET for info/tree)
- Reuses `utils.classify_upstream` so verdicts match production
- Captures **body preview** (UTF-8 decoded, 4 KB cap, binary-safe fallback) and curated **response headers** per attempt
- Sets `final_response` to the bound attempt's full shape so the UI can show what a production caller would receive
- 100% coverage in `test_core.py` (31 tests)
3. **Admin Chain Tester UI** — single-page panel on `fallback-sources.vue`:
- **System state — draft editor**. Independent of live config; ``Load from System`` deep-copies live sources into the draft; every edit toggles a "Draft modified" tag; ``Push to System`` runs strong-confirm + atomic `bulk-replace`; ``Discard Draft`` resets.
- **User state — simulation**. Identity radio (anonymous / username / user_id) + free-form Authorization-header-style per-URL token overrides (the `Bearer xxx|url,token|...` shape, pre-decoded).
- **Run + timeline**. Two run buttons: ``Simulate`` against the draft, ``Real`` against live config + impersonated identity. Result renders as final-outcome tag → bound source → per-attempt rows (status, X-Error-Code, curated headers, expandable body preview) → final response section.
- Op selector uses HF-API-equivalent labels (`Repo info` / `List files` / `Resolve file` / `paths_info`).
## New backend endpoints
| Method | Path | Purpose |
|---|---|---|
| `PUT` | `/admin/api/fallback/sources-bulk-replace` | Atomic transactional replace of every `FallbackSource` row. Triggers `cache.clear()`. |
| `POST` | `/admin/api/fallback/test/simulate` | Run `probe_chain` with operator-supplied source list + per-URL token overlay. No cache writes. |
| `POST` | `/admin/api/fallback/test/real` | Run `probe_chain` against live `get_enabled_sources(...)`, optionally impersonating a user (DB tokens layered with header-style overrides per the production `get_merged_external_tokens` precedence). |
## Files touched
| File | Change |
|---|---|
| `src/kohakuhub/api/fallback/cache.py` | Default TTL 300 → 30. Updated docstring. |
| `src/kohakuhub/api/fallback/core.py` | **NEW** — pure `probe_chain` + `ProbeAttempt` / `ProbeReport`. |
| `src/kohakuhub/api/admin/routers/fallback.py` | Three new endpoints (bulk-replace + test/simulate + test/real). |
| `src/kohakuhub/config.py` | Default 300 → 30. |
| `config-example.toml` | Default + comment. |
| `src/kohaku-hub-admin/src/utils/api.js` | Three wrappers (`bulkReplaceFallbackSources`, `testFallbackChainSimulate`, `testFallbackChainReal`). |
| `src/kohaku-hub-admin/src/pages/fallback-sources.vue` | Chain Tester card (~600 added lines). |
| `test/kohakuhub/api/fallback/test_core.py` | **NEW** — 31 tests at 100% line coverage. |
| `test/kohakuhub/api/admin/routers/test_fallback_debug.py` | **NEW** — 21 tests for the three admin endpoints. |
| `test/kohaku-hub-admin/pages/test_fallback_sources_page.test.js` | Extended from 60 to 91 tests covering every tester flow. |
| `test/kohaku-hub-admin/utils/test_api.test.js` | Invocation + URL asserts for the three new wrappers. |
## Test plan
- [x] `pytest test/kohakuhub/api/fallback/test_core.py` — 31/31 pass; `core.py` 100% line coverage
- [x] `pytest test/kohakuhub/api/admin/routers/test_fallback_debug.py` — 21/21 pass
- [x] Combined fallback + admin coverage — `core.py` 100%, admin router 91% (only pre-existing CRUD exception handlers uncovered)
- [x] `npm test` — 91/91 pass; `fallback-sources.vue` 100% lines + statements, 96.47% branches
- [x] Backend regression suite (changed-area scope): **477 passed, 1 skipped** in 239s
- [x] `pnpm build` — admin SPA bundles cleanly
- [ ] CI matrix (Python 3.10/3.11/3.12 × hf_hub versions)
## Notes for reviewers
- The chain tester's probe is intentionally simpler than the production binding path (no HEAD-then-GET commit for resolve; single per-source call). The verdict (BIND_AND_RESPOND / BIND_AND_PROPAGATE / TRY_NEXT_SOURCE) matches production via the shared `classify_upstream`, so a tester says-it-binds source will bind in production too.
- `final_response` is `None` for `CHAIN_EXHAUSTED` — the production aggregate is built from `utils.build_aggregate_failure_response` which depends on op-specific scope semantics the tester deliberately doesn't impose. The per-attempt list is sufficient to diagnose what each source said.
- "Push to system" replaces every row atomically. There's no per-row Save anymore for draft edits — the user explicitly asked for batched-edit + final-save semantics in lieu of inline auto-save (matches the workflow in their direction).
- Live-server browser verification deferred to the user's own `make reset-and-seed backend` per their stated workflow; the SPA infrastructure is identical to PR #81's admin frontend (already Playwright-verified) and the page builds cleanly.
Refs: #78, #77, #79
The previous commit (perf(tree): use LakeFS path-filtered logCommits ...)
relies on the ``objects=`` / ``prefixes=`` / ``limit=`` query parameters on
``logCommits``, introduced in LakeFS v0.54.0 (released 2021-11-08).
Pre-v0.54 servers silently ignore those parameters and would surface
incorrect ``lastCommit`` metadata on the file-list ``expand=true`` page.
The shipped docker bundle pins ``treeverse/lakefs:latest`` so default
deployments are always compatible. The constraint only matters for
self-deployments that pin an older LakeFS image.
Document the requirement in:
* README — Architecture/Stack line and the Quick Start "Prereq" callout.
* docs/setup.md — new "Prerequisites" section.
* docs/deployment.md — new "Component Version Requirements" section.
* docs/deployment/production.md — new "Component Versions" section.
* docs/deployment/docker.md — extended ``lakefs`` service description.
Code-side docstrings on ``LakeFSRestClient.log_commits`` and
``tree.resolve_last_commits_for_paths`` already carry the same note from
the original perf commit.
