Files
KohakuHub/docs/deployment/production.md
Naomi Rue Golding 4467133bf5 feat(fallback): TTL default 30s + chain-probe decoupling + admin chain-tester UI (#78)
## Summary

Closes #78. Stacks on PR #77 (which carries #79's strict-freshness cache architecture); base is `feat/fallback-repo-binding` so #78's cache-architecture deps are already merged in.

Three pieces:

1. **TTL default 300 → 30s** — `cfg.fallback.cache_ttl_seconds` and `RepoSourceCache.__init__(ttl_seconds=30)`. `config-example.toml` updated. Self-heal window for transient bound-source failures shrinks 10×; the 30s bound is still long enough for hf_hub session consistency.

2. **Chain-probe decoupling — `core.probe_chain` pure function**. New `src/kohakuhub/api/fallback/core.py` with:
   - `ProbeAttempt` / `ProbeReport` dataclasses
   - `probe_chain(op, repo_type, namespace, name, sources, ...)` — pure async, no cache writes, no binding lock, no FastAPI request context
   - Per-op HTTP method dispatch (HEAD for resolve, POST for paths_info, GET for info/tree)
   - Reuses `utils.classify_upstream` so verdicts match production
   - Captures **body preview** (UTF-8 decoded, 4 KB cap, binary-safe fallback) and curated **response headers** per attempt
   - Sets `final_response` to the bound attempt's full shape so the UI can show what a production caller would receive
   - 100% coverage in `test_core.py` (31 tests)

3. **Admin Chain Tester UI** — single-page panel on `fallback-sources.vue`:
   - **System state — draft editor**. Independent of live config; ``Load from System`` deep-copies live sources into the draft; every edit toggles a "Draft modified" tag; ``Push to System`` runs strong-confirm + atomic `bulk-replace`; ``Discard Draft`` resets.
   - **User state — simulation**. Identity radio (anonymous / username / user_id) + free-form Authorization-header-style per-URL token overrides (the `Bearer xxx|url,token|...` shape, pre-decoded).
   - **Run + timeline**. Two run buttons: ``Simulate`` against the draft, ``Real`` against live config + impersonated identity. Result renders as final-outcome tag → bound source → per-attempt rows (status, X-Error-Code, curated headers, expandable body preview) → final response section.
   - Op selector uses HF-API-equivalent labels (`Repo info` / `List files` / `Resolve file` / `paths_info`).

## New backend endpoints

| Method | Path | Purpose |
|---|---|---|
| `PUT` | `/admin/api/fallback/sources-bulk-replace` | Atomic transactional replace of every `FallbackSource` row. Triggers `cache.clear()`. |
| `POST` | `/admin/api/fallback/test/simulate` | Run `probe_chain` with operator-supplied source list + per-URL token overlay. No cache writes. |
| `POST` | `/admin/api/fallback/test/real` | Run `probe_chain` against live `get_enabled_sources(...)`, optionally impersonating a user (DB tokens layered with header-style overrides per the production `get_merged_external_tokens` precedence). |

## Files touched

| File | Change |
|---|---|
| `src/kohakuhub/api/fallback/cache.py` | Default TTL 300 → 30. Updated docstring. |
| `src/kohakuhub/api/fallback/core.py` | **NEW** — pure `probe_chain` + `ProbeAttempt` / `ProbeReport`. |
| `src/kohakuhub/api/admin/routers/fallback.py` | Three new endpoints (bulk-replace + test/simulate + test/real). |
| `src/kohakuhub/config.py` | Default 300 → 30. |
| `config-example.toml` | Default + comment. |
| `src/kohaku-hub-admin/src/utils/api.js` | Three wrappers (`bulkReplaceFallbackSources`, `testFallbackChainSimulate`, `testFallbackChainReal`). |
| `src/kohaku-hub-admin/src/pages/fallback-sources.vue` | Chain Tester card (~600 added lines). |
| `test/kohakuhub/api/fallback/test_core.py` | **NEW** — 31 tests at 100% line coverage. |
| `test/kohakuhub/api/admin/routers/test_fallback_debug.py` | **NEW** — 21 tests for the three admin endpoints. |
| `test/kohaku-hub-admin/pages/test_fallback_sources_page.test.js` | Extended from 60 to 91 tests covering every tester flow. |
| `test/kohaku-hub-admin/utils/test_api.test.js` | Invocation + URL asserts for the three new wrappers. |

## Test plan

- [x] `pytest test/kohakuhub/api/fallback/test_core.py` — 31/31 pass; `core.py` 100% line coverage
- [x] `pytest test/kohakuhub/api/admin/routers/test_fallback_debug.py` — 21/21 pass
- [x] Combined fallback + admin coverage — `core.py` 100%, admin router 91% (only pre-existing CRUD exception handlers uncovered)
- [x] `npm test` — 91/91 pass; `fallback-sources.vue` 100% lines + statements, 96.47% branches
- [x] Backend regression suite (changed-area scope): **477 passed, 1 skipped** in 239s
- [x] `pnpm build` — admin SPA bundles cleanly
- [ ] CI matrix (Python 3.10/3.11/3.12 × hf_hub versions)

## Notes for reviewers

- The chain tester's probe is intentionally simpler than the production binding path (no HEAD-then-GET commit for resolve; single per-source call). The verdict (BIND_AND_RESPOND / BIND_AND_PROPAGATE / TRY_NEXT_SOURCE) matches production via the shared `classify_upstream`, so a tester says-it-binds source will bind in production too.
- `final_response` is `None` for `CHAIN_EXHAUSTED` — the production aggregate is built from `utils.build_aggregate_failure_response` which depends on op-specific scope semantics the tester deliberately doesn't impose. The per-attempt list is sufficient to diagnose what each source said.
- "Push to system" replaces every row atomically. There's no per-row Save anymore for draft edits — the user explicitly asked for batched-edit + final-save semantics in lieu of inline auto-save (matches the workflow in their direction).
- Live-server browser verification deferred to the user's own `make reset-and-seed backend` per their stated workflow; the SPA infrastructure is identical to PR #81's admin frontend (already Playwright-verified) and the page builds cleanly.

Refs: #78, #77, #79
2026-05-04 18:15:06 +08:00

2.9 KiB

title, description, icon
title description icon
Production Deployment SSL, domain setup, external S3, security hardening i-carbon-cloud-upload

Production Deployment

Deploy KohakuHub for production use.

Component Versions

  • LakeFS ≥ v0.54.0 (released 2021-11-08). The bundled docker compose uses treeverse/lakefs:latest and is always compatible. If your production stack pins an older LakeFS image, upgrade before rolling out KohakuHub — the file-list expand=true endpoint depends on path-filtered logCommits (objects= / prefixes= / limit=) introduced in v0.54.0; pre-v0.54 servers silently drop those parameters and would surface wrong lastCommit metadata.

SSL & Domain

nginx config:

server {
    listen 443 ssl http2;
    server_name hub.yourdomain.com;
    ssl_certificate /path/to/cert.pem;
    ssl_certificate_key /path/to/key.pem;

    # Forward every KohakuHub path to the backend. The hf_hub-compatible
    # public URLs (no /api prefix) MUST be reachable at the root since
    # huggingface_hub clients hit them directly:
    #   /<repo_type>s/<ns>/<name>/resolve/<rev>/<path>   (HEAD/GET file)
    #   /<repo_type>s/<ns>/<name>/tree/<rev>/...         (list files)
    #   /<ns>/<name>/resolve/...                         (model default)
    # The chain tester in the admin SPA exercises these same routes
    # (see src/kohaku-hub-admin/vite.config.js for the dev-mode mirror)
    # so misconfigured nginx → CHAIN_EXHAUSTED on every probe.
    location / {
        proxy_pass http://kohakuhub-backend:48888;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    # Admin SPA static bundle — path-prefixed under /admin so it
    # coexists with the hf_hub-compat root paths.
    location /admin/ {
        alias /var/www/kohakuhub-admin/;
        try_files $uri $uri/ /admin/index.html;
    }
}

Update base URL:

KOHAKU_HUB_BASE_URL: https://hub.yourdomain.com

External S3

Cloudflare R2:

KOHAKU_HUB_S3_ENDPOINT: https://account.r2.cloudflarestorage.com
KOHAKU_HUB_S3_PUBLIC_ENDPOINT: https://pub.r2.dev
KOHAKU_HUB_S3_REGION: auto
KOHAKU_HUB_S3_SIGNATURE_VERSION: s3v4

AWS S3:

KOHAKU_HUB_S3_ENDPOINT: https://s3.amazonaws.com
KOHAKU_HUB_S3_REGION: us-east-1
KOHAKU_HUB_S3_FORCE_PATH_STYLE: false

Security

Change all secrets:

python scripts/generate_secret.py
# Update SESSION_SECRET, ADMIN_SECRET_TOKEN

Change passwords:

  • PostgreSQL
  • MinIO
  • LakeFS

Scaling

Multi-worker:

command: uvicorn kohakuhub.main:app --workers 8

Database uses db.atomic() for safety.

Backups

docker exec postgres pg_dump -U hub kohakuhub | gzip > backup.sql.gz

See Security for hardening guide.