mirror of
https://github.com/KohakuBlueleaf/KohakuHub.git
synced 2026-07-20 21:14:09 -05:00
narugo1992andClaude Opus 4.7
aff9fd47ef
Introduces the prerequisite cache layer tracked in #73. No business code yet consumes the helpers — this is plumbing only, gated by KOHAKU_HUB_CACHE_ENABLED (default: false). Subsequent issues will adopt specific cache patterns on top of this foundation. Why now: every hot read endpoint currently makes 1–3 LakeFS REST calls plus several Postgres queries per request, with the only existing cache being a per-process cachetools.TTLCache in fallback/cache.py — useless across the default 4-worker uvicorn deployment. Design highlights (full design in docs/development/cache.md): - Pure cache, no source-of-truth state. Silent-degradation contract: every cache call is wrapped in try/except and falls back to L3 when Valkey is unreachable. CI runs a dedicated cache-disabled job to regression-guard this. - L1 (per-worker cachetools) restricted to immutable / content-addressed data only — multi-worker uvicorn has no portable cross-worker invalidation channel, and constraining L1 to "key contains its own version" sidesteps that entirely. - L2 (Valkey) holds everything. Helpers ship with TTL jitter (±15% default), two-level singleflight (asyncio.Lock + Valkey SET NX EX), negative cache, generation counters, and per-namespace metrics. - Persistence: RDB on, AOF off, persistent volume. Mode-A (lakefs:commit, lakefs:stat, lakefs:list — commit_id-keyed) survives restart safely. Mode-B (mutable) namespaces are flushed on every Valkey restart by a run_id-based bootstrap coordinator that serializes the flush across workers. Includes: - src/kohakuhub/cache.py — the helper module (319 stmts, 83% coverage via the new test module). - src/kohakuhub/api/admin/routers/cache.py — admin endpoints exposing hit/miss/error counters, Valkey memory state, and bootstrap-flush metadata. - test/kohakuhub/test_cache.py — 34 tests against a real Valkey, covering: round-trips, TTL jitter spread, SCAN-based prefix delete over >SCAN_BATCH_SIZE keys, two-level singleflight (100 concurrent calls fold to 1 fetch), bootstrap flush selectivity (Mode-A survives, Mode-B is wiped, exactly), two-worker bootstrap coordination, silent degradation when Valkey is disabled OR unreachable, generation counters, negative cache, the Mode-B prefix list shape contract. - docker-compose.example.yml — adds the valkey service with RDB + LFU + bind-mounted hub-meta/valkey-data, mirroring the persistence pattern of the other stateful services. - scripts/dev/up_infra.sh / down_infra.sh / reset_local_data.sh — Valkey container plumbing for local dev (host port 26379). - .github/workflows/fullstack-tests.yml — adds valkey to the existing matrix services and adds a separate single-Python job (backend-tests-cache-disabled) running with KOHAKU_HUB_CACHE_ENABLED=false as the silent-degradation contract regression guard. - docs/development/cache.md — the design doc referenced by the cache module's docstrings. Refs: #73 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
78 lines
2.5 KiB
Bash
Executable File
78 lines
2.5 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
set -euo pipefail
|
|
|
|
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
|
ENV_FILE="${ROOT_DIR}/.env.dev"
|
|
LAKEFS_CREDENTIALS_FILE="${ROOT_DIR}/hub-meta/dev/lakefs/credentials.env"
|
|
|
|
warn_red() {
|
|
printf '\033[1;31m%s\033[0m\n' "$1"
|
|
}
|
|
|
|
warn_red "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
|
|
warn_red "!! DANGER: THIS IRREVERSIBLY CLEARS LOCAL KOHAKUHUB DEV DATA !!"
|
|
warn_red "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
|
|
warn_red "This clears through the local reset helper:"
|
|
warn_red " - application data in PostgreSQL"
|
|
warn_red " - all objects in the local S3 bucket"
|
|
warn_red " - all LakeFS repositories in the local dev instance"
|
|
warn_red " - the local demo seed manifest"
|
|
warn_red ""
|
|
warn_red "Consequence:"
|
|
warn_red " - all local accounts, repos, orgs, commits, likes, and download stats are lost"
|
|
warn_red " - the Docker bind-mount directories are kept in place"
|
|
warn_red " - local infra stays running so you can re-seed immediately"
|
|
warn_red ""
|
|
warn_red ".env.dev and persisted LakeFS credentials are NOT removed."
|
|
echo
|
|
|
|
read -r -p "Continue with local reset? [y/N]: " confirmation
|
|
if [[ ! "${confirmation}" =~ ^[Yy]$ ]]; then
|
|
echo "Aborted. Local data was not changed."
|
|
exit 0
|
|
fi
|
|
|
|
if [[ ! -f "${ENV_FILE}" ]]; then
|
|
echo "Missing ${ENV_FILE}"
|
|
echo "Create it first: cp .env.dev.example .env.dev"
|
|
exit 1
|
|
fi
|
|
|
|
if [[ -n "${VIRTUAL_ENV:-}" ]]; then
|
|
PYTHON_BIN="${PYTHON_BIN:-python}"
|
|
elif [[ -x "${ROOT_DIR}/venv/bin/python" ]]; then
|
|
PYTHON_BIN="${PYTHON_BIN:-${ROOT_DIR}/venv/bin/python}"
|
|
else
|
|
PYTHON_BIN="${PYTHON_BIN:-python3}"
|
|
fi
|
|
|
|
set -a
|
|
# shellcheck disable=SC1090
|
|
source "${ENV_FILE}"
|
|
set +a
|
|
|
|
"${ROOT_DIR}/scripts/dev/up_infra.sh"
|
|
"${ROOT_DIR}/scripts/dev/run_backend.sh" --prepare-only --skip-seed
|
|
|
|
if [[ ! -f "${LAKEFS_CREDENTIALS_FILE}" ]]; then
|
|
echo "Missing ${LAKEFS_CREDENTIALS_FILE}"
|
|
echo "LakeFS bootstrap did not produce reusable credentials."
|
|
exit 1
|
|
fi
|
|
|
|
set -a
|
|
# shellcheck disable=SC1090
|
|
source "${LAKEFS_CREDENTIALS_FILE}"
|
|
set +a
|
|
|
|
"${PYTHON_BIN}" "${ROOT_DIR}/scripts/dev/reset_local_data_direct.py"
|
|
|
|
# Wipe the L2 cache after the reset. The cache references repos / commits
|
|
# that no longer exist; leaving stale Mode-A entries (commit_id-keyed) in
|
|
# place is correctness-safe but wastes memory. FLUSHALL is targeted at the
|
|
# dev container only, never at production.
|
|
if docker ps --format '{{.Names}}' | grep -Fxq "kohakuhub-dev-valkey"; then
|
|
docker exec kohakuhub-dev-valkey valkey-cli FLUSHALL >/dev/null 2>&1 || true
|
|
echo "Flushed kohakuhub-dev-valkey contents"
|
|
fi
|