KohakuHub/docs/development/cache.md

Cache layer

KohakuHub uses a Valkey-based L2 cache (Redis-compatible) to absorb traffic that would otherwise hit Postgres or LakeFS REST repeatedly. The design has one non-negotiable rule:

The cache must never be on the correctness critical path.

If the cache is unavailable, the API stays correct — every read falls through to the source of truth. The cache is allowed to be late, but never wrong, and its absence is allowed to be slow, but never an outage.

This document covers what is in the cache, what is not, and the invariants that callers must respect when integrating with it. Implementation lives in src/kohakuhub/cache.py; tests in test/kohakuhub/test_cache.py.

Architecture

L1: per-worker cachetools.TTLCache (~µs)
    Immutable / content-addressed data ONLY.
    Mutable data is forbidden in L1 — no cross-worker invalidation channel.

L2: Valkey (~ms)
    All cacheable data (mutable + immutable).
    Cross-worker shared, cross-worker invalidation automatic.

L3: Postgres / LakeFS REST / S3 — source of truth.

L1 is intentionally narrow. The default 4-worker uvicorn deployment cannot synchronously invalidate per-process state across workers; rather than build a Pub/Sub fan-out (high complexity, low marginal benefit on top of a 1ms L2 hit), we restrict L1 to data that is provably safe to keep stale across all workers — namely content-addressed entries whose key contains its own version identifier.

Consistency model

Two patterns, applied per data class:

Mode A — Immutable / content-addressed

Key contains the version. Examples:

• lakefs:commit:{repo}:{commit_id} — LakeFS commits are SHA-addressed.
• lakefs:stat:{repo}:{commit_id}:{path} — same, scoped by commit.
• lakefs:list:{repo}:{commit_id}:{prefix} — same, scoped by commit.

These are correct by construction: the key cannot resolve to a different value over time. TTL is a memory-pressure tool, not a correctness tool. No active invalidation is needed; LFU eviction reclaims them when memory tightens.

Mode B — Mutable, real-time consistency required

Two sub-patterns, picked by fan-out:

B1. Generation counter — preferred for entities with many derived caches (Repository, User, Organization).

• The entity table has a cache_gen BIGINT NOT NULL DEFAULT 1 column.
• Every mutating db_operations.* path bumps cache_gen in the same transaction as the business write.
• Reads embed the current gen in their cache key: repo:info:{id}:g{gen}. Old gens are unreachable; LFU reclaims them.
• Avoids the bug class of "forgot to enumerate one of the derived keys during invalidation."

B2. Write-through DEL — used for narrow-fanout entities (Session, Token, Branch HEAD).

• The mutator path explicitly DELs the affected key(s) after DB commit.
• Order matters: DB commit → cache DEL → response. Inverting that ordering races concurrent reads back into stale state before the new value is durable.

Persistence and the boot-time flush

The cache container is started with RDB snapshotting enabled (--save 300 100, --appendonly no) and a persistent volume. Two reasons:

1. The bulk of cache value lives in Mode-A immutable keys. Discarding them on every Valkey restart turns routine maintenance into a synchronized 1k-fanout LakeFS REST burst — exactly the spike the cache exists to prevent.
2. Mode-B keys CAN go stale across a restart (a write whose DEL lands during Valkey downtime is silently lost), so RDB-restored Mode-B values are suspect.

Resolution: persist RDB, but flush all Mode-B namespaces on every Valkey restart. The list of Mode-B prefixes is the MODE_B_PREFIXES constant in cache.py:

Survives across restart (Mode A)	Flushed on every Valkey restart (Mode B)
lakefs:commit:	lakefs:branch:
lakefs:stat:	repo:gen:, repo:info:, org:gen:, org:info:, user:gen:, user:info:
lakefs:list:	uo:, sess:, tok:, list_gen:, list:, negative:

Restart detection uses Valkey's INFO server.run_id (changes on every Valkey process start). The API stores the last-seen run_id in Valkey itself (cache:bootstrap_run_id); on connect, the worker that wins SET cache:bootstrap_lock 1 NX EX 30 performs the flush and updates the marker. Other workers wait until the marker matches the current run_id before serving Mode-B reads.

This decouples cache flushes from API restarts: redeploying the API against a long-running Valkey is a no-op for the flush coordinator; a Valkey upgrade or pod-eviction triggers exactly one flush across all API workers.

Stampede / eviction defenses

All five must remain in place; each closes a different failure mode.

1. TTL jitter ±15% — applied inside cache_set_json, no opt-out. Without this, a synchronized burst of writes all expire at the same instant.
2. Two-level singleflight — asyncio.Lock per cache key inside the worker, plus Valkey SET sf:lock:{key} 1 NX EX 5 across workers. Both required: per-worker alone leaves N workers racing; cross-worker alone leaves intra-worker contention.
3. Negative cache — cache_set_negative() writes a sentinel under a short TTL (15s default). Prevents repeated lookups for nonexistent entities from hammering L3. Critical: when the entity is later created, the create path MUST cache_invalidate(...) the negative key — otherwise the negative result hides the new entity until TTL expiry.
4. Refresh-ahead — allowed only on Mode-A immutable entries with measured high QPS. Forbidden on Mode-B; serving a stale value while refreshing in the background defeats write-through consistency.
5. maxmemory + allkeys-lfu, no key allowed without TTL (even immutable entries: cap at 24h). Memory is bounded; eviction is automatic.

What is NOT cached

These are deliberately uncached, even though they would benefit from the performance:

Data	Why not
File row by (repo, path)	Preupload SHA256 dedup is strong-consistency-sensitive; a wrong-side cache miss causes silent skip-of-upload.
Quota / used_bytes	Mutated on every write; caching adds bug surface for no win.
Presigned S3 URLs	Already short-TTL by design, per-request, per-user — no shared key.
Likes / downloads counters	Already updated asynchronously; an extra cache layer adds inconsistency without latency win.

Adding a new cached read

1. Decide the mode (A vs B1 vs B2). If you can't decide, the data is probably not safe to cache without more thought.
2. Pick a key shape that reflects the mode:
   • Mode A: include the version in the key.
   • Mode B1: include :g{gen} and add a generation bump to every mutator.
   • Mode B2: pick a deterministic key and add cache_invalidate(key) to every mutator post-DB-commit.
3. Use cache_get_or_fetch(key, fetch_fn, ttl=...) from cache.py. Do not call redis.asyncio directly — singleflight, jitter, silent degradation, and metrics all live in the helper.
4. Add a test that writes the underlying entity and immediately reads through the cached path; assert the new value is returned. This is the regression guard for the real-time consistency contract.

Operating

• Admin endpoint GET /admin/api/cache/stats returns hit/miss/error counters per namespace plus Valkey memory state and the last-flush metadata.
• POST /admin/api/cache/metrics/reset zeros the counters without touching cache contents.
• KOHAKU_HUB_CACHE_ENABLED=false disables the cache layer at startup. All cache calls then degrade to "miss" silently. CI runs a dedicated job with this flag set, so the silent-degradation contract is regression-tested on every push.

Related issues

• #73 — design issue and TODO list for follow-up cache integrations beyond infrastructure.