mirror of
https://github.com/KohakuBlueleaf/KohakuHub.git
synced 2026-07-23 14:31:41 -05:00
narugo1992andClaude Opus 4.7
d0445282fb
Implements issue #27 v4: file-level HF-compatible metadata preview computed entirely in the browser via HTTP Range reads against the existing /resolve/ 302 → presigned S3/MinIO URL. Zero new backend preview code, zero LRU, zero precomputation, zero new DB state. Backend (minimal CORS plumbing only): - main.py CORSMiddleware: add `expose_headers` so browsers can read Content-Range / X-Linked-* / X-Repo-Commit / ETag / Location off the final 206 response that follows the /resolve/ 302. - docker-compose.example.yml + scripts/dev/up_infra.sh: wire `MINIO_API_CORS_ALLOW_ORIGIN` so the SPA can cross-origin Range-read presigned targets. Configurable via `DEV_MINIO_CORS_ALLOW_ORIGIN`. - docs/development/local-dev.md: MinIO CORS section explaining the hard prerequisite + smoke-test probe + how to recreate the container. Frontend: - utils/safetensors.js (~190 LOC): pure-JS parser mirroring huggingface_hub.parse_safetensors_file_metadata byte-for-byte (speculative 100 KB first read, two-read fallback for fat headers, SAFETENSORS_MAX_HEADER_LENGTH guard). Exposes parseSafetensorsMetadata + summarizeSafetensors. - utils/parquet.js: thin wrapper over hyparquet's asyncBufferFromUrl + parquetMetadataAsync with mode:"cors" + credentials:"omit" so cookies never leak onto presigned URLs. Normalizes BigInt row counts. - components/repo/preview/FilePreviewDialog.vue: ElDialog with per-phase spinner text (range-head → parsing → done for safetensors, head → footer → parsing → done for parquet), dtype/row-group tables, and an explicit "CORS likely misconfigured" placeholder on failure. - RepoViewer.vue: HF-style chart-line-data icon next to .safetensors and .parquet rows; click opens the modal with the resolved /resolve/ URL for the current branch. Tests + fixtures: - test_files.py::test_resolve_get_302_exposes_cors_headers_for_browser_preview pins the `Access-Control-Expose-Headers` list against regressions. - test/kohaku-hub-ui/utils/test_safetensors.test.js: 6 cases covering the real-HF-format fixture, dtype summary, progress phases, fat-header fallback, oversized-header guard, and non-206 error paths. - test/kohaku-hub-ui/utils/test_parquet.test.js: footer parse + progress phase assertions. - test/kohaku-hub-ui/fixtures/previews/{tiny.safetensors,tiny.parquet}: byte-identical-to-HF fixtures produced by the real safetensors / pyarrow libs via scripts/dev/generate_preview_test_fixtures.py (committed so tests stay offline per AGENTS.md §5.2). Seed: - seed_demo_data.py: add two RemoteAsset entries for real HF-hosted small fixtures pinned by sha256, and wire them into visible paths (open-media-lab/vision-language-assistant-3b/fixtures/hf-tiny-random-bert.safetensors, open-media-lab/multimodal-benchmark-suite/fixtures/hf-no-robots-test.parquet) so the preview can be exercised against files that actually came off huggingface.co rather than purely local pyarrow/safetensors output. SEED_VERSION bumped to local-dev-demo-v4. Verified end-to-end against the dev stack: safetensors parser output on the seeded fixtures matches huggingface_hub.parse_safetensors_file_metadata byte-for-byte on the same file (100 tensors, 126,851 params, I64=512 / F32=126,339, metadata `{format: pt, ...}`). Browser preview modal renders both file kinds correctly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
156 lines
6.2 KiB
YAML
156 lines
6.2 KiB
YAML
# docker-compose.yml - Example configuration for KohakuHub
|
|
# Copy this to docker-compose.yml and customize for your deployment
|
|
|
|
services:
|
|
hub-ui:
|
|
image: nginx:alpine
|
|
container_name: hub-ui
|
|
restart: always
|
|
ports:
|
|
- "28080:80" # Public web interface
|
|
volumes:
|
|
- ./src/kohaku-hub-ui/dist:/usr/share/nginx/html
|
|
- ./src/kohaku-hub-admin/dist:/usr/share/nginx/html-admin
|
|
- ./docker/nginx/default.conf:/etc/nginx/conf.d/default.conf
|
|
depends_on:
|
|
- hub-api
|
|
|
|
hub-api:
|
|
build: .
|
|
container_name: hub-api
|
|
restart: always
|
|
ports:
|
|
- "48888:48888" # Internal API port (optional, for debugging)
|
|
depends_on:
|
|
- postgres
|
|
- lakefs
|
|
- minio
|
|
environment:
|
|
## ===== CRITICAL: Endpoint Configuration (MUST CHANGE) =====
|
|
## These determine how users access your KohakuHub instance
|
|
- KOHAKU_HUB_BASE_URL=http://127.0.0.1:28080 # Change to your public URL (e.g., https://hub.example.com)
|
|
- KOHAKU_HUB_S3_PUBLIC_ENDPOINT=http://127.0.0.1:29001 # Change to your S3 public URL
|
|
|
|
## ===== CRITICAL: Security Configuration (MUST CHANGE) =====
|
|
- KOHAKU_HUB_SESSION_SECRET=change-this-to-random-string-in-production
|
|
- KOHAKU_HUB_ADMIN_SECRET_TOKEN=change-this-to-random-admin-token-in-production
|
|
- KOHAKU_HUB_DATABASE_KEY=change-this-to-random-encryption-key-in-production # For external tokens (generate with: openssl rand -hex 32)
|
|
|
|
## ===== Performance Configuration =====
|
|
- KOHAKU_HUB_WORKERS=4 # Number of worker processes (1-8, recommend: CPU cores)
|
|
|
|
## ===== Database Configuration =====
|
|
- KOHAKU_HUB_DB_BACKEND=postgres
|
|
- KOHAKU_HUB_DATABASE_URL=postgresql://<user>:<password>@postgres:5432/kohakuhub
|
|
- KOHAKU_HUB_AUTO_MIGRATE=true # Auto-confirm database migrations (required for Docker)
|
|
|
|
## ===== S3 Storage Configuration =====
|
|
- KOHAKU_HUB_S3_ENDPOINT=http://minio:9000
|
|
- KOHAKU_HUB_S3_ACCESS_KEY=minioadmin
|
|
- KOHAKU_HUB_S3_SECRET_KEY=minioadmin
|
|
- KOHAKU_HUB_S3_BUCKET=hub-storage
|
|
- KOHAKU_HUB_S3_REGION=us-east-1 # S3 region (us-east-1 for MinIO, auto for R2, or specific AWS region)
|
|
# - KOHAKU_HUB_S3_SIGNATURE_VERSION=s3v4 # Uncomment for R2/AWS S3 (leave commented for MinIO default)
|
|
|
|
## ===== LakeFS Configuration =====
|
|
- KOHAKU_HUB_LAKEFS_ENDPOINT=http://lakefs:28000
|
|
- KOHAKU_HUB_LAKEFS_REPO_NAMESPACE=hf
|
|
# LakeFS credentials auto-generated on first start
|
|
|
|
## ===== Application Configuration =====
|
|
- KOHAKU_HUB_SITE_NAME=KohakuHub
|
|
- KOHAKU_HUB_LFS_THRESHOLD_BYTES=1000000
|
|
- KOHAKU_HUB_LFS_MULTIPART_THRESHOLD_BYTES=100_000_000 # 100MB - use multipart for files larger than this
|
|
- KOHAKU_HUB_LFS_MULTIPART_CHUNK_SIZE_BYTES=50_000_000 # 50MB - size of each part (min 5MB except last)
|
|
- KOHAKU_HUB_LFS_KEEP_VERSIONS=5
|
|
- KOHAKU_HUB_LFS_AUTO_GC=true
|
|
|
|
## ===== Auth & SMTP Configuration =====
|
|
- KOHAKU_HUB_REQUIRE_EMAIL_VERIFICATION=false
|
|
- KOHAKU_HUB_INVITATION_ONLY=false # Set to true to require invitation for registration
|
|
- KOHAKU_HUB_SESSION_EXPIRE_HOURS=168
|
|
- KOHAKU_HUB_TOKEN_EXPIRE_DAYS=365
|
|
- KOHAKU_HUB_ADMIN_ENABLED=true
|
|
# SMTP (Optional - for email verification)
|
|
- KOHAKU_HUB_SMTP_ENABLED=false
|
|
- KOHAKU_HUB_SMTP_HOST=smtp.gmail.com
|
|
- KOHAKU_HUB_SMTP_PORT=587
|
|
- KOHAKU_HUB_SMTP_USERNAME=
|
|
- KOHAKU_HUB_SMTP_PASSWORD=
|
|
- KOHAKU_HUB_SMTP_FROM=noreply@kohakuhub.local
|
|
- KOHAKU_HUB_SMTP_TLS=true
|
|
|
|
## ===== Storage Quota Configuration (Optional) =====
|
|
- KOHAKU_HUB_DEFAULT_USER_PRIVATE_QUOTA_BYTES=10_000_000
|
|
- KOHAKU_HUB_DEFAULT_USER_PUBLIC_QUOTA_BYTES=100_000_000
|
|
- KOHAKU_HUB_DEFAULT_ORG_PRIVATE_QUOTA_BYTES=10_000_000
|
|
- KOHAKU_HUB_DEFAULT_ORG_PUBLIC_QUOTA_BYTES=100_000_000
|
|
|
|
## ===== Fallback Configuration (Optional) =====
|
|
# - KOHAKU_HUB_FALLBACK_ENABLED=true
|
|
# - KOHAKU_HUB_FALLBACK_REQUIRE_AUTH=false # Set true to require authentication for fallback access
|
|
# - KOHAKU_HUB_FALLBACK_SOURCES=[{"url":"https://huggingface.co","name":"HuggingFace","source_type":"huggingface","priority":1}]
|
|
volumes:
|
|
- ./hub-meta/hub-api:/hub-api-creds
|
|
|
|
minio:
|
|
image: quay.io/minio/minio:latest
|
|
container_name: minio
|
|
command: server /data --console-address ":29000"
|
|
environment:
|
|
- MINIO_ROOT_USER=minioadmin
|
|
- MINIO_ROOT_PASSWORD=minioadmin
|
|
# Required so the SPA can do cross-origin Range reads on presigned
|
|
# /resolve/ targets for the pure-client safetensors/parquet preview
|
|
# (deepghs/KohakuHub#27). Tighten to a specific origin list in prod.
|
|
- MINIO_API_CORS_ALLOW_ORIGIN=*
|
|
ports:
|
|
- "29001:9000" # S3 API
|
|
- "29000:29000" # Web Console
|
|
volumes:
|
|
- ./hub-storage/minio-data:/data
|
|
- ./hub-meta/minio-data:/root/.minio
|
|
|
|
lakefs:
|
|
build:
|
|
context: ./docker/lakefs
|
|
container_name: lakefs
|
|
environment:
|
|
- LAKEFS_DATABASE_TYPE=local
|
|
- LAKEFS_DATABASE_LOCAL_PATH=/var/lakefs/data/metadata.db
|
|
- LAKEFS_BLOCKSTORE_TYPE=s3
|
|
- LAKEFS_BLOCKSTORE_S3_ENDPOINT=http://minio:9000
|
|
- LAKEFS_BLOCKSTORE_S3_BUCKET=hub-storage
|
|
- LAKEFS_BLOCKSTORE_S3_FORCE_PATH_STYLE=true
|
|
- LAKEFS_BLOCKSTORE_S3_CREDENTIALS_ACCESS_KEY_ID=minioadmin
|
|
- LAKEFS_BLOCKSTORE_S3_CREDENTIALS_SECRET_ACCESS_KEY=minioadmin
|
|
- LAKEFS_BLOCKSTORE_S3_REGION=us-east-1 # S3 region (us-east-1 for MinIO, auto for R2, or specific AWS region)
|
|
- LAKEFS_AUTH_ENCRYPT_SECRET_KEY=change-me-in-production
|
|
- LAKEFS_LOGGING_FORMAT=text
|
|
- LAKEFS_LISTEN_ADDRESS=0.0.0.0:28000
|
|
ports:
|
|
- "28000:28000" # LakeFS admin UI (optional)
|
|
user: "${UID}:${GID}"
|
|
depends_on:
|
|
- minio
|
|
volumes:
|
|
- ./hub-meta/lakefs-data:/var/lakefs/data
|
|
- ./hub-meta/lakefs-cache:/lakefs/data/cache
|
|
|
|
postgres:
|
|
image: postgres:15
|
|
container_name: postgres
|
|
restart: always
|
|
environment:
|
|
- POSTGRES_USER=<user>
|
|
- POSTGRES_PASSWORD=<password>
|
|
- POSTGRES_DB=kohakuhub
|
|
ports:
|
|
- "25432:5432" # Optional: for external access
|
|
volumes:
|
|
- ./hub-meta/postgres-data:/var/lib/postgresql/data
|
|
|
|
networks:
|
|
default:
|
|
name: hub-net
|