mirror of https://github.com/KohakuBlueleaf/KohakuHub.git synced 2026-07-16 02:33:10 -05:00

Files

Kohaku-Blueleaf 88a2e3c328 update document for APIs

2025-10-24 18:45:55 +08:00

18 KiB

Raw Permalink Blame History

title, description, icon

title	description	icon
External Tokens API	Manage external fallback tokens for accessing remote sources	key

External Tokens API

Manage user-specific external tokens for accessing fallback sources like HuggingFace Hub.

Overview

KohakuHub supports external fallback sources for repositories not available locally. Users can provide their own tokens to access private repositories on external sources.

Token Storage:

Database: Encrypted tokens stored persistently (AES-256)
Authorization header: Temporary per-request tokens
Priority: Header tokens override database tokens

Security:

All tokens encrypted with DATABASE_KEY (Fernet/AES-256)
Tokens never logged in plaintext
Masked when displayed (e.g., hf_a***)

Endpoints

Get Available Fallback Sources

Get list of configured fallback sources (public endpoint).

Endpoint: GET /api/fallback-sources/available

Authentication: None (public)

Response:

[
  {
    "url": "https://huggingface.co",
    "name": "HuggingFace Hub",
    "source_type": "huggingface",
    "priority": 10
  },
  {
    "url": "https://custom-hub.example.com",
    "name": "Custom Hub",
    "source_type": "huggingface",
    "priority": 20
  }
]

Field Descriptions:

Field	Type	Description
`url`	string	Base URL of the fallback source
`name`	string	Display name
`source_type`	string	Source type (currently: `huggingface`)
`priority`	integer	Priority (lower = higher priority)

Use Cases:

Display available sources to users
Let users know which sources they can configure tokens for
Show source priorities

Example:

import requests

response = requests.get("http://localhost:28080/api/fallback-sources/available")
sources = response.json()

print("Available fallback sources:")
for source in sources:
    print(f"  {source['name']}: {source['url']}")

List User External Tokens

Get list of user's configured external tokens (tokens are masked).

Endpoint: GET /api/users/{username}/external-tokens

Parameters:

Parameter	Type	Location	Required	Description
`username`	string	path	Yes	Username (must match authenticated user)

Authentication: Required (can only view own tokens)

Response:

[
  {
    "url": "https://huggingface.co",
    "token_preview": "hf_a***",
    "created_at": "2025-01-15T10:30:00Z",
    "updated_at": "2025-01-15T10:30:00Z"
  },
  {
    "url": "https://custom-hub.example.com",
    "token_preview": "tok_x***",
    "created_at": "2025-01-10T08:00:00Z",
    "updated_at": "2025-01-12T14:30:00Z"
  }
]

Field Descriptions:

Field	Type	Description
`url`	string	Fallback source URL
`token_preview`	string	Masked token (first 4 chars + `***`)
`created_at`	string	ISO 8601 timestamp when token was added
`updated_at`	string	ISO 8601 timestamp when token was last updated

Security:

Full tokens are never returned in API responses
Only first 4 characters shown (e.g., hf_a***)
Tokens are encrypted in database

Example:

response = requests.get(
    "http://localhost:28080/api/users/alice/external-tokens",
    headers={"Authorization": "Bearer YOUR_TOKEN"}
)
tokens = response.json()

for token in tokens:
    print(f"{token['url']}: {token['token_preview']}")

Add or Update External Token

Add a new external token or update an existing one.

Endpoint: POST /api/users/{username}/external-tokens

Parameters:

Parameter	Type	Location	Required	Description
`username`	string	path	Yes	Username (must match authenticated user)

Request Body:

{
  "url": "https://huggingface.co",
  "token": "hf_abc123xyz789..."
}

Field	Type	Required	Description
`url`	string	Yes	Base URL of fallback source (must start with `http`)
`token`	string	Yes	Plain text token

Authentication: Required (can only manage own tokens)

Response:

{
  "success": true,
  "message": "External token saved"
}

Behavior:

If token exists for this URL, it will be updated
If token doesn't exist, it will be created
Token is encrypted before storage
URL must be a valid HTTP(S) URL

Example:

# Add HuggingFace token
response = requests.post(
    "http://localhost:28080/api/users/alice/external-tokens",
    headers={"Authorization": "Bearer YOUR_TOKEN"},
    json={
        "url": "https://huggingface.co",
        "token": "hf_abc123xyz789..."
    }
)

# Add custom hub token
response = requests.post(
    "http://localhost:28080/api/users/alice/external-tokens",
    headers={"Authorization": "Bearer YOUR_TOKEN"},
    json={
        "url": "https://custom-hub.example.com",
        "token": "custom_token_here"
    }
)

Delete External Token

Delete an external token for a specific URL.

Endpoint: DELETE /api/users/{username}/external-tokens/{url:path}

Parameters:

Parameter	Type	Location	Required	Description
`username`	string	path	Yes	Username (must match authenticated user)
`url`	string	path	Yes	URL of the fallback source (URL-encoded)

Authentication: Required (can only manage own tokens)

Response:

{
  "success": true,
  "message": "External token deleted"
}

Error Responses:

Status	Description
403	Not authorized to manage these tokens
404	Token not found for this URL

Example:

import urllib.parse

# URL-encode the source URL
url = urllib.parse.quote("https://huggingface.co", safe='')

response = requests.delete(
    f"http://localhost:28080/api/users/alice/external-tokens/{url}",
    headers={"Authorization": "Bearer YOUR_TOKEN"}
)

Bulk Update External Tokens

Replace all external tokens with a new set (useful for sync operations).

Endpoint: PUT /api/users/{username}/external-tokens/bulk

Parameters:

Parameter	Type	Location	Required	Description
`username`	string	path	Yes	Username (must match authenticated user)

Request Body:

{
  "tokens": [
    {
      "url": "https://huggingface.co",
      "token": "hf_abc123..."
    },
    {
      "url": "https://custom-hub.example.com",
      "token": "custom_token..."
    }
  ]
}

Field	Type	Required	Description
`tokens`	array	Yes	List of token objects
`tokens[].url`	string	Yes	Source URL
`tokens[].token`	string	Yes	Plain text token

Authentication: Required (can only manage own tokens)

Response:

{
  "success": true,
  "message": "Updated 2 external tokens"
}

Behavior:

Deletes tokens not in the new list
Adds new tokens
Updates existing tokens
Atomic operation (all or nothing)

Use Cases:

Sync tokens from frontend localStorage
Import/export token configurations
Reset all tokens

Example:

# Replace all tokens
response = requests.put(
    "http://localhost:28080/api/users/alice/external-tokens/bulk",
    headers={"Authorization": "Bearer YOUR_TOKEN"},
    json={
        "tokens": [
            {
                "url": "https://huggingface.co",
                "token": "hf_new_token_123"
            },
            {
                "url": "https://custom.example.com",
                "token": "custom_token_456"
            }
        ]
    }
)

Authorization Header Format

External tokens can also be passed per-request via the Authorization header.

Format: Bearer <auth_token>|<url1>,<token1>|<url2>,<token2>...

Examples:

# Regular authentication only
Authorization: Bearer hf_abc123

# With one external token
Authorization: Bearer hf_abc123|https://huggingface.co,hf_ext123

# With multiple external tokens
Authorization: Bearer hf_abc123|https://huggingface.co,hf_ext123|https://custom.com,tok_xyz

# Session-based auth with external token (no main token)
Authorization: Bearer |https://huggingface.co,hf_ext123

# Empty token allowed (for public sources with auth requirement)
Authorization: Bearer hf_abc123|https://example.com,

Priority:

Header tokens (highest) - Temporary, per-request
Database tokens - Persistent user preferences
Admin tokens (lowest) - Server-wide defaults

Use Cases:

Temporary token override without saving
Anonymous users passing tokens
CI/CD with different tokens per job

Usage Examples

Comprehensive Token Management

import requests
import urllib.parse

BASE_URL = "http://localhost:28080"
TOKEN = "YOUR_TOKEN"

class ExternalTokenManager:
    def __init__(self, base_url: str, token: str, username: str):
        self.base_url = base_url
        self.username = username
        self.headers = {"Authorization": f"Bearer {token}"}

    def list_sources(self):
        """List available fallback sources."""
        response = requests.get(
            f"{self.base_url}/api/fallback-sources/available"
        )
        response.raise_for_status()
        return response.json()

    def list_tokens(self):
        """List user's external tokens."""
        response = requests.get(
            f"{self.base_url}/api/users/{self.username}/external-tokens",
            headers=self.headers
        )
        response.raise_for_status()
        return response.json()

    def add_token(self, url: str, token: str):
        """Add or update external token."""
        response = requests.post(
            f"{self.base_url}/api/users/{self.username}/external-tokens",
            headers=self.headers,
            json={"url": url, "token": token}
        )
        response.raise_for_status()
        return response.json()

    def delete_token(self, url: str):
        """Delete external token."""
        encoded_url = urllib.parse.quote(url, safe='')
        response = requests.delete(
            f"{self.base_url}/api/users/{self.username}/external-tokens/{encoded_url}",
            headers=self.headers
        )
        response.raise_for_status()
        return response.json()

    def bulk_update(self, tokens: list[dict]):
        """Bulk update all tokens."""
        response = requests.put(
            f"{self.base_url}/api/users/{self.username}/external-tokens/bulk",
            headers=self.headers,
            json={"tokens": tokens}
        )
        response.raise_for_status()
        return response.json()

    def print_summary(self):
        """Print comprehensive token summary."""
        print(f"\n=== External Tokens for {self.username} ===\n")

        # Available sources
        sources = self.list_sources()
        print("Available sources:")
        for source in sources:
            print(f"  {source['name']}: {source['url']}")

        # User's tokens
        tokens = self.list_tokens()
        print(f"\nConfigured tokens ({len(tokens)}):")
        for token in tokens:
            print(f"  {token['url']}: {token['token_preview']}")
            print(f"    Updated: {token['updated_at']}")

# Usage
mgr = ExternalTokenManager(BASE_URL, TOKEN, "alice")

# Add tokens
mgr.add_token("https://huggingface.co", "hf_abc123xyz789...")
mgr.add_token("https://custom.example.com", "custom_token_here")

# List tokens
mgr.print_summary()

# Delete token
mgr.delete_token("https://custom.example.com")

# Bulk update (sync from another source)
mgr.bulk_update([
    {"url": "https://huggingface.co", "token": "hf_new_token"},
    {"url": "https://another.com", "token": "another_token"}
])

Using Header-Based Tokens

def make_request_with_external_token(repo_id: str, external_url: str, external_token: str):
    """Make request with temporary external token in header."""

    # Format: Bearer <main_token>|<external_url>,<external_token>
    auth_header = f"Bearer {TOKEN}|{external_url},{external_token}"

    response = requests.get(
        f"{BASE_URL}/models/{repo_id}/tree/main",
        headers={"Authorization": auth_header}
    )

    return response.json()

# Usage - temporary HF token for this request only
files = make_request_with_external_token(
    "openai/gpt-4",
    "https://huggingface.co",
    "hf_temporary_token_123"
)

Frontend Integration (JavaScript)

class ExternalTokenManager {
  constructor(baseURL, token, username) {
    this.baseURL = baseURL;
    this.username = username;
    this.headers = {
      'Authorization': `Bearer ${token}`,
      'Content-Type': 'application/json'
    };
  }

  async listSources() {
    const response = await fetch(
      `${this.baseURL}/api/fallback-sources/available`
    );
    return await response.json();
  }

  async listTokens() {
    const response = await fetch(
      `${this.baseURL}/api/users/${this.username}/external-tokens`,
      { headers: this.headers }
    );
    return await response.json();
  }

  async addToken(url, token) {
    const response = await fetch(
      `${this.baseURL}/api/users/${this.username}/external-tokens`,
      {
        method: 'POST',
        headers: this.headers,
        body: JSON.stringify({ url, token })
      }
    );
    return await response.json();
  }

  async deleteToken(url) {
    const encodedUrl = encodeURIComponent(url);
    const response = await fetch(
      `${this.baseURL}/api/users/${this.username}/external-tokens/${encodedUrl}`,
      {
        method: 'DELETE',
        headers: this.headers
      }
    );
    return await response.json();
  }

  async bulkUpdate(tokens) {
    const response = await fetch(
      `${this.baseURL}/api/users/${this.username}/external-tokens/bulk`,
      {
        method: 'PUT',
        headers: this.headers,
        body: JSON.stringify({ tokens })
      }
    );
    return await response.json();
  }

  // Format authorization header with external tokens
  formatAuthHeader(mainToken, externalTokens) {
    if (!externalTokens || externalTokens.length === 0) {
      return `Bearer ${mainToken}`;
    }

    const tokenPairs = externalTokens
      .map(t => `${t.url},${t.token}`)
      .join('|');

    return `Bearer ${mainToken}|${tokenPairs}`;
  }
}

// Usage
const tokenMgr = new ExternalTokenManager(
  'http://localhost:28080',
  'YOUR_TOKEN',
  'alice'
);

// Add token
await tokenMgr.addToken('https://huggingface.co', 'hf_abc123...');

// List tokens
const tokens = await tokenMgr.listTokens();
console.log('Configured tokens:', tokens);

// Make request with external tokens
const externalTokens = [
  { url: 'https://huggingface.co', token: 'hf_abc123' }
];
const authHeader = tokenMgr.formatAuthHeader('main_token', externalTokens);

const response = await fetch('/models/openai/gpt-4/tree/main', {
  headers: { 'Authorization': authHeader }
});

Sync Between LocalStorage and Database

// Frontend utility for dual storage
class TokenStorage {
  constructor(apiManager) {
    this.api = apiManager;
    this.localStorageKey = 'kohakuhub_external_tokens';
  }

  // Get from localStorage
  getLocal() {
    const stored = localStorage.getItem(this.localStorageKey);
    return stored ? JSON.parse(stored) : [];
  }

  // Save to localStorage
  saveLocal(tokens) {
    localStorage.setItem(this.localStorageKey, JSON.stringify(tokens));
  }

  // Add token to both localStorage and database
  async add(url, token) {
    // Update localStorage
    const tokens = this.getLocal();
    const existingIndex = tokens.findIndex(t => t.url === url);

    if (existingIndex >= 0) {
      tokens[existingIndex] = { url, token };
    } else {
      tokens.push({ url, token });
    }

    this.saveLocal(tokens);

    // Update database
    await this.api.addToken(url, token);
  }

  // Sync localStorage to database
  async syncToDatabase() {
    const localTokens = this.getLocal();
    await this.api.bulkUpdate(localTokens);
  }

  // Sync database to localStorage
  async syncFromDatabase() {
    const dbTokens = await this.api.listTokens();

    // Convert from preview format to full tokens (only if we have full tokens)
    // Note: We can't recover full tokens from preview, so this only works
    // if we're maintaining localStorage as source of truth
    const localTokens = this.getLocal();

    // Merge: keep local tokens that match DB URLs
    const merged = localTokens.filter(lt =>
      dbTokens.some(dt => dt.url === lt.url)
    );

    this.saveLocal(merged);
  }

  // Clear all tokens
  async clear() {
    localStorage.removeItem(this.localStorageKey);
    const dbTokens = await this.api.listTokens();
    for (const token of dbTokens) {
      await this.api.deleteToken(token.url);
    }
  }
}

// Usage
const storage = new TokenStorage(tokenMgr);

// Add token to both
await storage.add('https://huggingface.co', 'hf_abc123...');

// Sync localStorage to DB
await storage.syncToDatabase();

Security Best Practices

Token Encryption

All tokens encrypted with AES-256 (Fernet)
Encryption key derived from DATABASE_KEY environment variable
Never store tokens in plaintext
Tokens masked when displayed

Token Handling

from kohakuhub.crypto import encrypt_token, decrypt_token, mask_token

# Encrypt before storage
encrypted = encrypt_token("hf_abc123xyz789...")

# Decrypt when reading
plain = decrypt_token(encrypted)

# Mask for display
masked = mask_token("hf_abc123xyz789...", show_chars=4)
# Returns: "hf_a***"

Environment Setup

# Generate secure encryption key
export KOHAKU_HUB_DATABASE_KEY="$(openssl rand -hex 32)"

# Optional: Require authentication for fallback access
export KOHAKU_HUB_FALLBACK_REQUIRE_AUTH=false

CLI Usage

See CLI Documentation for command-line interface:

# List available sources
kohub-cli settings user external-tokens sources

# List configured tokens
kohub-cli settings user external-tokens list

# Add token
kohub-cli settings user external-tokens add \
  --url https://huggingface.co \
  --token hf_abc123...

# Delete token
kohub-cli settings user external-tokens delete \
  --url https://huggingface.co

# Bulk import from file
kohub-cli settings user external-tokens import tokens.json

Next Steps

Settings API - User and organization settings
Repository API - Access repositories
Authentication - Token-based authentication
Fallback System - Configure fallback sources

18 KiB Raw Permalink Blame History

External Tokens API

Overview

Endpoints

Get Available Fallback Sources

List User External Tokens

Add or Update External Token

Delete External Token

Bulk Update External Tokens

Authorization Header Format

Usage Examples

Comprehensive Token Management

Using Header-Based Tokens

Frontend Integration (JavaScript)

Sync Between LocalStorage and Database

Security Best Practices

Token Encryption

Token Handling

Environment Setup

CLI Usage

Next Steps

18 KiB

Raw Permalink Blame History