From 4cc300549d73b2dd761927427bb640b50809d204 Mon Sep 17 00:00:00 2001 From: Kohaku-Blueleaf <59680068+KohakuBlueleaf@users.noreply.github.com> Date: Sat, 4 Oct 2025 18:06:16 +0800 Subject: [PATCH] fix some documentation --- API.md | 28 ++++ README.md | 36 +++-- TODO.md | 170 ++++++++++++++++---- src/kohaku-hub-ui/src/components.d.ts | 17 -- src/kohaku-hub-ui/src/pages/get-started.vue | 65 ++++++-- 5 files changed, 246 insertions(+), 70 deletions(-) diff --git a/API.md b/API.md index 2c39246..4f6e32c 100644 --- a/API.md +++ b/API.md @@ -585,6 +585,7 @@ When uploading LFS file: | `/api/{type}s/{id}` | GET | ○ | Get repo info | | `/api/{type}s/{id}/tree/{rev}/{path}` | GET | ○ | List files | | `/api/{type}s/{id}/revision/{rev}` | GET | ○ | Get revision info | +| `/api/{type}s/{id}/paths-info/{rev}` | POST | ○ | Get info for specific paths | ### File Operations @@ -594,6 +595,8 @@ When uploading LFS file: | `/api/{type}s/{id}/commit/{rev}` | POST | ✓ | Atomic commit | | `/{id}/resolve/{rev}/{file}` | GET | ○ | Download file | | `/{id}/resolve/{rev}/{file}` | HEAD | ○ | Get file metadata | +| `/{type}s/{id}/resolve/{rev}/{file}` | GET | ○ | Download file (with type) | +| `/{type}s/{id}/resolve/{rev}/{file}` | HEAD | ○ | Get file metadata (with type) | ### LFS Operations @@ -602,6 +605,30 @@ When uploading LFS file: | `/{id}.git/info/lfs/objects/batch` | POST | ✓ | LFS batch API | | `/api/{id}.git/info/lfs/verify` | POST | ✓ | Verify upload | +### Authentication Operations + +| Endpoint | Method | Auth | Description | +|----------|--------|------|-------------| +| `/api/auth/register` | POST | ✗ | Register new user | +| `/api/auth/login` | POST | ✗ | Login and create session | +| `/api/auth/logout` | POST | ✓ | Logout and destroy session | +| `/api/auth/verify-email` | GET | ✗ | Verify email with token | +| `/api/auth/me` | GET | ✓ | Get current user info | +| `/api/auth/tokens` | GET | ✓ | List user's API tokens | +| `/api/auth/tokens/create` | POST | ✓ | Create new API token | +| `/api/auth/tokens/{id}` | DELETE | ✓ | Revoke API token | + +### Organization Operations + +| Endpoint | Method | Auth | Description | +|----------|--------|------|-------------| +| `/org/create` | POST | ✓ | Create new organization | +| `/org/{name}` | GET | ✗ | Get organization details | +| `/org/{name}/members` | POST | ✓ | Add member to organization | +| `/org/{name}/members/{username}` | DELETE | ✓ | Remove member from organization | +| `/org/{name}/members/{username}` | PUT | ✓ | Update member role | +| `/org/users/{username}/orgs` | GET | ✗ | List user's organizations | + ### Utility Operations | Endpoint | Method | Auth | Description | @@ -609,6 +636,7 @@ When uploading LFS file: | `/api/validate-yaml` | POST | ✗ | Validate YAML content | | `/api/whoami-v2` | GET | ✓ | Get user info | | `/health` | GET | ✗ | Health check | +| `/` | GET | ✗ | API information | **Auth Legend**: - ✓ = Required diff --git a/README.md b/README.md index e664e05..c2e90ff 100644 --- a/README.md +++ b/README.md @@ -28,10 +28,13 @@ KohakuHub provides a simple but functional solution for teams and individuals wh - ✅ **File Operations**: Upload, download, copy, and delete files with automatic deduplication - ✅ **Large File Support**: Handles files of any size with Git LFS protocol - ✅ **Version Control**: Git-like branching and commit history via LakeFS -- ✅ **Authentication & Authorization**: Secure user registration, session management, and API tokens -- ✅ **Organization Management**: Create organizations and manage member roles -- 🚧 **CLI Tool**: under development. -- 🚧 **Web UI**: under development. +- ✅ **Authentication & Authorization**: Complete user registration, session management, and API tokens +- ✅ **Organization Management**: Full organization support with member roles (admin, super-admin, member) +- ✅ **Permission System**: Namespace-based permissions for repositories and organizations +- ✅ **Web UI**: Modern Vue 3 interface with file browsing, editing, and repository management +- ✅ **Code Highlighting**: Syntax highlighting for code files with Monaco Editor integration +- ✅ **Markdown Support**: Built-in markdown rendering for documentation +- 🚧 **CLI Tool**: Basic functionality available (user/org management), more features in development ## Architecture @@ -305,14 +308,27 @@ See [TODO.md](./TODO.md) for detailed development status. - Some Path related API may not be 100% supported, report if they are important for you. - ✅ HuggingFace client compatibility - ✅ Large file support (Git LFS) -- ✅ Docker deployment -- 🚧 Authentication & Authorization - - Not all verification process are applied +- ✅ Docker deployment with docker-compose +- ✅ Authentication & Authorization + - User registration with email verification (optional) + - Session-based authentication with secure cookies + - API token generation and management + - Permission system for repositories and organizations - ✅ Organization Management + - Create/delete organizations + - Member management with roles (admin, super-admin, member) + - Organization-based namespaces for repositories +- ✅ Web User Interface + - Vue 3 + Vite frontend with modern UI + - Repository browsing and file viewing + - Code editor with syntax highlighting + - File upload/download interface + - Markdown documentation rendering + - User authentication pages (login/register) + - Settings and organization management pages - 🚧 CLI for administration - - Not fully functional, only core utils which usually require web interface. -- 🚧 Web user interface - - Under dev, not stable. + - User and organization management functional + - Additional administrative features in development ## Contributing diff --git a/TODO.md b/TODO.md index e92e660..e55f583 100644 --- a/TODO.md +++ b/TODO.md @@ -1,33 +1,147 @@ # TODO + Kohaku-Hub is a pretty large project and really hard to say where to start is better, but I will try to list all the known TODOs here with brief priority note - -- [ ] Basic Infra Structure +## Infrastructure +- [x] Basic Infra Structure - [x] LakeFS + MinIO deployment - [x] MinIO presigned URL - - [ ] ...TBA... -- [ ] API layer - - [x] Upload - - [x] Upload small file (not LFS) - - [x] Upload large file - - [x] Download - - [x] Direct http requests (with S3 presigned url) - - [ ] Other special interface (if have) - - [x] Deletion - - [ ] Repository Managements - - [x] Tree List - - [x] Creation - - [x] Deletion - - [ ] Move/Modify name - - [ ] ...TBA... - - [x] User Auth System - - [x] Organization - - [ ] Permission verification - - [ ] ...TBA... -- [x] Basic webUI - - [x] User system - - [x] Repo List - - [x] Repo creation - - [x] Repo Info page - - [x] Repo file list - - [ ] ...TBA... \ No newline at end of file + - [x] PostgreSQL database support + - [x] SQLite database support + - [x] Docker compose setup + - [x] Environment configuration system + +## API Layer +- [x] Core File Operations + - [x] Upload small files (not LFS) + - [x] Upload large files (LFS) + - [x] Download with S3 presigned URLs + - [x] File deletion + - [x] File copy operations + - [x] Content deduplication +- [x] Repository Management + - [x] Create repository + - [x] Delete repository + - [x] List repositories + - [x] Get repository info + - [x] Tree list (recursive & non-recursive) + - [x] Paths-info endpoint + - [ ] Move/Rename repository + - [ ] Repository transfer between namespaces +- [x] Authentication & Authorization + - [x] User registration + - [x] User login/logout + - [x] Email verification (optional) + - [x] Session management + - [x] API token generation + - [x] API token revocation + - [x] Permission system (namespace-based) + - [x] Repository access control (read/write/delete) + - [ ] More granular permissions + - [ ] Rate limiting +- [x] Organization Management + - [x] Create organization + - [x] Get organization details + - [x] Add/remove members + - [x] Update member roles + - [x] List user organizations + - [ ] Organization deletion + - [ ] Organization settings/metadata +- [ ] Advanced Features + - [ ] Repository branches (beyond main) + - [ ] Pull requests / merge requests + - [ ] Discussion/comments + - [ ] Repository stars/likes + - [ ] Download statistics + - [ ] Search functionality + - [ ] Tags/categories + +## Web UI (Vue 3 + Vite) +- [x] Core Pages + - [x] Home/landing page + - [x] User registration page + - [x] User login page + - [x] User settings page + - [x] About/docs pages +- [x] Repository Features + - [x] Repository list + - [x] Repository creation + - [x] Repository info page + - [x] File browser/tree view + - [x] File viewer with code highlighting + - [x] File editor (Monaco Editor) + - [x] File uploader + - [x] Markdown renderer + - [ ] Commit history view + - [ ] Branch management UI + - [ ] Diff viewer + - [ ] Repository settings + - [ ] Repository deletion UI +- [x] User/Organization UI + - [x] User profile view + - [x] Organization pages + - [ ] Organization member management UI + - [ ] Organization settings UI +- [ ] Additional Features + - [x] Theme support (dark/light) + - [ ] Search interface + - [ ] Notifications + - [ ] Activity feed + - [ ] File preview for images/media + +## CLI Tool +- [x] Basic User Management + - [x] User registration + - [x] User login + - [x] Token creation +- [x] Basic Organization Management + - [x] Create organization + - [x] Add/remove members + - [x] Update member roles +- [ ] Repository Management + - [ ] Create/delete repositories via CLI + - [ ] Upload/download files + - [ ] Manage branches +- [ ] Administrative Features + - [ ] User administration + - [ ] System statistics + - [ ] Backup/restore utilities + +## Documentation +- [x] API.md (comprehensive API documentation) +- [x] README.md (getting started guide) +- [x] TODO.md (this file) +- [x] config-example.toml +- [ ] Deployment guides + - [ ] Production deployment best practices + - [ ] Scaling guide + - [ ] Migration guide +- [ ] Developer documentation + - [ ] Architecture overview + - [ ] Contributing guidelines + - [ ] API client development guide + +## Testing & Quality +- [ ] Unit tests + - [ ] API endpoint tests + - [ ] Database model tests + - [ ] Authentication/authorization tests +- [ ] Integration tests + - [ ] E2E workflow tests + - [ ] HuggingFace client compatibility tests +- [ ] Performance testing + - [ ] Load testing + - [ ] Large file handling +- [ ] Security auditing + - [ ] Authentication security review + - [ ] SQL injection prevention + - [ ] XSS prevention in UI + +## Future Enhancements +- [ ] Multi-region/CDN support +- [ ] Webhook system +- [ ] Model card/dataset card templates +- [ ] Integration with CI/CD pipelines +- [ ] Automated model evaluation +- [ ] Dataset versioning with lineage tracking +- [ ] Model registry with deployment tracking \ No newline at end of file diff --git a/src/kohaku-hub-ui/src/components.d.ts b/src/kohaku-hub-ui/src/components.d.ts index 73a7984..892e56d 100644 --- a/src/kohaku-hub-ui/src/components.d.ts +++ b/src/kohaku-hub-ui/src/components.d.ts @@ -10,27 +10,10 @@ declare module 'vue' { export interface GlobalComponents { CodeEditor: typeof import('./components/common/CodeEditor.vue')['default'] CodeViewer: typeof import('./components/common/CodeViewer.vue')['default'] - ElBreadcrumb: typeof import('element-plus/es')['ElBreadcrumb'] - ElBreadcrumbItem: typeof import('element-plus/es')['ElBreadcrumbItem'] ElButton: typeof import('element-plus/es')['ElButton'] - ElCheckbox: typeof import('element-plus/es')['ElCheckbox'] - ElDialog: typeof import('element-plus/es')['ElDialog'] ElDropdown: typeof import('element-plus/es')['ElDropdown'] ElDropdownItem: typeof import('element-plus/es')['ElDropdownItem'] ElDropdownMenu: typeof import('element-plus/es')['ElDropdownMenu'] - ElForm: typeof import('element-plus/es')['ElForm'] - ElFormItem: typeof import('element-plus/es')['ElFormItem'] - ElIcon: typeof import('element-plus/es')['ElIcon'] - ElInput: typeof import('element-plus/es')['ElInput'] - ElOption: typeof import('element-plus/es')['ElOption'] - ElProgress: typeof import('element-plus/es')['ElProgress'] - ElRadio: typeof import('element-plus/es')['ElRadio'] - ElRadioButton: typeof import('element-plus/es')['ElRadioButton'] - ElRadioGroup: typeof import('element-plus/es')['ElRadioGroup'] - ElSelect: typeof import('element-plus/es')['ElSelect'] - ElSkeleton: typeof import('element-plus/es')['ElSkeleton'] - ElTabPane: typeof import('element-plus/es')['ElTabPane'] - ElTabs: typeof import('element-plus/es')['ElTabs'] ElTag: typeof import('element-plus/es')['ElTag'] FileUploader: typeof import('./components/repo/FileUploader.vue')['default'] HelloWorld: typeof import('./components/HelloWorld.vue')['default'] diff --git a/src/kohaku-hub-ui/src/pages/get-started.vue b/src/kohaku-hub-ui/src/pages/get-started.vue index 4c4a7ba..1c67b1e 100644 --- a/src/kohaku-hub-ui/src/pages/get-started.vue +++ b/src/kohaku-hub-ui/src/pages/get-started.vue @@ -89,7 +89,7 @@ huggingface-cli upload username/model-name ./my-model --repo-type model ### Option 2: hfutils -A lightweight alternative with similar functionality. +A powerful alternative from [deepghs/hfutils](https://github.com/deepghs/hfutils) with advanced features. #### Installation @@ -100,39 +100,74 @@ pip install hfutils #### Usage \`\`\`bash -# Set your token +# Set your token for private repositories export HF_TOKEN=your_access_token_here -# Download a repository -hf download username/model-name +# Download a single file +hfutils download -r username/model-name -o ./local-file.txt -f remote-file.txt -# Upload files -hf upload username/model-name /path/to/files +# Download entire directory +hfutils download -r username/model-name -o ./local-dir -d remote-dir + +# Download archived directory +hfutils download -r username/model-name -o ./local-dir -a archive.zip + +# Upload a single file +hfutils upload -r username/model-name -i ./local-file.txt -f remote-file.txt + +# Upload directory as tree +hfutils upload -r username/model-name -i ./local-dir -d remote-dir + +# Upload directory as archive +hfutils upload -r username/model-name -i ./local-dir -a archive.zip +\`\`\` + +#### Advanced Options + +\`\`\`bash +# Enable transfer acceleration +export HF_HUB_ENABLE_HF_TRANSFER=1 + +# Specify repository type +hfutils download -r username/dataset-name -t dataset -o ./output + +# Custom temporary directory +export TMPDIR=/path/to/tmp \`\`\` ### Option 3: kohub-cli (KohakuHub Native) -If available, kohub-cli provides KohakuHub-specific features. +KohakuHub's native CLI tool for user and organization management. #### Installation +Currently, you need to install from source: + \`\`\`bash -pip install kohub-cli +# Clone the repository +git clone https://github.com/KohakuBlueleaf/KohakuHub.git +cd KohakuHub + +# Install dependencies and the CLI +pip install -r requirements.txt +pip install -e . \`\`\` #### Usage +The CLI provides an interactive menu for managing users and organizations: + \`\`\`bash -# Login to KohakuHub -kohub login +# Run the interactive CLI +kohub-cli -# Download a repository -kohub download username/model-name - -# Upload files -kohub upload username/model-name /path/to/files +# Available operations: +# - User Management: Register, Login, Create Token +# - Organization Management: Create Organization, Manage Members \`\`\` +For file operations, use huggingface-cli or hfutils instead. + ## Using the Python API You can also use the HuggingFace Hub library directly in your Python code: