mirror of
https://github.com/KohakuBlueleaf/KohakuHub.git
synced 2026-03-11 17:34:08 -05:00
fix some documentation
This commit is contained in:
Kohaku-Blueleaf
28
API.md
28
API.md
@@ -585,6 +585,7 @@ When uploading LFS file:
|
||||
| `/api/{type}s/{id}` | GET | ○ | Get repo info |
|
||||
| `/api/{type}s/{id}/tree/{rev}/{path}` | GET | ○ | List files |
|
||||
| `/api/{type}s/{id}/revision/{rev}` | GET | ○ | Get revision info |
|
||||
| `/api/{type}s/{id}/paths-info/{rev}` | POST | ○ | Get info for specific paths |
|
||||
|
||||
### File Operations
|
||||
|
||||
@@ -594,6 +595,8 @@ When uploading LFS file:
|
||||
| `/api/{type}s/{id}/commit/{rev}` | POST | ✓ | Atomic commit |
|
||||
| `/{id}/resolve/{rev}/{file}` | GET | ○ | Download file |
|
||||
| `/{id}/resolve/{rev}/{file}` | HEAD | ○ | Get file metadata |
|
||||
| `/{type}s/{id}/resolve/{rev}/{file}` | GET | ○ | Download file (with type) |
|
||||
| `/{type}s/{id}/resolve/{rev}/{file}` | HEAD | ○ | Get file metadata (with type) |
|
||||
|
||||
### LFS Operations
|
||||
|
||||
@@ -602,6 +605,30 @@ When uploading LFS file:
|
||||
| `/{id}.git/info/lfs/objects/batch` | POST | ✓ | LFS batch API |
|
||||
| `/api/{id}.git/info/lfs/verify` | POST | ✓ | Verify upload |
|
||||
|
||||
### Authentication Operations
|
||||
|
||||
| Endpoint | Method | Auth | Description |
|
||||
|----------|--------|------|-------------|
|
||||
| `/api/auth/register` | POST | ✗ | Register new user |
|
||||
| `/api/auth/login` | POST | ✗ | Login and create session |
|
||||
| `/api/auth/logout` | POST | ✓ | Logout and destroy session |
|
||||
| `/api/auth/verify-email` | GET | ✗ | Verify email with token |
|
||||
| `/api/auth/me` | GET | ✓ | Get current user info |
|
||||
| `/api/auth/tokens` | GET | ✓ | List user's API tokens |
|
||||
| `/api/auth/tokens/create` | POST | ✓ | Create new API token |
|
||||
| `/api/auth/tokens/{id}` | DELETE | ✓ | Revoke API token |
|
||||
|
||||
### Organization Operations
|
||||
|
||||
| Endpoint | Method | Auth | Description |
|
||||
|----------|--------|------|-------------|
|
||||
| `/org/create` | POST | ✓ | Create new organization |
|
||||
| `/org/{name}` | GET | ✗ | Get organization details |
|
||||
| `/org/{name}/members` | POST | ✓ | Add member to organization |
|
||||
| `/org/{name}/members/{username}` | DELETE | ✓ | Remove member from organization |
|
||||
| `/org/{name}/members/{username}` | PUT | ✓ | Update member role |
|
||||
| `/org/users/{username}/orgs` | GET | ✗ | List user's organizations |
|
||||
|
||||
### Utility Operations
|
||||
|
||||
| Endpoint | Method | Auth | Description |
|
||||
@@ -609,6 +636,7 @@ When uploading LFS file:
|
||||
| `/api/validate-yaml` | POST | ✗ | Validate YAML content |
|
||||
| `/api/whoami-v2` | GET | ✓ | Get user info |
|
||||
| `/health` | GET | ✗ | Health check |
|
||||
| `/` | GET | ✗ | API information |
|
||||
|
||||
**Auth Legend**:
|
||||
- ✓ = Required
|
||||
|
||||
36
README.md
36
README.md
@@ -28,10 +28,13 @@ KohakuHub provides a simple but functional solution for teams and individuals wh
|
||||
- ✅ **File Operations**: Upload, download, copy, and delete files with automatic deduplication
|
||||
- ✅ **Large File Support**: Handles files of any size with Git LFS protocol
|
||||
- ✅ **Version Control**: Git-like branching and commit history via LakeFS
|
||||
- ✅ **Authentication & Authorization**: Secure user registration, session management, and API tokens
|
||||
- ✅ **Organization Management**: Create organizations and manage member roles
|
||||
- 🚧 **CLI Tool**: under development.
|
||||
- 🚧 **Web UI**: under development.
|
||||
- ✅ **Authentication & Authorization**: Complete user registration, session management, and API tokens
|
||||
- ✅ **Organization Management**: Full organization support with member roles (admin, super-admin, member)
|
||||
- ✅ **Permission System**: Namespace-based permissions for repositories and organizations
|
||||
- ✅ **Web UI**: Modern Vue 3 interface with file browsing, editing, and repository management
|
||||
- ✅ **Code Highlighting**: Syntax highlighting for code files with Monaco Editor integration
|
||||
- ✅ **Markdown Support**: Built-in markdown rendering for documentation
|
||||
- 🚧 **CLI Tool**: Basic functionality available (user/org management), more features in development
|
||||
|
||||
## Architecture
|
||||
|
||||
@@ -305,14 +308,27 @@ See [TODO.md](./TODO.md) for detailed development status.
|
||||
- Some Path related API may not be 100% supported, report if they are important for you.
|
||||
- ✅ HuggingFace client compatibility
|
||||
- ✅ Large file support (Git LFS)
|
||||
- ✅ Docker deployment
|
||||
- 🚧 Authentication & Authorization
|
||||
- Not all verification process are applied
|
||||
- ✅ Docker deployment with docker-compose
|
||||
- ✅ Authentication & Authorization
|
||||
- User registration with email verification (optional)
|
||||
- Session-based authentication with secure cookies
|
||||
- API token generation and management
|
||||
- Permission system for repositories and organizations
|
||||
- ✅ Organization Management
|
||||
- Create/delete organizations
|
||||
- Member management with roles (admin, super-admin, member)
|
||||
- Organization-based namespaces for repositories
|
||||
- ✅ Web User Interface
|
||||
- Vue 3 + Vite frontend with modern UI
|
||||
- Repository browsing and file viewing
|
||||
- Code editor with syntax highlighting
|
||||
- File upload/download interface
|
||||
- Markdown documentation rendering
|
||||
- User authentication pages (login/register)
|
||||
- Settings and organization management pages
|
||||
- 🚧 CLI for administration
|
||||
- Not fully functional, only core utils which usually require web interface.
|
||||
- 🚧 Web user interface
|
||||
- Under dev, not stable.
|
||||
- User and organization management functional
|
||||
- Additional administrative features in development
|
||||
|
||||
## Contributing
|
||||
|
||||
|
||||
170
TODO.md
170
TODO.md
@@ -1,33 +1,147 @@
|
||||
# TODO
|
||||
|
||||
Kohaku-Hub is a pretty large project and really hard to say where to start is better, but I will try to list all the known TODOs here with brief priority note
|
||||
|
||||
|
||||
- [ ] Basic Infra Structure
|
||||
## Infrastructure
|
||||
- [x] Basic Infra Structure
|
||||
- [x] LakeFS + MinIO deployment
|
||||
- [x] MinIO presigned URL
|
||||
- [ ] ...TBA...
|
||||
- [ ] API layer
|
||||
- [x] Upload
|
||||
- [x] Upload small file (not LFS)
|
||||
- [x] Upload large file
|
||||
- [x] Download
|
||||
- [x] Direct http requests (with S3 presigned url)
|
||||
- [ ] Other special interface (if have)
|
||||
- [x] Deletion
|
||||
- [ ] Repository Managements
|
||||
- [x] Tree List
|
||||
- [x] Creation
|
||||
- [x] Deletion
|
||||
- [ ] Move/Modify name
|
||||
- [ ] ...TBA...
|
||||
- [x] User Auth System
|
||||
- [x] Organization
|
||||
- [ ] Permission verification
|
||||
- [ ] ...TBA...
|
||||
- [x] Basic webUI
|
||||
- [x] User system
|
||||
- [x] Repo List
|
||||
- [x] Repo creation
|
||||
- [x] Repo Info page
|
||||
- [x] Repo file list
|
||||
- [ ] ...TBA...
|
||||
- [x] PostgreSQL database support
|
||||
- [x] SQLite database support
|
||||
- [x] Docker compose setup
|
||||
- [x] Environment configuration system
|
||||
|
||||
## API Layer
|
||||
- [x] Core File Operations
|
||||
- [x] Upload small files (not LFS)
|
||||
- [x] Upload large files (LFS)
|
||||
- [x] Download with S3 presigned URLs
|
||||
- [x] File deletion
|
||||
- [x] File copy operations
|
||||
- [x] Content deduplication
|
||||
- [x] Repository Management
|
||||
- [x] Create repository
|
||||
- [x] Delete repository
|
||||
- [x] List repositories
|
||||
- [x] Get repository info
|
||||
- [x] Tree list (recursive & non-recursive)
|
||||
- [x] Paths-info endpoint
|
||||
- [ ] Move/Rename repository
|
||||
- [ ] Repository transfer between namespaces
|
||||
- [x] Authentication & Authorization
|
||||
- [x] User registration
|
||||
- [x] User login/logout
|
||||
- [x] Email verification (optional)
|
||||
- [x] Session management
|
||||
- [x] API token generation
|
||||
- [x] API token revocation
|
||||
- [x] Permission system (namespace-based)
|
||||
- [x] Repository access control (read/write/delete)
|
||||
- [ ] More granular permissions
|
||||
- [ ] Rate limiting
|
||||
- [x] Organization Management
|
||||
- [x] Create organization
|
||||
- [x] Get organization details
|
||||
- [x] Add/remove members
|
||||
- [x] Update member roles
|
||||
- [x] List user organizations
|
||||
- [ ] Organization deletion
|
||||
- [ ] Organization settings/metadata
|
||||
- [ ] Advanced Features
|
||||
- [ ] Repository branches (beyond main)
|
||||
- [ ] Pull requests / merge requests
|
||||
- [ ] Discussion/comments
|
||||
- [ ] Repository stars/likes
|
||||
- [ ] Download statistics
|
||||
- [ ] Search functionality
|
||||
- [ ] Tags/categories
|
||||
|
||||
## Web UI (Vue 3 + Vite)
|
||||
- [x] Core Pages
|
||||
- [x] Home/landing page
|
||||
- [x] User registration page
|
||||
- [x] User login page
|
||||
- [x] User settings page
|
||||
- [x] About/docs pages
|
||||
- [x] Repository Features
|
||||
- [x] Repository list
|
||||
- [x] Repository creation
|
||||
- [x] Repository info page
|
||||
- [x] File browser/tree view
|
||||
- [x] File viewer with code highlighting
|
||||
- [x] File editor (Monaco Editor)
|
||||
- [x] File uploader
|
||||
- [x] Markdown renderer
|
||||
- [ ] Commit history view
|
||||
- [ ] Branch management UI
|
||||
- [ ] Diff viewer
|
||||
- [ ] Repository settings
|
||||
- [ ] Repository deletion UI
|
||||
- [x] User/Organization UI
|
||||
- [x] User profile view
|
||||
- [x] Organization pages
|
||||
- [ ] Organization member management UI
|
||||
- [ ] Organization settings UI
|
||||
- [ ] Additional Features
|
||||
- [x] Theme support (dark/light)
|
||||
- [ ] Search interface
|
||||
- [ ] Notifications
|
||||
- [ ] Activity feed
|
||||
- [ ] File preview for images/media
|
||||
|
||||
## CLI Tool
|
||||
- [x] Basic User Management
|
||||
- [x] User registration
|
||||
- [x] User login
|
||||
- [x] Token creation
|
||||
- [x] Basic Organization Management
|
||||
- [x] Create organization
|
||||
- [x] Add/remove members
|
||||
- [x] Update member roles
|
||||
- [ ] Repository Management
|
||||
- [ ] Create/delete repositories via CLI
|
||||
- [ ] Upload/download files
|
||||
- [ ] Manage branches
|
||||
- [ ] Administrative Features
|
||||
- [ ] User administration
|
||||
- [ ] System statistics
|
||||
- [ ] Backup/restore utilities
|
||||
|
||||
## Documentation
|
||||
- [x] API.md (comprehensive API documentation)
|
||||
- [x] README.md (getting started guide)
|
||||
- [x] TODO.md (this file)
|
||||
- [x] config-example.toml
|
||||
- [ ] Deployment guides
|
||||
- [ ] Production deployment best practices
|
||||
- [ ] Scaling guide
|
||||
- [ ] Migration guide
|
||||
- [ ] Developer documentation
|
||||
- [ ] Architecture overview
|
||||
- [ ] Contributing guidelines
|
||||
- [ ] API client development guide
|
||||
|
||||
## Testing & Quality
|
||||
- [ ] Unit tests
|
||||
- [ ] API endpoint tests
|
||||
- [ ] Database model tests
|
||||
- [ ] Authentication/authorization tests
|
||||
- [ ] Integration tests
|
||||
- [ ] E2E workflow tests
|
||||
- [ ] HuggingFace client compatibility tests
|
||||
- [ ] Performance testing
|
||||
- [ ] Load testing
|
||||
- [ ] Large file handling
|
||||
- [ ] Security auditing
|
||||
- [ ] Authentication security review
|
||||
- [ ] SQL injection prevention
|
||||
- [ ] XSS prevention in UI
|
||||
|
||||
## Future Enhancements
|
||||
- [ ] Multi-region/CDN support
|
||||
- [ ] Webhook system
|
||||
- [ ] Model card/dataset card templates
|
||||
- [ ] Integration with CI/CD pipelines
|
||||
- [ ] Automated model evaluation
|
||||
- [ ] Dataset versioning with lineage tracking
|
||||
- [ ] Model registry with deployment tracking
|
||||
17
src/kohaku-hub-ui/src/components.d.ts
vendored
17
src/kohaku-hub-ui/src/components.d.ts
vendored
@@ -10,27 +10,10 @@ declare module 'vue' {
|
||||
export interface GlobalComponents {
|
||||
CodeEditor: typeof import('./components/common/CodeEditor.vue')['default']
|
||||
CodeViewer: typeof import('./components/common/CodeViewer.vue')['default']
|
||||
ElBreadcrumb: typeof import('element-plus/es')['ElBreadcrumb']
|
||||
ElBreadcrumbItem: typeof import('element-plus/es')['ElBreadcrumbItem']
|
||||
ElButton: typeof import('element-plus/es')['ElButton']
|
||||
ElCheckbox: typeof import('element-plus/es')['ElCheckbox']
|
||||
ElDialog: typeof import('element-plus/es')['ElDialog']
|
||||
ElDropdown: typeof import('element-plus/es')['ElDropdown']
|
||||
ElDropdownItem: typeof import('element-plus/es')['ElDropdownItem']
|
||||
ElDropdownMenu: typeof import('element-plus/es')['ElDropdownMenu']
|
||||
ElForm: typeof import('element-plus/es')['ElForm']
|
||||
ElFormItem: typeof import('element-plus/es')['ElFormItem']
|
||||
ElIcon: typeof import('element-plus/es')['ElIcon']
|
||||
ElInput: typeof import('element-plus/es')['ElInput']
|
||||
ElOption: typeof import('element-plus/es')['ElOption']
|
||||
ElProgress: typeof import('element-plus/es')['ElProgress']
|
||||
ElRadio: typeof import('element-plus/es')['ElRadio']
|
||||
ElRadioButton: typeof import('element-plus/es')['ElRadioButton']
|
||||
ElRadioGroup: typeof import('element-plus/es')['ElRadioGroup']
|
||||
ElSelect: typeof import('element-plus/es')['ElSelect']
|
||||
ElSkeleton: typeof import('element-plus/es')['ElSkeleton']
|
||||
ElTabPane: typeof import('element-plus/es')['ElTabPane']
|
||||
ElTabs: typeof import('element-plus/es')['ElTabs']
|
||||
ElTag: typeof import('element-plus/es')['ElTag']
|
||||
FileUploader: typeof import('./components/repo/FileUploader.vue')['default']
|
||||
HelloWorld: typeof import('./components/HelloWorld.vue')['default']
|
||||
|
||||
@@ -89,7 +89,7 @@ huggingface-cli upload username/model-name ./my-model --repo-type model
|
||||
|
||||
### Option 2: hfutils
|
||||
|
||||
A lightweight alternative with similar functionality.
|
||||
A powerful alternative from [deepghs/hfutils](https://github.com/deepghs/hfutils) with advanced features.
|
||||
|
||||
#### Installation
|
||||
|
||||
@@ -100,39 +100,74 @@ pip install hfutils
|
||||
#### Usage
|
||||
|
||||
\`\`\`bash
|
||||
# Set your token
|
||||
# Set your token for private repositories
|
||||
export HF_TOKEN=your_access_token_here
|
||||
|
||||
# Download a repository
|
||||
hf download username/model-name
|
||||
# Download a single file
|
||||
hfutils download -r username/model-name -o ./local-file.txt -f remote-file.txt
|
||||
|
||||
# Upload files
|
||||
hf upload username/model-name /path/to/files
|
||||
# Download entire directory
|
||||
hfutils download -r username/model-name -o ./local-dir -d remote-dir
|
||||
|
||||
# Download archived directory
|
||||
hfutils download -r username/model-name -o ./local-dir -a archive.zip
|
||||
|
||||
# Upload a single file
|
||||
hfutils upload -r username/model-name -i ./local-file.txt -f remote-file.txt
|
||||
|
||||
# Upload directory as tree
|
||||
hfutils upload -r username/model-name -i ./local-dir -d remote-dir
|
||||
|
||||
# Upload directory as archive
|
||||
hfutils upload -r username/model-name -i ./local-dir -a archive.zip
|
||||
\`\`\`
|
||||
|
||||
#### Advanced Options
|
||||
|
||||
\`\`\`bash
|
||||
# Enable transfer acceleration
|
||||
export HF_HUB_ENABLE_HF_TRANSFER=1
|
||||
|
||||
# Specify repository type
|
||||
hfutils download -r username/dataset-name -t dataset -o ./output
|
||||
|
||||
# Custom temporary directory
|
||||
export TMPDIR=/path/to/tmp
|
||||
\`\`\`
|
||||
|
||||
### Option 3: kohub-cli (KohakuHub Native)
|
||||
|
||||
If available, kohub-cli provides KohakuHub-specific features.
|
||||
KohakuHub's native CLI tool for user and organization management.
|
||||
|
||||
#### Installation
|
||||
|
||||
Currently, you need to install from source:
|
||||
|
||||
\`\`\`bash
|
||||
pip install kohub-cli
|
||||
# Clone the repository
|
||||
git clone https://github.com/KohakuBlueleaf/KohakuHub.git
|
||||
cd KohakuHub
|
||||
|
||||
# Install dependencies and the CLI
|
||||
pip install -r requirements.txt
|
||||
pip install -e .
|
||||
\`\`\`
|
||||
|
||||
#### Usage
|
||||
|
||||
The CLI provides an interactive menu for managing users and organizations:
|
||||
|
||||
\`\`\`bash
|
||||
# Login to KohakuHub
|
||||
kohub login
|
||||
# Run the interactive CLI
|
||||
kohub-cli
|
||||
|
||||
# Download a repository
|
||||
kohub download username/model-name
|
||||
|
||||
# Upload files
|
||||
kohub upload username/model-name /path/to/files
|
||||
# Available operations:
|
||||
# - User Management: Register, Login, Create Token
|
||||
# - Organization Management: Create Organization, Manage Members
|
||||
\`\`\`
|
||||
|
||||
For file operations, use huggingface-cli or hfutils instead.
|
||||
|
||||
## Using the Python API
|
||||
|
||||
You can also use the HuggingFace Hub library directly in your Python code:
|
||||
|
||||
Reference in New Issue
Block a user