mirror of
https://github.com/open-webui/open-webui.git
synced 2026-08-29 08:07:34 -05:00
1cbfa1be06be67d5a0895cf3f2de1fd9feda91d5
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 PVBLIC Foundation
|
972b0d8f34 |
Merge upstream v0.9.5 into main
Resolved conflicts in ~47 files. Kept local customizations and integrated upstream security fixes, channels streaming, async migration, and new features. Backend - RAG / vector / audio / video / storage - retrieval.py, knowledge.py: kept two-namespace deletion pattern, expandable tree, Drive sync hooks, get_namespace_for_collection helper - pinecone.py + retrieval/vector/utils.py: kept namespace isolation, retry logic, PINECONE_DIMENSION PersistentConfig - retrieval/utils.py + loaders/main.py: kept RobustCSVLoader, transcript segments; took upstream Excel/PPTX loaders - audio.py: preserved results[idx] chunk ordering, HTTPException re-raise, audio/* + video/* MIME support - storage/provider.py: kept caching + signed URL changes Backend - core - main.py: combined local schedulers (temp file cleanup, Gmail sync, Drive sync) with upstream scheduler_worker_loop; kept gmail.router - config.py: kept Alembic multi-head cleanup + upgrade-to-heads, expanded GOOGLE_OAUTH_SCOPE (Gmail + Drive readonly), merged GOOGLE_OAUTH_AUTHORIZE_PARAMS, kept local RAG defaults + VIDEO_AUDIO_CHUNKING_STRATEGY - env.py: kept local WEBUI_NAME branding (no '(Open WebUI)' suffix) - middleware.py: combined UserUsages.record_usage with upstream outlet_filter_handler - oauth.py: awaited update_session_by_id, kept verbose token-refresh errors and trigger_gmail_sync_if_needed - pdf_generator.py: kept ReportLab implementation with custom fonts - task.py: made prompt_improvement_template async-aware - routers/tasks.py: awaited prompt_improvement_template - utils/models.py: kept missing_function_ids tracking - utils/misc.py: kept broader audio/video MIME support Backend - models - users.py: added Float import, kept UserSpendLimitForm, gmail_sync default, UserUsages cleanup; took upstream profile_image_url validator and async delete pattern - chats.py: kept JSONB indexes, archived/pinned, custom indexes, PG17 JSON_TABLE search optimization; took upstream async migration Backend - routers (upstream security fixes + local hooks) - channels.py: took upstream get_filtered_models + CHAT_COMPLETION_HANDLER; kept process_message_knowledge for RAG context injection - chats.py, files.py, users.py, memories.py, openai.py, auths.py: took upstream mass-assignment, SSRF, ownership, image URL validation fixes; kept spend-limit and external user ID hooks Alembic migrations - Renamed upstream e1f2a3b4c5d6_add_is_pinned_to_note.py revision id to e2f3a4b5c6d7 to resolve collision with our merge_all_heads migration - Added f7e8d9c0b1a2_merge_post_v0_9_5.py merging e1f2a3b4c5d6 + a0b1c2d3e4f5 into a single head Frontend (Svelte) - Chat.svelte: kept previousChatIdProp dedup + messageQueue for in-flight messages - CitationModal.svelte: kept WEBUI_BASE_URL import alongside upstream config + injectCsp - ContentRenderer.svelte: kept csv/json artifact detection alongside upstream tool-call/reasoning rendering - MarkdownTokens.svelte: kept tableToCSV alongside new GROUPABLE_DETAIL helpers - Placeholder.svelte: kept mobile-friendly model selection layout - channel/Messages/Message.svelte: re-inserted Knowledge Sources display; took upstream timestamp formatting - channel/MessageInput/InputMenu.svelte: removed bits-ui, kept knowledge attach tabs - workspace/Knowledge/KnowledgeBase/AddContentMenu.svelte: kept Connect Google Drive entry, normalized button styling - layout/Navbar/Menu.svelte + layout/Sidebar/ChatMenu.svelte: kept ReportLab backend PDF + Word doc export, layered upstream client-side PDF as alternative Translations + assets - en-US/translation.json: merged new upstream keys, kept local PPTX, Gmail sync, video chapters, spend-limit strings - favicons left as-is (kept ours) Dependencies - requirements.txt + pyproject.toml: harmonized redis 7.4.0, kept unstructured[all-docs] extra, took upstream pydub 0.25.1, ddgs 9.11.4 Dockerfile - Adopted upstream ENV UV_LINK_MODE=copy for QEMU arm64 cross-builds; kept local CUDA + model pre-download blocks Version bump - package.json + pyproject.toml -> 0.9.5 - CHANGELOG.md: added Fork Merge Summary section Made-with: Cursor |
||
|
PVBLIC Foundation
|
2bbcef5fe7 |
fix: Comprehensive HTML stripping for email indexing
Problem:
1. HTML garbage in keywords (border, padding, cellpadding)
2. DOCTYPE declarations appearing in snippets
3. MSO conditional comments ('[if mso]') not removed
4. Inline styles (style="...") left garbage text
Solution - Both files now strip:
- DOCTYPE and XML declarations (<!DOCTYPE...>, <?xml?>)
- CDATA sections and HTML comments
- MSO Office conditional comments ([if mso]...[endif])
- Script and style blocks with content
- Inline style attributes BEFORE removing tags
- All remaining HTML tags
Defense in depth:
- gmail_processor.py: First-pass HTML stripping
- gmail_indexer_v2.py: Secondary stripping in _deep_clean_for_embeddings
This ensures clean text for embeddings, keywords, and snippets.
|
||
|
PVBLIC Foundation
|
2dd6eff030 |
Add email attachment processing and text quality improvements
NEW FEATURE: Email Attachment Processing
=========================================
New File: gmail_attachment_processor.py (202 lines)
- Downloads and processes email attachments using unstructured.io
- Supports: PDF, DOCX, XLSX, PPTX, TXT, CSV, MD, HTML, EML
- Configurable size limits and file type filtering
- Integrates with Open WebUI's UnstructuredLoader
- Extracts text from attachments and combines with email body
- Makes attachment content fully searchable
Configuration (config.py + main.py):
- GMAIL_PROCESS_ATTACHMENTS (default: True)
- GMAIL_MAX_ATTACHMENT_SIZE_MB (default: 10MB)
- GMAIL_ATTACHMENT_TYPES (default: .pdf,.docx,.doc,.xlsx,.xls,.pptx,.ppt,.txt,.csv,.md,.html,.eml)
Email Text Quality Improvements (gmail_indexer.py):
=====================================
1. _deep_clean_for_search() - Deep text cleaning:
- Remove URLs → replace with [link]
- Remove email addresses → replace with [email] (privacy)
- Clean escape sequences (\n→\n, \"→", \/→/)
- Remove HTML entities
- Normalize whitespace (max 2 newlines)
- Remove orphaned punctuation
2. _is_notification_email() - Smart detection:
- Detect Google Drive/Docs/Sheets notifications
- Detect automated messages (no-reply, confirmations)
- Flag low-value emails for optimized storage
3. Smart Chunking Strategy:
- Notifications: summary only (1 vector)
- Short emails (<500 chars): summary only
- Substantial emails: summary + content chunks
- Min chunk size: 100 chars (was 50)
- Skip duplicate/empty chunks
Attachment Integration (gmail_fetcher.py, gmail_processor.py, gmail_indexer.py):
- fetch_attachment() method in GmailFetcher
- _get_attachments() method in GmailProcessor
- Attachment metadata tracking (count, names, processed count)
- Combines attachment text with email body before embedding
- Attachment content included in semantic search
Gmail Auto Sync Integration (gmail_auto_sync.py):
- Pass attachment config to indexer
- Pass fetcher to indexer for downloads
- Log attachment processing status
Benefits:
- Complete email search (body + attachments)
- Cleaner, more readable text in vectors
- Better metadata extraction (entities, topics)
- Privacy improvements (URLs/emails removed)
- Storage optimization (notifications = 1 vector)
- Find documents shared via email
- Configurable and safe (size/type limits)
Example: Email with PDF attachment
Before: Only email body indexed ("See attached report")
After: Email body + full PDF content indexed and searchable
|
||
|
PVBLIC Foundation
|
2f73d2b7b4 |
Add Gmail sync core files from feat/gmail-sync branch
Cherry-picked essential Gmail sync functionality: Core Files (7): - utils/gmail_fetcher.py - Fetches emails from Gmail API - utils/gmail_processor.py - Parses Gmail API responses - utils/email_cleaner.py - Email-specific text cleaning - utils/gmail_indexer.py - Creates vector embeddings using extract_enhanced_metadata - utils/gmail_auto_sync.py - Orchestrates sync process - routers/gmail.py - Gmail sync API endpoints - models/gmail_sync.py - Gmail sync status tracking Integration Files (3): - main.py - Register Gmail router and background tasks - oauth.py - Trigger Gmail sync after Google OAuth - config.py - Gmail sync configuration settings Database (2): - migrations/33cc3721a72_add_gmail_sync_status_table.py - migrations/0b80d222da03_merge_gmail_sync_and_main_branch_.py Dependencies: - requirements.txt - Updated unstructured[all-docs]==0.18.15 for full document support |