PVBLIC Foundation
2bbcef5fe7
fix: Comprehensive HTML stripping for email indexing
...
Problem:
1. HTML garbage in keywords (border, padding, cellpadding)
2. DOCTYPE declarations appearing in snippets
3. MSO conditional comments ('[if mso]') not removed
4. Inline styles (style="...") left garbage text
Solution - Both files now strip:
- DOCTYPE and XML declarations (<!DOCTYPE...>, <?xml?>)
- CDATA sections and HTML comments
- MSO Office conditional comments ([if mso]...[endif])
- Script and style blocks with content
- Inline style attributes BEFORE removing tags
- All remaining HTML tags
Defense in depth:
- gmail_processor.py: First-pass HTML stripping
- gmail_indexer_v2.py: Secondary stripping in _deep_clean_for_embeddings
This ensures clean text for embeddings, keywords, and snippets.
2025-11-27 12:02:33 -08:00
PVBLIC Foundation
2dd6eff030
Add email attachment processing and text quality improvements
...
NEW FEATURE: Email Attachment Processing
=========================================
New File: gmail_attachment_processor.py (202 lines)
- Downloads and processes email attachments using unstructured.io
- Supports: PDF, DOCX, XLSX, PPTX, TXT, CSV, MD, HTML, EML
- Configurable size limits and file type filtering
- Integrates with Open WebUI's UnstructuredLoader
- Extracts text from attachments and combines with email body
- Makes attachment content fully searchable
Configuration (config.py + main.py):
- GMAIL_PROCESS_ATTACHMENTS (default: True)
- GMAIL_MAX_ATTACHMENT_SIZE_MB (default: 10MB)
- GMAIL_ATTACHMENT_TYPES (default: .pdf,.docx,.doc,.xlsx,.xls,.pptx,.ppt,.txt,.csv,.md,.html,.eml)
Email Text Quality Improvements (gmail_indexer.py):
=====================================
1. _deep_clean_for_search() - Deep text cleaning:
- Remove URLs → replace with [link]
- Remove email addresses → replace with [email] (privacy)
- Clean escape sequences (\n→\n, \"→", \/→/)
- Remove HTML entities
- Normalize whitespace (max 2 newlines)
- Remove orphaned punctuation
2. _is_notification_email() - Smart detection:
- Detect Google Drive/Docs/Sheets notifications
- Detect automated messages (no-reply, confirmations)
- Flag low-value emails for optimized storage
3. Smart Chunking Strategy:
- Notifications: summary only (1 vector)
- Short emails (<500 chars): summary only
- Substantial emails: summary + content chunks
- Min chunk size: 100 chars (was 50)
- Skip duplicate/empty chunks
Attachment Integration (gmail_fetcher.py, gmail_processor.py, gmail_indexer.py):
- fetch_attachment() method in GmailFetcher
- _get_attachments() method in GmailProcessor
- Attachment metadata tracking (count, names, processed count)
- Combines attachment text with email body before embedding
- Attachment content included in semantic search
Gmail Auto Sync Integration (gmail_auto_sync.py):
- Pass attachment config to indexer
- Pass fetcher to indexer for downloads
- Log attachment processing status
Benefits:
- Complete email search (body + attachments)
- Cleaner, more readable text in vectors
- Better metadata extraction (entities, topics)
- Privacy improvements (URLs/emails removed)
- Storage optimization (notifications = 1 vector)
- Find documents shared via email
- Configurable and safe (size/type limits)
Example: Email with PDF attachment
Before: Only email body indexed ("See attached report")
After: Email body + full PDF content indexed and searchable
2025-11-06 12:20:48 -08:00
PVBLIC Foundation