mirror of
https://github.com/open-webui/open-webui.git
synced 2026-08-29 08:07:34 -05:00
Problem:
1. HTML garbage in keywords (border, padding, cellpadding)
2. DOCTYPE declarations appearing in snippets
3. MSO conditional comments ('[if mso]') not removed
4. Inline styles (style="...") left garbage text
Solution - Both files now strip:
- DOCTYPE and XML declarations (<!DOCTYPE...>, <?xml?>)
- CDATA sections and HTML comments
- MSO Office conditional comments ([if mso]...[endif])
- Script and style blocks with content
- Inline style attributes BEFORE removing tags
- All remaining HTML tags
Defense in depth:
- gmail_processor.py: First-pass HTML stripping
- gmail_indexer_v2.py: Secondary stripping in _deep_clean_for_embeddings
This ensures clean text for embeddings, keywords, and snippets.