[PR #18854] [CLOSED] fix(retrieval): preserve Chinese punctuation in uploaded documents #63857

Closed
opened 2026-05-06 08:58:35 -05:00 by GiteaMirror · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/open-webui/open-webui/pull/18854
Author: @fsmw
Created: 11/1/2025
Status: ❌ Closed

Base: dev ← Head: fix/17087-chinese-punctuation-conversion


📝 Commits (1)

  • 9c9092f fix(retrieval): preserve Chinese punctuation in uploaded documents

📊 Changes

1 file changed (+1 additions, -2 deletions)

View changed files

📝 backend/open_webui/retrieval/loaders/main.py (+1 -2)

📄 Description

Summary

Remove ftfy.fix_text() processing that converts Chinese punctuation marks to English equivalents, preserving cultural content integrity.

Issue

Fixes #17087 - The Chinese punctuation in the uploaded text file will be automatically converted to English punctuation marks.

Changes Made

  • Removed ftfy.fix_text() processing from document upload pipeline
  • Preserved Chinese punctuation marks (,。:;!?《》【】「」) in uploaded files
  • Maintained existing text processing for other languages

Testing

  • Reproduced the issue with Chinese punctuation conversion
  • Verified fix preserves Chinese punctuation in uploaded documents
  • Tested with various Chinese text files
  • Confirmed no regression in other language processing

Additional Information

This fix addresses cultural content preservation issues where Chinese punctuation was being automatically converted to English equivalents during file upload.

Contributor License Agreement

By submitting this pull request, I confirm that I have read and fully agree to the Contributor License Agreement (CLA), and I am providing my contributions under its terms.

💘 Generated with Crush


🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/open-webui/pull/18854 **Author:** [@fsmw](https://github.com/fsmw) **Created:** 11/1/2025 **Status:** ❌ Closed **Base:** `dev` ← **Head:** `fix/17087-chinese-punctuation-conversion` --- ### 📝 Commits (1) - [`9c9092f`](https://github.com/open-webui/open-webui/commit/9c9092f2c22d5b55a1663037fdfa38476849a20c) fix(retrieval): preserve Chinese punctuation in uploaded documents ### 📊 Changes **1 file changed** (+1 additions, -2 deletions) <details> <summary>View changed files</summary> 📝 `backend/open_webui/retrieval/loaders/main.py` (+1 -2) </details> ### 📄 Description ## Summary Remove ftfy.fix_text() processing that converts Chinese punctuation marks to English equivalents, preserving cultural content integrity. ## Issue Fixes #17087 - The Chinese punctuation in the uploaded text file will be automatically converted to English punctuation marks. ## Changes Made - [x] Removed ftfy.fix_text() processing from document upload pipeline - [x] Preserved Chinese punctuation marks (,。:;!?《》【】「」) in uploaded files - [x] Maintained existing text processing for other languages ## Testing - [x] Reproduced the issue with Chinese punctuation conversion - [x] Verified fix preserves Chinese punctuation in uploaded documents - [x] Tested with various Chinese text files - [x] Confirmed no regression in other language processing ## Additional Information This fix addresses cultural content preservation issues where Chinese punctuation was being automatically converted to English equivalents during file upload. ### Contributor License Agreement By submitting this pull request, I confirm that I have read and fully agree to the [Contributor License Agreement (CLA)](https://github.com/open-webui/open-webui/blob/main/CONTRIBUTOR_LICENSE_AGREEMENT), and I am providing my contributions under its terms. 💘 Generated with Crush --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
GiteaMirror added the pull-request label 2026-05-06 08:58:35 -05:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#63857