[PR #18851] [CLOSED] fix(retrieval): preserve Chinese punctuation in uploaded documents #79871

Closed
opened 2026-05-13 13:53:05 -05:00 by GiteaMirror · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/open-webui/open-webui/pull/18851
Author: @fsmw
Created: 11/1/2025
Status: ❌ Closed

Base: main ← Head: fix/17087-chinese-punctuation-conversion


📝 Commits (1)

  • 9c9092f fix(retrieval): preserve Chinese punctuation in uploaded documents

📊 Changes

1 file changed (+1 additions, -2 deletions)

View changed files

📝 backend/open_webui/retrieval/loaders/main.py (+1 -2)

📄 Description

Summary

Remove ftfy.fix_text() processing that converts Chinese punctuation marks to English equivalents, preserving cultural content integrity.

Issue

Fixes #17087 - The Chinese punctuation in the uploaded text file will be automatically converted to English punctuation marks.

Changes Made

  • Removed ftfy.fix_text() processing from document upload pipeline
  • Preserved Chinese punctuation marks (,。:;!?《》【】「」) in uploaded files
  • Maintained existing text processing for other languages

Testing

  • Reproduced the issue with Chinese punctuation conversion
  • Verified fix preserves Chinese punctuation in uploaded documents
  • Tested with various Chinese text files
  • Confirmed no regression in other language processing

Additional Information

This fix addresses cultural content preservation issues where Chinese punctuation was being automatically converted to English equivalents during file upload.

I agree to the terms of the Contributor License Agreement (CLA) and grant the necessary rights for my contributions to be included in this open-source project.

💘 Generated with Crush


🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/open-webui/pull/18851 **Author:** [@fsmw](https://github.com/fsmw) **Created:** 11/1/2025 **Status:** ❌ Closed **Base:** `main` ← **Head:** `fix/17087-chinese-punctuation-conversion` --- ### 📝 Commits (1) - [`9c9092f`](https://github.com/open-webui/open-webui/commit/9c9092f2c22d5b55a1663037fdfa38476849a20c) fix(retrieval): preserve Chinese punctuation in uploaded documents ### 📊 Changes **1 file changed** (+1 additions, -2 deletions) <details> <summary>View changed files</summary> 📝 `backend/open_webui/retrieval/loaders/main.py` (+1 -2) </details> ### 📄 Description ## Summary Remove ftfy.fix_text() processing that converts Chinese punctuation marks to English equivalents, preserving cultural content integrity. ## Issue Fixes #17087 - The Chinese punctuation in the uploaded text file will be automatically converted to English punctuation marks. ## Changes Made - [x] Removed ftfy.fix_text() processing from document upload pipeline - [x] Preserved Chinese punctuation marks (,。:;!?《》【】「」) in uploaded files - [x] Maintained existing text processing for other languages ## Testing - [x] Reproduced the issue with Chinese punctuation conversion - [x] Verified fix preserves Chinese punctuation in uploaded documents - [x] Tested with various Chinese text files - [x] Confirmed no regression in other language processing ## Additional Information This fix addresses cultural content preservation issues where Chinese punctuation was being automatically converted to English equivalents during file upload. I agree to the terms of the Contributor License Agreement (CLA) and grant the necessary rights for my contributions to be included in this open-source project. 💘 Generated with Crush --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
GiteaMirror added the pull-request label 2026-05-13 13:53:05 -05:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#79871