github-starred/open-webui
2
0
Fork 0
mirror of https://github.com/open-webui/open-webui.git synced 2026-11-09 10:50:35 -06:00
Code Issues 2.4k Packages Projects Releases 129 Wiki Activity

[GH-ISSUE #17605] issue: non_ASCII UTF-8 always encoded in payload sent to model (leading to bad tokenization) #33871

New Issue
Closed
opened 2026-04-25 07:44:46 -05:00 by GiteaMirror · 1 comment
GiteaMirror commented 2026-04-25 07:44:46 -05:00
Owner
Copy Link
Copy Source

Originally created by @mramendi on GitHub (Sep 19, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/17605

Check Existing Issues

  • I have searched for any existing and/or related issues.
  • I have searched for any existing and/or related discussions.
  • I am using the latest version of Open WebUI.

Installation Method

Docker

Open WebUI Version

v0.6.30

Ollama Version (if applicable)

No response

Operating System

RHEL 9

Browser (if applicable)

No response

Confirmation

  • I have read and followed all instructions in README.md.
  • I am using the latest version of both Open WebUI and Ollama.
  • I have included the browser console logs.
  • I have included the Docker container logs.
  • I have provided every relevant configuration, setting, and environment variable used in my setup.
  • I have clearly listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc).
  • I have documented step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation. My steps:
  • Start with the initial platform/version/OS and dependencies used,
  • Specify exact install/launch/configure commands,
  • List URLs visited, user input (incl. example values/emails/passwords if needed),
  • Describe all options and toggles enabled or changed,
  • Include any files or environmental changes,
  • Identify the expected and actual result at each stage,
  • Ensure any reasonably skilled user can follow and hit the same issue.

Expected Behavior

When I enter a message in non-ASCII UTF-8, for example, Cyrillic, it should be sent to the model unmodified as UTF8, for example, "привет!". Or at least there should be a switch to enable this behaviour.

Actual Behavior

When I enter a message in non-ASCII UTF-8, for example, Cyrillic, it is sent to the model encoded with \u, for example, "\u043f\u0440\u0438\u0432\u0435\u0442!"

Steps to Reproduce

This is my particular container setup. I don't believe this is really caused by Podman/Quadlet, but I am including the relevant config files anyway. The upload failed so I paste them at the end of this message. So, steps in my specific case:

  1. Start with RHEL9 - that is what I have, but Fedora, CentOS Stream, Rocky, Alma should all work. Just be sure you're in "Red Hat world" so you have Podman and Quadlet.
  2. Make sure the following packages are installed: podman, podman-compose, podman-plugins.
  3. Place the files that I pasted below into the $HOME/.config/containers/systemd directory. Modify for file paths and API/database endpoints/keys (note my setup injects API keys and the PostgreSQL password in an start-wrapper.sh script that is intentionally not pasted - it only sets the environment variables OPENAI_API_KEY and DATABASE_URL then execs the normal start.sh)
  4. start the service with "systtemctl --user start openwebui-with-playwright
  5. Either provide HTTPS with a reverse proxy, or for test purposes modify the files to publish the port on the host
  6. Access OpenWebUI through a browser normally and go through the setup
  7. Set up some way of intercepting and logging the request body. I use a simple LiteLLM callback; I can shjare it if necessary.
  8. Click New chat and select a model (your API endpoint must provide at least one model)
  9. Send the folowing message: "привет!"
  10. Review the logged message body and notice the message is encoded as "\u043f\u0440\u0438\u0432\u0435\u0442!"

openwebui-with-playwright.yaml

apiVersion: v1
kind: Pod
metadata:
name: openwebui-playwright
spec:
containers:

---------- OpenWebUI ----------

  • name: openwebui
    image: ghcr.io/open-webui/open-webui:main
    env:

    • name: OPENAI_API_BASE_URL
      value: "http://host.containers.internal:4000"
    • name: WEB_LOADER_ENGINE
      value: playwright
    • name: PLAYWRIGHT_WS_URL
      value: "ws://localhost:3000" # inside the pod => localhost
    • name: PYTHONPATH
      value: /app/backend/data/site-packages
    • name: USER_AGENT
      value: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36
    • name: PYTHONIOENCODING
      value: utf-8
    • name: JSON_ENSURE_ASCII
      value: False
      command: ["/config/start-wrapper.sh"]

    ports:

    • containerPort: 8080
      volumeMounts:
    • name: data
      mountPath: /app/backend/data
    • name: master-key
      mountPath: /run/secrets/litellm-master.key
      subPath: litellm-master.key
      readOnly: true
    • name: db-pass
      mountPath: /run/secrets/openwebui-db.pass
      subPath: openwebui.pass
      readOnly: true
    • name: config-files
      mountPath: /config/start-wrapper.sh
      subPath: start-wrapper.sh
      readOnly: true

---------- Playwright browser ----------

  • name: playwright
    image: mcr.microsoft.com/playwright:v1.49.1-noble
    command: [ "npx", "playwright@1.49.1", "run-server", "--port=3000", "--host=0.0.0.0" ]
    ports:
    • containerPort: 3000 # only inside the pod

volumes:

  • name: data
    hostPath:
    path: /srv/hdd/openwebui-files
    type: DirectoryOrCreate
  • name: master-key
    hostPath:
    path: /home/admin/.config/ai
    type: Directory
  • name: db-pass
    hostPath:
    path: /home/admin/.config/ai
    type: Directory
  • name: config-files
    hostPath:
    path: /home/admin/.config/ai
    type: Directory
    ===

openwebui-with-playwright.kube

[Unit]
Description=OpenWebUI + Playwright (internal only)
After=ai-network.service litellm.service postgresql.service
Wants=ai-network.service

[Kube]
Yaml=%h/.config/containers/systemd/openwebui-with-playwright.yaml
Network=ai.network
AutoUpdate=registry

[Install]
WantedBy=default.target

Logs & Screenshots

I see nothing relevant in the pod logs, just a series of GET and POST headers. The browser behaves normally.

Additional Information

No response

Originally created by @mramendi on GitHub (Sep 19, 2025). Original GitHub issue: https://github.com/open-webui/open-webui/issues/17605 ### Check Existing Issues - [x] I have searched for any existing and/or related issues. - [x] I have searched for any existing and/or related discussions. - [x] I am using the latest version of Open WebUI. ### Installation Method Docker ### Open WebUI Version v0.6.30 ### Ollama Version (if applicable) _No response_ ### Operating System RHEL 9 ### Browser (if applicable) _No response_ ### Confirmation - [x] I have read and followed all instructions in `README.md`. - [x] I am using the latest version of **both** Open WebUI and Ollama. - [x] I have included the browser console logs. - [x] I have included the Docker container logs. - [x] I have **provided every relevant configuration, setting, and environment variable used in my setup.** - [x] I have clearly **listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup** (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc). - [x] I have documented **step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation**. My steps: - Start with the initial platform/version/OS and dependencies used, - Specify exact install/launch/configure commands, - List URLs visited, user input (incl. example values/emails/passwords if needed), - Describe all options and toggles enabled or changed, - Include any files or environmental changes, - Identify the expected and actual result at each stage, - Ensure any reasonably skilled user can follow and hit the same issue. ### Expected Behavior When I enter a message in non-ASCII UTF-8, for example, Cyrillic, it should be sent to the model unmodified as UTF8, for example, "привет!". Or at least there should be a switch to enable this behaviour. ### Actual Behavior When I enter a message in non-ASCII UTF-8, for example, Cyrillic, it is sent to the model encoded with \u, for example, "\u043f\u0440\u0438\u0432\u0435\u0442!" ### Steps to Reproduce This is my particular container setup. I don't believe this is really caused by Podman/Quadlet, but I am including the relevant config files anyway. The upload failed so I paste them at the end of this message. So, steps in my specific case: 1. Start with RHEL9 - that is what I have, but Fedora, CentOS Stream, Rocky, Alma should all work. Just be sure you're in "Red Hat world" so you have Podman and Quadlet. 2. Make sure the following packages are installed: podman, podman-compose, podman-plugins. 3. Place the files that I pasted below into the $HOME/.config/containers/systemd directory. Modify for file paths and API/database endpoints/keys (note my setup injects API keys and the PostgreSQL password in an start-wrapper.sh script that is intentionally not pasted - it only sets the environment variables OPENAI_API_KEY and DATABASE_URL then execs the normal start.sh) 4. start the service with "systtemctl --user start openwebui-with-playwright 5. Either provide HTTPS with a reverse proxy, or for test purposes modify the files to publish the port on the host 6. Access OpenWebUI through a browser normally and go through the setup 7. Set up some way of intercepting and logging the request body. I use a simple LiteLLM callback; I can shjare it if necessary. 8. Click New chat and select a model (your API endpoint must provide at least one model) 9. Send the folowing message: "привет!" 10. Review the logged message body and notice the message is encoded as "\u043f\u0440\u0438\u0432\u0435\u0442!" openwebui-with-playwright.yaml === apiVersion: v1 kind: Pod metadata: name: openwebui-playwright spec: containers: # ---------- OpenWebUI ---------- - name: openwebui image: ghcr.io/open-webui/open-webui:main env: - name: OPENAI_API_BASE_URL value: "http://host.containers.internal:4000" - name: WEB_LOADER_ENGINE value: playwright - name: PLAYWRIGHT_WS_URL value: "ws://localhost:3000" # inside the pod => localhost - name: PYTHONPATH value: /app/backend/data/site-packages - name: USER_AGENT value: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36 - name: PYTHONIOENCODING value: utf-8 - name: JSON_ENSURE_ASCII value: False command: ["/config/start-wrapper.sh"] ports: - containerPort: 8080 volumeMounts: - name: data mountPath: /app/backend/data - name: master-key mountPath: /run/secrets/litellm-master.key subPath: litellm-master.key readOnly: true - name: db-pass mountPath: /run/secrets/openwebui-db.pass subPath: openwebui.pass readOnly: true - name: config-files mountPath: /config/start-wrapper.sh subPath: start-wrapper.sh readOnly: true # ---------- Playwright browser ---------- - name: playwright image: mcr.microsoft.com/playwright:v1.49.1-noble command: [ "npx", "playwright@1.49.1", "run-server", "--port=3000", "--host=0.0.0.0" ] ports: - containerPort: 3000 # only inside the pod volumes: - name: data hostPath: path: /srv/hdd/openwebui-files type: DirectoryOrCreate - name: master-key hostPath: path: /home/admin/.config/ai type: Directory - name: db-pass hostPath: path: /home/admin/.config/ai type: Directory - name: config-files hostPath: path: /home/admin/.config/ai type: Directory === openwebui-with-playwright.kube === [Unit] Description=OpenWebUI + Playwright (internal only) After=ai-network.service litellm.service postgresql.service Wants=ai-network.service [Kube] Yaml=%h/.config/containers/systemd/openwebui-with-playwright.yaml Network=ai.network AutoUpdate=registry [Install] WantedBy=default.target == ### Logs & Screenshots I see nothing relevant in the pod logs, just a series of GET and POST headers. The browser behaves normally. ### Additional Information _No response_
GiteaMirror added the bug label 2026-04-25 07:44:46 -05:00
GiteaMirror commented 2026-04-25 07:44:50 -05:00
Author
Owner
Copy Link
Copy Source

@tjbck commented on GitHub (Sep 23, 2025):

Unable to reproduce, most likely caused by the logging part. If not PR welcome!

<!-- gh-comment-id:3325648207 --> @tjbck commented on GitHub (Sep 23, 2025): Unable to reproduce, most likely caused by the logging part. If not PR welcome!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
confirmed
confirmed issue
core
documentation
enhancement
good first issue
help wanted
non-core
pull-request
Mirrored from GitHub Pull Request
 python
question
testing wanted
No labels bug
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
GiteaMirror ninjasurge (Kaehlen Rog)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#33871