issue: stale socket-connections in state CLOSE-WAIT when run with docker and nginx reverse-proxy #5688

Open
opened 2025-11-11 16:29:32 -06:00 by GiteaMirror · 11 comments
Owner

Originally created by @mx-tc on GitHub (Jul 2, 2025).

Check Existing Issues

  • I have searched the existing issues and discussions.
  • I am using the latest version of Open WebUI.

Installation Method

Docker

Open WebUI Version

v0.6.15

Ollama Version (if applicable)

0.9.2

Operating System

Linux Mint 22.1

Browser (if applicable)

No response

Confirmation

  • I have read and followed all instructions in README.md.
  • I am using the latest version of both Open WebUI and Ollama.
  • I have included the browser console logs.
  • I have included the Docker container logs.
  • I have provided every relevant configuration, setting, and environment variable used in my setup.
  • I have clearly listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc).
  • I have documented step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation. My steps:
  • Start with the initial platform/version/OS and dependencies used,
  • Specify exact install/launch/configure commands,
  • List URLs visited, user input (incl. example values/emails/passwords if needed),
  • Describe all options and toggles enabled or changed,
  • Include any files or environmental changes,
  • Identify the expected and actual result at each stage,
  • Ensure any reasonably skilled user can follow and hit the same issue.

Expected Behavior

Normal usage of AI-chat for multiple users should work reliable.

Actual Behavior

AI-chat works as expected just after fresh start of the open-webui docker-container, but after some time the system stops working; ~ 2 hours of concurrent usage by ~3 users.

After some investigation I observed an increased amount of open sockets in state CLOSE-WAIT owned by open-webui.
ATTENTION: using tools like ss or netstat on the host won't list those sockets, as they are scoped into docker-containers own network-namespace; to observe sockets use as root-user something like:

nsenter -t $(docker inspect -f '{{.State.Pid}}' open-webui-open-webui-1) -n ss --tcp -n

This results in something like

State                           Recv-Q                      Send-Q                                           Local Address:Port                                            Peer Address:Port                      Process                     
CLOSE-WAIT                      2263                        0                                                   172.18.0.3:8080                                              172.18.0.2:50612                                                 
CLOSE-WAIT                      1489                        0                                                   172.18.0.3:8080                                              172.18.0.2:49396                                                 
CLOSE-WAIT                      2278                        0                                                   172.18.0.3:8080                                              172.18.0.2:36992                                                 
CLOSE-WAIT                      1505                        0                                                   172.18.0.3:8080                                              172.18.0.2:49432                                                 
CLOSE-WAIT                      1615                        0                                                   172.18.0.3:8080                                              172.18.0.2:38194                                                 
CLOSE-WAIT                      3803                        0                                                   172.18.0.3:8080                                              172.18.0.2:49394                                                 
ESTAB                           0                           0                                                   172.18.0.3:34614                                             172.17.0.1:11434                                                 
ESTAB                           0                           0                                                   172.18.0.3:34596                                             172.17.0.1:11434                                                 

Also there are many curl and jq processes that can be seen with ps aux, propably originating from stuck docker HEALTHCHECK

Steps to Reproduce

Note: Not sure if it is precisely reproducable, following description only show the setup in which I observed it.

  1. Linux Mint 22.1 + Docker-CE + ollama (Linux native: curl -fsSL https://ollama.com/install.sh | sh)
  2. Use provided docker-compose.yml and NGINX reverse-proxy-configuration
  3. Start service-stack; docker compose up -d
  4. Use Webinterface for AI-chat for some time (about two hours)

Logs & Screenshots

Attached screenshot show Prometheus / Grafana metrics showing high allocated socket count:

Image

Additional Information

The setup used nearly fits the scheme Ollama on Host, Open WebUI in Container (Linux) from official documentation.

Additionally a reverse-proxy started as docker-container is configured in front of it for HTTPS-termination.

NOTE: the community-provided configuration HTTPS using Nginx wasn't sufficient and had to be supplemented with directives to allow for web-sockets being passed thru the reverse-proxy.

The whole stack is started using docker-compose.

docker-compose.yaml


# ATTENTION: for access to ollama on host it needs addidtional configuration: in /etc/systemd/system/ollama.service add followin env-variables:
#Environment="OLLAMA_HOST=0.0.0.0"
#Environment="OLLAMA_ORIGINS=http://localhost:11434,http://172.17.0.1"

# open-webui configuration by env-variables: https://docs.openwebui.com/getting-started/env-configuration/#webui_secret_key

services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    restart: always
# direct access to open-webui only during setup
#    ports:
#      - "8080:8080"
    volumes:
      - data:/app/backend/data
      - "/etc/ssl/certs/ldap-server-cert.pem:/etc/ssl/certs/ldap-server-cert.pem:ro"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    environment:
      WEBUI_URL: "https://ai-server.intranet"
      WEBUI_SECRET_KEY: "${WEBUI_SECRET_KEY}"
      ENABLE_OLLAMA_API: "true"
      OLLAMA_BASE_URLS: "http://host.docker.internal:11434"
      ENABLE_OPENAI_API: "false"
      ENABLE_SIGNUP: "false"
      DEFAULT_USER_ROLE: "admin"
      ENABLE_LDAP: "true"
      LDAP_APP_DN: "${LDAP_APP_DN}"
      LDAP_APP_PASSWORD: "${LDAP_APP_PASSWORD}"
      LDAP_SERVER_HOST: "ldap.intranet"
      LDAP_SERVER_PORT: "636"
      LDAP_SEARCH_BASE: "${LDAP_SEARCH_BASE}"
      LDAP_ATTRIBUTE_FOR_MAIL: "mail"
      LDAP_ATTRIBUTE_FOR_USERNAME: "uid"
      LDAP_CA_CERT_FILE: "/etc/ssl/certs/ldap-server-cert.pem"
    # FIXME: after some time the healtcheck stops working; in this state it seems there are many sockets in state CLOSE_WAIT and many stuck curl-instances can be seen
    # original healthcheck from Dockerfile here: https://github.com/open-webui/open-webui/blob/b5f4c85bb196c16a775802907aedd87366f58b0f/Dockerfile#L171
    # propably should use a curl-timeout to avoid spawned curl-instances that never finish:
    # "curl --silent --fail --max-time 5 http://localhost:${PORT:-8080}/health | jq -ne 'input.status == true' || exit 1"
    # disabled for now
    healthcheck:
      disable: true

  nginx:
    image: nginx:latest
    restart: always
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx.conf:/etc/nginx/conf.d/default.conf
    secrets:
      - source: openwebui_cert
        target: /etc/ssl/certs/ai-server.intranet-cert.pem
      - source: openwebui_cert_key
        target: /etc/ssl/private/ai-server.intranet-key.pem

volumes:
  data:

secrets:
  openwebui_cert:
    file: ./ai-server.intranet-cert.pem
  openwebui_cert_key:
    file: ./ai-server.intranet-key.pem

nginx.conf:

server {
    listen 80;
    listen 8080;
    server_name ai-server.intranet;

    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    server_name ai-server.intranet;

    ssl_certificate /etc/ssl/certs/ai-server.intranet-cert.pem;
    ssl_certificate_key /etc/ssl/private/ai-server.intranet-key.pem;
    ssl_protocols TLSv1.2 TLSv1.3;

    location / {
        proxy_pass http://open-webui:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # (Optional) Disable proxy buffering for better streaming response from models
        proxy_buffering off;

        # (Optional) Increase max request size for large attachments and long audio messages
        client_max_body_size 20M;
        proxy_read_timeout 10m;

        proxy_cache_bypass $http_upgrade;
    }
}
Originally created by @mx-tc on GitHub (Jul 2, 2025). ### Check Existing Issues - [x] I have searched the existing issues and discussions. - [x] I am using the latest version of Open WebUI. ### Installation Method Docker ### Open WebUI Version v0.6.15 ### Ollama Version (if applicable) 0.9.2 ### Operating System Linux Mint 22.1 ### Browser (if applicable) _No response_ ### Confirmation - [x] I have read and followed all instructions in `README.md`. - [x] I am using the latest version of **both** Open WebUI and Ollama. - [x] I have included the browser console logs. - [x] I have included the Docker container logs. - [x] I have **provided every relevant configuration, setting, and environment variable used in my setup.** - [x] I have clearly **listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup** (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc). - [x] I have documented **step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation**. My steps: - Start with the initial platform/version/OS and dependencies used, - Specify exact install/launch/configure commands, - List URLs visited, user input (incl. example values/emails/passwords if needed), - Describe all options and toggles enabled or changed, - Include any files or environmental changes, - Identify the expected and actual result at each stage, - Ensure any reasonably skilled user can follow and hit the same issue. ### Expected Behavior Normal usage of AI-chat for multiple users should work reliable. ### Actual Behavior AI-chat works as expected just after fresh start of the open-webui docker-container, but after some time the system stops working; ~ 2 hours of concurrent usage by ~3 users. After some investigation I observed an increased amount of open sockets in state CLOSE-WAIT owned by open-webui. ATTENTION: using tools like `ss` or `netstat` on the host won't list those sockets, as they are scoped into docker-containers own network-namespace; to observe sockets use as root-user something like: ``` nsenter -t $(docker inspect -f '{{.State.Pid}}' open-webui-open-webui-1) -n ss --tcp -n ``` This results in something like ``` State Recv-Q Send-Q Local Address:Port Peer Address:Port Process CLOSE-WAIT 2263 0 172.18.0.3:8080 172.18.0.2:50612 CLOSE-WAIT 1489 0 172.18.0.3:8080 172.18.0.2:49396 CLOSE-WAIT 2278 0 172.18.0.3:8080 172.18.0.2:36992 CLOSE-WAIT 1505 0 172.18.0.3:8080 172.18.0.2:49432 CLOSE-WAIT 1615 0 172.18.0.3:8080 172.18.0.2:38194 CLOSE-WAIT 3803 0 172.18.0.3:8080 172.18.0.2:49394 ESTAB 0 0 172.18.0.3:34614 172.17.0.1:11434 ESTAB 0 0 172.18.0.3:34596 172.17.0.1:11434 ``` Also there are many `curl` and `jq` processes that can be seen with `ps aux`, propably originating from stuck docker HEALTHCHECK ### Steps to Reproduce Note: Not sure if it is precisely reproducable, following description only show the setup in which I observed it. 1. Linux Mint 22.1 + Docker-CE + *ollama* (Linux native: `curl -fsSL https://ollama.com/install.sh | sh`) 3. Use provided `docker-compose.yml` and NGINX reverse-proxy-configuration 5. Start service-stack; ` docker compose up -d` 6. Use Webinterface for AI-chat for some time (about two hours) ### Logs & Screenshots Attached screenshot show *Prometheus* / *Grafana* metrics showing high allocated socket count: ![Image](https://github.com/user-attachments/assets/15615803-064b-4ca6-b32d-1c1469f9f7b2) ### Additional Information The setup used nearly fits the scheme [Ollama on Host, Open WebUI in Container (Linux)](https://docs.openwebui.com/getting-started/advanced-topics/network-diagrams#ollama-on-host-open-webui-in-container-linux) from official documentation. Additionally a reverse-proxy started as docker-container is configured in front of it for HTTPS-termination. NOTE: the community-provided configuration [HTTPS using Nginx](https://docs.openwebui.com/tutorials/https-nginx/) wasn't sufficient and had to be supplemented with directives to allow for web-sockets being passed thru the reverse-proxy. The whole stack is started using *docker-compose*. `docker-compose.yaml` ``` # ATTENTION: for access to ollama on host it needs addidtional configuration: in /etc/systemd/system/ollama.service add followin env-variables: #Environment="OLLAMA_HOST=0.0.0.0" #Environment="OLLAMA_ORIGINS=http://localhost:11434,http://172.17.0.1" # open-webui configuration by env-variables: https://docs.openwebui.com/getting-started/env-configuration/#webui_secret_key services: open-webui: image: ghcr.io/open-webui/open-webui:main restart: always # direct access to open-webui only during setup # ports: # - "8080:8080" volumes: - data:/app/backend/data - "/etc/ssl/certs/ldap-server-cert.pem:/etc/ssl/certs/ldap-server-cert.pem:ro" extra_hosts: - "host.docker.internal:host-gateway" environment: WEBUI_URL: "https://ai-server.intranet" WEBUI_SECRET_KEY: "${WEBUI_SECRET_KEY}" ENABLE_OLLAMA_API: "true" OLLAMA_BASE_URLS: "http://host.docker.internal:11434" ENABLE_OPENAI_API: "false" ENABLE_SIGNUP: "false" DEFAULT_USER_ROLE: "admin" ENABLE_LDAP: "true" LDAP_APP_DN: "${LDAP_APP_DN}" LDAP_APP_PASSWORD: "${LDAP_APP_PASSWORD}" LDAP_SERVER_HOST: "ldap.intranet" LDAP_SERVER_PORT: "636" LDAP_SEARCH_BASE: "${LDAP_SEARCH_BASE}" LDAP_ATTRIBUTE_FOR_MAIL: "mail" LDAP_ATTRIBUTE_FOR_USERNAME: "uid" LDAP_CA_CERT_FILE: "/etc/ssl/certs/ldap-server-cert.pem" # FIXME: after some time the healtcheck stops working; in this state it seems there are many sockets in state CLOSE_WAIT and many stuck curl-instances can be seen # original healthcheck from Dockerfile here: https://github.com/open-webui/open-webui/blob/b5f4c85bb196c16a775802907aedd87366f58b0f/Dockerfile#L171 # propably should use a curl-timeout to avoid spawned curl-instances that never finish: # "curl --silent --fail --max-time 5 http://localhost:${PORT:-8080}/health | jq -ne 'input.status == true' || exit 1" # disabled for now healthcheck: disable: true nginx: image: nginx:latest restart: always ports: - "80:80" - "443:443" volumes: - ./nginx.conf:/etc/nginx/conf.d/default.conf secrets: - source: openwebui_cert target: /etc/ssl/certs/ai-server.intranet-cert.pem - source: openwebui_cert_key target: /etc/ssl/private/ai-server.intranet-key.pem volumes: data: secrets: openwebui_cert: file: ./ai-server.intranet-cert.pem openwebui_cert_key: file: ./ai-server.intranet-key.pem ``` `nginx.conf`: ``` server { listen 80; listen 8080; server_name ai-server.intranet; return 301 https://$host$request_uri; } server { listen 443 ssl; server_name ai-server.intranet; ssl_certificate /etc/ssl/certs/ai-server.intranet-cert.pem; ssl_certificate_key /etc/ssl/private/ai-server.intranet-key.pem; ssl_protocols TLSv1.2 TLSv1.3; location / { proxy_pass http://open-webui:8080; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; # (Optional) Disable proxy buffering for better streaming response from models proxy_buffering off; # (Optional) Increase max request size for large attachments and long audio messages client_max_body_size 20M; proxy_read_timeout 10m; proxy_cache_bypass $http_upgrade; } } ```
GiteaMirror added the bug label 2025-11-11 16:29:32 -06:00
Author
Owner

@mx-tc commented on GitHub (Jul 3, 2025):

To further refine how we use AI-chat in case it matters:

  • direct chat using web-application
  • Visual Studio Code / IntelliJ using Continue-plugin, which communicates using ollama-API endpoint provided from open-webui: https://ai-server.intranet/ollama/
@mx-tc commented on GitHub (Jul 3, 2025): To further refine how we use AI-chat in case it matters: - direct chat using web-application - _Visual Studio Code_ / _IntelliJ_ using _Continue_-plugin, which communicates using _ollama_-API endpoint provided from open-webui: `https://ai-server.intranet/ollama/`
Author
Owner

@expilu commented on GitHub (Jul 3, 2025):

I'm having a similar problem with v0.6.15. With a similar set up using open webui as "proxy" for the ollama api for the Continue plugin.
It works for a while after restart, but then it stops working and the container is unhealthy.
In my case I have a traefik in front instead of nginx.

@expilu commented on GitHub (Jul 3, 2025): I'm having a similar problem with v0.6.15. With a similar set up using open webui as "proxy" for the ollama api for the Continue plugin. It works for a while after restart, but then it stops working and the container is unhealthy. In my case I have a traefik in front instead of nginx.
Author
Owner

@rgaricano commented on GitHub (Jul 3, 2025):

Related & addresed: https://github.com/open-webui/open-webui/issues/14945

@rgaricano commented on GitHub (Jul 3, 2025): Related & addresed: https://github.com/open-webui/open-webui/issues/14945
Author
Owner

@BAngelis commented on GitHub (Jul 7, 2025):

I'm having a very similar problem with v0.6.15. One user, 10 - 20 prompts, stops responding. Used both Caddy and nginx reverse proxys, similar behavior from the user's perspective.

@BAngelis commented on GitHub (Jul 7, 2025): I'm having a very similar problem with v0.6.15. One user, 10 - 20 prompts, stops responding. Used both Caddy and nginx reverse proxys, similar behavior from the user's perspective.
Author
Owner

@AlexanderZhk commented on GitHub (Oct 7, 2025):

+1

Same setup here, nginx as reverse proxy for https termination.
At point 1 OWUI became unresponsive, at point 2 we implemented restart on unhealthy, but the issue persists:

Image

relevant nginx.conf:

        location / {
            proxy_pass http://open-webui-ai;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
        }

We're running with postgres and redis. OWUI v0.6.32

@AlexanderZhk commented on GitHub (Oct 7, 2025): +1 Same setup here, nginx as reverse proxy for https termination. At point 1 OWUI became unresponsive, at point 2 we implemented restart on unhealthy, but the issue persists: <img width="800" height="427" alt="Image" src="https://github.com/user-attachments/assets/aab80908-c3e8-4256-9318-bde421b309e3" /> relevant nginx.conf: ``` location / { proxy_pass http://open-webui-ai; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; } ``` We're running with postgres and redis. OWUI v0.6.32
Author
Owner

@unixshaman commented on GitHub (Oct 13, 2025):

I think I've encountered the same issue. The OWUI version was v0.6.32, but then I updated to OWUI v0.6.33, and it's unclear whether it was an external influence or a version bug—but now it seems to be working.

@unixshaman commented on GitHub (Oct 13, 2025): I think I've encountered the same issue. The OWUI version was v0.6.32, but then I updated to OWUI v0.6.33, and it's unclear whether it was an external influence or a version bug—but now it seems to be working.
Author
Owner

@rgaricano commented on GitHub (Oct 13, 2025):

user & or session opened? in what conditions happend the peak? starting? with some specific request?
it seem a race condition of all clients trying to refresh its connections or massive requests (e.g. by the issued retrieving logic)

@rgaricano commented on GitHub (Oct 13, 2025): user & or session opened? in what conditions happend the peak? starting? with some specific request? it seem a race condition of all clients trying to refresh its connections or massive requests (e.g. by the issued retrieving logic)
Author
Owner

@AlexanderZhk commented on GitHub (Oct 13, 2025):

user & or session opened? in what conditions happend the peak? starting? with some specific request? it seem a race condition of all clients trying to refresh its connections or massive requests (e.g. by the issued retrieving logic)

In my case it was unclear, OWUI logs had nothing useful, but I didn't try changing log levels (this deployment, where the issue happened, can see ~200 real users peaks, uncontrolled environment). It's hard to reproduce, seems to happen sporadically for me. My wild guess is some clients' browser could be not closing websockets properly.

Wasn't able to narrow down or reproduce with sub ~25 users in a controlled environment.

Can you go into massive requests (e.g. by the issued retrieving logic) into more detail? We have RAG enabled, so multiple users using or indexing RAG is entirely possible. (No custom filters or pipes)

@AlexanderZhk commented on GitHub (Oct 13, 2025): > user & or session opened? in what conditions happend the peak? starting? with some specific request? it seem a race condition of all clients trying to refresh its connections or massive requests (e.g. by the issued retrieving logic) In my case it was unclear, OWUI logs had nothing useful, but I didn't try changing log levels (this deployment, where the issue happened, can see ~200 real users peaks, uncontrolled environment). It's hard to reproduce, seems to happen sporadically for me. My wild guess is some clients' browser could be not closing websockets properly. Wasn't able to narrow down or reproduce with sub ~25 users in a controlled environment. Can you go into massive requests (e.g. by the issued retrieving logic) into more detail? We have RAG enabled, so multiple users using or indexing RAG is entirely possible. (No custom filters or pipes)
Author
Owner

@rgaricano commented on GitHub (Oct 13, 2025):

Can you go into massive requests (e.g. by the issued retrieving logic) into more detail? We have RAG enabled, so multiple users using or indexing RAG is entirely possible. (No custom filters or pipes)

https://github.com/open-webui/open-webui/issues/18133

@rgaricano commented on GitHub (Oct 13, 2025): > Can you go into massive requests (e.g. by the issued retrieving logic) into more detail? We have RAG enabled, so multiple users using or indexing RAG is entirely possible. (No custom filters or pipes) https://github.com/open-webui/open-webui/issues/18133
Author
Owner

@Daikku commented on GitHub (Oct 22, 2025):

+1

same problem

@Daikku commented on GitHub (Oct 22, 2025): +1 same problem
Author
Owner

@nikita-makarevich commented on GitHub (Oct 27, 2025):

I have the same problem, I'm really looking forward to a solution

@nikita-makarevich commented on GitHub (Oct 27, 2025): I have the same problem, I'm really looking forward to a solution
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#5688