github-starred/open-webui
2
0
Fork 0
mirror of https://github.com/open-webui/open-webui.git synced 2026-07-16 23:21:44 -05:00
Code Issues 2.4k Packages Projects Releases 129 Wiki Activity

[GH-ISSUE #14208] issue: Slow query response from Open-webui #120492

New Issue
Closed
opened 2026-05-20 22:06:54 -05:00 by GiteaMirror · 0 comments
GiteaMirror commented 2026-05-20 22:06:54 -05:00
Owner
Copy Link

Originally created by @yoshihito-tatano on GitHub (May 23, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/14208

Check Existing Issues

  • I have searched the existing issues and discussions.
  • I am using the latest version of Open WebUI.

Installation Method

Git Clone

Open WebUI Version

ghcr.io/open-webui/open-webui:v0.6.10

Ollama Version (if applicable)

ollama/ollama:0.7.1-rc0

Operating System

Ubuntu 22.04

Browser (if applicable)

No response

Confirmation

  • I have read and followed all instructions in README.md.
  • I am using the latest version of both Open WebUI and Ollama.
  • I have included the browser console logs.
  • I have included the Docker container logs.
  • I have listed steps to reproduce the bug in detail.

Expected Behavior

kubectl apply -k ./kubernetes/manifest/gpu

I followed the documentation and deployed Open-webUI+Ollama using GPU
GPU information here.

`root@ollama-0:/# nvidia-smi
Fri May 23 00:17:52 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.144.03 Driver Version: 550.144.03 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA A100-PCIE-40GB On | 00000000:81:00.0 Off | 0 |
| N/A 34C P0 38W / 250W | 4MiB / 40960MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA A100-PCIE-40GB On | 00000000:A1:00.0 Off | 0 |
| N/A 34C P0 41W / 250W | 4MiB / 40960MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 2 NVIDIA A100-PCIE-40GB On | 00000000:C1:00.0 Off | 0 |
| N/A 35C P0 41W / 250W | 4MiB / 40960MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 3 NVIDIA A100-PCIE-40GB On | 00000000:E1:00.0 Off | 0 |
| N/A 33C P0 37W / 250W | 4MiB / 40960MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
`

Pulled the model on ollama and ran it, and it returns a response using the GPU as expected.
However, when I run the query from Open-WebUI, the results are quite slow. Also, it uses CPU as well as GPU.
root@ollama-0:/# ollama ps NAME ID SIZE PROCESSOR UNTIL llama-swallow:8b e4ef5a288c42 20 GB 33%/67% CPU/GPU 29 minutes from now

Actual Behavior

We expect Ollama to make full use of the GPU and return a fast response. However, I suspect that the CPU is actually being used unintentionally for some of the processing, which is slowing down the response.
This is only happening when executing from Open-webUI, and I would like to know if there is a way to control it.

Steps to Reproduce

  1. kubectl apply -k ./kubernetes/manifest/gpu
  2. pull the model ( I have tried several models llama3.3 and gemma3
  3. Use Open-webUI

400: Ollama: 400, message='Bad Request', url='http://ollama-service.open-webui.svc.cluster.local:11434/api/chat'
I could not run any of the Gemma3 models in my environment with the above error.

Logs & Screenshots

root@ollama-0:/# ollama ps NAME ID SIZE PROCESSOR UNTIL llama3.3:latest a6eb4748fd29 73 GB 64%/36% CPU/GPU 29 minutes from now

time=2025-05-23T00:34:02.124Z level=INFO source=routes.go:1205 msg="server config" env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:24574 OLLAMA_DEBUG:DEBUG OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:true OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:4 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:true ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" time=2025-05-23T00:34:02.129Z level=INFO source=images.go:463 msg="total blobs: 38" time=2025-05-23T00:34:02.130Z level=INFO source=images.go:470 msg="total unused blobs removed: 0" time=2025-05-23T00:34:02.131Z level=INFO source=routes.go:1258 msg="Listening on :11434 (version 0.7.1-rc0)" time=2025-05-23T00:34:02.131Z level=DEBUG source=sched.go:108 msg="starting llm scheduler" time=2025-05-23T00:34:02.131Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs" time=2025-05-23T00:34:02.160Z level=DEBUG source=gpu.go:98 msg="searching for GPU discovery libraries for NVIDIA" time=2025-05-23T00:34:02.160Z level=DEBUG source=gpu.go:501 msg="Searching for GPU library" name=libcuda.so* time=2025-05-23T00:34:02.160Z level=DEBUG source=gpu.go:525 msg="gpu library search" globs="[/usr/lib/ollama/libcuda.so* /usr/local/nvidia/lib/libcuda.so* /usr/local/nvidia/lib64/libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]" time=2025-05-23T00:34:02.162Z level=DEBUG source=gpu.go:558 msg="discovered GPU libraries" paths=[/usr/lib/x86_64-linux-gnu/libcuda.so.550.144.03] initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.144.03 dlsym: cuInit - 0x7fb2b16aebc0 dlsym: cuDriverGetVersion - 0x7fb2b16aebe0 dlsym: cuDeviceGetCount - 0x7fb2b16aec20 dlsym: cuDeviceGet - 0x7fb2b16aec00 dlsym: cuDeviceGetAttribute - 0x7fb2b16aed00 dlsym: cuDeviceGetUuid - 0x7fb2b16aec60 dlsym: cuDeviceGetName - 0x7fb2b16aec40 dlsym: cuCtxCreate_v3 - 0x7fb2b16aeee0 dlsym: cuMemGetInfo_v2 - 0x7fb2b16b8e20 dlsym: cuCtxDestroy - 0x7fb2b1713830 calling cuInit calling cuDriverGetVersion raw version 0x2f08 CUDA driver version: 12.4 calling cuDeviceGetCount device count 4 time=2025-05-23T00:34:02.474Z level=DEBUG source=gpu.go:125 msg="detected GPUs" count=4 library=/usr/lib/x86_64-linux-gnu/libcuda.so.550.144.03 [GPU-cf0928a8-fc00-9b78-4b6f-75b9f5fdfa2a] CUDA totalMem 40326mb [GPU-cf0928a8-fc00-9b78-4b6f-75b9f5fdfa2a] CUDA freeMem 39903mb [GPU-cf0928a8-fc00-9b78-4b6f-75b9f5fdfa2a] Compute Capability 8.0 [GPU-7274d657-2e08-df43-f06f-d4be16816916] CUDA totalMem 40326mb [GPU-7274d657-2e08-df43-f06f-d4be16816916] CUDA freeMem 39903mb [GPU-7274d657-2e08-df43-f06f-d4be16816916] Compute Capability 8.0 [GPU-d0f7b6db-6dcf-b4f2-f1a9-858a807e855e] CUDA totalMem 40326mb [GPU-d0f7b6db-6dcf-b4f2-f1a9-858a807e855e] CUDA freeMem 39903mb [GPU-d0f7b6db-6dcf-b4f2-f1a9-858a807e855e] Compute Capability 8.0 [GPU-2945b7e8-3bb0-14c1-99b9-986b6ce57629] CUDA totalMem 40326mb [GPU-2945b7e8-3bb0-14c1-99b9-986b6ce57629] CUDA freeMem 39903mb [GPU-2945b7e8-3bb0-14c1-99b9-986b6ce57629] Compute Capability 8.0 time=2025-05-23T00:34:03.689Z level=DEBUG source=amd_linux.go:419 msg="amdgpu driver not detected /sys/module/amdgpu" releasing cuda driver library time=2025-05-23T00:34:03.689Z level=INFO source=types.go:130 msg="inference compute" id=GPU-cf0928a8-fc00-9b78-4b6f-75b9f5fdfa2a library=cuda variant=v12 compute=8.0 driver=12.4 name="NVIDIA A100-PCIE-40GB" total="39.4 GiB" available="39.0 GiB" time=2025-05-23T00:34:03.689Z level=INFO source=types.go:130 msg="inference compute" id=GPU-7274d657-2e08-df43-f06f-d4be16816916 library=cuda variant=v12 compute=8.0 driver=12.4 name="NVIDIA A100-PCIE-40GB" total="39.4 GiB" available="39.0 GiB" time=2025-05-23T00:34:03.689Z level=INFO source=types.go:130 msg="inference compute" id=GPU-d0f7b6db-6dcf-b4f2-f1a9-858a807e855e library=cuda variant=v12 compute=8.0 driver=12.4 name="NVIDIA A100-PCIE-40GB" total="39.4 GiB" available="39.0 GiB" time=2025-05-23T00:34:03.689Z level=INFO source=types.go:130 msg="inference compute" id=GPU-2945b7e8-3bb0-14c1-99b9-986b6ce57629 library=cuda variant=v12 compute=8.0 driver=12.4 name="NVIDIA A100-PCIE-40GB" total="39.4 GiB" available="39.0 GiB" [GIN] 2025/05/23 - 00:34:27 | 200 | 72.489µs | 127.0.0.1 | HEAD "/" [GIN] 2025/05/23 - 00:34:27 | 200 | 236.388µs | 127.0.0.1 | GET "/api/ps" time=2025-05-23T00:34:51.505Z level=DEBUG source=ggml.go:155 msg="key not found" key=general.alignment default=32 time=2025-05-23T00:34:51.506Z level=WARN source=types.go:555 msg="invalid option provided" option=function_calling time=2025-05-23T00:34:51.506Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.7 GiB" before.free="489.1 GiB" before.free_swap="0 B" now.total="503.7 GiB" now.free="489.0 GiB" now.free_swap="0 B" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.144.03 dlsym: cuInit - 0x7fb2b16aebc0 dlsym: cuDriverGetVersion - 0x7fb2b16aebe0 dlsym: cuDeviceGetCount - 0x7fb2b16aec20 dlsym: cuDeviceGet - 0x7fb2b16aec00 dlsym: cuDeviceGetAttribute - 0x7fb2b16aed00 dlsym: cuDeviceGetUuid - 0x7fb2b16aec60 dlsym: cuDeviceGetName - 0x7fb2b16aec40 dlsym: cuCtxCreate_v3 - 0x7fb2b16aeee0 dlsym: cuMemGetInfo_v2 - 0x7fb2b16b8e20 dlsym: cuCtxDestroy - 0x7fb2b1713830 calling cuInit calling cuDriverGetVersion raw version 0x2f08 CUDA driver version: 12.4 calling cuDeviceGetCount device count 4 time=2025-05-23T00:34:51.885Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-cf0928a8-fc00-9b78-4b6f-75b9f5fdfa2a name="NVIDIA A100-PCIE-40GB" overhead="0 B" before.total="39.4 GiB" before.free="39.0 GiB" now.total="39.4 GiB" now.free="39.0 GiB" now.used="423.0 MiB" time=2025-05-23T00:34:52.261Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-7274d657-2e08-df43-f06f-d4be16816916 name="NVIDIA A100-PCIE-40GB" overhead="0 B" before.total="39.4 GiB" before.free="39.0 GiB" now.total="39.4 GiB" now.free="39.0 GiB" now.used="423.0 MiB" time=2025-05-23T00:34:52.638Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-d0f7b6db-6dcf-b4f2-f1a9-858a807e855e name="NVIDIA A100-PCIE-40GB" overhead="0 B" before.total="39.4 GiB" before.free="39.0 GiB" now.total="39.4 GiB" now.free="39.0 GiB" now.used="423.0 MiB" time=2025-05-23T00:34:53.018Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2945b7e8-3bb0-14c1-99b9-986b6ce57629 name="NVIDIA A100-PCIE-40GB" overhead="0 B" before.total="39.4 GiB" before.free="39.0 GiB" now.total="39.4 GiB" now.free="39.0 GiB" now.used="423.0 MiB" releasing cuda driver library time=2025-05-23T00:34:53.018Z level=DEBUG source=sched.go:185 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=12 gpu_count=4 time=2025-05-23T00:34:53.038Z level=DEBUG source=ggml.go:155 msg="key not found" key=general.alignment default=32 time=2025-05-23T00:34:53.057Z level=DEBUG source=ggml.go:155 msg="key not found" key=general.alignment default=32 time=2025-05-23T00:34:53.057Z level=DEBUG source=sched.go:228 msg="loading first model" model=/root/.ollama/models/blobs/sha256-0e56d3bc2b1ff6af8820beb958f869e4180d120fe195a1b3206afd4e3df37417 time=2025-05-23T00:34:53.057Z level=DEBUG source=memory.go:111 msg=evaluating library=cuda gpu_count=4 available="[39.0 GiB 39.0 GiB 39.0 GiB 39.0 GiB]" time=2025-05-23T00:34:53.057Z level=DEBUG source=ggml.go:155 msg="key not found" key=llama.vision.block_count default=0 time=2025-05-23T00:34:53.058Z level=INFO source=sched.go:793 msg="new model will fit in available VRAM, loading" model=/root/.ollama/models/blobs/sha256-0e56d3bc2b1ff6af8820beb958f869e4180d120fe195a1b3206afd4e3df37417 library=cuda parallel=4 required="12.6 GiB" time=2025-05-23T00:34:53.058Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.7 GiB" before.free="489.0 GiB" before.free_swap="0 B" now.total="503.7 GiB" now.free="489.0 GiB" now.free_swap="0 B" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.144.03 dlsym: cuInit - 0x7fb2b16aebc0 dlsym: cuDriverGetVersion - 0x7fb2b16aebe0 dlsym: cuDeviceGetCount - 0x7fb2b16aec20 dlsym: cuDeviceGet - 0x7fb2b16aec00 dlsym: cuDeviceGetAttribute - 0x7fb2b16aed00 dlsym: cuDeviceGetUuid - 0x7fb2b16aec60 dlsym: cuDeviceGetName - 0x7fb2b16aec40 dlsym: cuCtxCreate_v3 - 0x7fb2b16aeee0 dlsym: cuMemGetInfo_v2 - 0x7fb2b16b8e20 dlsym: cuCtxDestroy - 0x7fb2b1713830 calling cuInit calling cuDriverGetVersion raw version 0x2f08 CUDA driver version: 12.4 calling cuDeviceGetCount device count 4 time=2025-05-23T00:34:53.379Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-cf0928a8-fc00-9b78-4b6f-75b9f5fdfa2a name="NVIDIA A100-PCIE-40GB" overhead="0 B" before.total="39.4 GiB" before.free="39.0 GiB" now.total="39.4 GiB" now.free="39.0 GiB" now.used="423.0 MiB" time=2025-05-23T00:34:53.709Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-7274d657-2e08-df43-f06f-d4be16816916 name="NVIDIA A100-PCIE-40GB" overhead="0 B" before.total="39.4 GiB" before.free="39.0 GiB" now.total="39.4 GiB" now.free="39.0 GiB" now.used="423.0 MiB" time=2025-05-23T00:34:54.044Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-d0f7b6db-6dcf-b4f2-f1a9-858a807e855e name="NVIDIA A100-PCIE-40GB" overhead="0 B" before.total="39.4 GiB" before.free="39.0 GiB" now.total="39.4 GiB" now.free="39.0 GiB" now.used="423.0 MiB" time=2025-05-23T00:34:54.369Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2945b7e8-3bb0-14c1-99b9-986b6ce57629 name="NVIDIA A100-PCIE-40GB" overhead="0 B" before.total="39.4 GiB" before.free="39.0 GiB" now.total="39.4 GiB" now.free="39.0 GiB" now.used="423.0 MiB" releasing cuda driver library time=2025-05-23T00:34:54.369Z level=INFO source=server.go:135 msg="system memory" total="503.7 GiB" free="489.0 GiB" free_swap="0 B" time=2025-05-23T00:34:54.369Z level=DEBUG source=memory.go:111 msg=evaluating library=cuda gpu_count=4 available="[39.0 GiB 39.0 GiB 39.0 GiB 39.0 GiB]" time=2025-05-23T00:34:54.369Z level=DEBUG source=ggml.go:155 msg="key not found" key=llama.vision.block_count default=0 time=2025-05-23T00:34:54.370Z level=INFO source=server.go:168 msg=offload library=cuda layers.requested=7 layers.model=33 layers.offload=7 layers.split=1,2,2,2 memory.available="[39.0 GiB 39.0 GiB 39.0 GiB 39.0 GiB]" memory.gpu_overhead="0 B" memory.required.full="18.9 GiB" memory.required.partial="12.6 GiB" memory.required.kv="3.7 GiB" memory.required.allocations="[3.0 GiB 3.2 GiB 3.2 GiB 3.2 GiB]" memory.weights.total="4.3 GiB" memory.weights.repeating="3.9 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="2.0 GiB" memory.graph.partial="2.0 GiB" time=2025-05-23T00:34:54.370Z level=DEBUG source=server.go:284 msg="compatible gpu libraries" compatible="[cuda_v12 cuda_v11]" time=2025-05-23T00:34:54.438Z level=DEBUG source=ggml.go:155 msg="key not found" key=general.alignment default=32 time=2025-05-23T00:34:54.439Z level=DEBUG source=ggml.go:155 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+" time=2025-05-23T00:34:54.439Z level=DEBUG source=ggml.go:155 msg="key not found" key=tokenizer.ggml.add_bos_token default=true time=2025-05-23T00:34:54.439Z level=DEBUG source=ggml.go:155 msg="key not found" key=tokenizer.ggml.add_eos_token default=false time=2025-05-23T00:34:54.439Z level=DEBUG source=ggml.go:155 msg="key not found" key=tokenizer.ggml.eos_token_ids default="&{size:0 values:]}" time=2025-05-23T00:34:54.439Z level=DEBUG source=ggml.go:155 msg="key not found" key=llama.rope.freq_scale default=1 time=2025-05-23T00:34:54.440Z level=DEBUG source=server.go:360 msg="adding gpu library" path=/usr/lib/ollama/cuda_v12 time=2025-05-23T00:34:54.440Z level=DEBUG source=server.go:367 msg="adding gpu dependency paths" paths=[/usr/lib/ollama/cuda_v12] time=2025-05-23T00:34:54.440Z level=INFO source=server.go:431 msg="starting llama server" cmd="/usr/bin/ollama runner --ollama-engine --model /root/.ollama/models/blobs/sha256-0e56d3bc2b1ff6af8820beb958f869e4180d120fe195a1b3206afd4e3df37417 --ctx-size 30060 --batch-size 512 --n-gpu-layers 7 --threads 4 --parallel 4 --tensor-split 1,2,2,2 --port 37839" time=2025-05-23T00:34:54.440Z level=DEBUG source=server.go:432 msg=subprocess PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LD_LIBRARY_PATH=/usr/lib/ollama/cuda_v12:/usr/lib/ollama/cuda_v12:/usr/lib/ollama:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/lib/ollama OLLAMA_HOST=0.0.0.0:11434 OLLAMA_DEBUG=1 OLLAMA_CONTEXT_LENGTH=24574 OLLAMA_GPU_OVERHEAD=0 OLLAMA_SCHED_SPREAD=1 OLLAMA_NEW_ENGINE=1 OLLAMA_NUM_PARALLEL=4 OLLAMA_SERVICE_SERVICE_HOST=10.108.17.67 OLLAMA_SERVICE_SERVICE_PORT=11434 OLLAMA_SERVICE_PORT_11434_TCP=tcp://10.108.17.67:11434 OLLAMA_SERVICE_PORT_11434_TCP_ADDR=10.108.17.67 OLLAMA_SERVICE_PORT=tcp://10.108.17.67:11434 OLLAMA_SERVICE_PORT_11434_TCP_PROTO=tcp OLLAMA_SERVICE_PORT_11434_TCP_PORT=11434 OLLAMA_MAX_LOADED_MODELS=12 OLLAMA_LIBRARY_PATH=/usr/lib/ollama:/usr/lib/ollama/cuda_v12 CUDA_VISIBLE_DEVICES=GPU-cf0928a8-fc00-9b78-4b6f-75b9f5fdfa2a,GPU-7274d657-2e08-df43-f06f-d4be16816916,GPU-d0f7b6db-6dcf-b4f2-f1a9-858a807e855e,GPU-2945b7e8-3bb0-14c1-99b9-986b6ce57629 time=2025-05-23T00:34:54.441Z level=INFO source=sched.go:472 msg="loaded runners" count=1 time=2025-05-23T00:34:54.441Z level=INFO source=server.go:591 msg="waiting for llama runner to start responding" time=2025-05-23T00:34:54.441Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server not responding" time=2025-05-23T00:34:54.490Z level=INFO source=runner.go:917 msg="starting ollama engine" time=2025-05-23T00:34:54.492Z level=INFO source=runner.go:977 msg="Server listening on 127.0.0.1:37839" time=2025-05-23T00:34:54.555Z level=DEBUG source=ggml.go:155 msg="key not found" key=general.alignment default=32 time=2025-05-23T00:34:54.556Z level=DEBUG source=ggml.go:155 msg="key not found" key=general.description default="" time=2025-05-23T00:34:54.556Z level=INFO source=ggml.go:87 msg="" architecture=llama file_type=Q4_K_M name="Llama 3.1 Swallow 8B Instruct v0.3" description="" num_tensors=292 num_key_values=58 time=2025-05-23T00:34:54.556Z level=DEBUG source=ggml.go:94 msg="ggml backend load all from path" path=/usr/lib/ollama load_backend: loaded CPU backend from /usr/lib/ollama/libggml-cpu-haswell.so time=2025-05-23T00:34:54.598Z level=DEBUG source=ggml.go:94 msg="ggml backend load all from path" path=/usr/lib/ollama/cuda_v12 time=2025-05-23T00:34:54.693Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server loading model" ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 4 CUDA devices: Device 0: NVIDIA A100-PCIE-40GB, compute capability 8.0, VMM: yes Device 1: NVIDIA A100-PCIE-40GB, compute capability 8.0, VMM: yes Device 2: NVIDIA A100-PCIE-40GB, compute capability 8.0, VMM: yes Device 3: NVIDIA A100-PCIE-40GB, compute capability 8.0, VMM: yes load_backend: loaded CUDA backend from /usr/lib/ollama/cuda_v12/libggml-cuda.so time=2025-05-23T00:34:55.478Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 CUDA.1.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.1.USE_GRAPHS=1 CUDA.1.PEER_MAX_BATCH_SIZE=128 CUDA.2.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.2.USE_GRAPHS=1 CUDA.2.PEER_MAX_BATCH_SIZE=128 CUDA.3.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.3.USE_GRAPHS=1 CUDA.3.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc) time=2025-05-23T00:34:55.729Z level=INFO source=ggml.go:313 msg="model weights" buffer=CUDA2 size="265.0 MiB" time=2025-05-23T00:34:55.729Z level=INFO source=ggml.go:313 msg="model weights" buffer=CUDA3 size="265.0 MiB" time=2025-05-23T00:34:55.729Z level=INFO source=ggml.go:313 msg="model weights" buffer=CPU size="3.7 GiB" time=2025-05-23T00:34:55.729Z level=INFO source=ggml.go:313 msg="model weights" buffer=CUDA0 size="117.0 MiB" time=2025-05-23T00:34:55.729Z level=INFO source=ggml.go:313 msg="model weights" buffer=CUDA1 size="249.5 MiB" time=2025-05-23T00:34:55.730Z level=DEBUG source=ggml.go:155 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+" time=2025-05-23T00:34:55.730Z level=DEBUG source=ggml.go:155 msg="key not found" key=tokenizer.ggml.add_bos_token default=true time=2025-05-23T00:34:55.730Z level=DEBUG source=ggml.go:155 msg="key not found" key=tokenizer.ggml.add_eos_token default=false time=2025-05-23T00:34:55.730Z level=DEBUG source=ggml.go:155 msg="key not found" key=tokenizer.ggml.eos_token_ids default="&{size:0 values:]}" time=2025-05-23T00:34:55.730Z level=DEBUG source=ggml.go:155 msg="key not found" key=llama.rope.freq_scale default=1 time=2025-05-23T00:34:57.221Z level=DEBUG source=ggml.go:573 msg="compute graph" nodes=1126 splits=6 time=2025-05-23T00:34:57.221Z level=INFO source=ggml.go:576 msg="compute graph" backend=CUDA0 buffer_type=CUDA0 size="1.9 GiB" time=2025-05-23T00:34:57.221Z level=INFO source=ggml.go:576 msg="compute graph" backend=CUDA1 buffer_type=CUDA1 size="1.9 GiB" time=2025-05-23T00:34:57.221Z level=INFO source=ggml.go:576 msg="compute graph" backend=CUDA2 buffer_type=CUDA2 size="1.9 GiB" time=2025-05-23T00:34:57.221Z level=INFO source=ggml.go:576 msg="compute graph" backend=CUDA3 buffer_type=CUDA3 size="1.9 GiB" time=2025-05-23T00:34:57.221Z level=INFO source=ggml.go:576 msg="compute graph" backend=CPU buffer_type=CPU size="1.9 GiB" time=2025-05-23T00:34:57.489Z level=DEBUG source=server.go:636 msg="model load progress 0.02" time=2025-05-23T00:34:57.889Z level=DEBUG source=server.go:636 msg="model load progress 0.03" time=2025-05-23T00:34:58.189Z level=DEBUG source=server.go:636 msg="model load progress 0.04" time=2025-05-23T00:34:58.687Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server not responding" time=2025-05-23T00:34:58.988Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server loading model" time=2025-05-23T00:34:58.988Z level=DEBUG source=server.go:636 msg="model load progress 0.08" time=2025-05-23T00:34:59.588Z level=DEBUG source=server.go:636 msg="model load progress 0.11" time=2025-05-23T00:35:00.186Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server not responding" time=2025-05-23T00:35:00.488Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server loading model" time=2025-05-23T00:35:00.488Z level=DEBUG source=server.go:636 msg="model load progress 0.16"

Additional Information

No response

Originally created by @yoshihito-tatano on GitHub (May 23, 2025). Original GitHub issue: https://github.com/open-webui/open-webui/issues/14208 ### Check Existing Issues - [x] I have searched the existing issues and discussions. - [x] I am using the latest version of Open WebUI. ### Installation Method Git Clone ### Open WebUI Version ghcr.io/open-webui/open-webui:v0.6.10 ### Ollama Version (if applicable) ollama/ollama:0.7.1-rc0 ### Operating System Ubuntu 22.04 ### Browser (if applicable) _No response_ ### Confirmation - [x] I have read and followed all instructions in `README.md`. - [x] I am using the latest version of **both** Open WebUI and Ollama. - [x] I have included the browser console logs. - [x] I have included the Docker container logs. - [x] I have listed steps to reproduce the bug in detail. ### Expected Behavior `kubectl apply -k ./kubernetes/manifest/gpu` I followed the documentation and deployed Open-webUI+Ollama using GPU GPU information here. `root@ollama-0:/# nvidia-smi Fri May 23 00:17:52 2025 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 550.144.03 Driver Version: 550.144.03 CUDA Version: 12.4 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA A100-PCIE-40GB On | 00000000:81:00.0 Off | 0 | | N/A 34C P0 38W / 250W | 4MiB / 40960MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA A100-PCIE-40GB On | 00000000:A1:00.0 Off | 0 | | N/A 34C P0 41W / 250W | 4MiB / 40960MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 2 NVIDIA A100-PCIE-40GB On | 00000000:C1:00.0 Off | 0 | | N/A 35C P0 41W / 250W | 4MiB / 40960MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 3 NVIDIA A100-PCIE-40GB On | 00000000:E1:00.0 Off | 0 | | N/A 33C P0 37W / 250W | 4MiB / 40960MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+ ` Pulled the model on ollama and ran it, and it returns a response using the GPU as expected. However, when I run the query from Open-WebUI, the results are quite slow. Also, it uses CPU as well as GPU. `root@ollama-0:/# ollama ps NAME ID SIZE PROCESSOR UNTIL llama-swallow:8b e4ef5a288c42 20 GB 33%/67% CPU/GPU 29 minutes from now` ### Actual Behavior We expect Ollama to make full use of the GPU and return a fast response. However, I suspect that the CPU is actually being used unintentionally for some of the processing, which is slowing down the response. This is only happening when executing from Open-webUI, and I would like to know if there is a way to control it. ### Steps to Reproduce 1. `kubectl apply -k ./kubernetes/manifest/gpu` 2. pull the model ( I have tried several models llama3.3 and gemma3 3. Use Open-webUI `400: Ollama: 400, message='Bad Request', url='http://ollama-service.open-webui.svc.cluster.local:11434/api/chat'` I could not run any of the Gemma3 models in my environment with the above error. ### Logs & Screenshots `root@ollama-0:/# ollama ps NAME ID SIZE PROCESSOR UNTIL llama3.3:latest a6eb4748fd29 73 GB 64%/36% CPU/GPU 29 minutes from now` `time=2025-05-23T00:34:02.124Z level=INFO source=routes.go:1205 msg="server config" env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:24574 OLLAMA_DEBUG:DEBUG OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:true OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:4 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:true ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" time=2025-05-23T00:34:02.129Z level=INFO source=images.go:463 msg="total blobs: 38" time=2025-05-23T00:34:02.130Z level=INFO source=images.go:470 msg="total unused blobs removed: 0" time=2025-05-23T00:34:02.131Z level=INFO source=routes.go:1258 msg="Listening on :11434 (version 0.7.1-rc0)" time=2025-05-23T00:34:02.131Z level=DEBUG source=sched.go:108 msg="starting llm scheduler" time=2025-05-23T00:34:02.131Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs" time=2025-05-23T00:34:02.160Z level=DEBUG source=gpu.go:98 msg="searching for GPU discovery libraries for NVIDIA" time=2025-05-23T00:34:02.160Z level=DEBUG source=gpu.go:501 msg="Searching for GPU library" name=libcuda.so* time=2025-05-23T00:34:02.160Z level=DEBUG source=gpu.go:525 msg="gpu library search" globs="[/usr/lib/ollama/libcuda.so* /usr/local/nvidia/lib/libcuda.so* /usr/local/nvidia/lib64/libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]" time=2025-05-23T00:34:02.162Z level=DEBUG source=gpu.go:558 msg="discovered GPU libraries" paths=[/usr/lib/x86_64-linux-gnu/libcuda.so.550.144.03] initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.144.03 dlsym: cuInit - 0x7fb2b16aebc0 dlsym: cuDriverGetVersion - 0x7fb2b16aebe0 dlsym: cuDeviceGetCount - 0x7fb2b16aec20 dlsym: cuDeviceGet - 0x7fb2b16aec00 dlsym: cuDeviceGetAttribute - 0x7fb2b16aed00 dlsym: cuDeviceGetUuid - 0x7fb2b16aec60 dlsym: cuDeviceGetName - 0x7fb2b16aec40 dlsym: cuCtxCreate_v3 - 0x7fb2b16aeee0 dlsym: cuMemGetInfo_v2 - 0x7fb2b16b8e20 dlsym: cuCtxDestroy - 0x7fb2b1713830 calling cuInit calling cuDriverGetVersion raw version 0x2f08 CUDA driver version: 12.4 calling cuDeviceGetCount device count 4 time=2025-05-23T00:34:02.474Z level=DEBUG source=gpu.go:125 msg="detected GPUs" count=4 library=/usr/lib/x86_64-linux-gnu/libcuda.so.550.144.03 [GPU-cf0928a8-fc00-9b78-4b6f-75b9f5fdfa2a] CUDA totalMem 40326mb [GPU-cf0928a8-fc00-9b78-4b6f-75b9f5fdfa2a] CUDA freeMem 39903mb [GPU-cf0928a8-fc00-9b78-4b6f-75b9f5fdfa2a] Compute Capability 8.0 [GPU-7274d657-2e08-df43-f06f-d4be16816916] CUDA totalMem 40326mb [GPU-7274d657-2e08-df43-f06f-d4be16816916] CUDA freeMem 39903mb [GPU-7274d657-2e08-df43-f06f-d4be16816916] Compute Capability 8.0 [GPU-d0f7b6db-6dcf-b4f2-f1a9-858a807e855e] CUDA totalMem 40326mb [GPU-d0f7b6db-6dcf-b4f2-f1a9-858a807e855e] CUDA freeMem 39903mb [GPU-d0f7b6db-6dcf-b4f2-f1a9-858a807e855e] Compute Capability 8.0 [GPU-2945b7e8-3bb0-14c1-99b9-986b6ce57629] CUDA totalMem 40326mb [GPU-2945b7e8-3bb0-14c1-99b9-986b6ce57629] CUDA freeMem 39903mb [GPU-2945b7e8-3bb0-14c1-99b9-986b6ce57629] Compute Capability 8.0 time=2025-05-23T00:34:03.689Z level=DEBUG source=amd_linux.go:419 msg="amdgpu driver not detected /sys/module/amdgpu" releasing cuda driver library time=2025-05-23T00:34:03.689Z level=INFO source=types.go:130 msg="inference compute" id=GPU-cf0928a8-fc00-9b78-4b6f-75b9f5fdfa2a library=cuda variant=v12 compute=8.0 driver=12.4 name="NVIDIA A100-PCIE-40GB" total="39.4 GiB" available="39.0 GiB" time=2025-05-23T00:34:03.689Z level=INFO source=types.go:130 msg="inference compute" id=GPU-7274d657-2e08-df43-f06f-d4be16816916 library=cuda variant=v12 compute=8.0 driver=12.4 name="NVIDIA A100-PCIE-40GB" total="39.4 GiB" available="39.0 GiB" time=2025-05-23T00:34:03.689Z level=INFO source=types.go:130 msg="inference compute" id=GPU-d0f7b6db-6dcf-b4f2-f1a9-858a807e855e library=cuda variant=v12 compute=8.0 driver=12.4 name="NVIDIA A100-PCIE-40GB" total="39.4 GiB" available="39.0 GiB" time=2025-05-23T00:34:03.689Z level=INFO source=types.go:130 msg="inference compute" id=GPU-2945b7e8-3bb0-14c1-99b9-986b6ce57629 library=cuda variant=v12 compute=8.0 driver=12.4 name="NVIDIA A100-PCIE-40GB" total="39.4 GiB" available="39.0 GiB" [GIN] 2025/05/23 - 00:34:27 | 200 | 72.489µs | 127.0.0.1 | HEAD "/" [GIN] 2025/05/23 - 00:34:27 | 200 | 236.388µs | 127.0.0.1 | GET "/api/ps" time=2025-05-23T00:34:51.505Z level=DEBUG source=ggml.go:155 msg="key not found" key=general.alignment default=32 time=2025-05-23T00:34:51.506Z level=WARN source=types.go:555 msg="invalid option provided" option=function_calling time=2025-05-23T00:34:51.506Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.7 GiB" before.free="489.1 GiB" before.free_swap="0 B" now.total="503.7 GiB" now.free="489.0 GiB" now.free_swap="0 B" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.144.03 dlsym: cuInit - 0x7fb2b16aebc0 dlsym: cuDriverGetVersion - 0x7fb2b16aebe0 dlsym: cuDeviceGetCount - 0x7fb2b16aec20 dlsym: cuDeviceGet - 0x7fb2b16aec00 dlsym: cuDeviceGetAttribute - 0x7fb2b16aed00 dlsym: cuDeviceGetUuid - 0x7fb2b16aec60 dlsym: cuDeviceGetName - 0x7fb2b16aec40 dlsym: cuCtxCreate_v3 - 0x7fb2b16aeee0 dlsym: cuMemGetInfo_v2 - 0x7fb2b16b8e20 dlsym: cuCtxDestroy - 0x7fb2b1713830 calling cuInit calling cuDriverGetVersion raw version 0x2f08 CUDA driver version: 12.4 calling cuDeviceGetCount device count 4 time=2025-05-23T00:34:51.885Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-cf0928a8-fc00-9b78-4b6f-75b9f5fdfa2a name="NVIDIA A100-PCIE-40GB" overhead="0 B" before.total="39.4 GiB" before.free="39.0 GiB" now.total="39.4 GiB" now.free="39.0 GiB" now.used="423.0 MiB" time=2025-05-23T00:34:52.261Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-7274d657-2e08-df43-f06f-d4be16816916 name="NVIDIA A100-PCIE-40GB" overhead="0 B" before.total="39.4 GiB" before.free="39.0 GiB" now.total="39.4 GiB" now.free="39.0 GiB" now.used="423.0 MiB" time=2025-05-23T00:34:52.638Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-d0f7b6db-6dcf-b4f2-f1a9-858a807e855e name="NVIDIA A100-PCIE-40GB" overhead="0 B" before.total="39.4 GiB" before.free="39.0 GiB" now.total="39.4 GiB" now.free="39.0 GiB" now.used="423.0 MiB" time=2025-05-23T00:34:53.018Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2945b7e8-3bb0-14c1-99b9-986b6ce57629 name="NVIDIA A100-PCIE-40GB" overhead="0 B" before.total="39.4 GiB" before.free="39.0 GiB" now.total="39.4 GiB" now.free="39.0 GiB" now.used="423.0 MiB" releasing cuda driver library time=2025-05-23T00:34:53.018Z level=DEBUG source=sched.go:185 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=12 gpu_count=4 time=2025-05-23T00:34:53.038Z level=DEBUG source=ggml.go:155 msg="key not found" key=general.alignment default=32 time=2025-05-23T00:34:53.057Z level=DEBUG source=ggml.go:155 msg="key not found" key=general.alignment default=32 time=2025-05-23T00:34:53.057Z level=DEBUG source=sched.go:228 msg="loading first model" model=/root/.ollama/models/blobs/sha256-0e56d3bc2b1ff6af8820beb958f869e4180d120fe195a1b3206afd4e3df37417 time=2025-05-23T00:34:53.057Z level=DEBUG source=memory.go:111 msg=evaluating library=cuda gpu_count=4 available="[39.0 GiB 39.0 GiB 39.0 GiB 39.0 GiB]" time=2025-05-23T00:34:53.057Z level=DEBUG source=ggml.go:155 msg="key not found" key=llama.vision.block_count default=0 time=2025-05-23T00:34:53.058Z level=INFO source=sched.go:793 msg="new model will fit in available VRAM, loading" model=/root/.ollama/models/blobs/sha256-0e56d3bc2b1ff6af8820beb958f869e4180d120fe195a1b3206afd4e3df37417 library=cuda parallel=4 required="12.6 GiB" time=2025-05-23T00:34:53.058Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.7 GiB" before.free="489.0 GiB" before.free_swap="0 B" now.total="503.7 GiB" now.free="489.0 GiB" now.free_swap="0 B" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.144.03 dlsym: cuInit - 0x7fb2b16aebc0 dlsym: cuDriverGetVersion - 0x7fb2b16aebe0 dlsym: cuDeviceGetCount - 0x7fb2b16aec20 dlsym: cuDeviceGet - 0x7fb2b16aec00 dlsym: cuDeviceGetAttribute - 0x7fb2b16aed00 dlsym: cuDeviceGetUuid - 0x7fb2b16aec60 dlsym: cuDeviceGetName - 0x7fb2b16aec40 dlsym: cuCtxCreate_v3 - 0x7fb2b16aeee0 dlsym: cuMemGetInfo_v2 - 0x7fb2b16b8e20 dlsym: cuCtxDestroy - 0x7fb2b1713830 calling cuInit calling cuDriverGetVersion raw version 0x2f08 CUDA driver version: 12.4 calling cuDeviceGetCount device count 4 time=2025-05-23T00:34:53.379Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-cf0928a8-fc00-9b78-4b6f-75b9f5fdfa2a name="NVIDIA A100-PCIE-40GB" overhead="0 B" before.total="39.4 GiB" before.free="39.0 GiB" now.total="39.4 GiB" now.free="39.0 GiB" now.used="423.0 MiB" time=2025-05-23T00:34:53.709Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-7274d657-2e08-df43-f06f-d4be16816916 name="NVIDIA A100-PCIE-40GB" overhead="0 B" before.total="39.4 GiB" before.free="39.0 GiB" now.total="39.4 GiB" now.free="39.0 GiB" now.used="423.0 MiB" time=2025-05-23T00:34:54.044Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-d0f7b6db-6dcf-b4f2-f1a9-858a807e855e name="NVIDIA A100-PCIE-40GB" overhead="0 B" before.total="39.4 GiB" before.free="39.0 GiB" now.total="39.4 GiB" now.free="39.0 GiB" now.used="423.0 MiB" time=2025-05-23T00:34:54.369Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2945b7e8-3bb0-14c1-99b9-986b6ce57629 name="NVIDIA A100-PCIE-40GB" overhead="0 B" before.total="39.4 GiB" before.free="39.0 GiB" now.total="39.4 GiB" now.free="39.0 GiB" now.used="423.0 MiB" releasing cuda driver library time=2025-05-23T00:34:54.369Z level=INFO source=server.go:135 msg="system memory" total="503.7 GiB" free="489.0 GiB" free_swap="0 B" time=2025-05-23T00:34:54.369Z level=DEBUG source=memory.go:111 msg=evaluating library=cuda gpu_count=4 available="[39.0 GiB 39.0 GiB 39.0 GiB 39.0 GiB]" time=2025-05-23T00:34:54.369Z level=DEBUG source=ggml.go:155 msg="key not found" key=llama.vision.block_count default=0 time=2025-05-23T00:34:54.370Z level=INFO source=server.go:168 msg=offload library=cuda layers.requested=7 layers.model=33 layers.offload=7 layers.split=1,2,2,2 memory.available="[39.0 GiB 39.0 GiB 39.0 GiB 39.0 GiB]" memory.gpu_overhead="0 B" memory.required.full="18.9 GiB" memory.required.partial="12.6 GiB" memory.required.kv="3.7 GiB" memory.required.allocations="[3.0 GiB 3.2 GiB 3.2 GiB 3.2 GiB]" memory.weights.total="4.3 GiB" memory.weights.repeating="3.9 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="2.0 GiB" memory.graph.partial="2.0 GiB" time=2025-05-23T00:34:54.370Z level=DEBUG source=server.go:284 msg="compatible gpu libraries" compatible="[cuda_v12 cuda_v11]" time=2025-05-23T00:34:54.438Z level=DEBUG source=ggml.go:155 msg="key not found" key=general.alignment default=32 time=2025-05-23T00:34:54.439Z level=DEBUG source=ggml.go:155 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+" time=2025-05-23T00:34:54.439Z level=DEBUG source=ggml.go:155 msg="key not found" key=tokenizer.ggml.add_bos_token default=true time=2025-05-23T00:34:54.439Z level=DEBUG source=ggml.go:155 msg="key not found" key=tokenizer.ggml.add_eos_token default=false time=2025-05-23T00:34:54.439Z level=DEBUG source=ggml.go:155 msg="key not found" key=tokenizer.ggml.eos_token_ids default="&{size:0 values:]}" time=2025-05-23T00:34:54.439Z level=DEBUG source=ggml.go:155 msg="key not found" key=llama.rope.freq_scale default=1 time=2025-05-23T00:34:54.440Z level=DEBUG source=server.go:360 msg="adding gpu library" path=/usr/lib/ollama/cuda_v12 time=2025-05-23T00:34:54.440Z level=DEBUG source=server.go:367 msg="adding gpu dependency paths" paths=[/usr/lib/ollama/cuda_v12] time=2025-05-23T00:34:54.440Z level=INFO source=server.go:431 msg="starting llama server" cmd="/usr/bin/ollama runner --ollama-engine --model /root/.ollama/models/blobs/sha256-0e56d3bc2b1ff6af8820beb958f869e4180d120fe195a1b3206afd4e3df37417 --ctx-size 30060 --batch-size 512 --n-gpu-layers 7 --threads 4 --parallel 4 --tensor-split 1,2,2,2 --port 37839" time=2025-05-23T00:34:54.440Z level=DEBUG source=server.go:432 msg=subprocess PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LD_LIBRARY_PATH=/usr/lib/ollama/cuda_v12:/usr/lib/ollama/cuda_v12:/usr/lib/ollama:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/lib/ollama OLLAMA_HOST=0.0.0.0:11434 OLLAMA_DEBUG=1 OLLAMA_CONTEXT_LENGTH=24574 OLLAMA_GPU_OVERHEAD=0 OLLAMA_SCHED_SPREAD=1 OLLAMA_NEW_ENGINE=1 OLLAMA_NUM_PARALLEL=4 OLLAMA_SERVICE_SERVICE_HOST=10.108.17.67 OLLAMA_SERVICE_SERVICE_PORT=11434 OLLAMA_SERVICE_PORT_11434_TCP=tcp://10.108.17.67:11434 OLLAMA_SERVICE_PORT_11434_TCP_ADDR=10.108.17.67 OLLAMA_SERVICE_PORT=tcp://10.108.17.67:11434 OLLAMA_SERVICE_PORT_11434_TCP_PROTO=tcp OLLAMA_SERVICE_PORT_11434_TCP_PORT=11434 OLLAMA_MAX_LOADED_MODELS=12 OLLAMA_LIBRARY_PATH=/usr/lib/ollama:/usr/lib/ollama/cuda_v12 CUDA_VISIBLE_DEVICES=GPU-cf0928a8-fc00-9b78-4b6f-75b9f5fdfa2a,GPU-7274d657-2e08-df43-f06f-d4be16816916,GPU-d0f7b6db-6dcf-b4f2-f1a9-858a807e855e,GPU-2945b7e8-3bb0-14c1-99b9-986b6ce57629 time=2025-05-23T00:34:54.441Z level=INFO source=sched.go:472 msg="loaded runners" count=1 time=2025-05-23T00:34:54.441Z level=INFO source=server.go:591 msg="waiting for llama runner to start responding" time=2025-05-23T00:34:54.441Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server not responding" time=2025-05-23T00:34:54.490Z level=INFO source=runner.go:917 msg="starting ollama engine" time=2025-05-23T00:34:54.492Z level=INFO source=runner.go:977 msg="Server listening on 127.0.0.1:37839" time=2025-05-23T00:34:54.555Z level=DEBUG source=ggml.go:155 msg="key not found" key=general.alignment default=32 time=2025-05-23T00:34:54.556Z level=DEBUG source=ggml.go:155 msg="key not found" key=general.description default="" time=2025-05-23T00:34:54.556Z level=INFO source=ggml.go:87 msg="" architecture=llama file_type=Q4_K_M name="Llama 3.1 Swallow 8B Instruct v0.3" description="" num_tensors=292 num_key_values=58 time=2025-05-23T00:34:54.556Z level=DEBUG source=ggml.go:94 msg="ggml backend load all from path" path=/usr/lib/ollama load_backend: loaded CPU backend from /usr/lib/ollama/libggml-cpu-haswell.so time=2025-05-23T00:34:54.598Z level=DEBUG source=ggml.go:94 msg="ggml backend load all from path" path=/usr/lib/ollama/cuda_v12 time=2025-05-23T00:34:54.693Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server loading model" ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 4 CUDA devices: Device 0: NVIDIA A100-PCIE-40GB, compute capability 8.0, VMM: yes Device 1: NVIDIA A100-PCIE-40GB, compute capability 8.0, VMM: yes Device 2: NVIDIA A100-PCIE-40GB, compute capability 8.0, VMM: yes Device 3: NVIDIA A100-PCIE-40GB, compute capability 8.0, VMM: yes load_backend: loaded CUDA backend from /usr/lib/ollama/cuda_v12/libggml-cuda.so time=2025-05-23T00:34:55.478Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 CUDA.1.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.1.USE_GRAPHS=1 CUDA.1.PEER_MAX_BATCH_SIZE=128 CUDA.2.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.2.USE_GRAPHS=1 CUDA.2.PEER_MAX_BATCH_SIZE=128 CUDA.3.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.3.USE_GRAPHS=1 CUDA.3.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc) time=2025-05-23T00:34:55.729Z level=INFO source=ggml.go:313 msg="model weights" buffer=CUDA2 size="265.0 MiB" time=2025-05-23T00:34:55.729Z level=INFO source=ggml.go:313 msg="model weights" buffer=CUDA3 size="265.0 MiB" time=2025-05-23T00:34:55.729Z level=INFO source=ggml.go:313 msg="model weights" buffer=CPU size="3.7 GiB" time=2025-05-23T00:34:55.729Z level=INFO source=ggml.go:313 msg="model weights" buffer=CUDA0 size="117.0 MiB" time=2025-05-23T00:34:55.729Z level=INFO source=ggml.go:313 msg="model weights" buffer=CUDA1 size="249.5 MiB" time=2025-05-23T00:34:55.730Z level=DEBUG source=ggml.go:155 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+" time=2025-05-23T00:34:55.730Z level=DEBUG source=ggml.go:155 msg="key not found" key=tokenizer.ggml.add_bos_token default=true time=2025-05-23T00:34:55.730Z level=DEBUG source=ggml.go:155 msg="key not found" key=tokenizer.ggml.add_eos_token default=false time=2025-05-23T00:34:55.730Z level=DEBUG source=ggml.go:155 msg="key not found" key=tokenizer.ggml.eos_token_ids default="&{size:0 values:]}" time=2025-05-23T00:34:55.730Z level=DEBUG source=ggml.go:155 msg="key not found" key=llama.rope.freq_scale default=1 time=2025-05-23T00:34:57.221Z level=DEBUG source=ggml.go:573 msg="compute graph" nodes=1126 splits=6 time=2025-05-23T00:34:57.221Z level=INFO source=ggml.go:576 msg="compute graph" backend=CUDA0 buffer_type=CUDA0 size="1.9 GiB" time=2025-05-23T00:34:57.221Z level=INFO source=ggml.go:576 msg="compute graph" backend=CUDA1 buffer_type=CUDA1 size="1.9 GiB" time=2025-05-23T00:34:57.221Z level=INFO source=ggml.go:576 msg="compute graph" backend=CUDA2 buffer_type=CUDA2 size="1.9 GiB" time=2025-05-23T00:34:57.221Z level=INFO source=ggml.go:576 msg="compute graph" backend=CUDA3 buffer_type=CUDA3 size="1.9 GiB" time=2025-05-23T00:34:57.221Z level=INFO source=ggml.go:576 msg="compute graph" backend=CPU buffer_type=CPU size="1.9 GiB" time=2025-05-23T00:34:57.489Z level=DEBUG source=server.go:636 msg="model load progress 0.02" time=2025-05-23T00:34:57.889Z level=DEBUG source=server.go:636 msg="model load progress 0.03" time=2025-05-23T00:34:58.189Z level=DEBUG source=server.go:636 msg="model load progress 0.04" time=2025-05-23T00:34:58.687Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server not responding" time=2025-05-23T00:34:58.988Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server loading model" time=2025-05-23T00:34:58.988Z level=DEBUG source=server.go:636 msg="model load progress 0.08" time=2025-05-23T00:34:59.588Z level=DEBUG source=server.go:636 msg="model load progress 0.11" time=2025-05-23T00:35:00.186Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server not responding" time=2025-05-23T00:35:00.488Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server loading model" time=2025-05-23T00:35:00.488Z level=DEBUG source=server.go:636 msg="model load progress 0.16" ` ### Additional Information _No response_
GiteaMirror added the bug label 2026-05-20 22:06:54 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
confirmed
confirmed issue
core
documentation
enhancement
good first issue
help wanted
non-core
pull-request
Mirrored from GitHub Pull Request
 python
question
testing wanted
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#120492
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?