github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-15 20:54:54 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #12849] qwen3-vl parallel request failed #34274

New Issue
Open
opened 2026-04-22 17:42:54 -05:00 by GiteaMirror · 9 comments
GiteaMirror commented 2026-04-22 17:42:54 -05:00
Owner
Copy Link

Originally created by @Brian-209 on GitHub (Oct 30, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/12849

Originally assigned to: @jessegross on GitHub.

What is the issue?

qwen3-vl seems to have bug on handling parallel request.
When I try to run 2 requests for one qwen3-vl model, the second one just queue and wait the first request to finish.
Then I try to run 2 requests for one qwen3 model, it runs parallelly as normal.
I have set OLLAMA_NUM_PARALLEL to 6 and tested within the same server.

Relevant log output

time=2025-10-30T15:45:22.285+08:00 level=INFO source=routes.go:1524 msg="server config" env="map[CUDA_VISIBLE_DEVICES: GGML_VK_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:8192 OLLAMA_DEBUG:INFO OLLAMA_FLASH_ATTENTION:true OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE:q8_0 OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:2 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:C:\\Users\\.ollama\\models OLLAMA_MULTIUSER_CACHE:true OLLAMA_NEW_ENGINE:true OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:6 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_REMOTES:[ollama.com] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES:]"
time=2025-10-30T15:45:22.289+08:00 level=INFO source=images.go:522 msg="total blobs: 32"
time=2025-10-30T15:45:22.289+08:00 level=INFO source=images.go:529 msg="total unused blobs removed: 0"
time=2025-10-30T15:45:22.290+08:00 level=INFO source=routes.go:1577 msg="Listening on [::]:11434 (version 0.12.7)"
time=2025-10-30T15:45:22.291+08:00 level=INFO source=runner.go:76 msg="discovering available GPUs..."
time=2025-10-30T15:45:22.295+08:00 level=INFO source=server.go:385 msg="starting runner" cmd="C:\\Users\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --port 56716"
time=2025-10-30T15:45:22.400+08:00 level=INFO source=server.go:385 msg="starting runner" cmd="C:\\Users\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --port 56721"
time=2025-10-30T15:45:22.471+08:00 level=INFO source=server.go:385 msg="starting runner" cmd="C:\\Users\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --port 56726"
time=2025-10-30T15:45:22.720+08:00 level=INFO source=server.go:385 msg="starting runner" cmd="C:\\Users\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --port 56732"
time=2025-10-30T15:45:22.720+08:00 level=INFO source=server.go:385 msg="starting runner" cmd="C:\\Users\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --port 56733"
time=2025-10-30T15:45:22.952+08:00 level=INFO source=types.go:42 msg="inference compute" id=GPU-ca44d757-2cc4-a597-149c-30b12b5bacb4 filtered_id="" library=CUDA compute=12.0 name=CUDA1 description="NVIDIA GeForce RTX 5060 Ti" libdirs=ollama,cuda_v12 driver=12.9 pci_id=0000:05:00.0 type=discrete total="15.9 GiB" available="15.4 GiB"
time=2025-10-30T15:45:22.952+08:00 level=INFO source=types.go:42 msg="inference compute" id=GPU-fee6fb1c-00bf-c621-1289-0fbf6937dad0 filtered_id="" library=CUDA compute=12.0 name=CUDA0 description="NVIDIA GeForce RTX 5060 Ti" libdirs=ollama,cuda_v12 driver=12.9 pci_id=0000:01:00.0 type=discrete total="15.9 GiB" available="13.5 GiB"
[GIN] 2025/10/30 - 15:45:22 | 200 |            0s |       127.0.0.1 | HEAD     "/"
[GIN] 2025/10/30 - 15:45:22 | 200 |            0s |       127.0.0.1 | GET      "/api/ps"
[GIN] 2025/10/30 - 15:45:26 | 200 |            0s |  192.168.107.65 | GET      "/api/ps"
[GIN] 2025/10/30 - 15:45:26 | 200 |     12.2058ms |  192.168.107.65 | GET      "/api/tags"
[GIN] 2025/10/30 - 15:45:32 | 200 |            0s |       127.0.0.1 | HEAD     "/"
[GIN] 2025/10/30 - 15:45:32 | 200 |     33.7115ms |       127.0.0.1 | POST     "/api/show"
time=2025-10-30T15:45:32.395+08:00 level=INFO source=server.go:385 msg="starting runner" cmd="C:\\Users\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --port 56750"
[GIN] 2025/10/30 - 15:45:32 | 200 |            0s |       127.0.0.1 | HEAD     "/"
[GIN] 2025/10/30 - 15:45:32 | 200 |     34.1516ms |       127.0.0.1 | POST     "/api/show"
time=2025-10-30T15:45:32.673+08:00 level=INFO source=cpu_windows.go:139 msg=packages count=1
time=2025-10-30T15:45:32.673+08:00 level=INFO source=cpu_windows.go:155 msg="efficiency cores detected" maxEfficiencyClass=1
time=2025-10-30T15:45:32.673+08:00 level=INFO source=cpu_windows.go:186 msg="" package=0 cores=16 efficiency=8 threads=24
time=2025-10-30T15:45:32.702+08:00 level=WARN source=sched.go:397 msg="model architecture does not currently support parallel requests" architecture=qwen3vl
time=2025-10-30T15:45:32.720+08:00 level=INFO source=server.go:215 msg="enabling flash attention"
time=2025-10-30T15:45:32.720+08:00 level=INFO source=server.go:385 msg="starting runner" cmd="C:\\Users\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model C:\\Users\\.ollama\\models\\blobs\\sha256-ed12a4674d727a74ac4816c906094ea9d3119fbea46ca93288c3ce4ffbe38c55 --port 56756"
time=2025-10-30T15:45:32.720+08:00 level=INFO source=server.go:638 msg="loading model" "model layers"=37 requested=-1
time=2025-10-30T15:45:32.720+08:00 level=INFO source=server.go:643 msg="system memory" total="95.8 GiB" free="69.9 GiB" free_swap="63.9 GiB"
time=2025-10-30T15:45:32.720+08:00 level=INFO source=server.go:650 msg="gpu memory" id=GPU-ca44d757-2cc4-a597-149c-30b12b5bacb4 library=CUDA available="14.9 GiB" free="15.4 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-10-30T15:45:32.720+08:00 level=INFO source=server.go:650 msg="gpu memory" id=GPU-fee6fb1c-00bf-c621-1289-0fbf6937dad0 library=CUDA available="13.1 GiB" free="13.5 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-10-30T15:45:32.756+08:00 level=INFO source=runner.go:1337 msg="starting ollama engine"
time=2025-10-30T15:45:32.756+08:00 level=INFO source=runner.go:1372 msg="Server listening on 127.0.0.1:56756"
time=2025-10-30T15:45:32.766+08:00 level=INFO source=runner.go:1210 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:8192 KvCacheType:q8_0 NumThreads:8 GPULayers:37[ID:GPU-ca44d757-2cc4-a597-149c-30b12b5bacb4 Layers:37(0..36)] MultiUserCache:true ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-10-30T15:45:32.784+08:00 level=INFO source=ggml.go:135 msg="" architecture=qwen3vl file_type=Q4_K_M name="" description="" num_tensors=858 num_key_values=40
load_backend: loaded CPU backend from C:\Users\Matrix\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-alderlake.dll
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
  Device 0: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes, ID: GPU-fee6fb1c-00bf-c621-1289-0fbf6937dad0
  Device 1: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes, ID: GPU-ca44d757-2cc4-a597-149c-30b12b5bacb4
load_backend: loaded CUDA backend from C:\Users\Matrix\AppData\Local\Programs\Ollama\lib\ollama\cuda_v12\ggml-cuda.dll
time=2025-10-30T15:45:32.882+08:00 level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX_VNNI=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,520,600,610,700,750,800,860,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 CUDA.1.ARCHS=500,520,600,610,700,750,800,860,890,900,1200 CUDA.1.USE_GRAPHS=1 CUDA.1.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2025-10-30T15:45:33.275+08:00 level=INFO source=runner.go:1210 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:8192 KvCacheType:q8_0 NumThreads:8 GPULayers:37[ID:GPU-ca44d757-2cc4-a597-149c-30b12b5bacb4 Layers:37(0..36)] MultiUserCache:true ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-10-30T15:45:33.696+08:00 level=INFO source=runner.go:1210 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:8192 KvCacheType:q8_0 NumThreads:8 GPULayers:37[ID:GPU-ca44d757-2cc4-a597-149c-30b12b5bacb4 Layers:37(0..36)] MultiUserCache:true ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-10-30T15:45:33.696+08:00 level=INFO source=ggml.go:481 msg="offloading 36 repeating layers to GPU"
time=2025-10-30T15:45:33.696+08:00 level=INFO source=ggml.go:488 msg="offloading output layer to GPU"
time=2025-10-30T15:45:33.696+08:00 level=INFO source=ggml.go:493 msg="offloaded 37/37 layers to GPU"
time=2025-10-30T15:45:33.696+08:00 level=INFO source=device.go:212 msg="model weights" device=CUDA1 size="5.4 GiB"
time=2025-10-30T15:45:33.696+08:00 level=INFO source=device.go:217 msg="model weights" device=CPU size="333.8 MiB"
time=2025-10-30T15:45:33.696+08:00 level=INFO source=device.go:223 msg="kv cache" device=CUDA1 size="612.0 MiB"
time=2025-10-30T15:45:33.696+08:00 level=INFO source=device.go:234 msg="compute graph" device=CUDA1 size="4.3 GiB"
time=2025-10-30T15:45:33.696+08:00 level=INFO source=device.go:239 msg="compute graph" device=CPU size="63.3 MiB"
time=2025-10-30T15:45:33.696+08:00 level=INFO source=device.go:244 msg="total memory" size="10.7 GiB"
time=2025-10-30T15:45:33.696+08:00 level=INFO source=sched.go:493 msg="loaded runners" count=1
time=2025-10-30T15:45:33.696+08:00 level=INFO source=server.go:1236 msg="waiting for llama runner to start responding"
time=2025-10-30T15:45:33.696+08:00 level=INFO source=server.go:1270 msg="waiting for server to become available" status="llm server loading model"
time=2025-10-30T15:45:34.948+08:00 level=INFO source=server.go:1274 msg="llama runner started in 2.23 seconds"
[GIN] 2025/10/30 - 15:45:34 | 200 |    2.6122977s |       127.0.0.1 | POST     "/api/generate"
[GIN] 2025/10/30 - 15:45:34 | 200 |    2.3186151s |       127.0.0.1 | POST     "/api/generate"
[GIN] 2025/10/30 - 15:45:53 | 200 |    9.0552306s |       127.0.0.1 | POST     "/api/chat"
[GIN] 2025/10/30 - 15:45:56 | 200 |            0s |  192.168.107.65 | GET      "/api/ps"
[GIN] 2025/10/30 - 15:45:56 | 200 |       1.539ms |  192.168.107.65 | GET      "/api/tags"
[GIN] 2025/10/30 - 15:45:58 | 200 |   12.4868612s |       127.0.0.1 | POST     "/api/chat"
[GIN] 2025/10/30 - 15:46:01 | 200 |            0s |       127.0.0.1 | HEAD     "/"
[GIN] 2025/10/30 - 15:46:01 | 200 |       3.094ms |       127.0.0.1 | GET      "/api/tags"
[GIN] 2025/10/30 - 15:46:15 | 200 |            0s |       127.0.0.1 | HEAD     "/"
[GIN] 2025/10/30 - 15:46:15 | 200 |     42.2701ms |       127.0.0.1 | POST     "/api/show"
ggml_backend_cuda_device_get_memory device GPU-ca44d757-2cc4-a597-149c-30b12b5bacb4 utilizing NVML memory reporting free: 5337882624 total: 17103323136
time=2025-10-30T15:46:15.475+08:00 level=INFO source=server.go:385 msg="starting runner" cmd="C:\\Users\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --port 55042"
time=2025-10-30T15:46:15.769+08:00 level=INFO source=cpu_windows.go:139 msg=packages count=1
time=2025-10-30T15:46:15.769+08:00 level=INFO source=cpu_windows.go:155 msg="efficiency cores detected" maxEfficiencyClass=1
time=2025-10-30T15:46:15.769+08:00 level=INFO source=cpu_windows.go:186 msg="" package=0 cores=16 efficiency=8 threads=24
time=2025-10-30T15:46:15.779+08:00 level=INFO source=sched.go:559 msg="updated VRAM based on existing loaded models" gpu=GPU-ca44d757-2cc4-a597-149c-30b12b5bacb4 library=CUDA total="15.9 GiB" available="4.8 GiB"
time=2025-10-30T15:46:15.779+08:00 level=INFO source=sched.go:559 msg="updated VRAM based on existing loaded models" gpu=GPU-fee6fb1c-00bf-c621-1289-0fbf6937dad0 library=CUDA total="15.9 GiB" available="13.4 GiB"
time=2025-10-30T15:46:15.811+08:00 level=INFO source=server.go:215 msg="enabling flash attention"
time=2025-10-30T15:46:15.811+08:00 level=INFO source=server.go:385 msg="starting runner" cmd="C:\\Users\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model C:\\Users\\.ollama\\models\\blobs\\sha256-ad7f12578413f17ce506bbf2809c3c7b8fd27bb1cf81acb3ac133a7538fc4259 --port 55047"
time=2025-10-30T15:46:15.816+08:00 level=INFO source=server.go:638 msg="loading model" "model layers"=37 requested=-1
time=2025-10-30T15:46:15.816+08:00 level=INFO source=server.go:643 msg="system memory" total="95.8 GiB" free="68.9 GiB" free_swap="52.0 GiB"
time=2025-10-30T15:46:15.817+08:00 level=INFO source=server.go:650 msg="gpu memory" id=GPU-ca44d757-2cc4-a597-149c-30b12b5bacb4 library=CUDA available="4.4 GiB" free="4.8 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-10-30T15:46:15.817+08:00 level=INFO source=server.go:650 msg="gpu memory" id=GPU-fee6fb1c-00bf-c621-1289-0fbf6937dad0 library=CUDA available="13.0 GiB" free="13.4 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-10-30T15:46:15.846+08:00 level=INFO source=runner.go:1337 msg="starting ollama engine"
time=2025-10-30T15:46:15.847+08:00 level=INFO source=runner.go:1372 msg="Server listening on 127.0.0.1:55047"
time=2025-10-30T15:46:15.850+08:00 level=INFO source=runner.go:1210 msg=load request="{Operation:fit LoraPath:[] Parallel:6 BatchSize:512 FlashAttention:true KvSize:49152 KvCacheType:q8_0 NumThreads:8 GPULayers:37[ID:GPU-fee6fb1c-00bf-c621-1289-0fbf6937dad0 Layers:37(0..36)] MultiUserCache:true ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-10-30T15:46:15.878+08:00 level=INFO source=ggml.go:135 msg="" architecture=qwen3 file_type=F16 name="Qwen3 4B Instruct 2507" description="" num_tensors=398 num_key_values=33
load_backend: loaded CPU backend from C:\Users\Matrix\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-alderlake.dll
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
  Device 0: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes, ID: GPU-fee6fb1c-00bf-c621-1289-0fbf6937dad0
  Device 1: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes, ID: GPU-ca44d757-2cc4-a597-149c-30b12b5bacb4
load_backend: loaded CUDA backend from C:\Users\Matrix\AppData\Local\Programs\Ollama\lib\ollama\cuda_v12\ggml-cuda.dll
time=2025-10-30T15:46:15.950+08:00 level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX_VNNI=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,520,600,610,700,750,800,860,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 CUDA.1.ARCHS=500,520,600,610,700,750,800,860,890,900,1200 CUDA.1.USE_GRAPHS=1 CUDA.1.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2025-10-30T15:46:16.108+08:00 level=INFO source=runner.go:1210 msg=load request="{Operation:alloc LoraPath:[] Parallel:6 BatchSize:512 FlashAttention:true KvSize:49152 KvCacheType:q8_0 NumThreads:8 GPULayers:37[ID:GPU-fee6fb1c-00bf-c621-1289-0fbf6937dad0 Layers:37(0..36)] MultiUserCache:true ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-10-30T15:46:16.366+08:00 level=INFO source=runner.go:1210 msg=load request="{Operation:commit LoraPath:[] Parallel:6 BatchSize:512 FlashAttention:true KvSize:49152 KvCacheType:q8_0 NumThreads:8 GPULayers:37[ID:GPU-fee6fb1c-00bf-c621-1289-0fbf6937dad0 Layers:37(0..36)] MultiUserCache:true ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-10-30T15:46:16.367+08:00 level=INFO source=ggml.go:481 msg="offloading 36 repeating layers to GPU"
time=2025-10-30T15:46:16.367+08:00 level=INFO source=device.go:212 msg="model weights" device=CUDA0 size="7.5 GiB"
time=2025-10-30T15:46:16.367+08:00 level=INFO source=ggml.go:488 msg="offloading output layer to GPU"
time=2025-10-30T15:46:16.367+08:00 level=INFO source=ggml.go:493 msg="offloaded 37/37 layers to GPU"
time=2025-10-30T15:46:16.367+08:00 level=INFO source=device.go:217 msg="model weights" device=CPU size="741.9 MiB"
time=2025-10-30T15:46:16.367+08:00 level=INFO source=device.go:223 msg="kv cache" device=CUDA0 size="3.6 GiB"
time=2025-10-30T15:46:16.367+08:00 level=INFO source=device.go:234 msg="compute graph" device=CUDA0 size="382.0 MiB"
time=2025-10-30T15:46:16.367+08:00 level=INFO source=device.go:239 msg="compute graph" device=CPU size="5.0 MiB"
time=2025-10-30T15:46:16.367+08:00 level=INFO source=device.go:244 msg="total memory" size="12.2 GiB"
time=2025-10-30T15:46:16.367+08:00 level=INFO source=sched.go:493 msg="loaded runners" count=2
time=2025-10-30T15:46:16.367+08:00 level=INFO source=server.go:1236 msg="waiting for llama runner to start responding"
time=2025-10-30T15:46:16.367+08:00 level=INFO source=server.go:1270 msg="waiting for server to become available" status="llm server loading model"
time=2025-10-30T15:46:18.121+08:00 level=INFO source=server.go:1274 msg="llama runner started in 2.30 seconds"
[GIN] 2025/10/30 - 15:46:18 | 200 |    2.7521383s |       127.0.0.1 | POST     "/api/generate"
[GIN] 2025/10/30 - 15:46:18 | 200 |    2.7301815s |       127.0.0.1 | POST     "/api/chat"
[GIN] 2025/10/30 - 15:46:21 | 200 |            0s |       127.0.0.1 | HEAD     "/"
[GIN] 2025/10/30 - 15:46:21 | 200 |      29.398ms |       127.0.0.1 | POST     "/api/show"
[GIN] 2025/10/30 - 15:46:21 | 200 |     59.0449ms |       127.0.0.1 | POST     "/api/generate"
[GIN] 2025/10/30 - 15:46:24 | 200 |    435.2697ms |       127.0.0.1 | POST     "/api/chat"
[GIN] 2025/10/30 - 15:46:25 | 200 |    378.0352ms |       127.0.0.1 | POST     "/api/chat"
[GIN] 2025/10/30 - 15:46:27 | 200 |            0s |  192.168.107.65 | GET      "/api/ps"
[GIN] 2025/10/30 - 15:46:27 | 200 |      4.8246ms |  192.168.107.65 | GET      "/api/tags"
[GIN] 2025/10/30 - 15:46:45 | 200 |     7.327683s |       127.0.0.1 | POST     "/api/chat"
[GIN] 2025/10/30 - 15:46:47 | 200 |    9.1881319s |       127.0.0.1 | POST     "/api/chat"

OS

Windows

GPU

Nvidia

CPU

Intel

Ollama version

0.12.7

Originally created by @Brian-209 on GitHub (Oct 30, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/12849 Originally assigned to: @jessegross on GitHub. ### What is the issue? qwen3-vl seems to have bug on handling parallel request. When I try to run 2 requests for one qwen3-vl model, the second one just queue and wait the first request to finish. Then I try to run 2 requests for one qwen3 model, it runs parallelly as normal. I have set OLLAMA_NUM_PARALLEL to 6 and tested within the same server. ### Relevant log output ```shell time=2025-10-30T15:45:22.285+08:00 level=INFO source=routes.go:1524 msg="server config" env="map[CUDA_VISIBLE_DEVICES: GGML_VK_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:8192 OLLAMA_DEBUG:INFO OLLAMA_FLASH_ATTENTION:true OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE:q8_0 OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:2 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:C:\\Users\\.ollama\\models OLLAMA_MULTIUSER_CACHE:true OLLAMA_NEW_ENGINE:true OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:6 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_REMOTES:[ollama.com] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES:]" time=2025-10-30T15:45:22.289+08:00 level=INFO source=images.go:522 msg="total blobs: 32" time=2025-10-30T15:45:22.289+08:00 level=INFO source=images.go:529 msg="total unused blobs removed: 0" time=2025-10-30T15:45:22.290+08:00 level=INFO source=routes.go:1577 msg="Listening on [::]:11434 (version 0.12.7)" time=2025-10-30T15:45:22.291+08:00 level=INFO source=runner.go:76 msg="discovering available GPUs..." time=2025-10-30T15:45:22.295+08:00 level=INFO source=server.go:385 msg="starting runner" cmd="C:\\Users\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --port 56716" time=2025-10-30T15:45:22.400+08:00 level=INFO source=server.go:385 msg="starting runner" cmd="C:\\Users\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --port 56721" time=2025-10-30T15:45:22.471+08:00 level=INFO source=server.go:385 msg="starting runner" cmd="C:\\Users\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --port 56726" time=2025-10-30T15:45:22.720+08:00 level=INFO source=server.go:385 msg="starting runner" cmd="C:\\Users\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --port 56732" time=2025-10-30T15:45:22.720+08:00 level=INFO source=server.go:385 msg="starting runner" cmd="C:\\Users\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --port 56733" time=2025-10-30T15:45:22.952+08:00 level=INFO source=types.go:42 msg="inference compute" id=GPU-ca44d757-2cc4-a597-149c-30b12b5bacb4 filtered_id="" library=CUDA compute=12.0 name=CUDA1 description="NVIDIA GeForce RTX 5060 Ti" libdirs=ollama,cuda_v12 driver=12.9 pci_id=0000:05:00.0 type=discrete total="15.9 GiB" available="15.4 GiB" time=2025-10-30T15:45:22.952+08:00 level=INFO source=types.go:42 msg="inference compute" id=GPU-fee6fb1c-00bf-c621-1289-0fbf6937dad0 filtered_id="" library=CUDA compute=12.0 name=CUDA0 description="NVIDIA GeForce RTX 5060 Ti" libdirs=ollama,cuda_v12 driver=12.9 pci_id=0000:01:00.0 type=discrete total="15.9 GiB" available="13.5 GiB" [GIN] 2025/10/30 - 15:45:22 | 200 | 0s | 127.0.0.1 | HEAD "/" [GIN] 2025/10/30 - 15:45:22 | 200 | 0s | 127.0.0.1 | GET "/api/ps" [GIN] 2025/10/30 - 15:45:26 | 200 | 0s | 192.168.107.65 | GET "/api/ps" [GIN] 2025/10/30 - 15:45:26 | 200 | 12.2058ms | 192.168.107.65 | GET "/api/tags" [GIN] 2025/10/30 - 15:45:32 | 200 | 0s | 127.0.0.1 | HEAD "/" [GIN] 2025/10/30 - 15:45:32 | 200 | 33.7115ms | 127.0.0.1 | POST "/api/show" time=2025-10-30T15:45:32.395+08:00 level=INFO source=server.go:385 msg="starting runner" cmd="C:\\Users\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --port 56750" [GIN] 2025/10/30 - 15:45:32 | 200 | 0s | 127.0.0.1 | HEAD "/" [GIN] 2025/10/30 - 15:45:32 | 200 | 34.1516ms | 127.0.0.1 | POST "/api/show" time=2025-10-30T15:45:32.673+08:00 level=INFO source=cpu_windows.go:139 msg=packages count=1 time=2025-10-30T15:45:32.673+08:00 level=INFO source=cpu_windows.go:155 msg="efficiency cores detected" maxEfficiencyClass=1 time=2025-10-30T15:45:32.673+08:00 level=INFO source=cpu_windows.go:186 msg="" package=0 cores=16 efficiency=8 threads=24 time=2025-10-30T15:45:32.702+08:00 level=WARN source=sched.go:397 msg="model architecture does not currently support parallel requests" architecture=qwen3vl time=2025-10-30T15:45:32.720+08:00 level=INFO source=server.go:215 msg="enabling flash attention" time=2025-10-30T15:45:32.720+08:00 level=INFO source=server.go:385 msg="starting runner" cmd="C:\\Users\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model C:\\Users\\.ollama\\models\\blobs\\sha256-ed12a4674d727a74ac4816c906094ea9d3119fbea46ca93288c3ce4ffbe38c55 --port 56756" time=2025-10-30T15:45:32.720+08:00 level=INFO source=server.go:638 msg="loading model" "model layers"=37 requested=-1 time=2025-10-30T15:45:32.720+08:00 level=INFO source=server.go:643 msg="system memory" total="95.8 GiB" free="69.9 GiB" free_swap="63.9 GiB" time=2025-10-30T15:45:32.720+08:00 level=INFO source=server.go:650 msg="gpu memory" id=GPU-ca44d757-2cc4-a597-149c-30b12b5bacb4 library=CUDA available="14.9 GiB" free="15.4 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-10-30T15:45:32.720+08:00 level=INFO source=server.go:650 msg="gpu memory" id=GPU-fee6fb1c-00bf-c621-1289-0fbf6937dad0 library=CUDA available="13.1 GiB" free="13.5 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-10-30T15:45:32.756+08:00 level=INFO source=runner.go:1337 msg="starting ollama engine" time=2025-10-30T15:45:32.756+08:00 level=INFO source=runner.go:1372 msg="Server listening on 127.0.0.1:56756" time=2025-10-30T15:45:32.766+08:00 level=INFO source=runner.go:1210 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:8192 KvCacheType:q8_0 NumThreads:8 GPULayers:37[ID:GPU-ca44d757-2cc4-a597-149c-30b12b5bacb4 Layers:37(0..36)] MultiUserCache:true ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-10-30T15:45:32.784+08:00 level=INFO source=ggml.go:135 msg="" architecture=qwen3vl file_type=Q4_K_M name="" description="" num_tensors=858 num_key_values=40 load_backend: loaded CPU backend from C:\Users\Matrix\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-alderlake.dll ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 2 CUDA devices: Device 0: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes, ID: GPU-fee6fb1c-00bf-c621-1289-0fbf6937dad0 Device 1: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes, ID: GPU-ca44d757-2cc4-a597-149c-30b12b5bacb4 load_backend: loaded CUDA backend from C:\Users\Matrix\AppData\Local\Programs\Ollama\lib\ollama\cuda_v12\ggml-cuda.dll time=2025-10-30T15:45:32.882+08:00 level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX_VNNI=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,520,600,610,700,750,800,860,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 CUDA.1.ARCHS=500,520,600,610,700,750,800,860,890,900,1200 CUDA.1.USE_GRAPHS=1 CUDA.1.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2025-10-30T15:45:33.275+08:00 level=INFO source=runner.go:1210 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:8192 KvCacheType:q8_0 NumThreads:8 GPULayers:37[ID:GPU-ca44d757-2cc4-a597-149c-30b12b5bacb4 Layers:37(0..36)] MultiUserCache:true ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-10-30T15:45:33.696+08:00 level=INFO source=runner.go:1210 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:8192 KvCacheType:q8_0 NumThreads:8 GPULayers:37[ID:GPU-ca44d757-2cc4-a597-149c-30b12b5bacb4 Layers:37(0..36)] MultiUserCache:true ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-10-30T15:45:33.696+08:00 level=INFO source=ggml.go:481 msg="offloading 36 repeating layers to GPU" time=2025-10-30T15:45:33.696+08:00 level=INFO source=ggml.go:488 msg="offloading output layer to GPU" time=2025-10-30T15:45:33.696+08:00 level=INFO source=ggml.go:493 msg="offloaded 37/37 layers to GPU" time=2025-10-30T15:45:33.696+08:00 level=INFO source=device.go:212 msg="model weights" device=CUDA1 size="5.4 GiB" time=2025-10-30T15:45:33.696+08:00 level=INFO source=device.go:217 msg="model weights" device=CPU size="333.8 MiB" time=2025-10-30T15:45:33.696+08:00 level=INFO source=device.go:223 msg="kv cache" device=CUDA1 size="612.0 MiB" time=2025-10-30T15:45:33.696+08:00 level=INFO source=device.go:234 msg="compute graph" device=CUDA1 size="4.3 GiB" time=2025-10-30T15:45:33.696+08:00 level=INFO source=device.go:239 msg="compute graph" device=CPU size="63.3 MiB" time=2025-10-30T15:45:33.696+08:00 level=INFO source=device.go:244 msg="total memory" size="10.7 GiB" time=2025-10-30T15:45:33.696+08:00 level=INFO source=sched.go:493 msg="loaded runners" count=1 time=2025-10-30T15:45:33.696+08:00 level=INFO source=server.go:1236 msg="waiting for llama runner to start responding" time=2025-10-30T15:45:33.696+08:00 level=INFO source=server.go:1270 msg="waiting for server to become available" status="llm server loading model" time=2025-10-30T15:45:34.948+08:00 level=INFO source=server.go:1274 msg="llama runner started in 2.23 seconds" [GIN] 2025/10/30 - 15:45:34 | 200 | 2.6122977s | 127.0.0.1 | POST "/api/generate" [GIN] 2025/10/30 - 15:45:34 | 200 | 2.3186151s | 127.0.0.1 | POST "/api/generate" [GIN] 2025/10/30 - 15:45:53 | 200 | 9.0552306s | 127.0.0.1 | POST "/api/chat" [GIN] 2025/10/30 - 15:45:56 | 200 | 0s | 192.168.107.65 | GET "/api/ps" [GIN] 2025/10/30 - 15:45:56 | 200 | 1.539ms | 192.168.107.65 | GET "/api/tags" [GIN] 2025/10/30 - 15:45:58 | 200 | 12.4868612s | 127.0.0.1 | POST "/api/chat" [GIN] 2025/10/30 - 15:46:01 | 200 | 0s | 127.0.0.1 | HEAD "/" [GIN] 2025/10/30 - 15:46:01 | 200 | 3.094ms | 127.0.0.1 | GET "/api/tags" [GIN] 2025/10/30 - 15:46:15 | 200 | 0s | 127.0.0.1 | HEAD "/" [GIN] 2025/10/30 - 15:46:15 | 200 | 42.2701ms | 127.0.0.1 | POST "/api/show" ggml_backend_cuda_device_get_memory device GPU-ca44d757-2cc4-a597-149c-30b12b5bacb4 utilizing NVML memory reporting free: 5337882624 total: 17103323136 time=2025-10-30T15:46:15.475+08:00 level=INFO source=server.go:385 msg="starting runner" cmd="C:\\Users\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --port 55042" time=2025-10-30T15:46:15.769+08:00 level=INFO source=cpu_windows.go:139 msg=packages count=1 time=2025-10-30T15:46:15.769+08:00 level=INFO source=cpu_windows.go:155 msg="efficiency cores detected" maxEfficiencyClass=1 time=2025-10-30T15:46:15.769+08:00 level=INFO source=cpu_windows.go:186 msg="" package=0 cores=16 efficiency=8 threads=24 time=2025-10-30T15:46:15.779+08:00 level=INFO source=sched.go:559 msg="updated VRAM based on existing loaded models" gpu=GPU-ca44d757-2cc4-a597-149c-30b12b5bacb4 library=CUDA total="15.9 GiB" available="4.8 GiB" time=2025-10-30T15:46:15.779+08:00 level=INFO source=sched.go:559 msg="updated VRAM based on existing loaded models" gpu=GPU-fee6fb1c-00bf-c621-1289-0fbf6937dad0 library=CUDA total="15.9 GiB" available="13.4 GiB" time=2025-10-30T15:46:15.811+08:00 level=INFO source=server.go:215 msg="enabling flash attention" time=2025-10-30T15:46:15.811+08:00 level=INFO source=server.go:385 msg="starting runner" cmd="C:\\Users\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model C:\\Users\\.ollama\\models\\blobs\\sha256-ad7f12578413f17ce506bbf2809c3c7b8fd27bb1cf81acb3ac133a7538fc4259 --port 55047" time=2025-10-30T15:46:15.816+08:00 level=INFO source=server.go:638 msg="loading model" "model layers"=37 requested=-1 time=2025-10-30T15:46:15.816+08:00 level=INFO source=server.go:643 msg="system memory" total="95.8 GiB" free="68.9 GiB" free_swap="52.0 GiB" time=2025-10-30T15:46:15.817+08:00 level=INFO source=server.go:650 msg="gpu memory" id=GPU-ca44d757-2cc4-a597-149c-30b12b5bacb4 library=CUDA available="4.4 GiB" free="4.8 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-10-30T15:46:15.817+08:00 level=INFO source=server.go:650 msg="gpu memory" id=GPU-fee6fb1c-00bf-c621-1289-0fbf6937dad0 library=CUDA available="13.0 GiB" free="13.4 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-10-30T15:46:15.846+08:00 level=INFO source=runner.go:1337 msg="starting ollama engine" time=2025-10-30T15:46:15.847+08:00 level=INFO source=runner.go:1372 msg="Server listening on 127.0.0.1:55047" time=2025-10-30T15:46:15.850+08:00 level=INFO source=runner.go:1210 msg=load request="{Operation:fit LoraPath:[] Parallel:6 BatchSize:512 FlashAttention:true KvSize:49152 KvCacheType:q8_0 NumThreads:8 GPULayers:37[ID:GPU-fee6fb1c-00bf-c621-1289-0fbf6937dad0 Layers:37(0..36)] MultiUserCache:true ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-10-30T15:46:15.878+08:00 level=INFO source=ggml.go:135 msg="" architecture=qwen3 file_type=F16 name="Qwen3 4B Instruct 2507" description="" num_tensors=398 num_key_values=33 load_backend: loaded CPU backend from C:\Users\Matrix\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-alderlake.dll ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 2 CUDA devices: Device 0: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes, ID: GPU-fee6fb1c-00bf-c621-1289-0fbf6937dad0 Device 1: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes, ID: GPU-ca44d757-2cc4-a597-149c-30b12b5bacb4 load_backend: loaded CUDA backend from C:\Users\Matrix\AppData\Local\Programs\Ollama\lib\ollama\cuda_v12\ggml-cuda.dll time=2025-10-30T15:46:15.950+08:00 level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX_VNNI=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,520,600,610,700,750,800,860,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 CUDA.1.ARCHS=500,520,600,610,700,750,800,860,890,900,1200 CUDA.1.USE_GRAPHS=1 CUDA.1.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2025-10-30T15:46:16.108+08:00 level=INFO source=runner.go:1210 msg=load request="{Operation:alloc LoraPath:[] Parallel:6 BatchSize:512 FlashAttention:true KvSize:49152 KvCacheType:q8_0 NumThreads:8 GPULayers:37[ID:GPU-fee6fb1c-00bf-c621-1289-0fbf6937dad0 Layers:37(0..36)] MultiUserCache:true ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-10-30T15:46:16.366+08:00 level=INFO source=runner.go:1210 msg=load request="{Operation:commit LoraPath:[] Parallel:6 BatchSize:512 FlashAttention:true KvSize:49152 KvCacheType:q8_0 NumThreads:8 GPULayers:37[ID:GPU-fee6fb1c-00bf-c621-1289-0fbf6937dad0 Layers:37(0..36)] MultiUserCache:true ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-10-30T15:46:16.367+08:00 level=INFO source=ggml.go:481 msg="offloading 36 repeating layers to GPU" time=2025-10-30T15:46:16.367+08:00 level=INFO source=device.go:212 msg="model weights" device=CUDA0 size="7.5 GiB" time=2025-10-30T15:46:16.367+08:00 level=INFO source=ggml.go:488 msg="offloading output layer to GPU" time=2025-10-30T15:46:16.367+08:00 level=INFO source=ggml.go:493 msg="offloaded 37/37 layers to GPU" time=2025-10-30T15:46:16.367+08:00 level=INFO source=device.go:217 msg="model weights" device=CPU size="741.9 MiB" time=2025-10-30T15:46:16.367+08:00 level=INFO source=device.go:223 msg="kv cache" device=CUDA0 size="3.6 GiB" time=2025-10-30T15:46:16.367+08:00 level=INFO source=device.go:234 msg="compute graph" device=CUDA0 size="382.0 MiB" time=2025-10-30T15:46:16.367+08:00 level=INFO source=device.go:239 msg="compute graph" device=CPU size="5.0 MiB" time=2025-10-30T15:46:16.367+08:00 level=INFO source=device.go:244 msg="total memory" size="12.2 GiB" time=2025-10-30T15:46:16.367+08:00 level=INFO source=sched.go:493 msg="loaded runners" count=2 time=2025-10-30T15:46:16.367+08:00 level=INFO source=server.go:1236 msg="waiting for llama runner to start responding" time=2025-10-30T15:46:16.367+08:00 level=INFO source=server.go:1270 msg="waiting for server to become available" status="llm server loading model" time=2025-10-30T15:46:18.121+08:00 level=INFO source=server.go:1274 msg="llama runner started in 2.30 seconds" [GIN] 2025/10/30 - 15:46:18 | 200 | 2.7521383s | 127.0.0.1 | POST "/api/generate" [GIN] 2025/10/30 - 15:46:18 | 200 | 2.7301815s | 127.0.0.1 | POST "/api/chat" [GIN] 2025/10/30 - 15:46:21 | 200 | 0s | 127.0.0.1 | HEAD "/" [GIN] 2025/10/30 - 15:46:21 | 200 | 29.398ms | 127.0.0.1 | POST "/api/show" [GIN] 2025/10/30 - 15:46:21 | 200 | 59.0449ms | 127.0.0.1 | POST "/api/generate" [GIN] 2025/10/30 - 15:46:24 | 200 | 435.2697ms | 127.0.0.1 | POST "/api/chat" [GIN] 2025/10/30 - 15:46:25 | 200 | 378.0352ms | 127.0.0.1 | POST "/api/chat" [GIN] 2025/10/30 - 15:46:27 | 200 | 0s | 192.168.107.65 | GET "/api/ps" [GIN] 2025/10/30 - 15:46:27 | 200 | 4.8246ms | 192.168.107.65 | GET "/api/tags" [GIN] 2025/10/30 - 15:46:45 | 200 | 7.327683s | 127.0.0.1 | POST "/api/chat" [GIN] 2025/10/30 - 15:46:47 | 200 | 9.1881319s | 127.0.0.1 | POST "/api/chat" ``` ### OS Windows ### GPU Nvidia ### CPU Intel ### Ollama version 0.12.7
GiteaMirror added the bug label 2026-04-22 17:42:54 -05:00
GiteaMirror commented 2026-04-22 17:42:56 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Oct 30, 2025):

time=2025-10-30T15:45:32.702+08:00 level=WARN source=sched.go:397 msg="model architecture does not currently support parallel requests" architecture=qwen3vl

0a2d92081b/server/sched.go (L393-L398)

<!-- gh-comment-id:3467670652 --> @rick-github commented on GitHub (Oct 30, 2025): ``` time=2025-10-30T15:45:32.702+08:00 level=WARN source=sched.go:397 msg="model architecture does not currently support parallel requests" architecture=qwen3vl ``` https://github.com/ollama/ollama/blob/0a2d92081bb6b6b2d3eab5908fce08cfcf736e1d/server/sched.go#L393-L398
GiteaMirror commented 2026-04-22 17:42:56 -05:00
Author
Owner
Copy Link

@XiaZixun commented on GitHub (Nov 4, 2025):

Hello. I would like to ask whether this issue should be considered a feature, or can we expect it to be fixed in a future version?

<!-- gh-comment-id:3486426557 --> @XiaZixun commented on GitHub (Nov 4, 2025): Hello. I would like to ask whether this issue should be considered a feature, or can we expect it to be fixed in a future version?
GiteaMirror commented 2026-04-22 17:42:56 -05:00
Author
Owner
Copy Link

@youyuzzg commented on GitHub (Nov 11, 2025):

I have also noticed this bug. Waiting for it to be fixed.

<!-- gh-comment-id:3514626966 --> @youyuzzg commented on GitHub (Nov 11, 2025): I have also noticed this bug. Waiting for it to be fixed.
GiteaMirror commented 2026-04-22 17:42:57 -05:00
Author
Owner
Copy Link

@wenerme commented on GitHub (Nov 14, 2025):

switch to llamacpp for this, super simple, super fast

mkdir models
curl -L -o models/qwen3-vl-4b-mmproj-F16.gguf "https://hf-mirror.com/unsloth/Qwen3-VL-4B-Instruct-GGUF/resolve/main/mmproj-F16.gguf"


docker run --rm -it \
  --gpus all \
  -p 8080:8080 \
  -v $PWD/models:/models \
  -v $PWD/models:/root/.cache/llama.cpp \
  -e HF_ENDPOINT=https://modelscope.cn \
  --name llama.cpp ghcr.m.daocloud.io/ggml-org/llama.cpp:server-cuda \
  -hf unsloth/Qwen3-VL-4B-Instruct-GGUF:Q4_K_M -np 32 -c 131072 -ngl 99 -cb --mmproj /models/qwen3-vl-mmproj-F16.gguf -ngl 99

now i can have 32 concurrent with 4k (131072/32) context

<!-- gh-comment-id:3531970724 --> @wenerme commented on GitHub (Nov 14, 2025): switch to llamacpp for this, super simple, super fast ```bash mkdir models curl -L -o models/qwen3-vl-4b-mmproj-F16.gguf "https://hf-mirror.com/unsloth/Qwen3-VL-4B-Instruct-GGUF/resolve/main/mmproj-F16.gguf" docker run --rm -it \ --gpus all \ -p 8080:8080 \ -v $PWD/models:/models \ -v $PWD/models:/root/.cache/llama.cpp \ -e HF_ENDPOINT=https://modelscope.cn \ --name llama.cpp ghcr.m.daocloud.io/ggml-org/llama.cpp:server-cuda \ -hf unsloth/Qwen3-VL-4B-Instruct-GGUF:Q4_K_M -np 32 -c 131072 -ngl 99 -cb --mmproj /models/qwen3-vl-mmproj-F16.gguf -ngl 99 ``` now i can have 32 concurrent with 4k (131072/32) context
GiteaMirror commented 2026-04-22 17:42:57 -05:00
Author
Owner
Copy Link

@aas72 commented on GitHub (Jan 30, 2026):

I have also noticed this bug. Is there any chance this will be fixed any time soon?

<!-- gh-comment-id:3823085620 --> @aas72 commented on GitHub (Jan 30, 2026): I have also noticed this bug. Is there any chance this will be fixed any time soon?
GiteaMirror commented 2026-04-22 17:42:57 -05:00
Author
Owner
Copy Link

@youyuzzg commented on GitHub (Feb 5, 2026):

I have also noticed this bug. Is there any chance this will be fixed any time soon?

Same here, still waiting for a fix.

<!-- gh-comment-id:3850951557 --> @youyuzzg commented on GitHub (Feb 5, 2026): > I have also noticed this bug. Is there any chance this will be fixed any time soon? Same here, still waiting for a fix.
GiteaMirror commented 2026-04-22 17:42:58 -05:00
Author
Owner
Copy Link

@XiaZixun commented on GitHub (Feb 27, 2026):

// Some architectures are not safe with num_parallel > 1.
// ref: https://github.com/ollama/ollama/issues/4165
if slices.Contains([]string{"mllama", "qwen3vl", "qwen3vlmoe", "qwen35", "qwen35moe", "qwen3next", "lfm2", "lfm2moe", "nemotron_h", "nemotron_h_moe"}, req.model.Config.ModelFamily) && numParallel != 1 {
	numParallel = 1
	slog.Warn("model architecture does not currently support parallel requests", "architecture", req.model.Config.ModelFamily)
	}

The list of models that cannot be called in parallel is getting longer and longer, especially evident in the latest VLMs.
Sincerely hope this issue can be resolved.

Additionally, LlamaBarn seems to avoid this issue.

<!-- gh-comment-id:3973621407 --> @XiaZixun commented on GitHub (Feb 27, 2026): ``` // Some architectures are not safe with num_parallel > 1. // ref: https://github.com/ollama/ollama/issues/4165 if slices.Contains([]string{"mllama", "qwen3vl", "qwen3vlmoe", "qwen35", "qwen35moe", "qwen3next", "lfm2", "lfm2moe", "nemotron_h", "nemotron_h_moe"}, req.model.Config.ModelFamily) && numParallel != 1 { numParallel = 1 slog.Warn("model architecture does not currently support parallel requests", "architecture", req.model.Config.ModelFamily) } ``` The list of models that cannot be called in parallel is getting longer and longer, especially evident in the latest VLMs. Sincerely hope this issue can be resolved. Additionally, [LlamaBarn](https://github.com/ggml-org/LlamaBarn) seems to avoid this issue.
GiteaMirror commented 2026-04-22 17:42:59 -05:00
Author
Owner
Copy Link

@prcoe1 commented on GitHub (Mar 1, 2026):

Same just noticed this for qwen35 27b and 35b, i noticed lower than expected mem use and perf and then logs confirmed just sticks to parallel:1

<!-- gh-comment-id:3980813398 --> @prcoe1 commented on GitHub (Mar 1, 2026): Same just noticed this for qwen35 27b and 35b, i noticed lower than expected mem use and perf and then logs confirmed just sticks to parallel:1
GiteaMirror commented 2026-04-22 17:42:59 -05:00
Author
Owner
Copy Link

@charlesdrakon-cmyk commented on GitHub (Mar 19, 2026):

I am experiencing the exact same situation on qwen3.5 35b and 122b. Any idea when we might see some motion on this?

<!-- gh-comment-id:4092995445 --> @charlesdrakon-cmyk commented on GitHub (Mar 19, 2026): I am experiencing the exact same situation on qwen3.5 35b and 122b. Any idea when we might see some motion on this?
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#34274
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?