github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-08 00:51:34 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #10297] the issue of multiple GPU inference models running to the CPU for gemma3 27b #53274

New Issue
Closed
opened 2026-04-29 02:26:30 -05:00 by GiteaMirror · 4 comments
GiteaMirror commented 2026-04-29 02:26:30 -05:00
Owner
Copy Link

Originally created by @leizhu1989 on GitHub (Apr 16, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/10297

What is the issue?

system：Ubuntu
gpu：T4*2

When I run the gemma3:27b model on two T4 graphics cards, the weights are assigned to the two cards, but the inference seems to run to the CPU, and the speed is very slow. The usage rate of the two GPUs is about 30%, and sometimes only one card is used.Does it not support multi graphics card inference for a large model

docker logs:
ollama-container-0 | 2025/04/16 08:50:23 routes.go:1231: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:2048 OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:true OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:1 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:4 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
ollama-container-0 | time=2025-04-16T08:50:23.586Z level=INFO source=images.go:458 msg="total blobs: 5"
ollama-container-0 | time=2025-04-16T08:50:23.587Z level=INFO source=images.go:465 msg="total unused blobs removed: 0"
ollama-container-0 | time=2025-04-16T08:50:23.587Z level=INFO source=routes.go:1298 msg="Listening on [::]:11434 (version 0.6.5)"
ollama-container-0 | time=2025-04-16T08:50:23.588Z level=DEBUG source=sched.go:107 msg="starting llm scheduler"
ollama-container-0 | time=2025-04-16T08:50:23.588Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
ollama-container-0 | time=2025-04-16T08:50:23.590Z level=DEBUG source=gpu.go:98 msg="searching for GPU discovery libraries for NVIDIA"
ollama-container-0 | time=2025-04-16T08:50:23.590Z level=DEBUG source=gpu.go:501 msg="Searching for GPU library" name=libcuda.so
ollama-container-0 | time=2025-04-16T08:50:23.590Z level=DEBUG source=gpu.go:525 msg="gpu library search" globs="[/usr/lib/ollama/libcuda.so* /usr/local/nvidia/lib/libcuda.so* /usr/local/nvidia/lib64/libcuda.so* /usr/local/cuda*/targets//lib/libcuda.so /usr/lib/-linux-gnu/nvidia/current/libcuda.so /usr/lib/-linux-gnu/libcuda.so /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers//libcuda.so /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]"
ollama-container-0 | time=2025-04-16T08:50:23.591Z level=DEBUG source=gpu.go:558 msg="discovered GPU libraries" paths=[/usr/lib/x86_64-linux-gnu/libcuda.so.550.54.15]
ollama-container-0 | initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.54.15
ollama-container-0 | dlsym: cuInit - 0x7fc5fe6feef0
ollama-container-0 | dlsym: cuDriverGetVersion - 0x7fc5fe6fef10
ollama-container-0 | dlsym: cuDeviceGetCount - 0x7fc5fe6fef50
ollama-container-0 | dlsym: cuDeviceGet - 0x7fc5fe6fef30
ollama-container-0 | dlsym: cuDeviceGetAttribute - 0x7fc5fe6ff030
ollama-container-0 | dlsym: cuDeviceGetUuid - 0x7fc5fe6fef90
ollama-container-0 | dlsym: cuDeviceGetName - 0x7fc5fe6fef70
ollama-container-0 | dlsym: cuCtxCreate_v3 - 0x7fc5fe6ff210
ollama-container-0 | dlsym: cuMemGetInfo_v2 - 0x7fc5fe709190
ollama-container-0 | dlsym: cuCtxDestroy - 0x7fc5fe7637f0
ollama-container-0 | calling cuInit
ollama-container-0 | calling cuDriverGetVersion
ollama-container-0 | raw version 0x2f08
ollama-container-0 | CUDA driver version: 12.4
ollama-container-0 | calling cuDeviceGetCount
ollama-container-0 | device count 2
ollama-container-0 | time=2025-04-16T08:50:23.865Z level=DEBUG source=gpu.go:125 msg="detected GPUs" count=2 library=/usr/lib/x86_64-linux-gnu/libcuda.so.550.54.15
ollama-container-0 | [GPU-201dbf06-7d5c-1810-1a5a-c5af998610da] CUDA totalMem 14917 mb
ollama-container-0 | [GPU-201dbf06-7d5c-1810-1a5a-c5af998610da] CUDA freeMem 14814 mb
ollama-container-0 | [GPU-201dbf06-7d5c-1810-1a5a-c5af998610da] Compute Capability 7.5
ollama-container-0 | [GPU-e2bd2759-00e8-a0df-89f5-daa13e4b13d4] CUDA totalMem 14917 mb
ollama-container-0 | [GPU-e2bd2759-00e8-a0df-89f5-daa13e4b13d4] CUDA freeMem 14814 mb
ollama-container-0 | [GPU-e2bd2759-00e8-a0df-89f5-daa13e4b13d4] Compute Capability 7.5
ollama-container-0 | time=2025-04-16T08:50:24.521Z level=DEBUG source=amd_linux.go:419 msg="amdgpu driver not detected /sys/module/amdgpu"
ollama-container-0 | releasing cuda driver library
ollama-container-0 | time=2025-04-16T08:50:24.522Z level=INFO source=types.go:130 msg="inference compute" id=GPU-201dbf06-7d5c-1810-1a5a-c5af998610da library=cuda variant=v12 compute=7.5 driver=12.4 name="Tesla T4" total="14.6 GiB" available="14.5 GiB"
ollama-container-0 | time=2025-04-16T08:50:24.522Z level=INFO source=types.go:130 msg="inference compute" id=GPU-e2bd2759-00e8-a0df-89f5-daa13e4b13d4 library=cuda variant=v12 compute=7.5 driver=12.4 name="Tesla T4" total="14.6 GiB" available="14.5 GiB"
ollama-container-0 | [GIN] 2025/04/16 - 08:50:26 | 200 | 1.57596432s | 10.1.10.32 | POST "/api/pull"
ollama-container-0 | time=2025-04-16T08:50:26.336Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="125.9 GiB" before.free="119.3 GiB" before.free_swap="7.0 GiB" now.total="125.9 GiB" now.free="119.2 GiB" now.free_swap="7.0 GiB"
ollama-container-0 | initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.54.15
ollama-container-0 | dlsym: cuInit - 0x7fc5fe6feef0
ollama-container-0 | dlsym: cuDriverGetVersion - 0x7fc5fe6fef10
ollama-container-0 | dlsym: cuDeviceGetCount - 0x7fc5fe6fef50
ollama-container-0 | dlsym: cuDeviceGet - 0x7fc5fe6fef30
ollama-container-0 | dlsym: cuDeviceGetAttribute - 0x7fc5fe6ff030
ollama-container-0 | dlsym: cuDeviceGetUuid - 0x7fc5fe6fef90
ollama-container-0 | dlsym: cuDeviceGetName - 0x7fc5fe6fef70
ollama-container-0 | dlsym: cuCtxCreate_v3 - 0x7fc5fe6ff210
ollama-container-0 | dlsym: cuMemGetInfo_v2 - 0x7fc5fe709190
ollama-container-0 | dlsym: cuCtxDestroy - 0x7fc5fe7637f0
ollama-container-0 | calling cuInit
ollama-container-0 | calling cuDriverGetVersion
ollama-container-0 | raw version 0x2f08
ollama-container-0 | CUDA driver version: 12.4
ollama-container-0 | calling cuDeviceGetCount
ollama-container-0 | device count 2
ollama-container-0 | time=2025-04-16T08:50:26.945Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-201dbf06-7d5c-1810-1a5a-c5af998610da name="Tesla T4" overhead="0 B" before.total="14.6 GiB" before.free="14.5 GiB" now.total="14.6 GiB" now.free="14.5 GiB" now.used="102.9 MiB"
ollama-container-0 | time=2025-04-16T08:50:27.271Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-e2bd2759-00e8-a0df-89f5-daa13e4b13d4 name="Tesla T4" overhead="0 B" before.total="14.6 GiB" before.free="14.5 GiB" now.total="14.6 GiB" now.free="14.5 GiB" now.used="102.9 MiB"
ollama-container-0 | releasing cuda driver library
ollama-container-0 | time=2025-04-16T08:50:27.395Z level=DEBUG source=sched.go:226 msg="loading first model" model=/root/.ollama/models/blobs/sha256-e796792eba26c4d3b04b0ac5adb01a453dd9ec2dfd83b6c59cbf6fe5f30b0f68
ollama-container-0 | time=2025-04-16T08:50:27.396Z level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=1 available="[14.5 GiB]"
ollama-container-0 | time=2025-04-16T08:50:27.398Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="125.9 GiB" before.free="119.2 GiB" before.free_swap="7.0 GiB" now.total="125.9 GiB" now.free="119.2 GiB" now.free_swap="7.0 GiB"
ollama-container-0 | initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.54.15
ollama-container-0 | dlsym: cuInit - 0x7fc5fe6feef0
ollama-container-0 | dlsym: cuDriverGetVersion - 0x7fc5fe6fef10
ollama-container-0 | dlsym: cuDeviceGetCount - 0x7fc5fe6fef50
ollama-container-0 | dlsym: cuDeviceGet - 0x7fc5fe6fef30
ollama-container-0 | dlsym: cuDeviceGetAttribute - 0x7fc5fe6ff030
ollama-container-0 | dlsym: cuDeviceGetUuid - 0x7fc5fe6fef90
ollama-container-0 | dlsym: cuDeviceGetName - 0x7fc5fe6fef70
ollama-container-0 | dlsym: cuCtxCreate_v3 - 0x7fc5fe6ff210
ollama-container-0 | dlsym: cuMemGetInfo_v2 - 0x7fc5fe709190
ollama-container-0 | dlsym: cuCtxDestroy - 0x7fc5fe7637f0
ollama-container-0 | calling cuInit
ollama-container-0 | calling cuDriverGetVersion
ollama-container-0 | raw version 0x2f08
ollama-container-0 | CUDA driver version: 12.4
ollama-container-0 | calling cuDeviceGetCount
ollama-container-0 | device count 2
ollama-container-0 | time=2025-04-16T08:50:27.702Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-201dbf06-7d5c-1810-1a5a-c5af998610da name="Tesla T4" overhead="0 B" before.total="14.6 GiB" before.free="14.5 GiB" now.total="14.6 GiB" now.free="14.5 GiB" now.used="102.9 MiB"
ollama-container-0 | time=2025-04-16T08:50:27.965Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-e2bd2759-00e8-a0df-89f5-daa13e4b13d4 name="Tesla T4" overhead="0 B" before.total="14.6 GiB" before.free="14.5 GiB" now.total="14.6 GiB" now.free="14.5 GiB" now.used="102.9 MiB"
ollama-container-0 | releasing cuda driver library
ollama-container-0 | time=2025-04-16T08:50:27.966Z level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=1 available="[14.5 GiB]"
ollama-container-0 | time=2025-04-16T08:50:27.971Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="125.9 GiB" before.free="119.2 GiB" before.free_swap="7.0 GiB" now.total="125.9 GiB" now.free="119.2 GiB" now.free_swap="7.0 GiB"
ollama-container-0 | initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.54.15
ollama-container-0 | dlsym: cuInit - 0x7fc5fe6feef0
ollama-container-0 | dlsym: cuDriverGetVersion - 0x7fc5fe6fef10
ollama-container-0 | dlsym: cuDeviceGetCount - 0x7fc5fe6fef50
ollama-container-0 | dlsym: cuDeviceGet - 0x7fc5fe6fef30
ollama-container-0 | dlsym: cuDeviceGetAttribute - 0x7fc5fe6ff030
ollama-container-0 | dlsym: cuDeviceGetUuid - 0x7fc5fe6fef90
ollama-container-0 | dlsym: cuDeviceGetName - 0x7fc5fe6fef70
ollama-container-0 | dlsym: cuCtxCreate_v3 - 0x7fc5fe6ff210
ollama-container-0 | dlsym: cuMemGetInfo_v2 - 0x7fc5fe709190
ollama-container-0 | dlsym: cuCtxDestroy - 0x7fc5fe7637f0
ollama-container-0 | calling cuInit
ollama-container-0 | calling cuDriverGetVersion
ollama-container-0 | raw version 0x2f08
ollama-container-0 | CUDA driver version: 12.4
ollama-container-0 | calling cuDeviceGetCount
ollama-container-0 | device count 2
ollama-container-0 | time=2025-04-16T08:50:28.277Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-201dbf06-7d5c-1810-1a5a-c5af998610da name="Tesla T4" overhead="0 B" before.total="14.6 GiB" before.free="14.5 GiB" now.total="14.6 GiB" now.free="14.5 GiB" now.used="102.9 MiB"
ollama-container-0 | time=2025-04-16T08:50:28.553Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-e2bd2759-00e8-a0df-89f5-daa13e4b13d4 name="Tesla T4" overhead="0 B" before.total="14.6 GiB" before.free="14.5 GiB" now.total="14.6 GiB" now.free="14.5 GiB" now.used="102.9 MiB"
ollama-container-0 | releasing cuda driver library
ollama-container-0 | time=2025-04-16T08:50:28.555Z level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=2 available="[14.5 GiB 14.5 GiB]"
ollama-container-0 | time=2025-04-16T08:50:28.557Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="125.9 GiB" before.free="119.2 GiB" before.free_swap="7.0 GiB" now.total="125.9 GiB" now.free="119.2 GiB" now.free_swap="7.0 GiB"
ollama-container-0 | initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.54.15
ollama-container-0 | dlsym: cuInit - 0x7fc5fe6feef0
ollama-container-0 | dlsym: cuDriverGetVersion - 0x7fc5fe6fef10
ollama-container-0 | dlsym: cuDeviceGetCount - 0x7fc5fe6fef50
ollama-container-0 | dlsym: cuDeviceGet - 0x7fc5fe6fef30
ollama-container-0 | dlsym: cuDeviceGetAttribute - 0x7fc5fe6ff030
ollama-container-0 | dlsym: cuDeviceGetUuid - 0x7fc5fe6fef90
ollama-container-0 | dlsym: cuDeviceGetName - 0x7fc5fe6fef70
ollama-container-0 | dlsym: cuCtxCreate_v3 - 0x7fc5fe6ff210
ollama-container-0 | dlsym: cuMemGetInfo_v2 - 0x7fc5fe709190
ollama-container-0 | dlsym: cuCtxDestroy - 0x7fc5fe7637f0
ollama-container-0 | calling cuInit
ollama-container-0 | calling cuDriverGetVersion
ollama-container-0 | raw version 0x2f08
ollama-container-0 | CUDA driver version: 12.4
ollama-container-0 | calling cuDeviceGetCount
ollama-container-0 | device count 2
ollama-container-0 | time=2025-04-16T08:50:28.847Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-201dbf06-7d5c-1810-1a5a-c5af998610da name="Tesla T4" overhead="0 B" before.total="14.6 GiB" before.free="14.5 GiB" now.total="14.6 GiB" now.free="14.5 GiB" now.used="102.9 MiB"
ollama-container-0 | time=2025-04-16T08:50:29.108Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-e2bd2759-00e8-a0df-89f5-daa13e4b13d4 name="Tesla T4" overhead="0 B" before.total="14.6 GiB" before.free="14.5 GiB" now.total="14.6 GiB" now.free="14.5 GiB" now.used="102.9 MiB"
ollama-container-0 | releasing cuda driver library
ollama-container-0 | time=2025-04-16T08:50:29.109Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="125.9 GiB" before.free="119.2 GiB" before.free_swap="7.0 GiB" now.total="125.9 GiB" now.free="119.2 GiB" now.free_swap="7.0 GiB"
ollama-container-0 | initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.54.15
ollama-container-0 | dlsym: cuInit - 0x7fc5fe6feef0
ollama-container-0 | dlsym: cuDriverGetVersion - 0x7fc5fe6fef10
ollama-container-0 | dlsym: cuDeviceGetCount - 0x7fc5fe6fef50
ollama-container-0 | dlsym: cuDeviceGet - 0x7fc5fe6fef30
ollama-container-0 | dlsym: cuDeviceGetAttribute - 0x7fc5fe6ff030
ollama-container-0 | dlsym: cuDeviceGetUuid - 0x7fc5fe6fef90
ollama-container-0 | dlsym: cuDeviceGetName - 0x7fc5fe6fef70
ollama-container-0 | dlsym: cuCtxCreate_v3 - 0x7fc5fe6ff210
ollama-container-0 | dlsym: cuMemGetInfo_v2 - 0x7fc5fe709190
ollama-container-0 | dlsym: cuCtxDestroy - 0x7fc5fe7637f0
ollama-container-0 | calling cuInit
ollama-container-0 | calling cuDriverGetVersion
ollama-container-0 | raw version 0x2f08
ollama-container-0 | CUDA driver version: 12.4
ollama-container-0 | calling cuDeviceGetCount
ollama-container-0 | device count 2
ollama-container-0 | time=2025-04-16T08:50:29.391Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-201dbf06-7d5c-1810-1a5a-c5af998610da name="Tesla T4" overhead="0 B" before.total="14.6 GiB" before.free="14.5 GiB" now.total="14.6 GiB" now.free="14.5 GiB" now.used="102.9 MiB"
ollama-container-0 | time=2025-04-16T08:50:29.626Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-e2bd2759-00e8-a0df-89f5-daa13e4b13d4 name="Tesla T4" overhead="0 B" before.total="14.6 GiB" before.free="14.5 GiB" now.total="14.6 GiB" now.free="14.5 GiB" now.used="102.9 MiB"
ollama-container-0 | releasing cuda driver library
ollama-container-0 | time=2025-04-16T08:50:29.626Z level=INFO source=server.go:105 msg="system memory" total="125.9 GiB" free="119.2 GiB" free_swap="7.0 GiB"
ollama-container-0 | time=2025-04-16T08:50:29.626Z level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=2 available="[14.5 GiB 14.5 GiB]"
ollama-container-0 | time=2025-04-16T08:50:29.632Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="125.9 GiB" before.free="119.2 GiB" before.free_swap="7.0 GiB" now.total="125.9 GiB" now.free="119.1 GiB" now.free_swap="7.0 GiB"
ollama-container-0 | initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.54.15
ollama-container-0 | dlsym: cuInit - 0x7fc5fe6feef0
ollama-container-0 | dlsym: cuDriverGetVersion - 0x7fc5fe6fef10
ollama-container-0 | dlsym: cuDeviceGetCount - 0x7fc5fe6fef50
ollama-container-0 | dlsym: cuDeviceGet - 0x7fc5fe6fef30
ollama-container-0 | dlsym: cuDeviceGetAttribute - 0x7fc5fe6ff030
ollama-container-0 | dlsym: cuDeviceGetUuid - 0x7fc5fe6fef90
ollama-container-0 | dlsym: cuDeviceGetName - 0x7fc5fe6fef70
ollama-container-0 | dlsym: cuCtxCreate_v3 - 0x7fc5fe6ff210
ollama-container-0 | dlsym: cuMemGetInfo_v2 - 0x7fc5fe709190
ollama-container-0 | dlsym: cuCtxDestroy - 0x7fc5fe7637f0
ollama-container-0 | calling cuInit
ollama-container-0 | calling cuDriverGetVersion
ollama-container-0 | raw version 0x2f08
ollama-container-0 | CUDA driver version: 12.4
ollama-container-0 | calling cuDeviceGetCount
ollama-container-0 | device count 2
ollama-container-0 | time=2025-04-16T08:50:29.888Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-201dbf06-7d5c-1810-1a5a-c5af998610da name="Tesla T4" overhead="0 B" before.total="14.6 GiB" before.free="14.5 GiB" now.total="14.6 GiB" now.free="14.5 GiB" now.used="102.9 MiB"
ollama-container-0 | time=2025-04-16T08:50:30.136Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-e2bd2759-00e8-a0df-89f5-daa13e4b13d4 name="Tesla T4" overhead="0 B" before.total="14.6 GiB" before.free="14.5 GiB" now.total="14.6 GiB" now.free="14.5 GiB" now.used="102.9 MiB"
ollama-container-0 | releasing cuda driver library
ollama-container-0 | time=2025-04-16T08:50:30.138Z level=INFO source=server.go:138 msg=offload library=cuda layers.requested=-1 layers.model=63 layers.offload=62 layers.split=31,31 memory.available="[14.5 GiB 14.5 GiB]" memory.gpu_overhead="0 B" memory.required.full="29.0 GiB" memory.required.partial="27.9 GiB" memory.required.kv="4.9 GiB" memory.required.allocations="[13.5 GiB 14.4 GiB]" memory.weights.total="15.4 GiB" memory.weights.repeating="14.3 GiB" memory.weights.nonrepeating="1.1 GiB" memory.graph.full="2.7 GiB" memory.graph.partial="2.7 GiB" projector.weights="795.9 MiB" projector.graph="1.0 GiB"
ollama-container-0 | time=2025-04-16T08:50:30.138Z level=INFO source=server.go:185 msg="enabling flash attention"
ollama-container-0 | time=2025-04-16T08:50:30.138Z level=WARN source=server.go:193 msg="kv cache type not supported by model" type=""
ollama-container-0 | time=2025-04-16T08:50:30.138Z level=DEBUG source=server.go:262 msg="compatible gpu libraries" compatible="[cuda_v12 cuda_v11]"
ollama-container-0 | time=2025-04-16T08:50:30.324Z level=WARN source=ggml.go:152 msg="key not found" key=tokenizer.ggml.add_eot_token default=false
ollama-container-0 | time=2025-04-16T08:50:30.324Z level=DEBUG source=process_text_spm.go:21 msg=Tokens "num tokens"=262145 vals="[ ]" scores="[0 0 0 0 0]" types="[3 3 3 2 1]"
ollama-container-0 | time=2025-04-16T08:50:30.328Z level=DEBUG source=process_text_spm.go:35 msg="Token counts" normal=261882 unknown=1 control=5 "user defined"=1 unused=0 byte=256 "max token len"=93
ollama-container-0 | time=2025-04-16T08:50:30.336Z level=DEBUG source=process_text_spm.go:21 msg=Tokens "num tokens"=262145 vals="[ ]" scores="[0 0 0 0 0]" types="[3 3 3 2 1]"
ollama-container-0 | time=2025-04-16T08:50:30.339Z level=DEBUG source=process_text_spm.go:35 msg="Token counts" normal=261882 unknown=1 control=5 "user defined"=1 unused=0 byte=256 "max token len"=93
ollama-container-0 | time=2025-04-16T08:50:30.339Z level=WARN source=ggml.go:152 msg="key not found" key=gemma3.attention.layer_norm_rms_epsilon default=9.999999974752427e-07
ollama-container-0 | time=2025-04-16T08:50:30.339Z level=WARN source=ggml.go:152 msg="key not found" key=gemma3.rope.local.freq_base default=10000
ollama-container-0 | time=2025-04-16T08:50:30.339Z level=WARN source=ggml.go:152 msg="key not found" key=gemma3.rope.global.freq_base default=1e+06
ollama-container-0 | time=2025-04-16T08:50:30.339Z level=WARN source=ggml.go:152 msg="key not found" key=gemma3.rope.freq_scale default=1
ollama-container-0 | time=2025-04-16T08:50:30.339Z level=WARN source=ggml.go:152 msg="key not found" key=gemma3.mm_tokens_per_image default=256
ollama-container-0 | time=2025-04-16T08:50:30.340Z level=DEBUG source=server.go:335 msg="adding gpu library" path=/usr/lib/ollama/cuda_v12
ollama-container-0 | time=2025-04-16T08:50:30.340Z level=DEBUG source=server.go:343 msg="adding gpu dependency paths" paths=[/usr/lib/ollama/cuda_v12]
ollama-container-0 | time=2025-04-16T08:50:30.340Z level=INFO source=server.go:405 msg="starting llama server" cmd="/usr/bin/ollama runner --ollama-engine --model /root/.ollama/models/blobs/sha256-e796792eba26c4d3b04b0ac5adb01a453dd9ec2dfd83b6c59cbf6fe5f30b0f68 --ctx-size 40000 --batch-size 512 --n-gpu-layers 62 --verbose --threads 32 --flash-attn --parallel 4 --tensor-split 31,31 --port 42511"
ollama-container-0 | time=2025-04-16T08:50:30.340Z level=DEBUG source=server.go:423 msg=subprocess environment="[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LD_LIBRARY_PATH=/usr/lib/ollama/cuda_v12:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/lib/ollama/cuda_v12:/usr/lib/ollama CUDA_VISIBLE_DEVICES=GPU-201dbf06-7d5c-1810-1a5a-c5af998610da,GPU-e2bd2759-00e8-a0df-89f5-daa13e4b13d4]"
ollama-container-0 | time=2025-04-16T08:50:30.341Z level=INFO source=sched.go:451 msg="loaded runners" count=1
ollama-container-0 | time=2025-04-16T08:50:30.341Z level=INFO source=server.go:580 msg="waiting for llama runner to start responding"
ollama-container-0 | time=2025-04-16T08:50:30.342Z level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server error"
ollama-container-0 | time=2025-04-16T08:50:30.366Z level=INFO source=runner.go:816 msg="starting ollama engine"
ollama-container-0 | time=2025-04-16T08:50:30.367Z level=INFO source=runner.go:879 msg="Server listening on 127.0.0.1:42511"
ollama-container-0 | time=2025-04-16T08:50:30.548Z level=WARN source=ggml.go:152 msg="key not found" key=general.name default=""
ollama-container-0 | time=2025-04-16T08:50:30.548Z level=WARN source=ggml.go:152 msg="key not found" key=general.description default=""
ollama-container-0 | time=2025-04-16T08:50:30.548Z level=INFO source=ggml.go:67 msg="" architecture=gemma3 file_type=Q4_K_M name="" description="" num_tensors=1247 num_key_values=37
ollama-container-0 | time=2025-04-16T08:50:30.548Z level=DEBUG source=ggml.go:99 msg="ggml backend load all from path" path=/usr/lib/ollama/cuda_v12
ollama-container-0 | time=2025-04-16T08:50:30.595Z level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server loading model"
ollama-container-0 | ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ollama-container-0 | ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ollama-container-0 | ggml_cuda_init: found 2 CUDA devices:
ollama-container-0 | Device 0: Tesla T4, compute capability 7.5, VMM: yes
ollama-container-0 | Device 1: Tesla T4, compute capability 7.5, VMM: yes
ollama-container-0 | load_backend: loaded CUDA backend from /usr/lib/ollama/cuda_v12/libggml-cuda.so
ollama-container-0 | time=2025-04-16T08:50:30.851Z level=DEBUG source=ggml.go:93 msg="skipping path which is not part of ollama" path=/usr/local/nvidia/lib
ollama-container-0 | time=2025-04-16T08:50:30.851Z level=DEBUG source=ggml.go:93 msg="skipping path which is not part of ollama" path=/usr/local/nvidia/lib64
ollama-container-0 | time=2025-04-16T08:50:30.851Z level=DEBUG source=ggml.go:99 msg="ggml backend load all from path" path=/usr/lib/ollama
ollama-container-0 | ggml_backend_load_best: /usr/lib/ollama/libggml-cpu-haswell.so score: 55
ollama-container-0 | ggml_backend_load_best: /usr/lib/ollama/libggml-cpu-skylakex.so score: 183
ollama-container-0 | ggml_backend_load_best: /usr/lib/ollama/libggml-cpu-icelake.so score: 0
ollama-container-0 | ggml_backend_load_best: /usr/lib/ollama/libggml-cpu-sandybridge.so score: 20
ollama-container-0 | ggml_backend_load_best: /usr/lib/ollama/libggml-cpu-alderlake.so score: 0
ollama-container-0 | load_backend: loaded CPU backend from /usr/lib/ollama/libggml-cpu-skylakex.so
ollama-container-0 | time=2025-04-16T08:50:30.856Z level=INFO source=ggml.go:109 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.AVX512=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 CUDA.1.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.1.USE_GRAPHS=1 CUDA.1.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc)
ollama-container-0 | time=2025-04-16T08:50:30.856Z level=DEBUG source=ggml.go:220 msg="created tensor" name=mm.mm_input_projection.weight shape="[5376 1152]" dtype=1 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.856Z level=DEBUG source=ggml.go:220 msg="created tensor" name=mm.mm_soft_emb_norm.weight shape=[1152] dtype=0 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.856Z level=DEBUG source=ggml.go:220 msg="created tensor" name=output_norm.weight shape=[5376] dtype=0 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.856Z level=DEBUG source=ggml.go:220 msg="created tensor" name=token_embd.weight shape="[5376 262144]" dtype=14 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.857Z level=DEBUG source=ggml.go:220 msg="created tensor" name=output.weight shape="[5376 262144]" dtype=14 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.857Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.0.attn_k.bias shape=[1152] dtype=0 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.857Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.0.attn_k.weight shape="[1152 1152]" dtype=1 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.857Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.0.attn_output.bias shape=[1152] dtype=0 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.857Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.0.attn_output.weight shape="[1152 1152]" dtype=1 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.857Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.0.attn_q.bias shape=[1152] dtype=0 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.857Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.0.attn_q.weight shape="[1152 1152]" dtype=1 buffer_type=CPU
......
ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.7.mlp.fc2.bias shape=[1152] dtype=0 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.7.mlp.fc2.weight shape="[4304 1152]" dtype=1 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.8.attn_k.bias shape=[1152] dtype=0 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.8.attn_k.weight shape="[1152 1152]" dtype=1 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.8.attn_output.bias shape=[1152] dtype=0 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.8.attn_output.weight shape="[1152 1152]" dtype=1 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.8.attn_q.bias shape=[1152] dtype=0 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.8.attn_q.weight shape="[1152 1152]" dtype=1 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.8.attn_v.bias shape=[1152] dtype=0 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.8.attn_v.weight shape="[1152 1152]" dtype=1 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.8.layer_norm1.bias shape=[1152] dtype=0 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.8.layer_norm1.weight shape=[1152] dtype=0 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.8.layer_norm2.bias shape=[1152] dtype=0 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.8.layer_norm2.weight shape=[1152] dtype=0 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.8.mlp.fc1.bias shape=[4304] dtype=0 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.8.mlp.fc1.weight shape="[1152 4304]" dtype=1 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.8.mlp.fc2.bias shape=[1152] dtype=0 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.8.mlp.fc2.weight shape="[4304 1152]" dtype=1 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.9.attn_k.bias shape=[1152] dtype=0 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.9.attn_k.weight shape="[1152 1152]" dtype=1 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.9.attn_output.bias shape=[1152] dtype=0 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.9.attn_output.weight shape="[1152 1152]" dtype=1 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.9.attn_q.bias shape=[1152] dtype=0 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.9.attn_q.weight shape="[1152 1152]" dtype=1 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.9.attn_v.bias shape=[1152] dtype=0 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.9.attn_v.weight shape="[1152 1152]" dtype=1 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.9.layer_norm1.bias shape=[1152] dtype=0 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.9.layer_norm1.weight shape=[1152] dtype=0 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.9.layer_norm2.bias shape=[1152] dtype=0 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.9.layer_norm2.weight shape=[1152] dtype=0 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.9.mlp.fc1.bias shape=[4304] dtype=0 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.9.mlp.fc1.weight shape="[1152 4304]" dtype=1 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.9.mlp.fc2.bias shape=[1152] dtype=0 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.9.mlp.fc2.weight shape="[4304 1152]" dtype=1 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.patch_embedding.bias shape=[1152] dtype=0 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.patch_embedding.weight shape="[14 14 3 1152]" dtype=1 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.position_embedding.weight shape="[1152 4096]" dtype=1 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.post_layernorm.bias shape=[1152] dtype=0 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.post_layernorm.weight shape=[1152] dtype=0 buffer_type=CPU
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.0.attn_k.weight shape="[5376 2048]" dtype=12 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.0.attn_k_norm.weight shape=[128] dtype=0 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.0.attn_norm.weight shape=[5376] dtype=0 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.0.attn_output.weight shape="[4096 5376]" dtype=12 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.0.attn_q.weight shape="[5376 4096]" dtype=12 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.0.attn_q_norm.weight shape=[128] dtype=0 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.0.attn_v.weight shape="[5376 2048]" dtype=14 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.0.ffn_down.weight shape="[21504 5376]" dtype=14 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.0.ffn_gate.weight shape="[5376 21504]" dtype=12 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.0.ffn_norm.weight shape=[5376] dtype=0 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.0.ffn_up.weight shape="[5376 21504]" dtype=12 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.0.post_attention_norm.weight shape=[5376] dtype=0 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.0.post_ffw_norm.weight shape=[5376] dtype=0 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.1.attn_k.weight shape="[5376 2048]" dtype=12 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.1.attn_k_norm.weight shape=[128] dtype=0 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.1.attn_norm.weight shape=[5376] dtype=0 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.1.attn_output.weight shape="[4096 5376]" dtype=12 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.1.attn_q.weight shape="[5376 4096]" dtype=12 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.1.attn_q_norm.weight shape=[128] dtype=0 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.1.attn_v.weight shape="[5376 2048]" dtype=14 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.1.ffn_down.weight shape="[21504 5376]" dtype=14 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.1.ffn_gate.weight shape="[5376 21504]" dtype=12 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.1.ffn_norm.weight shape=[5376] dtype=0 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.1.ffn_up.weight shape="[5376 21504]" dtype=12 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.1.post_attention_norm.weight shape=[5376] dtype=0 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.1.post_ffw_norm.weight shape=[5376] dtype=0 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.2.attn_k.weight shape="[5376 2048]" dtype=12 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.2.attn_k_norm.weight shape=[128] dtype=0 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.2.attn_norm.weight shape=[5376] dtype=0 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.2.attn_output.weight shape="[4096 5376]" dtype=12 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.2.attn_q.weight shape="[5376 4096]" dtype=12 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:30.870Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.2.attn_q_norm.weight shape=[128] dtype=0 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:30.870Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.2.attn_v.weight shape="[5376 2048]" dtype=14 buffer_type=CUDA0
......
......
......
ollama-container-0 | time=2025-04-16T08:50:30.886Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.61.attn_output.weight shape="[4096 5376]" dtype=12 buffer_type=CUDA1
ollama-container-0 | time=2025-04-16T08:50:30.886Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.61.attn_q.weight shape="[5376 4096]" dtype=12 buffer_type=CUDA1
ollama-container-0 | time=2025-04-16T08:50:30.886Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.61.attn_q_norm.weight shape=[128] dtype=0 buffer_type=CUDA1
ollama-container-0 | time=2025-04-16T08:50:30.886Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.61.attn_v.weight shape="[5376 2048]" dtype=14 buffer_type=CUDA1
ollama-container-0 | time=2025-04-16T08:50:30.886Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.61.ffn_down.weight shape="[21504 5376]" dtype=14 buffer_type=CUDA1
ollama-container-0 | time=2025-04-16T08:50:30.886Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.61.ffn_gate.weight shape="[5376 21504]" dtype=12 buffer_type=CUDA1
ollama-container-0 | time=2025-04-16T08:50:30.887Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.61.ffn_norm.weight shape=[5376] dtype=0 buffer_type=CUDA1
ollama-container-0 | time=2025-04-16T08:50:30.887Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.61.ffn_up.weight shape="[5376 21504]" dtype=12 buffer_type=CUDA1
ollama-container-0 | time=2025-04-16T08:50:30.887Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.61.post_attention_norm.weight shape=[5376] dtype=0 buffer_type=CUDA1
ollama-container-0 | time=2025-04-16T08:50:30.887Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.61.post_ffw_norm.weight shape=[5376] dtype=0 buffer_type=CUDA1
ollama-container-0 | time=2025-04-16T08:50:31.048Z level=INFO source=ggml.go:289 msg="model weights" buffer=CUDA0 size="7.2 GiB"
ollama-container-0 | time=2025-04-16T08:50:31.048Z level=INFO source=ggml.go:289 msg="model weights" buffer=CUDA1 size="7.2 GiB"
ollama-container-0 | time=2025-04-16T08:50:31.048Z level=INFO source=ggml.go:289 msg="model weights" buffer=CPU size="2.9 GiB"
ollama-container-0 | time=2025-04-16T08:50:31.099Z level=DEBUG source=server.go:625 msg="model load progress 0.01"
ollama-container-0 | time=2025-04-16T08:50:31.351Z level=DEBUG source=server.go:625 msg="model load progress 0.07"
ollama-container-0 | time=2025-04-16T08:50:31.602Z level=DEBUG source=server.go:625 msg="model load progress 0.11"
ollama-container-0 | time=2025-04-16T08:50:31.854Z level=DEBUG source=server.go:625 msg="model load progress 0.14"
ollama-container-0 | time=2025-04-16T08:50:32.106Z level=DEBUG source=server.go:625 msg="model load progress 0.18"
.......
ollama-container-0 | time=2025-04-16T08:50:38.407Z level=DEBUG source=server.go:625 msg="model load progress 0.98"
ollama-container-0 | time=2025-04-16T08:50:38.561Z level=INFO source=ggml.go:388 msg="compute graph" backend=CUDA0 buffer_type=CUDA0
ollama-container-0 | time=2025-04-16T08:50:38.561Z level=INFO source=ggml.go:388 msg="compute graph" backend=CUDA1 buffer_type=CUDA1
ollama-container-0 | time=2025-04-16T08:50:38.561Z level=INFO source=ggml.go:388 msg="compute graph" backend=CPU buffer_type=CUDA_Host
ollama-container-0 | time=2025-04-16T08:50:38.589Z level=WARN source=ggml.go:152 msg="key not found" key=tokenizer.ggml.add_eot_token default=false
ollama-container-0 | time=2025-04-16T08:50:38.589Z level=DEBUG source=process_text_spm.go:21 msg=Tokens "num tokens"=262145 vals="[ ]" scores="[0 0 0 0 0]" types="[3 3 3 2 1]"
ollama-container-0 | time=2025-04-16T08:50:38.593Z level=DEBUG source=process_text_spm.go:35 msg="Token counts" normal=261882 unknown=1 control=5 "user defined"=1 unused=0 byte=256 "max token len"=93
ollama-container-0 | time=2025-04-16T08:50:38.597Z level=DEBUG source=process_text_spm.go:21 msg=Tokens "num tokens"=262145 vals="[ ]" scores="[0 0 0 0 0]" types="[3 3 3 2 1]"
ollama-container-0 | time=2025-04-16T08:50:38.600Z level=DEBUG source=process_text_spm.go:35 msg="Token counts" normal=261882 unknown=1 control=5 "user defined"=1 unused=0 byte=256 "max token len"=93
ollama-container-0 | time=2025-04-16T08:50:38.600Z level=WARN source=ggml.go:152 msg="key not found" key=gemma3.attention.layer_norm_rms_epsilon default=9.999999974752427e-07
ollama-container-0 | time=2025-04-16T08:50:38.600Z level=WARN source=ggml.go:152 msg="key not found" key=gemma3.rope.local.freq_base default=10000
ollama-container-0 | time=2025-04-16T08:50:38.600Z level=WARN source=ggml.go:152 msg="key not found" key=gemma3.rope.global.freq_base default=1e+06
ollama-container-0 | time=2025-04-16T08:50:38.600Z level=WARN source=ggml.go:152 msg="key not found" key=gemma3.rope.freq_scale default=1
ollama-container-0 | time=2025-04-16T08:50:38.600Z level=WARN source=ggml.go:152 msg="key not found" key=gemma3.mm_tokens_per_image default=256
ollama-container-0 | time=2025-04-16T08:50:38.600Z level=DEBUG source=model.go:205 msg="found tensor" name=v.patch_embedding.weight type=f16 shape="[14 14 3 1152]"
ollama-container-0 | time=2025-04-16T08:50:38.600Z level=DEBUG source=model.go:205 msg="found tensor" name=v.position_embedding.weight type=f16 shape="[1152 4096]"
ollama-container-0 | time=2025-04-16T08:50:38.600Z level=DEBUG source=model.go:205 msg="found tensor" name=v.post_layernorm.weight type=f32 shape=[1152]
ollama-container-0 | time=2025-04-16T08:50:38.600Z level=DEBUG source=model.go:205 msg="found tensor" name=v.post_layernorm.bias type=f32 shape=[1152]
ollama-container-0 | time=2025-04-16T08:50:38.600Z level=DEBUG source=model.go:205 msg="found tensor" name=v.blk.0.layer_norm1.weight type=f32 shape=[1152]
.......
ollama-container-0 | time=2025-04-16T08:50:38.663Z level=DEBUG source=model.go:205 msg="found tensor" name=output_norm.weight type=f32 shape=[5376]
ollama-container-0 | time=2025-04-16T08:50:38.663Z level=DEBUG source=model.go:205 msg="found tensor" name=output.weight type=q6_K shape="[5376 262144]"
ollama-container-0 | time=2025-04-16T08:50:38.663Z level=DEBUG source=model.go:205 msg="found tensor" name=mm.mm_soft_emb_norm.weight type=f32 shape=[1152]
ollama-container-0 | time=2025-04-16T08:50:38.663Z level=DEBUG source=model.go:205 msg="found tensor" name=mm.mm_input_projection.weight type=f16 shape="[5376 1152]"
ollama-container-0 | time=2025-04-16T08:50:38.910Z level=INFO source=server.go:619 msg="llama runner started in 8.57 seconds"
ollama-container-0 | time=2025-04-16T08:50:38.910Z level=DEBUG source=sched.go:464 msg="finished setting up runner" model=/root/.ollama/models/blobs/sha256-e796792eba26c4d3b04b0ac5adb01a453dd9ec2dfd83b6c59cbf6fe5f30b0f68
ollama-container-0 | time=2025-04-16T08:50:38.910Z level=WARN source=routes.go:280 msg="the context field is deprecated and will be removed in a future version of Ollama"
ollama-container-0 | time=2025-04-16T08:50:38.910Z level=DEBUG source=routes.go:297 msg="generate request" images=0 prompt="<start_of_turn>user\nhello<end_of_turn>\n<start_of_turn>model\n"
ollama-container-0 | time=2025-04-16T08:50:38.971Z level=DEBUG source=process_text_spm.go:184 msg="adding bos token to prompt" id=2
ollama-container-0 | time=2025-04-16T08:50:38.972Z level=DEBUG source=cache.go:136 msg="loading cache slot" id=0 cache=0 prompt=10 used=0 remaining=10
ollama-container-0 | [GIN] 2025/04/16 - 08:51:10 | 200 | 44.20092233s | 10.1.10.32 | POST "/api/generate"
ollama-container-0 | time=2025-04-16T08:51:10.464Z level=DEBUG source=sched.go:468 msg="context for request finished"
ollama-container-0 | time=2025-04-16T08:51:10.465Z level=DEBUG source=sched.go:341 msg="runner with non-zero duration has gone idle, adding timer" modelPath=/root/.ollama/models/blobs/sha256-e796792eba26c4d3b04b0ac5adb01a453dd9ec2dfd83b6c59cbf6fe5f30b0f68 duration=2562047h47m16.854775807s
ollama-container-0 | time=2025-04-16T08:51:10.465Z level=DEBUG source=sched.go:359 msg="after processing request finished event" modelPath=/root/.ollama/models/blobs/sha256-e796792eba26c4d3b04b0ac5adb01a453dd9ec2dfd83b6c59cbf6fe5f30b0f68 refCount=0

Relevant log output

OS

No response

GPU

No response

CPU

No response

Ollama version

No response

Originally created by @leizhu1989 on GitHub (Apr 16, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/10297 ### What is the issue? system：Ubuntu gpu：T4*2 When I run the gemma3:27b model on two T4 graphics cards, the weights are assigned to the two cards, but the inference seems to run to the CPU, and the speed is very slow. The usage rate of the two GPUs is about 30%, and sometimes only one card is used.Does it not support multi graphics card inference for a large model docker logs: ollama-container-0 | 2025/04/16 08:50:23 routes.go:1231: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:2048 OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:true OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:1 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:4 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" ollama-container-0 | time=2025-04-16T08:50:23.586Z level=INFO source=images.go:458 msg="total blobs: 5" ollama-container-0 | time=2025-04-16T08:50:23.587Z level=INFO source=images.go:465 msg="total unused blobs removed: 0" ollama-container-0 | time=2025-04-16T08:50:23.587Z level=INFO source=routes.go:1298 msg="Listening on [::]:11434 (version 0.6.5)" ollama-container-0 | time=2025-04-16T08:50:23.588Z level=DEBUG source=sched.go:107 msg="starting llm scheduler" ollama-container-0 | time=2025-04-16T08:50:23.588Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs" ollama-container-0 | time=2025-04-16T08:50:23.590Z level=DEBUG source=gpu.go:98 msg="searching for GPU discovery libraries for NVIDIA" ollama-container-0 | time=2025-04-16T08:50:23.590Z level=DEBUG source=gpu.go:501 msg="Searching for GPU library" name=libcuda.so* ollama-container-0 | time=2025-04-16T08:50:23.590Z level=DEBUG source=gpu.go:525 msg="gpu library search" globs="[/usr/lib/ollama/libcuda.so* /usr/local/nvidia/lib/libcuda.so* /usr/local/nvidia/lib64/libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]" ollama-container-0 | time=2025-04-16T08:50:23.591Z level=DEBUG source=gpu.go:558 msg="discovered GPU libraries" paths=[/usr/lib/x86_64-linux-gnu/libcuda.so.550.54.15] ollama-container-0 | initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.54.15 ollama-container-0 | dlsym: cuInit - 0x7fc5fe6feef0 ollama-container-0 | dlsym: cuDriverGetVersion - 0x7fc5fe6fef10 ollama-container-0 | dlsym: cuDeviceGetCount - 0x7fc5fe6fef50 ollama-container-0 | dlsym: cuDeviceGet - 0x7fc5fe6fef30 ollama-container-0 | dlsym: cuDeviceGetAttribute - 0x7fc5fe6ff030 ollama-container-0 | dlsym: cuDeviceGetUuid - 0x7fc5fe6fef90 ollama-container-0 | dlsym: cuDeviceGetName - 0x7fc5fe6fef70 ollama-container-0 | dlsym: cuCtxCreate_v3 - 0x7fc5fe6ff210 ollama-container-0 | dlsym: cuMemGetInfo_v2 - 0x7fc5fe709190 ollama-container-0 | dlsym: cuCtxDestroy - 0x7fc5fe7637f0 ollama-container-0 | calling cuInit ollama-container-0 | calling cuDriverGetVersion ollama-container-0 | raw version 0x2f08 ollama-container-0 | CUDA driver version: 12.4 ollama-container-0 | calling cuDeviceGetCount ollama-container-0 | device count 2 ollama-container-0 | time=2025-04-16T08:50:23.865Z level=DEBUG source=gpu.go:125 msg="detected GPUs" count=2 library=/usr/lib/x86_64-linux-gnu/libcuda.so.550.54.15 ollama-container-0 | [GPU-201dbf06-7d5c-1810-1a5a-c5af998610da] CUDA totalMem 14917 mb ollama-container-0 | [GPU-201dbf06-7d5c-1810-1a5a-c5af998610da] CUDA freeMem 14814 mb ollama-container-0 | [GPU-201dbf06-7d5c-1810-1a5a-c5af998610da] Compute Capability 7.5 ollama-container-0 | [GPU-e2bd2759-00e8-a0df-89f5-daa13e4b13d4] CUDA totalMem 14917 mb ollama-container-0 | [GPU-e2bd2759-00e8-a0df-89f5-daa13e4b13d4] CUDA freeMem 14814 mb ollama-container-0 | [GPU-e2bd2759-00e8-a0df-89f5-daa13e4b13d4] Compute Capability 7.5 ollama-container-0 | time=2025-04-16T08:50:24.521Z level=DEBUG source=amd_linux.go:419 msg="amdgpu driver not detected /sys/module/amdgpu" ollama-container-0 | releasing cuda driver library ollama-container-0 | time=2025-04-16T08:50:24.522Z level=INFO source=types.go:130 msg="inference compute" id=GPU-201dbf06-7d5c-1810-1a5a-c5af998610da library=cuda variant=v12 compute=7.5 driver=12.4 name="Tesla T4" total="14.6 GiB" available="14.5 GiB" ollama-container-0 | time=2025-04-16T08:50:24.522Z level=INFO source=types.go:130 msg="inference compute" id=GPU-e2bd2759-00e8-a0df-89f5-daa13e4b13d4 library=cuda variant=v12 compute=7.5 driver=12.4 name="Tesla T4" total="14.6 GiB" available="14.5 GiB" ollama-container-0 | [GIN] 2025/04/16 - 08:50:26 | 200 | 1.57596432s | 10.1.10.32 | POST "/api/pull" ollama-container-0 | time=2025-04-16T08:50:26.336Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="125.9 GiB" before.free="119.3 GiB" before.free_swap="7.0 GiB" now.total="125.9 GiB" now.free="119.2 GiB" now.free_swap="7.0 GiB" ollama-container-0 | initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.54.15 ollama-container-0 | dlsym: cuInit - 0x7fc5fe6feef0 ollama-container-0 | dlsym: cuDriverGetVersion - 0x7fc5fe6fef10 ollama-container-0 | dlsym: cuDeviceGetCount - 0x7fc5fe6fef50 ollama-container-0 | dlsym: cuDeviceGet - 0x7fc5fe6fef30 ollama-container-0 | dlsym: cuDeviceGetAttribute - 0x7fc5fe6ff030 ollama-container-0 | dlsym: cuDeviceGetUuid - 0x7fc5fe6fef90 ollama-container-0 | dlsym: cuDeviceGetName - 0x7fc5fe6fef70 ollama-container-0 | dlsym: cuCtxCreate_v3 - 0x7fc5fe6ff210 ollama-container-0 | dlsym: cuMemGetInfo_v2 - 0x7fc5fe709190 ollama-container-0 | dlsym: cuCtxDestroy - 0x7fc5fe7637f0 ollama-container-0 | calling cuInit ollama-container-0 | calling cuDriverGetVersion ollama-container-0 | raw version 0x2f08 ollama-container-0 | CUDA driver version: 12.4 ollama-container-0 | calling cuDeviceGetCount ollama-container-0 | device count 2 ollama-container-0 | time=2025-04-16T08:50:26.945Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-201dbf06-7d5c-1810-1a5a-c5af998610da name="Tesla T4" overhead="0 B" before.total="14.6 GiB" before.free="14.5 GiB" now.total="14.6 GiB" now.free="14.5 GiB" now.used="102.9 MiB" ollama-container-0 | time=2025-04-16T08:50:27.271Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-e2bd2759-00e8-a0df-89f5-daa13e4b13d4 name="Tesla T4" overhead="0 B" before.total="14.6 GiB" before.free="14.5 GiB" now.total="14.6 GiB" now.free="14.5 GiB" now.used="102.9 MiB" ollama-container-0 | releasing cuda driver library ollama-container-0 | time=2025-04-16T08:50:27.395Z level=DEBUG source=sched.go:226 msg="loading first model" model=/root/.ollama/models/blobs/sha256-e796792eba26c4d3b04b0ac5adb01a453dd9ec2dfd83b6c59cbf6fe5f30b0f68 ollama-container-0 | time=2025-04-16T08:50:27.396Z level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=1 available="[14.5 GiB]" ollama-container-0 | time=2025-04-16T08:50:27.398Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="125.9 GiB" before.free="119.2 GiB" before.free_swap="7.0 GiB" now.total="125.9 GiB" now.free="119.2 GiB" now.free_swap="7.0 GiB" ollama-container-0 | initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.54.15 ollama-container-0 | dlsym: cuInit - 0x7fc5fe6feef0 ollama-container-0 | dlsym: cuDriverGetVersion - 0x7fc5fe6fef10 ollama-container-0 | dlsym: cuDeviceGetCount - 0x7fc5fe6fef50 ollama-container-0 | dlsym: cuDeviceGet - 0x7fc5fe6fef30 ollama-container-0 | dlsym: cuDeviceGetAttribute - 0x7fc5fe6ff030 ollama-container-0 | dlsym: cuDeviceGetUuid - 0x7fc5fe6fef90 ollama-container-0 | dlsym: cuDeviceGetName - 0x7fc5fe6fef70 ollama-container-0 | dlsym: cuCtxCreate_v3 - 0x7fc5fe6ff210 ollama-container-0 | dlsym: cuMemGetInfo_v2 - 0x7fc5fe709190 ollama-container-0 | dlsym: cuCtxDestroy - 0x7fc5fe7637f0 ollama-container-0 | calling cuInit ollama-container-0 | calling cuDriverGetVersion ollama-container-0 | raw version 0x2f08 ollama-container-0 | CUDA driver version: 12.4 ollama-container-0 | calling cuDeviceGetCount ollama-container-0 | device count 2 ollama-container-0 | time=2025-04-16T08:50:27.702Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-201dbf06-7d5c-1810-1a5a-c5af998610da name="Tesla T4" overhead="0 B" before.total="14.6 GiB" before.free="14.5 GiB" now.total="14.6 GiB" now.free="14.5 GiB" now.used="102.9 MiB" ollama-container-0 | time=2025-04-16T08:50:27.965Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-e2bd2759-00e8-a0df-89f5-daa13e4b13d4 name="Tesla T4" overhead="0 B" before.total="14.6 GiB" before.free="14.5 GiB" now.total="14.6 GiB" now.free="14.5 GiB" now.used="102.9 MiB" ollama-container-0 | releasing cuda driver library ollama-container-0 | time=2025-04-16T08:50:27.966Z level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=1 available="[14.5 GiB]" ollama-container-0 | time=2025-04-16T08:50:27.971Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="125.9 GiB" before.free="119.2 GiB" before.free_swap="7.0 GiB" now.total="125.9 GiB" now.free="119.2 GiB" now.free_swap="7.0 GiB" ollama-container-0 | initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.54.15 ollama-container-0 | dlsym: cuInit - 0x7fc5fe6feef0 ollama-container-0 | dlsym: cuDriverGetVersion - 0x7fc5fe6fef10 ollama-container-0 | dlsym: cuDeviceGetCount - 0x7fc5fe6fef50 ollama-container-0 | dlsym: cuDeviceGet - 0x7fc5fe6fef30 ollama-container-0 | dlsym: cuDeviceGetAttribute - 0x7fc5fe6ff030 ollama-container-0 | dlsym: cuDeviceGetUuid - 0x7fc5fe6fef90 ollama-container-0 | dlsym: cuDeviceGetName - 0x7fc5fe6fef70 ollama-container-0 | dlsym: cuCtxCreate_v3 - 0x7fc5fe6ff210 ollama-container-0 | dlsym: cuMemGetInfo_v2 - 0x7fc5fe709190 ollama-container-0 | dlsym: cuCtxDestroy - 0x7fc5fe7637f0 ollama-container-0 | calling cuInit ollama-container-0 | calling cuDriverGetVersion ollama-container-0 | raw version 0x2f08 ollama-container-0 | CUDA driver version: 12.4 ollama-container-0 | calling cuDeviceGetCount ollama-container-0 | device count 2 ollama-container-0 | time=2025-04-16T08:50:28.277Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-201dbf06-7d5c-1810-1a5a-c5af998610da name="Tesla T4" overhead="0 B" before.total="14.6 GiB" before.free="14.5 GiB" now.total="14.6 GiB" now.free="14.5 GiB" now.used="102.9 MiB" ollama-container-0 | time=2025-04-16T08:50:28.553Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-e2bd2759-00e8-a0df-89f5-daa13e4b13d4 name="Tesla T4" overhead="0 B" before.total="14.6 GiB" before.free="14.5 GiB" now.total="14.6 GiB" now.free="14.5 GiB" now.used="102.9 MiB" ollama-container-0 | releasing cuda driver library ollama-container-0 | time=2025-04-16T08:50:28.555Z level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=2 available="[14.5 GiB 14.5 GiB]" ollama-container-0 | time=2025-04-16T08:50:28.557Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="125.9 GiB" before.free="119.2 GiB" before.free_swap="7.0 GiB" now.total="125.9 GiB" now.free="119.2 GiB" now.free_swap="7.0 GiB" ollama-container-0 | initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.54.15 ollama-container-0 | dlsym: cuInit - 0x7fc5fe6feef0 ollama-container-0 | dlsym: cuDriverGetVersion - 0x7fc5fe6fef10 ollama-container-0 | dlsym: cuDeviceGetCount - 0x7fc5fe6fef50 ollama-container-0 | dlsym: cuDeviceGet - 0x7fc5fe6fef30 ollama-container-0 | dlsym: cuDeviceGetAttribute - 0x7fc5fe6ff030 ollama-container-0 | dlsym: cuDeviceGetUuid - 0x7fc5fe6fef90 ollama-container-0 | dlsym: cuDeviceGetName - 0x7fc5fe6fef70 ollama-container-0 | dlsym: cuCtxCreate_v3 - 0x7fc5fe6ff210 ollama-container-0 | dlsym: cuMemGetInfo_v2 - 0x7fc5fe709190 ollama-container-0 | dlsym: cuCtxDestroy - 0x7fc5fe7637f0 ollama-container-0 | calling cuInit ollama-container-0 | calling cuDriverGetVersion ollama-container-0 | raw version 0x2f08 ollama-container-0 | CUDA driver version: 12.4 ollama-container-0 | calling cuDeviceGetCount ollama-container-0 | device count 2 ollama-container-0 | time=2025-04-16T08:50:28.847Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-201dbf06-7d5c-1810-1a5a-c5af998610da name="Tesla T4" overhead="0 B" before.total="14.6 GiB" before.free="14.5 GiB" now.total="14.6 GiB" now.free="14.5 GiB" now.used="102.9 MiB" ollama-container-0 | time=2025-04-16T08:50:29.108Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-e2bd2759-00e8-a0df-89f5-daa13e4b13d4 name="Tesla T4" overhead="0 B" before.total="14.6 GiB" before.free="14.5 GiB" now.total="14.6 GiB" now.free="14.5 GiB" now.used="102.9 MiB" ollama-container-0 | releasing cuda driver library ollama-container-0 | time=2025-04-16T08:50:29.109Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="125.9 GiB" before.free="119.2 GiB" before.free_swap="7.0 GiB" now.total="125.9 GiB" now.free="119.2 GiB" now.free_swap="7.0 GiB" ollama-container-0 | initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.54.15 ollama-container-0 | dlsym: cuInit - 0x7fc5fe6feef0 ollama-container-0 | dlsym: cuDriverGetVersion - 0x7fc5fe6fef10 ollama-container-0 | dlsym: cuDeviceGetCount - 0x7fc5fe6fef50 ollama-container-0 | dlsym: cuDeviceGet - 0x7fc5fe6fef30 ollama-container-0 | dlsym: cuDeviceGetAttribute - 0x7fc5fe6ff030 ollama-container-0 | dlsym: cuDeviceGetUuid - 0x7fc5fe6fef90 ollama-container-0 | dlsym: cuDeviceGetName - 0x7fc5fe6fef70 ollama-container-0 | dlsym: cuCtxCreate_v3 - 0x7fc5fe6ff210 ollama-container-0 | dlsym: cuMemGetInfo_v2 - 0x7fc5fe709190 ollama-container-0 | dlsym: cuCtxDestroy - 0x7fc5fe7637f0 ollama-container-0 | calling cuInit ollama-container-0 | calling cuDriverGetVersion ollama-container-0 | raw version 0x2f08 ollama-container-0 | CUDA driver version: 12.4 ollama-container-0 | calling cuDeviceGetCount ollama-container-0 | device count 2 ollama-container-0 | time=2025-04-16T08:50:29.391Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-201dbf06-7d5c-1810-1a5a-c5af998610da name="Tesla T4" overhead="0 B" before.total="14.6 GiB" before.free="14.5 GiB" now.total="14.6 GiB" now.free="14.5 GiB" now.used="102.9 MiB" ollama-container-0 | time=2025-04-16T08:50:29.626Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-e2bd2759-00e8-a0df-89f5-daa13e4b13d4 name="Tesla T4" overhead="0 B" before.total="14.6 GiB" before.free="14.5 GiB" now.total="14.6 GiB" now.free="14.5 GiB" now.used="102.9 MiB" ollama-container-0 | releasing cuda driver library ollama-container-0 | time=2025-04-16T08:50:29.626Z level=INFO source=server.go:105 msg="system memory" total="125.9 GiB" free="119.2 GiB" free_swap="7.0 GiB" ollama-container-0 | time=2025-04-16T08:50:29.626Z level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=2 available="[14.5 GiB 14.5 GiB]" ollama-container-0 | time=2025-04-16T08:50:29.632Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="125.9 GiB" before.free="119.2 GiB" before.free_swap="7.0 GiB" now.total="125.9 GiB" now.free="119.1 GiB" now.free_swap="7.0 GiB" ollama-container-0 | initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.54.15 ollama-container-0 | dlsym: cuInit - 0x7fc5fe6feef0 ollama-container-0 | dlsym: cuDriverGetVersion - 0x7fc5fe6fef10 ollama-container-0 | dlsym: cuDeviceGetCount - 0x7fc5fe6fef50 ollama-container-0 | dlsym: cuDeviceGet - 0x7fc5fe6fef30 ollama-container-0 | dlsym: cuDeviceGetAttribute - 0x7fc5fe6ff030 ollama-container-0 | dlsym: cuDeviceGetUuid - 0x7fc5fe6fef90 ollama-container-0 | dlsym: cuDeviceGetName - 0x7fc5fe6fef70 ollama-container-0 | dlsym: cuCtxCreate_v3 - 0x7fc5fe6ff210 ollama-container-0 | dlsym: cuMemGetInfo_v2 - 0x7fc5fe709190 ollama-container-0 | dlsym: cuCtxDestroy - 0x7fc5fe7637f0 ollama-container-0 | calling cuInit ollama-container-0 | calling cuDriverGetVersion ollama-container-0 | raw version 0x2f08 ollama-container-0 | CUDA driver version: 12.4 ollama-container-0 | calling cuDeviceGetCount ollama-container-0 | device count 2 ollama-container-0 | time=2025-04-16T08:50:29.888Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-201dbf06-7d5c-1810-1a5a-c5af998610da name="Tesla T4" overhead="0 B" before.total="14.6 GiB" before.free="14.5 GiB" now.total="14.6 GiB" now.free="14.5 GiB" now.used="102.9 MiB" ollama-container-0 | time=2025-04-16T08:50:30.136Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-e2bd2759-00e8-a0df-89f5-daa13e4b13d4 name="Tesla T4" overhead="0 B" before.total="14.6 GiB" before.free="14.5 GiB" now.total="14.6 GiB" now.free="14.5 GiB" now.used="102.9 MiB" ollama-container-0 | releasing cuda driver library ollama-container-0 | time=2025-04-16T08:50:30.138Z level=INFO source=server.go:138 msg=offload library=cuda layers.requested=-1 layers.model=63 layers.offload=62 layers.split=31,31 memory.available="[14.5 GiB 14.5 GiB]" memory.gpu_overhead="0 B" memory.required.full="29.0 GiB" memory.required.partial="27.9 GiB" memory.required.kv="4.9 GiB" memory.required.allocations="[13.5 GiB 14.4 GiB]" memory.weights.total="15.4 GiB" memory.weights.repeating="14.3 GiB" memory.weights.nonrepeating="1.1 GiB" memory.graph.full="2.7 GiB" memory.graph.partial="2.7 GiB" projector.weights="795.9 MiB" projector.graph="1.0 GiB" ollama-container-0 | time=2025-04-16T08:50:30.138Z level=INFO source=server.go:185 msg="enabling flash attention" ollama-container-0 | time=2025-04-16T08:50:30.138Z level=WARN source=server.go:193 msg="kv cache type not supported by model" type="" ollama-container-0 | time=2025-04-16T08:50:30.138Z level=DEBUG source=server.go:262 msg="compatible gpu libraries" compatible="[cuda_v12 cuda_v11]" ollama-container-0 | time=2025-04-16T08:50:30.324Z level=WARN source=ggml.go:152 msg="key not found" key=tokenizer.ggml.add_eot_token default=false ollama-container-0 | time=2025-04-16T08:50:30.324Z level=DEBUG source=process_text_spm.go:21 msg=Tokens "num tokens"=262145 vals="[<pad> <eos> <bos> <unk> <mask>]" scores="[0 0 0 0 0]" types="[3 3 3 2 1]" ollama-container-0 | time=2025-04-16T08:50:30.328Z level=DEBUG source=process_text_spm.go:35 msg="Token counts" normal=261882 unknown=1 control=5 "user defined"=1 unused=0 byte=256 "max token len"=93 ollama-container-0 | time=2025-04-16T08:50:30.336Z level=DEBUG source=process_text_spm.go:21 msg=Tokens "num tokens"=262145 vals="[<pad> <eos> <bos> <unk> <mask>]" scores="[0 0 0 0 0]" types="[3 3 3 2 1]" ollama-container-0 | time=2025-04-16T08:50:30.339Z level=DEBUG source=process_text_spm.go:35 msg="Token counts" normal=261882 unknown=1 control=5 "user defined"=1 unused=0 byte=256 "max token len"=93 ollama-container-0 | time=2025-04-16T08:50:30.339Z level=WARN source=ggml.go:152 msg="key not found" key=gemma3.attention.layer_norm_rms_epsilon default=9.999999974752427e-07 ollama-container-0 | time=2025-04-16T08:50:30.339Z level=WARN source=ggml.go:152 msg="key not found" key=gemma3.rope.local.freq_base default=10000 ollama-container-0 | time=2025-04-16T08:50:30.339Z level=WARN source=ggml.go:152 msg="key not found" key=gemma3.rope.global.freq_base default=1e+06 ollama-container-0 | time=2025-04-16T08:50:30.339Z level=WARN source=ggml.go:152 msg="key not found" key=gemma3.rope.freq_scale default=1 ollama-container-0 | time=2025-04-16T08:50:30.339Z level=WARN source=ggml.go:152 msg="key not found" key=gemma3.mm_tokens_per_image default=256 ollama-container-0 | time=2025-04-16T08:50:30.340Z level=DEBUG source=server.go:335 msg="adding gpu library" path=/usr/lib/ollama/cuda_v12 ollama-container-0 | time=2025-04-16T08:50:30.340Z level=DEBUG source=server.go:343 msg="adding gpu dependency paths" paths=[/usr/lib/ollama/cuda_v12] ollama-container-0 | time=2025-04-16T08:50:30.340Z level=INFO source=server.go:405 msg="starting llama server" cmd="/usr/bin/ollama runner --ollama-engine --model /root/.ollama/models/blobs/sha256-e796792eba26c4d3b04b0ac5adb01a453dd9ec2dfd83b6c59cbf6fe5f30b0f68 --ctx-size 40000 --batch-size 512 --n-gpu-layers 62 --verbose --threads 32 --flash-attn --parallel 4 --tensor-split 31,31 --port 42511" ollama-container-0 | time=2025-04-16T08:50:30.340Z level=DEBUG source=server.go:423 msg=subprocess environment="[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LD_LIBRARY_PATH=/usr/lib/ollama/cuda_v12:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/lib/ollama/cuda_v12:/usr/lib/ollama CUDA_VISIBLE_DEVICES=GPU-201dbf06-7d5c-1810-1a5a-c5af998610da,GPU-e2bd2759-00e8-a0df-89f5-daa13e4b13d4]" ollama-container-0 | time=2025-04-16T08:50:30.341Z level=INFO source=sched.go:451 msg="loaded runners" count=1 ollama-container-0 | time=2025-04-16T08:50:30.341Z level=INFO source=server.go:580 msg="waiting for llama runner to start responding" ollama-container-0 | time=2025-04-16T08:50:30.342Z level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server error" ollama-container-0 | time=2025-04-16T08:50:30.366Z level=INFO source=runner.go:816 msg="starting ollama engine" ollama-container-0 | time=2025-04-16T08:50:30.367Z level=INFO source=runner.go:879 msg="Server listening on 127.0.0.1:42511" ollama-container-0 | time=2025-04-16T08:50:30.548Z level=WARN source=ggml.go:152 msg="key not found" key=general.name default="" ollama-container-0 | time=2025-04-16T08:50:30.548Z level=WARN source=ggml.go:152 msg="key not found" key=general.description default="" ollama-container-0 | time=2025-04-16T08:50:30.548Z level=INFO source=ggml.go:67 msg="" architecture=gemma3 file_type=Q4_K_M name="" description="" num_tensors=1247 num_key_values=37 ollama-container-0 | time=2025-04-16T08:50:30.548Z level=DEBUG source=ggml.go:99 msg="ggml backend load all from path" path=/usr/lib/ollama/cuda_v12 ollama-container-0 | time=2025-04-16T08:50:30.595Z level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server loading model" ollama-container-0 | ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ollama-container-0 | ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ollama-container-0 | ggml_cuda_init: found 2 CUDA devices: ollama-container-0 | Device 0: Tesla T4, compute capability 7.5, VMM: yes ollama-container-0 | Device 1: Tesla T4, compute capability 7.5, VMM: yes ollama-container-0 | load_backend: loaded CUDA backend from /usr/lib/ollama/cuda_v12/libggml-cuda.so ollama-container-0 | time=2025-04-16T08:50:30.851Z level=DEBUG source=ggml.go:93 msg="skipping path which is not part of ollama" path=/usr/local/nvidia/lib ollama-container-0 | time=2025-04-16T08:50:30.851Z level=DEBUG source=ggml.go:93 msg="skipping path which is not part of ollama" path=/usr/local/nvidia/lib64 ollama-container-0 | time=2025-04-16T08:50:30.851Z level=DEBUG source=ggml.go:99 msg="ggml backend load all from path" path=/usr/lib/ollama ollama-container-0 | ggml_backend_load_best: /usr/lib/ollama/libggml-cpu-haswell.so score: 55 ollama-container-0 | ggml_backend_load_best: /usr/lib/ollama/libggml-cpu-skylakex.so score: 183 ollama-container-0 | ggml_backend_load_best: /usr/lib/ollama/libggml-cpu-icelake.so score: 0 ollama-container-0 | ggml_backend_load_best: /usr/lib/ollama/libggml-cpu-sandybridge.so score: 20 ollama-container-0 | ggml_backend_load_best: /usr/lib/ollama/libggml-cpu-alderlake.so score: 0 ollama-container-0 | load_backend: loaded CPU backend from /usr/lib/ollama/libggml-cpu-skylakex.so ollama-container-0 | time=2025-04-16T08:50:30.856Z level=INFO source=ggml.go:109 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.AVX512=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 CUDA.1.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.1.USE_GRAPHS=1 CUDA.1.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc) ollama-container-0 | time=2025-04-16T08:50:30.856Z level=DEBUG source=ggml.go:220 msg="created tensor" name=mm.mm_input_projection.weight shape="[5376 1152]" dtype=1 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.856Z level=DEBUG source=ggml.go:220 msg="created tensor" name=mm.mm_soft_emb_norm.weight shape=[1152] dtype=0 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.856Z level=DEBUG source=ggml.go:220 msg="created tensor" name=output_norm.weight shape=[5376] dtype=0 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.856Z level=DEBUG source=ggml.go:220 msg="created tensor" name=token_embd.weight shape="[5376 262144]" dtype=14 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.857Z level=DEBUG source=ggml.go:220 msg="created tensor" name=output.weight shape="[5376 262144]" dtype=14 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.857Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.0.attn_k.bias shape=[1152] dtype=0 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.857Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.0.attn_k.weight shape="[1152 1152]" dtype=1 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.857Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.0.attn_output.bias shape=[1152] dtype=0 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.857Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.0.attn_output.weight shape="[1152 1152]" dtype=1 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.857Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.0.attn_q.bias shape=[1152] dtype=0 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.857Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.0.attn_q.weight shape="[1152 1152]" dtype=1 buffer_type=CPU ...... ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.7.mlp.fc2.bias shape=[1152] dtype=0 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.7.mlp.fc2.weight shape="[4304 1152]" dtype=1 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.8.attn_k.bias shape=[1152] dtype=0 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.8.attn_k.weight shape="[1152 1152]" dtype=1 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.8.attn_output.bias shape=[1152] dtype=0 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.8.attn_output.weight shape="[1152 1152]" dtype=1 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.8.attn_q.bias shape=[1152] dtype=0 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.8.attn_q.weight shape="[1152 1152]" dtype=1 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.8.attn_v.bias shape=[1152] dtype=0 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.8.attn_v.weight shape="[1152 1152]" dtype=1 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.8.layer_norm1.bias shape=[1152] dtype=0 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.8.layer_norm1.weight shape=[1152] dtype=0 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.8.layer_norm2.bias shape=[1152] dtype=0 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.8.layer_norm2.weight shape=[1152] dtype=0 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.8.mlp.fc1.bias shape=[4304] dtype=0 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.8.mlp.fc1.weight shape="[1152 4304]" dtype=1 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.8.mlp.fc2.bias shape=[1152] dtype=0 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.8.mlp.fc2.weight shape="[4304 1152]" dtype=1 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.9.attn_k.bias shape=[1152] dtype=0 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.9.attn_k.weight shape="[1152 1152]" dtype=1 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.9.attn_output.bias shape=[1152] dtype=0 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.9.attn_output.weight shape="[1152 1152]" dtype=1 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.9.attn_q.bias shape=[1152] dtype=0 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.9.attn_q.weight shape="[1152 1152]" dtype=1 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.9.attn_v.bias shape=[1152] dtype=0 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.9.attn_v.weight shape="[1152 1152]" dtype=1 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.9.layer_norm1.bias shape=[1152] dtype=0 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.868Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.9.layer_norm1.weight shape=[1152] dtype=0 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.9.layer_norm2.bias shape=[1152] dtype=0 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.9.layer_norm2.weight shape=[1152] dtype=0 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.9.mlp.fc1.bias shape=[4304] dtype=0 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.9.mlp.fc1.weight shape="[1152 4304]" dtype=1 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.9.mlp.fc2.bias shape=[1152] dtype=0 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.blk.9.mlp.fc2.weight shape="[4304 1152]" dtype=1 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.patch_embedding.bias shape=[1152] dtype=0 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.patch_embedding.weight shape="[14 14 3 1152]" dtype=1 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.position_embedding.weight shape="[1152 4096]" dtype=1 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.post_layernorm.bias shape=[1152] dtype=0 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=v.post_layernorm.weight shape=[1152] dtype=0 buffer_type=CPU ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.0.attn_k.weight shape="[5376 2048]" dtype=12 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.0.attn_k_norm.weight shape=[128] dtype=0 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.0.attn_norm.weight shape=[5376] dtype=0 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.0.attn_output.weight shape="[4096 5376]" dtype=12 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.0.attn_q.weight shape="[5376 4096]" dtype=12 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.0.attn_q_norm.weight shape=[128] dtype=0 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.0.attn_v.weight shape="[5376 2048]" dtype=14 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.0.ffn_down.weight shape="[21504 5376]" dtype=14 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.0.ffn_gate.weight shape="[5376 21504]" dtype=12 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.0.ffn_norm.weight shape=[5376] dtype=0 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.0.ffn_up.weight shape="[5376 21504]" dtype=12 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.0.post_attention_norm.weight shape=[5376] dtype=0 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.0.post_ffw_norm.weight shape=[5376] dtype=0 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.1.attn_k.weight shape="[5376 2048]" dtype=12 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.1.attn_k_norm.weight shape=[128] dtype=0 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.1.attn_norm.weight shape=[5376] dtype=0 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.1.attn_output.weight shape="[4096 5376]" dtype=12 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.1.attn_q.weight shape="[5376 4096]" dtype=12 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.1.attn_q_norm.weight shape=[128] dtype=0 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.1.attn_v.weight shape="[5376 2048]" dtype=14 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.1.ffn_down.weight shape="[21504 5376]" dtype=14 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.1.ffn_gate.weight shape="[5376 21504]" dtype=12 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.1.ffn_norm.weight shape=[5376] dtype=0 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.1.ffn_up.weight shape="[5376 21504]" dtype=12 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.1.post_attention_norm.weight shape=[5376] dtype=0 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.1.post_ffw_norm.weight shape=[5376] dtype=0 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.2.attn_k.weight shape="[5376 2048]" dtype=12 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.2.attn_k_norm.weight shape=[128] dtype=0 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.2.attn_norm.weight shape=[5376] dtype=0 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.2.attn_output.weight shape="[4096 5376]" dtype=12 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:30.869Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.2.attn_q.weight shape="[5376 4096]" dtype=12 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:30.870Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.2.attn_q_norm.weight shape=[128] dtype=0 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:30.870Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.2.attn_v.weight shape="[5376 2048]" dtype=14 buffer_type=CUDA0 ...... ...... ...... ollama-container-0 | time=2025-04-16T08:50:30.886Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.61.attn_output.weight shape="[4096 5376]" dtype=12 buffer_type=CUDA1 ollama-container-0 | time=2025-04-16T08:50:30.886Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.61.attn_q.weight shape="[5376 4096]" dtype=12 buffer_type=CUDA1 ollama-container-0 | time=2025-04-16T08:50:30.886Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.61.attn_q_norm.weight shape=[128] dtype=0 buffer_type=CUDA1 ollama-container-0 | time=2025-04-16T08:50:30.886Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.61.attn_v.weight shape="[5376 2048]" dtype=14 buffer_type=CUDA1 ollama-container-0 | time=2025-04-16T08:50:30.886Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.61.ffn_down.weight shape="[21504 5376]" dtype=14 buffer_type=CUDA1 ollama-container-0 | time=2025-04-16T08:50:30.886Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.61.ffn_gate.weight shape="[5376 21504]" dtype=12 buffer_type=CUDA1 ollama-container-0 | time=2025-04-16T08:50:30.887Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.61.ffn_norm.weight shape=[5376] dtype=0 buffer_type=CUDA1 ollama-container-0 | time=2025-04-16T08:50:30.887Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.61.ffn_up.weight shape="[5376 21504]" dtype=12 buffer_type=CUDA1 ollama-container-0 | time=2025-04-16T08:50:30.887Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.61.post_attention_norm.weight shape=[5376] dtype=0 buffer_type=CUDA1 ollama-container-0 | time=2025-04-16T08:50:30.887Z level=DEBUG source=ggml.go:220 msg="created tensor" name=blk.61.post_ffw_norm.weight shape=[5376] dtype=0 buffer_type=CUDA1 ollama-container-0 | time=2025-04-16T08:50:31.048Z level=INFO source=ggml.go:289 msg="model weights" buffer=CUDA0 size="7.2 GiB" ollama-container-0 | time=2025-04-16T08:50:31.048Z level=INFO source=ggml.go:289 msg="model weights" buffer=CUDA1 size="7.2 GiB" ollama-container-0 | time=2025-04-16T08:50:31.048Z level=INFO source=ggml.go:289 msg="model weights" buffer=CPU size="2.9 GiB" ollama-container-0 | time=2025-04-16T08:50:31.099Z level=DEBUG source=server.go:625 msg="model load progress 0.01" ollama-container-0 | time=2025-04-16T08:50:31.351Z level=DEBUG source=server.go:625 msg="model load progress 0.07" ollama-container-0 | time=2025-04-16T08:50:31.602Z level=DEBUG source=server.go:625 msg="model load progress 0.11" ollama-container-0 | time=2025-04-16T08:50:31.854Z level=DEBUG source=server.go:625 msg="model load progress 0.14" ollama-container-0 | time=2025-04-16T08:50:32.106Z level=DEBUG source=server.go:625 msg="model load progress 0.18" ....... ollama-container-0 | time=2025-04-16T08:50:38.407Z level=DEBUG source=server.go:625 msg="model load progress 0.98" ollama-container-0 | time=2025-04-16T08:50:38.561Z level=INFO source=ggml.go:388 msg="compute graph" backend=CUDA0 buffer_type=CUDA0 ollama-container-0 | time=2025-04-16T08:50:38.561Z level=INFO source=ggml.go:388 msg="compute graph" backend=CUDA1 buffer_type=CUDA1 ollama-container-0 | time=2025-04-16T08:50:38.561Z level=INFO source=ggml.go:388 msg="compute graph" backend=CPU buffer_type=CUDA_Host ollama-container-0 | time=2025-04-16T08:50:38.589Z level=WARN source=ggml.go:152 msg="key not found" key=tokenizer.ggml.add_eot_token default=false ollama-container-0 | time=2025-04-16T08:50:38.589Z level=DEBUG source=process_text_spm.go:21 msg=Tokens "num tokens"=262145 vals="[<pad> <eos> <bos> <unk> <mask>]" scores="[0 0 0 0 0]" types="[3 3 3 2 1]" ollama-container-0 | time=2025-04-16T08:50:38.593Z level=DEBUG source=process_text_spm.go:35 msg="Token counts" normal=261882 unknown=1 control=5 "user defined"=1 unused=0 byte=256 "max token len"=93 ollama-container-0 | time=2025-04-16T08:50:38.597Z level=DEBUG source=process_text_spm.go:21 msg=Tokens "num tokens"=262145 vals="[<pad> <eos> <bos> <unk> <mask>]" scores="[0 0 0 0 0]" types="[3 3 3 2 1]" ollama-container-0 | time=2025-04-16T08:50:38.600Z level=DEBUG source=process_text_spm.go:35 msg="Token counts" normal=261882 unknown=1 control=5 "user defined"=1 unused=0 byte=256 "max token len"=93 ollama-container-0 | time=2025-04-16T08:50:38.600Z level=WARN source=ggml.go:152 msg="key not found" key=gemma3.attention.layer_norm_rms_epsilon default=9.999999974752427e-07 ollama-container-0 | time=2025-04-16T08:50:38.600Z level=WARN source=ggml.go:152 msg="key not found" key=gemma3.rope.local.freq_base default=10000 ollama-container-0 | time=2025-04-16T08:50:38.600Z level=WARN source=ggml.go:152 msg="key not found" key=gemma3.rope.global.freq_base default=1e+06 ollama-container-0 | time=2025-04-16T08:50:38.600Z level=WARN source=ggml.go:152 msg="key not found" key=gemma3.rope.freq_scale default=1 ollama-container-0 | time=2025-04-16T08:50:38.600Z level=WARN source=ggml.go:152 msg="key not found" key=gemma3.mm_tokens_per_image default=256 ollama-container-0 | time=2025-04-16T08:50:38.600Z level=DEBUG source=model.go:205 msg="found tensor" name=v.patch_embedding.weight type=f16 shape="[14 14 3 1152]" ollama-container-0 | time=2025-04-16T08:50:38.600Z level=DEBUG source=model.go:205 msg="found tensor" name=v.position_embedding.weight type=f16 shape="[1152 4096]" ollama-container-0 | time=2025-04-16T08:50:38.600Z level=DEBUG source=model.go:205 msg="found tensor" name=v.post_layernorm.weight type=f32 shape=[1152] ollama-container-0 | time=2025-04-16T08:50:38.600Z level=DEBUG source=model.go:205 msg="found tensor" name=v.post_layernorm.bias type=f32 shape=[1152] ollama-container-0 | time=2025-04-16T08:50:38.600Z level=DEBUG source=model.go:205 msg="found tensor" name=v.blk.0.layer_norm1.weight type=f32 shape=[1152] ....... ollama-container-0 | time=2025-04-16T08:50:38.663Z level=DEBUG source=model.go:205 msg="found tensor" name=output_norm.weight type=f32 shape=[5376] ollama-container-0 | time=2025-04-16T08:50:38.663Z level=DEBUG source=model.go:205 msg="found tensor" name=output.weight type=q6_K shape="[5376 262144]" ollama-container-0 | time=2025-04-16T08:50:38.663Z level=DEBUG source=model.go:205 msg="found tensor" name=mm.mm_soft_emb_norm.weight type=f32 shape=[1152] ollama-container-0 | time=2025-04-16T08:50:38.663Z level=DEBUG source=model.go:205 msg="found tensor" name=mm.mm_input_projection.weight type=f16 shape="[5376 1152]" ollama-container-0 | time=2025-04-16T08:50:38.910Z level=INFO source=server.go:619 msg="llama runner started in 8.57 seconds" ollama-container-0 | time=2025-04-16T08:50:38.910Z level=DEBUG source=sched.go:464 msg="finished setting up runner" model=/root/.ollama/models/blobs/sha256-e796792eba26c4d3b04b0ac5adb01a453dd9ec2dfd83b6c59cbf6fe5f30b0f68 ollama-container-0 | time=2025-04-16T08:50:38.910Z level=WARN source=routes.go:280 msg="the context field is deprecated and will be removed in a future version of Ollama" ollama-container-0 | time=2025-04-16T08:50:38.910Z level=DEBUG source=routes.go:297 msg="generate request" images=0 prompt="<start_of_turn>user\nhello<end_of_turn>\n<start_of_turn>model\n" ollama-container-0 | time=2025-04-16T08:50:38.971Z level=DEBUG source=process_text_spm.go:184 msg="adding bos token to prompt" id=2 ollama-container-0 | time=2025-04-16T08:50:38.972Z level=DEBUG source=cache.go:136 msg="loading cache slot" id=0 cache=0 prompt=10 used=0 remaining=10 ollama-container-0 | [GIN] 2025/04/16 - 08:51:10 | 200 | 44.20092233s | 10.1.10.32 | POST "/api/generate" ollama-container-0 | time=2025-04-16T08:51:10.464Z level=DEBUG source=sched.go:468 msg="context for request finished" ollama-container-0 | time=2025-04-16T08:51:10.465Z level=DEBUG source=sched.go:341 msg="runner with non-zero duration has gone idle, adding timer" modelPath=/root/.ollama/models/blobs/sha256-e796792eba26c4d3b04b0ac5adb01a453dd9ec2dfd83b6c59cbf6fe5f30b0f68 duration=2562047h47m16.854775807s ollama-container-0 | time=2025-04-16T08:51:10.465Z level=DEBUG source=sched.go:359 msg="after processing request finished event" modelPath=/root/.ollama/models/blobs/sha256-e796792eba26c4d3b04b0ac5adb01a453dd9ec2dfd83b6c59cbf6fe5f30b0f68 refCount=0 ### Relevant log output ```shell ``` ### OS _No response_ ### GPU _No response_ ### CPU _No response_ ### Ollama version _No response_
GiteaMirror added the bug label 2026-04-29 02:26:30 -05:00
GiteaMirror commented 2026-04-29 02:26:31 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Apr 16, 2025):

https://github.com/ollama/ollama/issues/7648#issuecomment-2473561990

<!-- gh-comment-id:2809707874 --> @rick-github commented on GitHub (Apr 16, 2025): https://github.com/ollama/ollama/issues/7648#issuecomment-2473561990
GiteaMirror commented 2026-04-29 02:26:32 -05:00
Author
Owner
Copy Link

@leizhu1989 commented on GitHub (Apr 17, 2025):

@rick-github Thank you for your reply. I have looked at the link you provided and it seems that Olama cannot solve this problem. I tried the 12b-q4 model again. A single graphics card T4 can run, but only input text without images, which is relatively slow. Is it should set some parameters to remove the image encoding part when starting Docker for a multimodal model？

<!-- gh-comment-id:2812235585 --> @leizhu1989 commented on GitHub (Apr 17, 2025): @rick-github Thank you for your reply. I have looked at the link you provided and it seems that Olama cannot solve this problem. I tried the 12b-q4 model again. A single graphics card T4 can run, but only input text without images, which is relatively slow. Is it should set some parameters to remove the image encoding part when starting Docker for a multimodal model？
GiteaMirror commented 2026-04-29 02:26:32 -05:00
Author
Owner
Copy Link

@leizhu1989 commented on GitHub (Apr 17, 2025):

It seems that the CPU and GPU are reasoning together, not purely GPU reasoning

<!-- gh-comment-id:2812249010 --> @leizhu1989 commented on GitHub (Apr 17, 2025): It seems that the CPU and GPU are reasoning together, not purely GPU reasoning
GiteaMirror commented 2026-04-29 02:26:32 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Apr 17, 2025):

Is it should set some parameters to remove the image encoding part when starting Docker for a multimodal model？

No.

It seems that the CPU and GPU are reasoning together, not purely GPU reasoning

If the model doesn't fit in the available VRAM, part of it will run in system RAM. Even if the whole model fits in VRAM, CPU is still used for providing instructions to the GPU, so there will still be CPU usage.

<!-- gh-comment-id:2812308358 --> @rick-github commented on GitHub (Apr 17, 2025): > Is it should set some parameters to remove the image encoding part when starting Docker for a multimodal model？ No. > It seems that the CPU and GPU are reasoning together, not purely GPU reasoning If the model doesn't fit in the available VRAM, part of it will run in system RAM. Even if the whole model fits in VRAM, CPU is still used for providing instructions to the GPU, so there will still be CPU usage.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#53274
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?