github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-22 13:42:25 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #9514] Granite-vision compute on CPU when FA is turned on. #6202

New Issue
Open
opened 2026-04-12 17:35:20 -05:00 by GiteaMirror · 7 comments
GiteaMirror commented 2026-04-12 17:35:20 -05:00
Owner
Copy Link

Originally created by @sammyf on GitHub (Mar 5, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/9514

What is the issue?

if Flash Attention is turned on Granite-Vision will be loaded on GPU if one is present, but the compute will be done on CPU.

Image
Image

Relevant log output

░░ The job identifier is 7479 and the job result is done.
Mar 05 08:49:01 neo-bandito systemd[1]: ollama.service: Consumed 590ms CPU time, 75.3M memory peak.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit ollama.service completed and consumed the indicated resources.
Mar 05 08:49:01 neo-bandito systemd[1]: Started Ollama Service.
░░ Subject: A start job for unit ollama.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit ollama.service has finished successfully.
░░ 
░░ The job identifier is 7479.
Mar 05 08:49:02 neo-bandito ollama[7692]: 2025/03/05 08:49:02 routes.go:1215: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:2048 OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:true OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:2562047h47m16.854775807s OLLAMA_KV_CACHE_TYPE:q8_0 OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:4 OLLAMA_MAX_QUEUE:10 OLLAMA_MODELS:/media/GLIMSPANKY/ollama/models/ OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:2 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
Mar 05 08:49:02 neo-bandito ollama[7692]: time=2025-03-05T08:49:02.017+01:00 level=INFO source=images.go:432 msg="total blobs: 178"
Mar 05 08:49:02 neo-bandito ollama[7692]: time=2025-03-05T08:49:02.018+01:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0"
Mar 05 08:49:02 neo-bandito ollama[7692]: time=2025-03-05T08:49:02.019+01:00 level=INFO source=routes.go:1277 msg="Listening on [::]:11434 (version 0.5.13)"
Mar 05 08:49:02 neo-bandito ollama[7692]: time=2025-03-05T08:49:02.019+01:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
Mar 05 08:49:02 neo-bandito ollama[7692]: time=2025-03-05T08:49:02.506+01:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-a9be7ece-3ea9-9a38-3a55-5aad9943f497 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA GeForce RTX 3090" total="23.6 GiB" available="22.6 GiB"
Mar 05 08:49:02 neo-bandito ollama[7692]: time=2025-03-05T08:49:02.506+01:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-7db6777e-b194-eee2-c132-cea3c32e6d0a library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA RTX A1000" total="7.7 GiB" available="7.6 GiB"
Mar 05 08:49:02 neo-bandito ollama[7692]: [GIN] 2025/03/05 - 08:49:02 | 200 |     357.985µs |   192.168.0.100 | GET      "/api/ps"
Mar 05 08:49:03 neo-bandito ollama[7692]: [GIN] 2025/03/05 - 08:49:03 | 200 |      16.937µs |   192.168.0.100 | GET      "/api/ps"
Mar 05 08:49:04 neo-bandito ollama[7692]: [GIN] 2025/03/05 - 08:49:04 | 200 |      63.065µs |   192.168.0.100 | GET      "/api/ps"
Mar 05 08:49:05 neo-bandito ollama[7692]: [GIN] 2025/03/05 - 08:49:05 | 200 |      59.069µs |   192.168.0.100 | GET      "/api/ps"
Mar 05 08:49:06 neo-bandito ollama[7692]: [GIN] 2025/03/05 - 08:49:06 | 200 |      67.917µs |   192.168.0.100 | GET      "/api/ps"
Mar 05 08:49:07 neo-bandito ollama[7692]: [GIN] 2025/03/05 - 08:49:07 | 200 |      22.628µs |   192.168.0.100 | GET      "/api/ps"
Mar 05 08:49:08 neo-bandito ollama[7692]: [GIN] 2025/03/05 - 08:49:08 | 200 |      61.197µs |   192.168.0.100 | GET      "/api/ps"
Mar 05 08:49:09 neo-bandito ollama[7692]: [GIN] 2025/03/05 - 08:49:09 | 200 |      96.138µs |   192.168.0.100 | GET      "/api/ps"
Mar 05 08:49:10 neo-bandito systemd[1]: /etc/systemd/system/ollama.service.d/override.conf:1: Assignment outside of section. Ignoring.
Mar 05 08:49:10 neo-bandito systemd[1]: /etc/systemd/system/ollama.service.d/override.conf:2: Assignment outside of section. Ignoring.
Mar 05 08:49:10 neo-bandito systemd[1]: /etc/systemd/system/ollama.service.d/override.conf:3: Assignment outside of section. Ignoring.
Mar 05 08:49:10 neo-bandito systemd[1]: /etc/systemd/system/ollama.service.d/override.conf:4: Assignment outside of section. Ignoring.
Mar 05 08:49:10 neo-bandito ollama[7692]: [GIN] 2025/03/05 - 08:49:10 | 200 |      19.044µs |   192.168.0.100 | GET      "/api/ps"
Mar 05 08:49:11 neo-bandito systemd[1]: Stopping Ollama Service...
░░ Subject: A stop job for unit ollama.service has begun execution
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A stop job for unit ollama.service has begun execution.
░░ 
░░ The job identifier is 7726.
Mar 05 08:49:11 neo-bandito systemd[1]: ollama.service: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit ollama.service has successfully entered the 'dead' state.
Mar 05 08:49:11 neo-bandito systemd[1]: Stopped Ollama Service.
░░ Subject: A stop job for unit ollama.service has finished
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A stop job for unit ollama.service has finished.
░░ 
░░ The job identifier is 7726 and the job result is done.
Mar 05 08:49:11 neo-bandito systemd[1]: Started Ollama Service.
░░ Subject: A start job for unit ollama.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit ollama.service has finished successfully.
░░ 
░░ The job identifier is 7726.
Mar 05 08:49:11 neo-bandito ollama[8024]: 2025/03/05 08:49:11 routes.go:1215: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:2048 OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:2562047h47m16.854775807s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:4 OLLAMA_MAX_QUEUE:10 OLLAMA_MODELS:/media/GLIMSPANKY/ollama/models/ OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:2 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
Mar 05 08:49:11 neo-bandito ollama[8024]: time=2025-03-05T08:49:11.335+01:00 level=INFO source=images.go:432 msg="total blobs: 178"
Mar 05 08:49:11 neo-bandito ollama[8024]: time=2025-03-05T08:49:11.336+01:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0"
Mar 05 08:49:11 neo-bandito ollama[8024]: time=2025-03-05T08:49:11.337+01:00 level=INFO source=routes.go:1277 msg="Listening on [::]:11434 (version 0.5.13)"
Mar 05 08:49:11 neo-bandito ollama[8024]: time=2025-03-05T08:49:11.337+01:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
Mar 05 08:49:11 neo-bandito ollama[8024]: time=2025-03-05T08:49:11.847+01:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-a9be7ece-3ea9-9a38-3a55-5aad9943f497 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA GeForce RTX 3090" total="23.6 GiB" available="22.6 GiB"
Mar 05 08:49:11 neo-bandito ollama[8024]: time=2025-03-05T08:49:11.847+01:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-7db6777e-b194-eee2-c132-cea3c32e6d0a library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA RTX A1000" total="7.7 GiB" available="7.6 GiB"
Mar 05 08:49:11 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:11 | 200 |      71.813µs |   192.168.0.100 | GET      "/api/ps"
Mar 05 08:49:12 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:12 | 200 |      65.689µs |   192.168.0.100 | GET      "/api/ps"
Mar 05 08:49:13 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:13 | 200 |      17.185µs |   192.168.0.100 | GET      "/api/ps"
Mar 05 08:49:14 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:14 | 200 |      59.232µs |   192.168.0.100 | GET      "/api/ps"
Mar 05 08:49:15 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:15 | 200 |      19.849µs |   192.168.0.100 | GET      "/api/ps"
Mar 05 08:49:16 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:16 | 200 |     112.109µs |   192.168.0.100 | GET      "/api/ps"
Mar 05 08:49:17 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:17 | 200 |      68.044µs |   192.168.0.100 | GET      "/api/ps"
Mar 05 08:49:18 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:18 | 200 |      63.217µs |   192.168.0.100 | GET      "/api/ps"
Mar 05 08:49:19 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:19 | 200 |      61.115µs |   192.168.0.100 | GET      "/api/ps"
Mar 05 08:49:20 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:20 | 200 |      58.772µs |   192.168.0.100 | GET      "/api/ps"
Mar 05 08:49:21 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:21 | 200 |      16.367µs |   192.168.0.100 | GET      "/api/ps"
Mar 05 08:49:22 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:22 | 200 |      69.372µs |   192.168.0.100 | GET      "/api/ps"
Mar 05 08:49:23 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:23 | 200 |      60.084µs |   192.168.0.100 | GET      "/api/ps"
Mar 05 08:49:24 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:24 | 200 |      13.857µs |       127.0.0.1 | HEAD     "/"
Mar 05 08:49:24 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:24 | 200 |   42.679002ms |       127.0.0.1 | POST     "/api/show"
Mar 05 08:49:24 neo-bandito ollama[8024]: time=2025-03-05T08:49:24.693+01:00 level=WARN source=ggml.go:136 msg="key not found" key=granite.attention.key_length default=64
Mar 05 08:49:24 neo-bandito ollama[8024]: time=2025-03-05T08:49:24.693+01:00 level=WARN source=ggml.go:136 msg="key not found" key=granite.attention.value_length default=64
Mar 05 08:49:24 neo-bandito ollama[8024]: time=2025-03-05T08:49:24.693+01:00 level=INFO source=sched.go:715 msg="new model will fit in available VRAM in single GPU, loading" model=/media/GLIMSPANKY/ollama/models/blobs/sha256-1aefcd9a8a15091b670951963b5f8a7e6653bb1350345e9621e179685ac9bc5f gpu=GPU-a9be7ece-3ea9-9a38-3a55-5aad9943f497 parallel=2 available=24303894528 required="7.0 GiB"
Mar 05 08:49:24 neo-bandito ollama[8024]: time=2025-03-05T08:49:24.937+01:00 level=INFO source=server.go:97 msg="system memory" total="62.6 GiB" free="57.8 GiB" free_swap="123.0 GiB"
Mar 05 08:49:24 neo-bandito ollama[8024]: time=2025-03-05T08:49:24.938+01:00 level=WARN source=ggml.go:136 msg="key not found" key=granite.attention.key_length default=64
Mar 05 08:49:24 neo-bandito ollama[8024]: time=2025-03-05T08:49:24.938+01:00 level=WARN source=ggml.go:136 msg="key not found" key=granite.attention.value_length default=64
Mar 05 08:49:24 neo-bandito ollama[8024]: time=2025-03-05T08:49:24.938+01:00 level=INFO source=server.go:130 msg=offload library=cuda layers.requested=-1 layers.model=41 layers.offload=41 layers.split="" memory.available="[22.6 GiB]" memory.gpu_overhead="0 B" memory.required.full="7.0 GiB" memory.required.partial="7.0 GiB" memory.required.kv="2.5 GiB" memory.required.allocations="[7.0 GiB]" memory.weights.total="3.9 GiB" memory.weights.repeating="3.8 GiB" memory.weights.nonrepeating="78.8 MiB" memory.graph.full="1.7 GiB" memory.graph.partial="1.7 GiB" projector.weights="851.2 MiB" projector.graph="0 B"
Mar 05 08:49:24 neo-bandito ollama[8024]: time=2025-03-05T08:49:24.939+01:00 level=INFO source=server.go:380 msg="starting llama server" cmd="/usr/local/bin/ollama runner --model /media/GLIMSPANKY/ollama/models/blobs/sha256-1aefcd9a8a15091b670951963b5f8a7e6653bb1350345e9621e179685ac9bc5f --ctx-size 32768 --batch-size 512 --n-gpu-layers 41 --mmproj /media/GLIMSPANKY/ollama/models/blobs/sha256-4d464be24899cf8dc1862945432e0cef4366c4181fa38b14754cc9279b727608 --threads 8 --parallel 2 --port 35673"
Mar 05 08:49:24 neo-bandito ollama[8024]: time=2025-03-05T08:49:24.939+01:00 level=INFO source=sched.go:450 msg="loaded runners" count=1
Mar 05 08:49:24 neo-bandito ollama[8024]: time=2025-03-05T08:49:24.939+01:00 level=INFO source=server.go:557 msg="waiting for llama runner to start responding"
Mar 05 08:49:24 neo-bandito ollama[8024]: time=2025-03-05T08:49:24.939+01:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server error"
Mar 05 08:49:24 neo-bandito ollama[8024]: time=2025-03-05T08:49:24.952+01:00 level=INFO source=runner.go:931 msg="starting go runner"
Mar 05 08:49:24 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:24 | 200 |       45.92µs |   192.168.0.100 | GET      "/api/ps"
Mar 05 08:49:25 neo-bandito ollama[8024]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
Mar 05 08:49:25 neo-bandito ollama[8024]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
Mar 05 08:49:25 neo-bandito ollama[8024]: ggml_cuda_init: found 1 CUDA devices:
Mar 05 08:49:25 neo-bandito ollama[8024]:   Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Mar 05 08:49:25 neo-bandito ollama[8024]: load_backend: loaded CUDA backend from /usr/local/lib/ollama/cuda_v12/libggml-cuda.so
Mar 05 08:49:25 neo-bandito ollama[8024]: load_backend: loaded CPU backend from /usr/local/lib/ollama/libggml-cpu-haswell.so
Mar 05 08:49:25 neo-bandito ollama[8024]: time=2025-03-05T08:49:25.235+01:00 level=INFO source=runner.go:934 msg=system info="CPU : LLAMAFILE = 1 | CUDA : ARCHS = 500,600,610,700,750,800,860,870,890,900,1200 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | LLAMAFILE = 1 | cgo(gcc)" threads=8
Mar 05 08:49:25 neo-bandito ollama[8024]: time=2025-03-05T08:49:25.236+01:00 level=INFO source=runner.go:992 msg="Server listening on 127.0.0.1:35673"
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 3090) - 23178 MiB free
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: loaded meta data with 32 key-value pairs and 362 tensors from /media/GLIMSPANKY/ollama/models/blobs/sha256-1aefcd9a8a15091b670951963b5f8a7e6653bb1350345e9621e179685ac9bc5f (version GGUF V3 (latest))
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv   0:                       general.architecture str              = granite
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv   1:                               general.type str              = model
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv   2:                               general.name str              = Granite_Vision_Llm
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv   3:                         general.size_label str              = 2.5B
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv   4:                        granite.block_count u32              = 40
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv   5:                     granite.context_length u32              = 16384
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv   6:                   granite.embedding_length u32              = 2048
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv   7:                granite.feed_forward_length u32              = 8192
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv   8:               granite.attention.head_count u32              = 32
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv   9:            granite.attention.head_count_kv u32              = 8
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv  10:                     granite.rope.freq_base f32              = 300000.000000
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv  11:   granite.attention.layer_norm_rms_epsilon f32              = 0.000010
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv  12:                         granite.vocab_size u32              = 49156
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv  13:               granite.rope.dimension_count u32              = 64
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv  14:                    granite.attention.scale f32              = 0.015625
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv  15:                    granite.embedding_scale f32              = 12.000000
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv  16:                     granite.residual_scale f32              = 0.220000
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv  17:                        granite.logit_scale f32              = 8.000000
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv  18:                       tokenizer.ggml.model str              = gpt2
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv  19:                         tokenizer.ggml.pre str              = refact
Mar 05 08:49:25 neo-bandito ollama[8024]: time=2025-03-05T08:49:25.447+01:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server loading model"
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv  20:                      tokenizer.ggml.tokens arr[str,49156]   = ["<|end_of_text|>", "<fim_prefix>", "...
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv  21:                  tokenizer.ggml.token_type arr[i32,49156]   = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv  22:                      tokenizer.ggml.merges arr[str,48891]   = ["Ġ Ġ", "ĠĠ ĠĠ", "ĠĠĠĠ ĠĠ...
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv  23:                tokenizer.ggml.bos_token_id u32              = 0
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv  24:                tokenizer.ggml.eos_token_id u32              = 0
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv  25:            tokenizer.ggml.unknown_token_id u32              = 0
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv  26:            tokenizer.ggml.padding_token_id u32              = 0
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv  27:               tokenizer.ggml.add_bos_token bool             = false
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv  28:                    tokenizer.chat_template str              = {%- if tools %}\n    {{- '<|start_of_r...
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv  29:            tokenizer.ggml.add_space_prefix bool             = false
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv  30:               general.quantization_version u32              = 2
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv  31:                          general.file_type u32              = 15
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - type  f32:   81 tensors
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - type q4_K:  240 tensors
Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - type q6_K:   41 tensors
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: file format = GGUF V3 (latest)
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: file type   = Q4_K - Medium
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: file size   = 1.44 GiB (4.87 BPW)
Mar 05 08:49:25 neo-bandito ollama[8024]: load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
Mar 05 08:49:25 neo-bandito ollama[8024]: load: special tokens cache size = 23
Mar 05 08:49:25 neo-bandito ollama[8024]: load: token to piece cache size = 0.2826 MB
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: arch             = granite
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: vocab_only       = 0
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_ctx_train      = 16384
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_embd           = 2048
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_layer          = 40
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_head           = 32
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_head_kv        = 8
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_rot            = 64
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_swa            = 0
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_embd_head_k    = 64
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_embd_head_v    = 64
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_gqa            = 4
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_embd_k_gqa     = 512
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_embd_v_gqa     = 512
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: f_norm_eps       = 0.0e+00
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: f_norm_rms_eps   = 1.0e-05
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: f_clamp_kqv      = 0.0e+00
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: f_max_alibi_bias = 0.0e+00
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: f_logit_scale    = 8.0e+00
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_ff             = 8192
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_expert         = 0
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_expert_used    = 0
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: causal attn      = 1
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: pooling type     = 0
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: rope type        = 0
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: rope scaling     = linear
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: freq_base_train  = 300000.0
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: freq_scale_train = 1
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_ctx_orig_yarn  = 16384
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: rope_finetuned   = unknown
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: ssm_d_conv       = 0
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: ssm_d_inner      = 0
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: ssm_d_state      = 0
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: ssm_dt_rank      = 0
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: ssm_dt_b_c_rms   = 0
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: model type       = 3B
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: model params     = 2.53 B
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: general.name     = Granite_Vision_Llm
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: f_embedding_scale = 12.000000
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: f_residual_scale  = 0.220000
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: f_attention_scale = 0.015625
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: vocab type       = BPE
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_vocab          = 49156
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_merges         = 48891
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: BOS token        = 0 '<|end_of_text|>'
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: EOS token        = 0 '<|end_of_text|>'
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: UNK token        = 0 '<|end_of_text|>'
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: PAD token        = 0 '<|end_of_text|>'
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: LF token         = 203 'Ċ'
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: EOG token        = 0 '<|end_of_text|>'
Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: max token length = 512
Mar 05 08:49:25 neo-bandito ollama[8024]: load_tensors: loading model tensors, this can take a while... (mmap = true)
Mar 05 08:49:25 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:25 | 200 |      43.164µs |   192.168.0.100 | GET      "/api/ps"
Mar 05 08:49:26 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:26 | 200 |      56.211µs |   192.168.0.100 | GET      "/api/ps"
Mar 05 08:49:27 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:27 | 200 |      92.251µs |   192.168.0.100 | GET      "/api/ps"
Mar 05 08:49:28 neo-bandito ollama[8024]: load_tensors: offloading 40 repeating layers to GPU
Mar 05 08:49:28 neo-bandito ollama[8024]: load_tensors: offloading output layer to GPU
Mar 05 08:49:28 neo-bandito ollama[8024]: load_tensors: offloaded 41/41 layers to GPU
Mar 05 08:49:28 neo-bandito ollama[8024]: load_tensors:        CUDA0 model buffer size =  1472.05 MiB
Mar 05 08:49:28 neo-bandito ollama[8024]: load_tensors:   CPU_Mapped model buffer size =    78.76 MiB
Mar 05 08:49:28 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:28 | 200 |      21.768µs |   192.168.0.100 | GET      "/api/ps"
Mar 05 08:49:29 neo-bandito ollama[8024]: llama_init_from_model: n_seq_max     = 2
Mar 05 08:49:29 neo-bandito ollama[8024]: llama_init_from_model: n_ctx         = 32768
Mar 05 08:49:29 neo-bandito ollama[8024]: llama_init_from_model: n_ctx_per_seq = 16384
Mar 05 08:49:29 neo-bandito ollama[8024]: llama_init_from_model: n_batch       = 1024
Mar 05 08:49:29 neo-bandito ollama[8024]: llama_init_from_model: n_ubatch      = 512
Mar 05 08:49:29 neo-bandito ollama[8024]: llama_init_from_model: flash_attn    = 0
Mar 05 08:49:29 neo-bandito ollama[8024]: llama_init_from_model: freq_base     = 300000.0
Mar 05 08:49:29 neo-bandito ollama[8024]: llama_init_from_model: freq_scale    = 1
Mar 05 08:49:29 neo-bandito ollama[8024]: llama_kv_cache_init: kv_size = 32768, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 40, can_shift = 1
Mar 05 08:49:29 neo-bandito ollama[8024]: llama_kv_cache_init:      CUDA0 KV buffer size =  2560.00 MiB
Mar 05 08:49:29 neo-bandito ollama[8024]: llama_init_from_model: KV self size  = 2560.00 MiB, K (f16): 1280.00 MiB, V (f16): 1280.00 MiB
Mar 05 08:49:29 neo-bandito ollama[8024]: llama_init_from_model:  CUDA_Host  output buffer size =     0.39 MiB
Mar 05 08:49:29 neo-bandito ollama[8024]: llama_init_from_model:      CUDA0 compute buffer size =  2128.00 MiB
Mar 05 08:49:29 neo-bandito ollama[8024]: llama_init_from_model:  CUDA_Host compute buffer size =    68.01 MiB
Mar 05 08:49:29 neo-bandito ollama[8024]: llama_init_from_model: graph nodes  = 1368
Mar 05 08:49:29 neo-bandito ollama[8024]: llama_init_from_model: graph splits = 2
Mar 05 08:49:29 neo-bandito ollama[8024]: key clip.use_silu not found in file
Mar 05 08:49:29 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:29 | 200 |     111.595µs |   192.168.0.100 | GET      "/api/ps"
Mar 05 08:49:30 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:30 | 200 |      59.625µs |   192.168.0.100 | GET      "/api/ps"
Mar 05 08:49:31 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:31 | 200 |      24.563µs |   192.168.0.100 | GET      "/api/ps"
Mar 05 08:49:32 neo-bandito ollama[8024]: key clip.vision.image_crop_resolution not found in file
Mar 05 08:49:32 neo-bandito ollama[8024]: time=2025-03-05T08:49:32.224+01:00 level=INFO source=server.go:596 msg="llama runner started in 7.29 seconds"
Mar 05 08:49:32 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:32 | 200 |   7.81541621s |       127.0.0.1 | POST     "/api/generate"

(EDIT: filled the systems specs)

OS

Linux

GPU

RTX3090

CPU

Intel

Ollama version

0.5.13

Originally created by @sammyf on GitHub (Mar 5, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/9514 ### What is the issue? if Flash Attention is turned on Granite-Vision will be loaded on GPU if one is present, but the compute will be done on CPU. ![Image](https://github.com/user-attachments/assets/0e88ebe4-9def-47f6-a877-f45b6c862be3) ![Image](https://github.com/user-attachments/assets/cb63a8fa-8288-4f79-8865-6732168e75ec) ### Relevant log output ```shell ░░ The job identifier is 7479 and the job result is done. Mar 05 08:49:01 neo-bandito systemd[1]: ollama.service: Consumed 590ms CPU time, 75.3M memory peak. ░░ Subject: Resources consumed by unit runtime ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ The unit ollama.service completed and consumed the indicated resources. Mar 05 08:49:01 neo-bandito systemd[1]: Started Ollama Service. ░░ Subject: A start job for unit ollama.service has finished successfully ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ A start job for unit ollama.service has finished successfully. ░░ ░░ The job identifier is 7479. Mar 05 08:49:02 neo-bandito ollama[7692]: 2025/03/05 08:49:02 routes.go:1215: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:2048 OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:true OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:2562047h47m16.854775807s OLLAMA_KV_CACHE_TYPE:q8_0 OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:4 OLLAMA_MAX_QUEUE:10 OLLAMA_MODELS:/media/GLIMSPANKY/ollama/models/ OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:2 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" Mar 05 08:49:02 neo-bandito ollama[7692]: time=2025-03-05T08:49:02.017+01:00 level=INFO source=images.go:432 msg="total blobs: 178" Mar 05 08:49:02 neo-bandito ollama[7692]: time=2025-03-05T08:49:02.018+01:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0" Mar 05 08:49:02 neo-bandito ollama[7692]: time=2025-03-05T08:49:02.019+01:00 level=INFO source=routes.go:1277 msg="Listening on [::]:11434 (version 0.5.13)" Mar 05 08:49:02 neo-bandito ollama[7692]: time=2025-03-05T08:49:02.019+01:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs" Mar 05 08:49:02 neo-bandito ollama[7692]: time=2025-03-05T08:49:02.506+01:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-a9be7ece-3ea9-9a38-3a55-5aad9943f497 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA GeForce RTX 3090" total="23.6 GiB" available="22.6 GiB" Mar 05 08:49:02 neo-bandito ollama[7692]: time=2025-03-05T08:49:02.506+01:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-7db6777e-b194-eee2-c132-cea3c32e6d0a library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA RTX A1000" total="7.7 GiB" available="7.6 GiB" Mar 05 08:49:02 neo-bandito ollama[7692]: [GIN] 2025/03/05 - 08:49:02 | 200 | 357.985µs | 192.168.0.100 | GET "/api/ps" Mar 05 08:49:03 neo-bandito ollama[7692]: [GIN] 2025/03/05 - 08:49:03 | 200 | 16.937µs | 192.168.0.100 | GET "/api/ps" Mar 05 08:49:04 neo-bandito ollama[7692]: [GIN] 2025/03/05 - 08:49:04 | 200 | 63.065µs | 192.168.0.100 | GET "/api/ps" Mar 05 08:49:05 neo-bandito ollama[7692]: [GIN] 2025/03/05 - 08:49:05 | 200 | 59.069µs | 192.168.0.100 | GET "/api/ps" Mar 05 08:49:06 neo-bandito ollama[7692]: [GIN] 2025/03/05 - 08:49:06 | 200 | 67.917µs | 192.168.0.100 | GET "/api/ps" Mar 05 08:49:07 neo-bandito ollama[7692]: [GIN] 2025/03/05 - 08:49:07 | 200 | 22.628µs | 192.168.0.100 | GET "/api/ps" Mar 05 08:49:08 neo-bandito ollama[7692]: [GIN] 2025/03/05 - 08:49:08 | 200 | 61.197µs | 192.168.0.100 | GET "/api/ps" Mar 05 08:49:09 neo-bandito ollama[7692]: [GIN] 2025/03/05 - 08:49:09 | 200 | 96.138µs | 192.168.0.100 | GET "/api/ps" Mar 05 08:49:10 neo-bandito systemd[1]: /etc/systemd/system/ollama.service.d/override.conf:1: Assignment outside of section. Ignoring. Mar 05 08:49:10 neo-bandito systemd[1]: /etc/systemd/system/ollama.service.d/override.conf:2: Assignment outside of section. Ignoring. Mar 05 08:49:10 neo-bandito systemd[1]: /etc/systemd/system/ollama.service.d/override.conf:3: Assignment outside of section. Ignoring. Mar 05 08:49:10 neo-bandito systemd[1]: /etc/systemd/system/ollama.service.d/override.conf:4: Assignment outside of section. Ignoring. Mar 05 08:49:10 neo-bandito ollama[7692]: [GIN] 2025/03/05 - 08:49:10 | 200 | 19.044µs | 192.168.0.100 | GET "/api/ps" Mar 05 08:49:11 neo-bandito systemd[1]: Stopping Ollama Service... ░░ Subject: A stop job for unit ollama.service has begun execution ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ A stop job for unit ollama.service has begun execution. ░░ ░░ The job identifier is 7726. Mar 05 08:49:11 neo-bandito systemd[1]: ollama.service: Deactivated successfully. ░░ Subject: Unit succeeded ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ The unit ollama.service has successfully entered the 'dead' state. Mar 05 08:49:11 neo-bandito systemd[1]: Stopped Ollama Service. ░░ Subject: A stop job for unit ollama.service has finished ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ A stop job for unit ollama.service has finished. ░░ ░░ The job identifier is 7726 and the job result is done. Mar 05 08:49:11 neo-bandito systemd[1]: Started Ollama Service. ░░ Subject: A start job for unit ollama.service has finished successfully ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ A start job for unit ollama.service has finished successfully. ░░ ░░ The job identifier is 7726. Mar 05 08:49:11 neo-bandito ollama[8024]: 2025/03/05 08:49:11 routes.go:1215: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:2048 OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:2562047h47m16.854775807s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:4 OLLAMA_MAX_QUEUE:10 OLLAMA_MODELS:/media/GLIMSPANKY/ollama/models/ OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:2 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" Mar 05 08:49:11 neo-bandito ollama[8024]: time=2025-03-05T08:49:11.335+01:00 level=INFO source=images.go:432 msg="total blobs: 178" Mar 05 08:49:11 neo-bandito ollama[8024]: time=2025-03-05T08:49:11.336+01:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0" Mar 05 08:49:11 neo-bandito ollama[8024]: time=2025-03-05T08:49:11.337+01:00 level=INFO source=routes.go:1277 msg="Listening on [::]:11434 (version 0.5.13)" Mar 05 08:49:11 neo-bandito ollama[8024]: time=2025-03-05T08:49:11.337+01:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs" Mar 05 08:49:11 neo-bandito ollama[8024]: time=2025-03-05T08:49:11.847+01:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-a9be7ece-3ea9-9a38-3a55-5aad9943f497 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA GeForce RTX 3090" total="23.6 GiB" available="22.6 GiB" Mar 05 08:49:11 neo-bandito ollama[8024]: time=2025-03-05T08:49:11.847+01:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-7db6777e-b194-eee2-c132-cea3c32e6d0a library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA RTX A1000" total="7.7 GiB" available="7.6 GiB" Mar 05 08:49:11 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:11 | 200 | 71.813µs | 192.168.0.100 | GET "/api/ps" Mar 05 08:49:12 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:12 | 200 | 65.689µs | 192.168.0.100 | GET "/api/ps" Mar 05 08:49:13 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:13 | 200 | 17.185µs | 192.168.0.100 | GET "/api/ps" Mar 05 08:49:14 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:14 | 200 | 59.232µs | 192.168.0.100 | GET "/api/ps" Mar 05 08:49:15 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:15 | 200 | 19.849µs | 192.168.0.100 | GET "/api/ps" Mar 05 08:49:16 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:16 | 200 | 112.109µs | 192.168.0.100 | GET "/api/ps" Mar 05 08:49:17 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:17 | 200 | 68.044µs | 192.168.0.100 | GET "/api/ps" Mar 05 08:49:18 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:18 | 200 | 63.217µs | 192.168.0.100 | GET "/api/ps" Mar 05 08:49:19 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:19 | 200 | 61.115µs | 192.168.0.100 | GET "/api/ps" Mar 05 08:49:20 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:20 | 200 | 58.772µs | 192.168.0.100 | GET "/api/ps" Mar 05 08:49:21 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:21 | 200 | 16.367µs | 192.168.0.100 | GET "/api/ps" Mar 05 08:49:22 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:22 | 200 | 69.372µs | 192.168.0.100 | GET "/api/ps" Mar 05 08:49:23 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:23 | 200 | 60.084µs | 192.168.0.100 | GET "/api/ps" Mar 05 08:49:24 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:24 | 200 | 13.857µs | 127.0.0.1 | HEAD "/" Mar 05 08:49:24 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:24 | 200 | 42.679002ms | 127.0.0.1 | POST "/api/show" Mar 05 08:49:24 neo-bandito ollama[8024]: time=2025-03-05T08:49:24.693+01:00 level=WARN source=ggml.go:136 msg="key not found" key=granite.attention.key_length default=64 Mar 05 08:49:24 neo-bandito ollama[8024]: time=2025-03-05T08:49:24.693+01:00 level=WARN source=ggml.go:136 msg="key not found" key=granite.attention.value_length default=64 Mar 05 08:49:24 neo-bandito ollama[8024]: time=2025-03-05T08:49:24.693+01:00 level=INFO source=sched.go:715 msg="new model will fit in available VRAM in single GPU, loading" model=/media/GLIMSPANKY/ollama/models/blobs/sha256-1aefcd9a8a15091b670951963b5f8a7e6653bb1350345e9621e179685ac9bc5f gpu=GPU-a9be7ece-3ea9-9a38-3a55-5aad9943f497 parallel=2 available=24303894528 required="7.0 GiB" Mar 05 08:49:24 neo-bandito ollama[8024]: time=2025-03-05T08:49:24.937+01:00 level=INFO source=server.go:97 msg="system memory" total="62.6 GiB" free="57.8 GiB" free_swap="123.0 GiB" Mar 05 08:49:24 neo-bandito ollama[8024]: time=2025-03-05T08:49:24.938+01:00 level=WARN source=ggml.go:136 msg="key not found" key=granite.attention.key_length default=64 Mar 05 08:49:24 neo-bandito ollama[8024]: time=2025-03-05T08:49:24.938+01:00 level=WARN source=ggml.go:136 msg="key not found" key=granite.attention.value_length default=64 Mar 05 08:49:24 neo-bandito ollama[8024]: time=2025-03-05T08:49:24.938+01:00 level=INFO source=server.go:130 msg=offload library=cuda layers.requested=-1 layers.model=41 layers.offload=41 layers.split="" memory.available="[22.6 GiB]" memory.gpu_overhead="0 B" memory.required.full="7.0 GiB" memory.required.partial="7.0 GiB" memory.required.kv="2.5 GiB" memory.required.allocations="[7.0 GiB]" memory.weights.total="3.9 GiB" memory.weights.repeating="3.8 GiB" memory.weights.nonrepeating="78.8 MiB" memory.graph.full="1.7 GiB" memory.graph.partial="1.7 GiB" projector.weights="851.2 MiB" projector.graph="0 B" Mar 05 08:49:24 neo-bandito ollama[8024]: time=2025-03-05T08:49:24.939+01:00 level=INFO source=server.go:380 msg="starting llama server" cmd="/usr/local/bin/ollama runner --model /media/GLIMSPANKY/ollama/models/blobs/sha256-1aefcd9a8a15091b670951963b5f8a7e6653bb1350345e9621e179685ac9bc5f --ctx-size 32768 --batch-size 512 --n-gpu-layers 41 --mmproj /media/GLIMSPANKY/ollama/models/blobs/sha256-4d464be24899cf8dc1862945432e0cef4366c4181fa38b14754cc9279b727608 --threads 8 --parallel 2 --port 35673" Mar 05 08:49:24 neo-bandito ollama[8024]: time=2025-03-05T08:49:24.939+01:00 level=INFO source=sched.go:450 msg="loaded runners" count=1 Mar 05 08:49:24 neo-bandito ollama[8024]: time=2025-03-05T08:49:24.939+01:00 level=INFO source=server.go:557 msg="waiting for llama runner to start responding" Mar 05 08:49:24 neo-bandito ollama[8024]: time=2025-03-05T08:49:24.939+01:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server error" Mar 05 08:49:24 neo-bandito ollama[8024]: time=2025-03-05T08:49:24.952+01:00 level=INFO source=runner.go:931 msg="starting go runner" Mar 05 08:49:24 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:24 | 200 | 45.92µs | 192.168.0.100 | GET "/api/ps" Mar 05 08:49:25 neo-bandito ollama[8024]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no Mar 05 08:49:25 neo-bandito ollama[8024]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no Mar 05 08:49:25 neo-bandito ollama[8024]: ggml_cuda_init: found 1 CUDA devices: Mar 05 08:49:25 neo-bandito ollama[8024]: Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Mar 05 08:49:25 neo-bandito ollama[8024]: load_backend: loaded CUDA backend from /usr/local/lib/ollama/cuda_v12/libggml-cuda.so Mar 05 08:49:25 neo-bandito ollama[8024]: load_backend: loaded CPU backend from /usr/local/lib/ollama/libggml-cpu-haswell.so Mar 05 08:49:25 neo-bandito ollama[8024]: time=2025-03-05T08:49:25.235+01:00 level=INFO source=runner.go:934 msg=system info="CPU : LLAMAFILE = 1 | CUDA : ARCHS = 500,600,610,700,750,800,860,870,890,900,1200 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | LLAMAFILE = 1 | cgo(gcc)" threads=8 Mar 05 08:49:25 neo-bandito ollama[8024]: time=2025-03-05T08:49:25.236+01:00 level=INFO source=runner.go:992 msg="Server listening on 127.0.0.1:35673" Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 3090) - 23178 MiB free Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: loaded meta data with 32 key-value pairs and 362 tensors from /media/GLIMSPANKY/ollama/models/blobs/sha256-1aefcd9a8a15091b670951963b5f8a7e6653bb1350345e9621e179685ac9bc5f (version GGUF V3 (latest)) Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv 0: general.architecture str = granite Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv 1: general.type str = model Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv 2: general.name str = Granite_Vision_Llm Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv 3: general.size_label str = 2.5B Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv 4: granite.block_count u32 = 40 Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv 5: granite.context_length u32 = 16384 Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv 6: granite.embedding_length u32 = 2048 Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv 7: granite.feed_forward_length u32 = 8192 Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv 8: granite.attention.head_count u32 = 32 Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv 9: granite.attention.head_count_kv u32 = 8 Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv 10: granite.rope.freq_base f32 = 300000.000000 Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv 11: granite.attention.layer_norm_rms_epsilon f32 = 0.000010 Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv 12: granite.vocab_size u32 = 49156 Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv 13: granite.rope.dimension_count u32 = 64 Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv 14: granite.attention.scale f32 = 0.015625 Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv 15: granite.embedding_scale f32 = 12.000000 Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv 16: granite.residual_scale f32 = 0.220000 Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv 17: granite.logit_scale f32 = 8.000000 Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv 18: tokenizer.ggml.model str = gpt2 Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv 19: tokenizer.ggml.pre str = refact Mar 05 08:49:25 neo-bandito ollama[8024]: time=2025-03-05T08:49:25.447+01:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server loading model" Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv 20: tokenizer.ggml.tokens arr[str,49156] = ["<|end_of_text|>", "<fim_prefix>", "... Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv 21: tokenizer.ggml.token_type arr[i32,49156] = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ... Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv 22: tokenizer.ggml.merges arr[str,48891] = ["Ġ Ġ", "ĠĠ ĠĠ", "ĠĠĠĠ ĠĠ... Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv 23: tokenizer.ggml.bos_token_id u32 = 0 Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv 24: tokenizer.ggml.eos_token_id u32 = 0 Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv 25: tokenizer.ggml.unknown_token_id u32 = 0 Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv 26: tokenizer.ggml.padding_token_id u32 = 0 Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv 27: tokenizer.ggml.add_bos_token bool = false Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv 28: tokenizer.chat_template str = {%- if tools %}\n {{- '<|start_of_r... Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv 29: tokenizer.ggml.add_space_prefix bool = false Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv 30: general.quantization_version u32 = 2 Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - kv 31: general.file_type u32 = 15 Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - type f32: 81 tensors Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - type q4_K: 240 tensors Mar 05 08:49:25 neo-bandito ollama[8024]: llama_model_loader: - type q6_K: 41 tensors Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: file format = GGUF V3 (latest) Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: file type = Q4_K - Medium Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: file size = 1.44 GiB (4.87 BPW) Mar 05 08:49:25 neo-bandito ollama[8024]: load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect Mar 05 08:49:25 neo-bandito ollama[8024]: load: special tokens cache size = 23 Mar 05 08:49:25 neo-bandito ollama[8024]: load: token to piece cache size = 0.2826 MB Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: arch = granite Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: vocab_only = 0 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_ctx_train = 16384 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_embd = 2048 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_layer = 40 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_head = 32 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_head_kv = 8 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_rot = 64 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_swa = 0 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_embd_head_k = 64 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_embd_head_v = 64 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_gqa = 4 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_embd_k_gqa = 512 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_embd_v_gqa = 512 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: f_norm_eps = 0.0e+00 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: f_norm_rms_eps = 1.0e-05 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: f_clamp_kqv = 0.0e+00 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: f_max_alibi_bias = 0.0e+00 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: f_logit_scale = 8.0e+00 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_ff = 8192 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_expert = 0 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_expert_used = 0 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: causal attn = 1 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: pooling type = 0 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: rope type = 0 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: rope scaling = linear Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: freq_base_train = 300000.0 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: freq_scale_train = 1 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_ctx_orig_yarn = 16384 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: rope_finetuned = unknown Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: ssm_d_conv = 0 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: ssm_d_inner = 0 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: ssm_d_state = 0 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: ssm_dt_rank = 0 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: ssm_dt_b_c_rms = 0 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: model type = 3B Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: model params = 2.53 B Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: general.name = Granite_Vision_Llm Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: f_embedding_scale = 12.000000 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: f_residual_scale = 0.220000 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: f_attention_scale = 0.015625 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: vocab type = BPE Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_vocab = 49156 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: n_merges = 48891 Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: BOS token = 0 '<|end_of_text|>' Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: EOS token = 0 '<|end_of_text|>' Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: UNK token = 0 '<|end_of_text|>' Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: PAD token = 0 '<|end_of_text|>' Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: LF token = 203 'Ċ' Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: EOG token = 0 '<|end_of_text|>' Mar 05 08:49:25 neo-bandito ollama[8024]: print_info: max token length = 512 Mar 05 08:49:25 neo-bandito ollama[8024]: load_tensors: loading model tensors, this can take a while... (mmap = true) Mar 05 08:49:25 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:25 | 200 | 43.164µs | 192.168.0.100 | GET "/api/ps" Mar 05 08:49:26 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:26 | 200 | 56.211µs | 192.168.0.100 | GET "/api/ps" Mar 05 08:49:27 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:27 | 200 | 92.251µs | 192.168.0.100 | GET "/api/ps" Mar 05 08:49:28 neo-bandito ollama[8024]: load_tensors: offloading 40 repeating layers to GPU Mar 05 08:49:28 neo-bandito ollama[8024]: load_tensors: offloading output layer to GPU Mar 05 08:49:28 neo-bandito ollama[8024]: load_tensors: offloaded 41/41 layers to GPU Mar 05 08:49:28 neo-bandito ollama[8024]: load_tensors: CUDA0 model buffer size = 1472.05 MiB Mar 05 08:49:28 neo-bandito ollama[8024]: load_tensors: CPU_Mapped model buffer size = 78.76 MiB Mar 05 08:49:28 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:28 | 200 | 21.768µs | 192.168.0.100 | GET "/api/ps" Mar 05 08:49:29 neo-bandito ollama[8024]: llama_init_from_model: n_seq_max = 2 Mar 05 08:49:29 neo-bandito ollama[8024]: llama_init_from_model: n_ctx = 32768 Mar 05 08:49:29 neo-bandito ollama[8024]: llama_init_from_model: n_ctx_per_seq = 16384 Mar 05 08:49:29 neo-bandito ollama[8024]: llama_init_from_model: n_batch = 1024 Mar 05 08:49:29 neo-bandito ollama[8024]: llama_init_from_model: n_ubatch = 512 Mar 05 08:49:29 neo-bandito ollama[8024]: llama_init_from_model: flash_attn = 0 Mar 05 08:49:29 neo-bandito ollama[8024]: llama_init_from_model: freq_base = 300000.0 Mar 05 08:49:29 neo-bandito ollama[8024]: llama_init_from_model: freq_scale = 1 Mar 05 08:49:29 neo-bandito ollama[8024]: llama_kv_cache_init: kv_size = 32768, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 40, can_shift = 1 Mar 05 08:49:29 neo-bandito ollama[8024]: llama_kv_cache_init: CUDA0 KV buffer size = 2560.00 MiB Mar 05 08:49:29 neo-bandito ollama[8024]: llama_init_from_model: KV self size = 2560.00 MiB, K (f16): 1280.00 MiB, V (f16): 1280.00 MiB Mar 05 08:49:29 neo-bandito ollama[8024]: llama_init_from_model: CUDA_Host output buffer size = 0.39 MiB Mar 05 08:49:29 neo-bandito ollama[8024]: llama_init_from_model: CUDA0 compute buffer size = 2128.00 MiB Mar 05 08:49:29 neo-bandito ollama[8024]: llama_init_from_model: CUDA_Host compute buffer size = 68.01 MiB Mar 05 08:49:29 neo-bandito ollama[8024]: llama_init_from_model: graph nodes = 1368 Mar 05 08:49:29 neo-bandito ollama[8024]: llama_init_from_model: graph splits = 2 Mar 05 08:49:29 neo-bandito ollama[8024]: key clip.use_silu not found in file Mar 05 08:49:29 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:29 | 200 | 111.595µs | 192.168.0.100 | GET "/api/ps" Mar 05 08:49:30 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:30 | 200 | 59.625µs | 192.168.0.100 | GET "/api/ps" Mar 05 08:49:31 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:31 | 200 | 24.563µs | 192.168.0.100 | GET "/api/ps" Mar 05 08:49:32 neo-bandito ollama[8024]: key clip.vision.image_crop_resolution not found in file Mar 05 08:49:32 neo-bandito ollama[8024]: time=2025-03-05T08:49:32.224+01:00 level=INFO source=server.go:596 msg="llama runner started in 7.29 seconds" Mar 05 08:49:32 neo-bandito ollama[8024]: [GIN] 2025/03/05 - 08:49:32 | 200 | 7.81541621s | 127.0.0.1 | POST "/api/generate" ``` (EDIT: filled the systems specs) ### OS Linux ### GPU RTX3090 ### CPU Intel ### Ollama version 0.5.13
GiteaMirror added the bug label 2026-04-12 17:35:20 -05:00
GiteaMirror commented 2026-04-12 17:35:21 -05:00
Author
Owner
Copy Link

@RadEdje commented on GitHub (Mar 5, 2025):

this seems to be happening for all my LLMs.... ollama logs that the GPU is detected (it names my rtx 4070 super). using Ollama ps.... it says 15% on cpu and 85% on gpu.... but if I look at my system tray.... my GPU vram usage dosen't budge but my CPU RAM gets full. all my LLM's also run slow... i think this could be an nvidia driver update issue?

based on this thread:
https://github.com/ollama/ollama/issues/4563

this has happened before.
I remember updating nvidia drivers recently.

Could it be the nvidia driver update?

<!-- gh-comment-id:2700421838 --> @RadEdje commented on GitHub (Mar 5, 2025): this seems to be happening for all my LLMs.... ollama logs that the GPU is detected (it names my rtx 4070 super). using Ollama ps.... it says 15% on cpu and 85% on gpu.... but if I look at my system tray.... my GPU vram usage dosen't budge but my CPU RAM gets full. all my LLM's also run slow... i think this could be an nvidia driver update issue? based on this thread: https://github.com/ollama/ollama/issues/4563 this has happened before. I remember updating nvidia drivers recently. Could it be the nvidia driver update?
GiteaMirror commented 2026-04-12 17:35:22 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 5, 2025):

@RadEdje Likely a different problem, create a new issue and add logs.

<!-- gh-comment-id:2700468921 --> @rick-github commented on GitHub (Mar 5, 2025): @RadEdje Likely a different problem, create a new issue and add [logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues).
GiteaMirror commented 2026-04-12 17:35:23 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 5, 2025):

I'm unable to replicate this. ollama 0.5.13 and granite3.2-vision:2b-q4_K_M always only uses one CPU thread, irrespective of whether OLLAMA_FLASH_ATTENTION is true or false.

Mar 05 08:49:29 neo-bandito ollama[8024]: llama_init_from_model: flash_attn    = 0

The log is from OLLAMA_FLASH_ATTENTION:false, do you have a log that shows model loading when OLLAMA_FLASH_ATTENTION:true?

<!-- gh-comment-id:2700512152 --> @rick-github commented on GitHub (Mar 5, 2025): I'm unable to replicate this. ollama 0.5.13 and granite3.2-vision:2b-q4_K_M always only uses one CPU thread, irrespective of whether `OLLAMA_FLASH_ATTENTION` is true or false. ``` Mar 05 08:49:29 neo-bandito ollama[8024]: llama_init_from_model: flash_attn = 0 ``` The log is from `OLLAMA_FLASH_ATTENTION:false`, do you have a log that shows model loading when `OLLAMA_FLASH_ATTENTION:true`?
GiteaMirror commented 2026-04-12 17:35:24 -05:00
Author
Owner
Copy Link

@sammyf commented on GitHub (Mar 5, 2025):

oops ...sorry ...

Here are the env variables for the systemd unit

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"
Environment="OLLAMA_KEEP_ALIVE=-1"
Environment="OLLAMA_MAX_LOADED_MODELS=4"
Environment="OLLAMA_MAX_QUEUE=10"
Environment="OLLAMA_NUM_PARALLEL=2"
Environment="OLLAMA_MODELS=/media/GLIMSPANKY/ollama/models/"

here is the log ( I didn't edit it, but a grep shows

llama_init_from_model: flash_attn = 1

granite.txt

<!-- gh-comment-id:2700556308 --> @sammyf commented on GitHub (Mar 5, 2025): oops ...sorry ... Here are the env variables for the systemd unit [Service] Environment="OLLAMA_HOST=0.0.0.0:11434" Environment="OLLAMA_FLASH_ATTENTION=1" Environment="OLLAMA_KV_CACHE_TYPE=q8_0" Environment="OLLAMA_KEEP_ALIVE=-1" Environment="OLLAMA_MAX_LOADED_MODELS=4" Environment="OLLAMA_MAX_QUEUE=10" Environment="OLLAMA_NUM_PARALLEL=2" Environment="OLLAMA_MODELS=/media/GLIMSPANKY/ollama/models/" here is the log ( I didn't edit it, but a grep shows ` llama_init_from_model: flash_attn = 1` [granite.txt](https://github.com/user-attachments/files/19087456/granite.txt)
GiteaMirror commented 2026-04-12 17:35:24 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 5, 2025):

Thanks, setting OLLAMA_KV_CACHE_TYPE to q4_0 or q8_0 does cause the CPU to be used when processing an image. Non-image or cache type f16 works fine. minicpm-v and llava, which also use clip projectors, don't use the CPU.

@alex-jw-brooks @gabe-l-hart

<!-- gh-comment-id:2700672208 --> @rick-github commented on GitHub (Mar 5, 2025): Thanks, setting `OLLAMA_KV_CACHE_TYPE` to `q4_0` or `q8_0` does cause the CPU to be used when processing an image. Non-image or cache type `f16` works fine. minicpm-v and llava, which also use clip projectors, don't use the CPU. @alex-jw-brooks @gabe-l-hart
GiteaMirror commented 2026-04-12 17:35:25 -05:00
Author
Owner
Copy Link

@RadEdje commented on GitHub (Mar 5, 2025):

@RadEdje Likely a different problem, create a new issue and add logs.

ok will do thank you.

<!-- gh-comment-id:2700738876 --> @RadEdje commented on GitHub (Mar 5, 2025): > [@RadEdje](https://github.com/RadEdje) Likely a different problem, create a new issue and add [logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues). ok will do thank you.
GiteaMirror commented 2026-04-12 17:35:26 -05:00
Author
Owner
Copy Link

@sammyf commented on GitHub (Mar 5, 2025):

I can confirm that setting kv_cache f16 'solved' the problem.

<!-- gh-comment-id:2700782849 --> @sammyf commented on GitHub (Mar 5, 2025): I can confirm that setting kv_cache f16 'solved' the problem.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#6202
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?