github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 05:05:37 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #10409] Error looking up nvidia GPU memory & error cuda driver library failed to get device context 801 & CUDA error: all CUDA-capable devices are busy or unavailable #53353

New Issue
Closed
opened 2026-04-29 02:42:07 -05:00 by GiteaMirror · 4 comments
GiteaMirror commented 2026-04-29 02:42:07 -05:00
Owner
Copy Link

Originally created by @Sven1403 on GitHub (Apr 25, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/10409

What is the issue?

Hello,

i want to use ollama with a Nvidia A16 Grid vGPU over ESXI VMware. When i dont declare the cuda lib path from ollama appdata location in the environment variables, the models (i tried deepseek-r1 1.5b to 8b) use 100% CPU. In the server log i already get this:

time=2025-04-25T11:23:03.078+02:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
time=2025-04-25T11:23:03.078+02:00 level=INFO source=gpu_windows.go:167 msg=packages count=1
time=2025-04-25T11:23:03.078+02:00 level=INFO source=gpu_windows.go:214 msg="" package=0 cores=4 efficiency=0 threads=4
time=2025-04-25T11:23:03.125+02:00 level=INFO source=gpu.go:272 msg="error looking up nvidia GPU memory" error="cuda driver library failed to get device context 801"
time=2025-04-25T11:23:03.126+02:00 level=INFO source=gpu.go:377 msg="no compatible GPUs were discovered"

Then i configured the environment variables with that

Image

Image

Image
(I also tried the cuda_v12 files)

That is resulting into this error:

Image

I also get more log details which i post in the log output area. You can find the "error looking up nvidia GPU memory here too.

I searched alot topics here and in other forums. I never found that exact problem. Anyone here can help? :)

Edit: Will check next week if the vGPU is configured with a A-Profile. Looks like this profile cant work:

Image

Btw the nvidia-smi output:

Image

Relevant log output

2025/04/25 12:46:31 routes.go:1232: INFO server config env="map[CUDA_VISIBLE_DEVICES:GPU-0d1397db-6149-11b2-b139-80f1d9baaab7 GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:2048 OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:C:\\Users\\User\\.ollama\\models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:true OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES:]"
time=2025-04-25T12:46:31.406+02:00 level=INFO source=images.go:458 msg="total blobs: 13"
time=2025-04-25T12:46:31.407+02:00 level=INFO source=images.go:465 msg="total unused blobs removed: 0"
time=2025-04-25T12:46:31.408+02:00 level=INFO source=routes.go:1299 msg="Listening on 127.0.0.1:11434 (version 0.6.6)"
time=2025-04-25T12:46:31.408+02:00 level=DEBUG source=sched.go:107 msg="starting llm scheduler"
time=2025-04-25T12:46:31.408+02:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
time=2025-04-25T12:46:31.408+02:00 level=INFO source=gpu_windows.go:167 msg=packages count=1
time=2025-04-25T12:46:31.408+02:00 level=INFO source=gpu_windows.go:214 msg="" package=0 cores=4 efficiency=0 threads=4
time=2025-04-25T12:46:31.408+02:00 level=DEBUG source=gpu.go:98 msg="searching for GPU discovery libraries for NVIDIA"
time=2025-04-25T12:46:31.408+02:00 level=DEBUG source=gpu.go:501 msg="Searching for GPU library" name=nvml.dll
time=2025-04-25T12:46:31.408+02:00 level=DEBUG source=gpu.go:525 msg="gpu library search" globs="[C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\nvml.dll C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cuda_v11\\nvml.dll C:\\Windows\\system32\\nvml.dll C:\\Windows\\nvml.dll C:\\Windows\\System32\\Wbem\\nvml.dll C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\nvml.dll C:\\Windows\\System32\\OpenSSH\\nvml.dll C:\\Program Files\\NVIDIA Corporation\\NVIDIA App\\NvDLISR\\nvml.dll C:\\Program Files\\NVIDIA Corporation\\Nsight Compute 2024.1.0\\nvml.dll C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common\\nvml.dll C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\nvml.dll C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cuda_v11\\nvml.dll C:\\Users\\User\\AppData\\Local\\Microsoft\\WindowsApps\\nvml.dll C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\nvml.dll C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\nvml.dll c:\\Windows\\System32\\nvml.dll]"
time=2025-04-25T12:46:31.408+02:00 level=DEBUG source=gpu.go:529 msg="skipping PhysX cuda library path" path="C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common\\nvml.dll"
time=2025-04-25T12:46:31.408+02:00 level=DEBUG source=gpu.go:558 msg="discovered GPU libraries" paths="[C:\\Windows\\system32\\nvml.dll c:\\Windows\\System32\\nvml.dll]"
time=2025-04-25T12:46:31.428+02:00 level=DEBUG source=gpu.go:111 msg="nvidia-ml loaded" library=C:\Windows\system32\nvml.dll
time=2025-04-25T12:46:31.428+02:00 level=DEBUG source=gpu.go:501 msg="Searching for GPU library" name=nvcuda.dll
time=2025-04-25T12:46:31.428+02:00 level=DEBUG source=gpu.go:525 msg="gpu library search" globs="[C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\nvcuda.dll C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cuda_v11\\nvcuda.dll C:\\Windows\\system32\\nvcuda.dll C:\\Windows\\nvcuda.dll C:\\Windows\\System32\\Wbem\\nvcuda.dll C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\nvcuda.dll C:\\Windows\\System32\\OpenSSH\\nvcuda.dll C:\\Program Files\\NVIDIA Corporation\\NVIDIA App\\NvDLISR\\nvcuda.dll C:\\Program Files\\NVIDIA Corporation\\Nsight Compute 2024.1.0\\nvcuda.dll C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common\\nvcuda.dll C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\nvcuda.dll C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cuda_v11\\nvcuda.dll C:\\Users\\User\\AppData\\Local\\Microsoft\\WindowsApps\\nvcuda.dll C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\nvcuda.dll C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\nvcuda.dll c:\\windows\\system*\\nvcuda.dll]"
time=2025-04-25T12:46:31.428+02:00 level=DEBUG source=gpu.go:529 msg="skipping PhysX cuda library path" path="C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common\\nvcuda.dll"
time=2025-04-25T12:46:31.429+02:00 level=DEBUG source=gpu.go:558 msg="discovered GPU libraries" paths=[C:\Windows\system32\nvcuda.dll]
initializing C:\Windows\system32\nvcuda.dll
dlsym: cuInit - 00007FFE815BFDE0
dlsym: cuDriverGetVersion - 00007FFE815BFE80
dlsym: cuDeviceGetCount - 00007FFE815C0676
dlsym: cuDeviceGet - 00007FFE815C0670
dlsym: cuDeviceGetAttribute - 00007FFE815BFFD0
dlsym: cuDeviceGetUuid - 00007FFE815C0682
dlsym: cuDeviceGetName - 00007FFE815C067C
dlsym: cuCtxCreate_v3 - 00007FFE815C06F4
dlsym: cuMemGetInfo_v2 - 00007FFE815C07F0
dlsym: cuCtxDestroy - 00007FFE815C0700
calling cuInit
calling cuDriverGetVersion
raw version 0x2f08
CUDA driver version: 12.4
calling cuDeviceGetCount
device count 1
time=2025-04-25T12:46:31.471+02:00 level=DEBUG source=gpu.go:125 msg="detected GPUs" count=1 library=C:\Windows\system32\nvcuda.dll
time=2025-04-25T12:46:31.471+02:00 level=INFO source=gpu.go:272 msg="error looking up nvidia GPU memory" error="cuda driver library failed to get device context 801"
time=2025-04-25T12:46:31.471+02:00 level=DEBUG source=amd_windows.go:34 msg="unable to load amdhip64_6.dll, please make sure to upgrade to the latest amd driver: Das angegebene Modul wurde nicht gefunden."
time=2025-04-25T12:46:31.471+02:00 level=INFO source=gpu.go:377 msg="no compatible GPUs were discovered"
releasing cuda driver library
releasing nvml library
time=2025-04-25T12:46:31.473+02:00 level=INFO source=types.go:130 msg="inference compute" id=0 library=cpu variant="" compute="" driver=0.0 name="" total="64.0 GiB" available="60.0 GiB"
[GIN] 2025/04/25 - 12:52:46 | 200 |            0s |       127.0.0.1 | GET      "/api/version"
[GIN] 2025/04/25 - 12:52:59 | 200 |      1.6369ms |       127.0.0.1 | HEAD     "/"
time=2025-04-25T12:53:00.041+02:00 level=WARN source=ggml.go:152 msg="key not found" key=general.alignment default=32
time=2025-04-25T12:53:00.065+02:00 level=WARN source=ggml.go:152 msg="key not found" key=general.alignment default=32
[GIN] 2025/04/25 - 12:53:00 | 200 |    116.3694ms |       127.0.0.1 | POST     "/api/show"
time=2025-04-25T12:53:00.116+02:00 level=WARN source=ggml.go:152 msg="key not found" key=general.alignment default=32
time=2025-04-25T12:53:00.118+02:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="64.0 GiB" before.free="60.0 GiB" before.free_swap="68.9 GiB" now.total="64.0 GiB" now.free="59.6 GiB" now.free_swap="68.3 GiB"
time=2025-04-25T12:53:00.120+02:00 level=DEBUG source=sched.go:183 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=3 gpu_count=1
time=2025-04-25T12:53:00.134+02:00 level=WARN source=ggml.go:152 msg="key not found" key=general.alignment default=32
time=2025-04-25T12:53:00.149+02:00 level=WARN source=ggml.go:152 msg="key not found" key=general.alignment default=32
time=2025-04-25T12:53:00.150+02:00 level=DEBUG source=sched.go:213 msg="cpu mode with first model, loading"
time=2025-04-25T12:53:00.150+02:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="64.0 GiB" before.free="59.6 GiB" before.free_swap="68.3 GiB" now.total="64.0 GiB" now.free="59.6 GiB" now.free_swap="68.3 GiB"
time=2025-04-25T12:53:00.155+02:00 level=INFO source=server.go:105 msg="system memory" total="64.0 GiB" free="59.6 GiB" free_swap="68.3 GiB"
time=2025-04-25T12:53:00.156+02:00 level=DEBUG source=memory.go:108 msg=evaluating library=cpu gpu_count=1 available="[59.6 GiB]"
time=2025-04-25T12:53:00.156+02:00 level=WARN source=ggml.go:152 msg="key not found" key=qwen2.vision.block_count default=0
time=2025-04-25T12:53:00.157+02:00 level=WARN source=ggml.go:152 msg="key not found" key=qwen2.attention.key_length default=128
time=2025-04-25T12:53:00.157+02:00 level=WARN source=ggml.go:152 msg="key not found" key=qwen2.attention.value_length default=128
time=2025-04-25T12:53:00.158+02:00 level=INFO source=server.go:138 msg=offload library=cpu layers.requested=-1 layers.model=29 layers.offload=0 layers.split="" memory.available="[59.6 GiB]" memory.gpu_overhead="0 B" memory.required.full="5.1 GiB" memory.required.partial="0 B" memory.required.kv="448.0 MiB" memory.required.allocations="[5.1 GiB]" memory.weights.total="4.1 GiB" memory.weights.repeating="3.7 GiB" memory.weights.nonrepeating="426.4 MiB" memory.graph.full="478.0 MiB" memory.graph.partial="730.4 MiB"
time=2025-04-25T12:53:00.162+02:00 level=DEBUG source=server.go:262 msg="compatible gpu libraries" compatible=[]
time=2025-04-25T12:53:00.208+02:00 level=WARN source=ggml.go:152 msg="key not found" key=general.alignment default=32
time=2025-04-25T12:53:00.209+02:00 level=DEBUG source=server.go:278 msg="model not yet supported by Ollama engine, switching to compatibility mode" model=C:\Users\User\.ollama\models\blobs\sha256-96c415656d377afbff962f6cdb2394ab092ccbcbaab4b82525bc4ca800fe8a49 error="unsupported model architecture \"qwen2\""
llama_model_loader: loaded meta data with 26 key-value pairs and 339 tensors from C:\Users\User\.ollama\models\blobs\sha256-96c415656d377afbff962f6cdb2394ab092ccbcbaab4b82525bc4ca800fe8a49 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen2
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = DeepSeek R1 Distill Qwen 7B
llama_model_loader: - kv   3:                           general.basename str              = DeepSeek-R1-Distill-Qwen
llama_model_loader: - kv   4:                         general.size_label str              = 7B
llama_model_loader: - kv   5:                          qwen2.block_count u32              = 28
llama_model_loader: - kv   6:                       qwen2.context_length u32              = 131072
llama_model_loader: - kv   7:                     qwen2.embedding_length u32              = 3584
llama_model_loader: - kv   8:                  qwen2.feed_forward_length u32              = 18944
llama_model_loader: - kv   9:                 qwen2.attention.head_count u32              = 28
llama_model_loader: - kv  10:              qwen2.attention.head_count_kv u32              = 4
llama_model_loader: - kv  11:                       qwen2.rope.freq_base f32              = 10000.000000
llama_model_loader: - kv  12:     qwen2.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  13:                          general.file_type u32              = 15
llama_model_loader: - kv  14:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  15:                         tokenizer.ggml.pre str              = qwen2
llama_model_loader: - kv  16:                      tokenizer.ggml.tokens arr[str,152064]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  17:                  tokenizer.ggml.token_type arr[i32,152064]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  18:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  19:                tokenizer.ggml.bos_token_id u32              = 151646
llama_model_loader: - kv  20:                tokenizer.ggml.eos_token_id u32              = 151643
llama_model_loader: - kv  21:            tokenizer.ggml.padding_token_id u32              = 151643
llama_model_loader: - kv  22:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  23:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  24:                    tokenizer.chat_template str              = {% if not add_generation_prompt is de...
llama_model_loader: - kv  25:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  141 tensors
llama_model_loader: - type q4_K:  169 tensors
llama_model_loader: - type q6_K:   29 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_K - Medium
print_info: file size   = 4.36 GiB (4.91 BPW) 
init_tokenizer: initializing tokenizer for type 2
load: control token: 151659 '<|fim_prefix|>' is not marked as EOG
load: control token: 151656 '<|video_pad|>' is not marked as EOG
load: control token: 151655 '<|image_pad|>' is not marked as EOG
load: control token: 151653 '<|vision_end|>' is not marked as EOG
load: control token: 151652 '<|vision_start|>' is not marked as EOG
load: control token: 151651 '<|quad_end|>' is not marked as EOG
load: control token: 151646 '<｜begin▁of▁sentence｜>' is not marked as EOG
load: control token: 151644 '<｜User｜>' is not marked as EOG
load: control token: 151661 '<|fim_suffix|>' is not marked as EOG
load: control token: 151660 '<|fim_middle|>' is not marked as EOG
load: control token: 151654 '<|vision_pad|>' is not marked as EOG
load: control token: 151650 '<|quad_start|>' is not marked as EOG
load: control token: 151647 '<|EOT|>' is not marked as EOG
load: control token: 151643 '<｜end▁of▁sentence｜>' is not marked as EOG
load: control token: 151645 '<｜Assistant｜>' is not marked as EOG
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: special tokens cache size = 22
load: token to piece cache size = 0.9310 MB
print_info: arch             = qwen2
print_info: vocab_only       = 1
print_info: model type       = ?B
print_info: model params     = 7.62 B
print_info: general.name     = DeepSeek R1 Distill Qwen 7B
print_info: vocab type       = BPE
print_info: n_vocab          = 152064
print_info: n_merges         = 151387
print_info: BOS token        = 151646 '<｜begin▁of▁sentence｜>'
print_info: EOS token        = 151643 '<｜end▁of▁sentence｜>'
print_info: EOT token        = 151643 '<｜end▁of▁sentence｜>'
print_info: PAD token        = 151643 '<｜end▁of▁sentence｜>'
print_info: LF token         = 198 'Ċ'
print_info: FIM PRE token    = 151659 '<|fim_prefix|>'
print_info: FIM SUF token    = 151661 '<|fim_suffix|>'
print_info: FIM MID token    = 151660 '<|fim_middle|>'
print_info: FIM PAD token    = 151662 '<|fim_pad|>'
print_info: FIM REP token    = 151663 '<|repo_name|>'
print_info: FIM SEP token    = 151664 '<|file_sep|>'
print_info: EOG token        = 151643 '<｜end▁of▁sentence｜>'
print_info: EOG token        = 151662 '<|fim_pad|>'
print_info: EOG token        = 151663 '<|repo_name|>'
print_info: EOG token        = 151664 '<|file_sep|>'
print_info: max token length = 256
llama_model_load: vocab only - skipping tensors
time=2025-04-25T12:53:00.453+02:00 level=DEBUG source=gpu.go:695 msg="no filter required for library cpu"
time=2025-04-25T12:53:00.453+02:00 level=INFO source=server.go:405 msg="starting llama server" cmd="C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --model C:\\Users\\User\\.ollama\\models\\blobs\\sha256-96c415656d377afbff962f6cdb2394ab092ccbcbaab4b82525bc4ca800fe8a49 --ctx-size 8192 --batch-size 512 --verbose --threads 4 --no-mmap --parallel 4 --port 49765"
time=2025-04-25T12:53:00.454+02:00 level=DEBUG source=server.go:423 msg=subprocess environment="[CUDA_ERROR_LEVEL=50 CUDA_PATH=C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.4 CUDA_PATH_V12_4=C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.4 CUDA_VISIBLE_DEVICES=GPU-0d1397db-6149-11b2-b139-80f1d9baaab7 PATH=C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cuda_v11;C:\\Windows\\system32;C:\\Windows;C:\\Windows\\System32\\Wbem;C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\;C:\\Windows\\System32\\OpenSSH\\;C:\\Program Files\\NVIDIA Corporation\\NVIDIA App\\NvDLISR;C:\\Program Files\\NVIDIA Corporation\\Nsight Compute 2024.1.0\\;C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common;C:\\Users\\User\\AppData\\Local\\Programs\\Ollama;C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cuda_v11;C:\\Users\\User\\AppData\\Local\\Microsoft\\WindowsApps;C:\\Users\\User\\AppData\\Local\\Programs\\Ollama;;C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\lib\\ollama]"
time=2025-04-25T12:53:00.461+02:00 level=INFO source=sched.go:451 msg="loaded runners" count=1
time=2025-04-25T12:53:00.461+02:00 level=INFO source=server.go:580 msg="waiting for llama runner to start responding"
time=2025-04-25T12:53:00.462+02:00 level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server error"
time=2025-04-25T12:53:00.482+02:00 level=INFO source=runner.go:853 msg="starting go runner"
time=2025-04-25T12:53:00.512+02:00 level=DEBUG source=ggml.go:99 msg="ggml backend load all from path" path=C:\Users\User\AppData\Local\Programs\Ollama\lib\ollama\cuda_v11
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA A16-16A, compute capability 8.6, VMM: no
load_backend: loaded CUDA backend from C:\Users\User\AppData\Local\Programs\Ollama\lib\ollama\cuda_v11\ggml-cuda.dll
time=2025-04-25T12:53:38.145+02:00 level=DEBUG source=ggml.go:93 msg="skipping path which is not part of ollama" path=C:\Windows\system32
time=2025-04-25T12:53:38.145+02:00 level=DEBUG source=ggml.go:93 msg="skipping path which is not part of ollama" path=C:\Windows
time=2025-04-25T12:53:38.145+02:00 level=DEBUG source=ggml.go:93 msg="skipping path which is not part of ollama" path=C:\Windows\System32\Wbem
time=2025-04-25T12:53:38.145+02:00 level=DEBUG source=ggml.go:93 msg="skipping path which is not part of ollama" path=C:\Windows\System32\WindowsPowerShell\v1.0
time=2025-04-25T12:53:38.145+02:00 level=DEBUG source=ggml.go:93 msg="skipping path which is not part of ollama" path=C:\Windows\System32\OpenSSH
time=2025-04-25T12:53:38.145+02:00 level=DEBUG source=ggml.go:93 msg="skipping path which is not part of ollama" path="C:\\Program Files\\NVIDIA Corporation\\NVIDIA App\\NvDLISR"
time=2025-04-25T12:53:38.145+02:00 level=DEBUG source=ggml.go:93 msg="skipping path which is not part of ollama" path="C:\\Program Files\\NVIDIA Corporation\\Nsight Compute 2024.1.0"
time=2025-04-25T12:53:38.145+02:00 level=DEBUG source=ggml.go:93 msg="skipping path which is not part of ollama" path="C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common"
time=2025-04-25T12:53:38.145+02:00 level=DEBUG source=ggml.go:99 msg="ggml backend load all from path" path=C:\Users\User\AppData\Local\Programs\Ollama
time=2025-04-25T12:53:38.146+02:00 level=DEBUG source=ggml.go:93 msg="skipping path which is not part of ollama" path=C:\Users\User\AppData\Local\Microsoft\WindowsApps
time=2025-04-25T12:53:38.146+02:00 level=DEBUG source=ggml.go:99 msg="ggml backend load all from path" path=C:\Users\User\AppData\Local\Programs\Ollama\lib\ollama
load_backend: loaded CPU backend from C:\Users\User\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-icelake.dll
time=2025-04-25T12:53:38.370+02:00 level=INFO source=ggml.go:109 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.AVX512=1 CPU.0.AVX512_VBMI=1 CPU.0.AVX512_VNNI=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
CUDA error: all CUDA-capable devices are busy or unavailable
  current device: 0, in function ggml_backend_cuda_device_get_memory at C:\a\ollama\ollama\ml\backend\ggml\ggml\src\ggml-cuda\ggml-cuda.cu:2959
  cudaMemGetInfo(free, total)
C:\a\ollama\ollama\ml\backend\ggml\ggml\src\ggml-cuda\ggml-cuda.cu:75: CUDA error
time=2025-04-25T12:53:38.536+02:00 level=ERROR source=server.go:449 msg="llama runner terminated" error="exit status 0xc0000409"
time=2025-04-25T12:53:38.579+02:00 level=ERROR source=sched.go:457 msg="error loading llama server" error="llama runner process has terminated: CUDA error"
time=2025-04-25T12:53:38.579+02:00 level=DEBUG source=sched.go:460 msg="triggering expiration for failed load" model=C:\Users\User\.ollama\models\blobs\sha256-96c415656d377afbff962f6cdb2394ab092ccbcbaab4b82525bc4ca800fe8a49
[GIN] 2025/04/25 - 12:53:38 | 500 |   38.4837386s |       127.0.0.1 | POST     "/api/generate"
time=2025-04-25T12:53:38.579+02:00 level=DEBUG source=sched.go:362 msg="runner expired event received" modelPath=C:\Users\User\.ollama\models\blobs\sha256-96c415656d377afbff962f6cdb2394ab092ccbcbaab4b82525bc4ca800fe8a49
time=2025-04-25T12:53:38.581+02:00 level=DEBUG source=sched.go:377 msg="got lock to unload" modelPath=C:\Users\User\.ollama\models\blobs\sha256-96c415656d377afbff962f6cdb2394ab092ccbcbaab4b82525bc4ca800fe8a49
time=2025-04-25T12:53:38.628+02:00 level=DEBUG source=server.go:1001 msg="stopping llama server"
time=2025-04-25T12:53:38.628+02:00 level=DEBUG source=sched.go:382 msg="runner released" modelPath=C:\Users\User\.ollama\models\blobs\sha256-96c415656d377afbff962f6cdb2394ab092ccbcbaab4b82525bc4ca800fe8a49
time=2025-04-25T12:53:38.628+02:00 level=DEBUG source=sched.go:386 msg="sending an unloaded event" modelPath=C:\Users\User\.ollama\models\blobs\sha256-96c415656d377afbff962f6cdb2394ab092ccbcbaab4b82525bc4ca800fe8a49
time=2025-04-25T12:53:38.628+02:00 level=DEBUG source=sched.go:310 msg="ignoring unload event with no pending requests"

OS

Windows 11

GPU

Nvidia A16

CPU

Intel

Ollama version

0.6.6

Originally created by @Sven1403 on GitHub (Apr 25, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/10409 ### What is the issue? Hello, i want to use ollama with a Nvidia A16 Grid vGPU over ESXI VMware. When i dont declare the cuda lib path from ollama appdata location in the environment variables, the models (i tried deepseek-r1 1.5b to 8b) use 100% CPU. In the server log i already get this: time=2025-04-25T11:23:03.078+02:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs" time=2025-04-25T11:23:03.078+02:00 level=INFO source=gpu_windows.go:167 msg=packages count=1 time=2025-04-25T11:23:03.078+02:00 level=INFO source=gpu_windows.go:214 msg="" package=0 cores=4 efficiency=0 threads=4 time=2025-04-25T11:23:03.125+02:00 level=INFO source=gpu.go:272 msg="error looking up nvidia GPU memory" error="cuda driver library failed to get device context 801" time=2025-04-25T11:23:03.126+02:00 level=INFO source=gpu.go:377 msg="no compatible GPUs were discovered" Then i configured the environment variables with that ![Image](https://github.com/user-attachments/assets/562d0853-cf00-4a22-bc6d-4e4ba4d6b06b) ![Image](https://github.com/user-attachments/assets/139143f5-06a9-4611-bbe5-03205fa6c85a) ![Image](https://github.com/user-attachments/assets/67892683-0b6c-4a0e-be3e-a6e01fcacd48) (I also tried the cuda_v12 files) That is resulting into this error: ![Image](https://github.com/user-attachments/assets/4a315921-1930-46c8-9b9c-c2f2f65f48c0) I also get more log details which i post in the log output area. You can find the "error looking up nvidia GPU memory here too. I searched alot topics here and in other forums. I never found that exact problem. Anyone here can help? :) Edit: Will check next week if the vGPU is configured with a A-Profile. Looks like this profile cant work: ![Image](https://github.com/user-attachments/assets/40d1bdf8-54e1-407c-b36c-04f4ba20c54e) Btw the nvidia-smi output: ![Image](https://github.com/user-attachments/assets/54887e86-b6c5-47ee-9b6b-d879ecfad52f) ### Relevant log output ```shell 2025/04/25 12:46:31 routes.go:1232: INFO server config env="map[CUDA_VISIBLE_DEVICES:GPU-0d1397db-6149-11b2-b139-80f1d9baaab7 GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:2048 OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:C:\\Users\\User\\.ollama\\models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:true OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES:]" time=2025-04-25T12:46:31.406+02:00 level=INFO source=images.go:458 msg="total blobs: 13" time=2025-04-25T12:46:31.407+02:00 level=INFO source=images.go:465 msg="total unused blobs removed: 0" time=2025-04-25T12:46:31.408+02:00 level=INFO source=routes.go:1299 msg="Listening on 127.0.0.1:11434 (version 0.6.6)" time=2025-04-25T12:46:31.408+02:00 level=DEBUG source=sched.go:107 msg="starting llm scheduler" time=2025-04-25T12:46:31.408+02:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs" time=2025-04-25T12:46:31.408+02:00 level=INFO source=gpu_windows.go:167 msg=packages count=1 time=2025-04-25T12:46:31.408+02:00 level=INFO source=gpu_windows.go:214 msg="" package=0 cores=4 efficiency=0 threads=4 time=2025-04-25T12:46:31.408+02:00 level=DEBUG source=gpu.go:98 msg="searching for GPU discovery libraries for NVIDIA" time=2025-04-25T12:46:31.408+02:00 level=DEBUG source=gpu.go:501 msg="Searching for GPU library" name=nvml.dll time=2025-04-25T12:46:31.408+02:00 level=DEBUG source=gpu.go:525 msg="gpu library search" globs="[C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\nvml.dll C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cuda_v11\\nvml.dll C:\\Windows\\system32\\nvml.dll C:\\Windows\\nvml.dll C:\\Windows\\System32\\Wbem\\nvml.dll C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\nvml.dll C:\\Windows\\System32\\OpenSSH\\nvml.dll C:\\Program Files\\NVIDIA Corporation\\NVIDIA App\\NvDLISR\\nvml.dll C:\\Program Files\\NVIDIA Corporation\\Nsight Compute 2024.1.0\\nvml.dll C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common\\nvml.dll C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\nvml.dll C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cuda_v11\\nvml.dll C:\\Users\\User\\AppData\\Local\\Microsoft\\WindowsApps\\nvml.dll C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\nvml.dll C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\nvml.dll c:\\Windows\\System32\\nvml.dll]" time=2025-04-25T12:46:31.408+02:00 level=DEBUG source=gpu.go:529 msg="skipping PhysX cuda library path" path="C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common\\nvml.dll" time=2025-04-25T12:46:31.408+02:00 level=DEBUG source=gpu.go:558 msg="discovered GPU libraries" paths="[C:\\Windows\\system32\\nvml.dll c:\\Windows\\System32\\nvml.dll]" time=2025-04-25T12:46:31.428+02:00 level=DEBUG source=gpu.go:111 msg="nvidia-ml loaded" library=C:\Windows\system32\nvml.dll time=2025-04-25T12:46:31.428+02:00 level=DEBUG source=gpu.go:501 msg="Searching for GPU library" name=nvcuda.dll time=2025-04-25T12:46:31.428+02:00 level=DEBUG source=gpu.go:525 msg="gpu library search" globs="[C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\nvcuda.dll C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cuda_v11\\nvcuda.dll C:\\Windows\\system32\\nvcuda.dll C:\\Windows\\nvcuda.dll C:\\Windows\\System32\\Wbem\\nvcuda.dll C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\nvcuda.dll C:\\Windows\\System32\\OpenSSH\\nvcuda.dll C:\\Program Files\\NVIDIA Corporation\\NVIDIA App\\NvDLISR\\nvcuda.dll C:\\Program Files\\NVIDIA Corporation\\Nsight Compute 2024.1.0\\nvcuda.dll C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common\\nvcuda.dll C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\nvcuda.dll C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cuda_v11\\nvcuda.dll C:\\Users\\User\\AppData\\Local\\Microsoft\\WindowsApps\\nvcuda.dll C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\nvcuda.dll C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\nvcuda.dll c:\\windows\\system*\\nvcuda.dll]" time=2025-04-25T12:46:31.428+02:00 level=DEBUG source=gpu.go:529 msg="skipping PhysX cuda library path" path="C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common\\nvcuda.dll" time=2025-04-25T12:46:31.429+02:00 level=DEBUG source=gpu.go:558 msg="discovered GPU libraries" paths=[C:\Windows\system32\nvcuda.dll] initializing C:\Windows\system32\nvcuda.dll dlsym: cuInit - 00007FFE815BFDE0 dlsym: cuDriverGetVersion - 00007FFE815BFE80 dlsym: cuDeviceGetCount - 00007FFE815C0676 dlsym: cuDeviceGet - 00007FFE815C0670 dlsym: cuDeviceGetAttribute - 00007FFE815BFFD0 dlsym: cuDeviceGetUuid - 00007FFE815C0682 dlsym: cuDeviceGetName - 00007FFE815C067C dlsym: cuCtxCreate_v3 - 00007FFE815C06F4 dlsym: cuMemGetInfo_v2 - 00007FFE815C07F0 dlsym: cuCtxDestroy - 00007FFE815C0700 calling cuInit calling cuDriverGetVersion raw version 0x2f08 CUDA driver version: 12.4 calling cuDeviceGetCount device count 1 time=2025-04-25T12:46:31.471+02:00 level=DEBUG source=gpu.go:125 msg="detected GPUs" count=1 library=C:\Windows\system32\nvcuda.dll time=2025-04-25T12:46:31.471+02:00 level=INFO source=gpu.go:272 msg="error looking up nvidia GPU memory" error="cuda driver library failed to get device context 801" time=2025-04-25T12:46:31.471+02:00 level=DEBUG source=amd_windows.go:34 msg="unable to load amdhip64_6.dll, please make sure to upgrade to the latest amd driver: Das angegebene Modul wurde nicht gefunden." time=2025-04-25T12:46:31.471+02:00 level=INFO source=gpu.go:377 msg="no compatible GPUs were discovered" releasing cuda driver library releasing nvml library time=2025-04-25T12:46:31.473+02:00 level=INFO source=types.go:130 msg="inference compute" id=0 library=cpu variant="" compute="" driver=0.0 name="" total="64.0 GiB" available="60.0 GiB" [GIN] 2025/04/25 - 12:52:46 | 200 | 0s | 127.0.0.1 | GET "/api/version" [GIN] 2025/04/25 - 12:52:59 | 200 | 1.6369ms | 127.0.0.1 | HEAD "/" time=2025-04-25T12:53:00.041+02:00 level=WARN source=ggml.go:152 msg="key not found" key=general.alignment default=32 time=2025-04-25T12:53:00.065+02:00 level=WARN source=ggml.go:152 msg="key not found" key=general.alignment default=32 [GIN] 2025/04/25 - 12:53:00 | 200 | 116.3694ms | 127.0.0.1 | POST "/api/show" time=2025-04-25T12:53:00.116+02:00 level=WARN source=ggml.go:152 msg="key not found" key=general.alignment default=32 time=2025-04-25T12:53:00.118+02:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="64.0 GiB" before.free="60.0 GiB" before.free_swap="68.9 GiB" now.total="64.0 GiB" now.free="59.6 GiB" now.free_swap="68.3 GiB" time=2025-04-25T12:53:00.120+02:00 level=DEBUG source=sched.go:183 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=3 gpu_count=1 time=2025-04-25T12:53:00.134+02:00 level=WARN source=ggml.go:152 msg="key not found" key=general.alignment default=32 time=2025-04-25T12:53:00.149+02:00 level=WARN source=ggml.go:152 msg="key not found" key=general.alignment default=32 time=2025-04-25T12:53:00.150+02:00 level=DEBUG source=sched.go:213 msg="cpu mode with first model, loading" time=2025-04-25T12:53:00.150+02:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="64.0 GiB" before.free="59.6 GiB" before.free_swap="68.3 GiB" now.total="64.0 GiB" now.free="59.6 GiB" now.free_swap="68.3 GiB" time=2025-04-25T12:53:00.155+02:00 level=INFO source=server.go:105 msg="system memory" total="64.0 GiB" free="59.6 GiB" free_swap="68.3 GiB" time=2025-04-25T12:53:00.156+02:00 level=DEBUG source=memory.go:108 msg=evaluating library=cpu gpu_count=1 available="[59.6 GiB]" time=2025-04-25T12:53:00.156+02:00 level=WARN source=ggml.go:152 msg="key not found" key=qwen2.vision.block_count default=0 time=2025-04-25T12:53:00.157+02:00 level=WARN source=ggml.go:152 msg="key not found" key=qwen2.attention.key_length default=128 time=2025-04-25T12:53:00.157+02:00 level=WARN source=ggml.go:152 msg="key not found" key=qwen2.attention.value_length default=128 time=2025-04-25T12:53:00.158+02:00 level=INFO source=server.go:138 msg=offload library=cpu layers.requested=-1 layers.model=29 layers.offload=0 layers.split="" memory.available="[59.6 GiB]" memory.gpu_overhead="0 B" memory.required.full="5.1 GiB" memory.required.partial="0 B" memory.required.kv="448.0 MiB" memory.required.allocations="[5.1 GiB]" memory.weights.total="4.1 GiB" memory.weights.repeating="3.7 GiB" memory.weights.nonrepeating="426.4 MiB" memory.graph.full="478.0 MiB" memory.graph.partial="730.4 MiB" time=2025-04-25T12:53:00.162+02:00 level=DEBUG source=server.go:262 msg="compatible gpu libraries" compatible=[] time=2025-04-25T12:53:00.208+02:00 level=WARN source=ggml.go:152 msg="key not found" key=general.alignment default=32 time=2025-04-25T12:53:00.209+02:00 level=DEBUG source=server.go:278 msg="model not yet supported by Ollama engine, switching to compatibility mode" model=C:\Users\User\.ollama\models\blobs\sha256-96c415656d377afbff962f6cdb2394ab092ccbcbaab4b82525bc4ca800fe8a49 error="unsupported model architecture \"qwen2\"" llama_model_loader: loaded meta data with 26 key-value pairs and 339 tensors from C:\Users\User\.ollama\models\blobs\sha256-96c415656d377afbff962f6cdb2394ab092ccbcbaab4b82525bc4ca800fe8a49 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = qwen2 llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = DeepSeek R1 Distill Qwen 7B llama_model_loader: - kv 3: general.basename str = DeepSeek-R1-Distill-Qwen llama_model_loader: - kv 4: general.size_label str = 7B llama_model_loader: - kv 5: qwen2.block_count u32 = 28 llama_model_loader: - kv 6: qwen2.context_length u32 = 131072 llama_model_loader: - kv 7: qwen2.embedding_length u32 = 3584 llama_model_loader: - kv 8: qwen2.feed_forward_length u32 = 18944 llama_model_loader: - kv 9: qwen2.attention.head_count u32 = 28 llama_model_loader: - kv 10: qwen2.attention.head_count_kv u32 = 4 llama_model_loader: - kv 11: qwen2.rope.freq_base f32 = 10000.000000 llama_model_loader: - kv 12: qwen2.attention.layer_norm_rms_epsilon f32 = 0.000001 llama_model_loader: - kv 13: general.file_type u32 = 15 llama_model_loader: - kv 14: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 15: tokenizer.ggml.pre str = qwen2 llama_model_loader: - kv 16: tokenizer.ggml.tokens arr[str,152064] = ["!", "\"", "#", "$", "%", "&", "'", ... llama_model_loader: - kv 17: tokenizer.ggml.token_type arr[i32,152064] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 18: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",... llama_model_loader: - kv 19: tokenizer.ggml.bos_token_id u32 = 151646 llama_model_loader: - kv 20: tokenizer.ggml.eos_token_id u32 = 151643 llama_model_loader: - kv 21: tokenizer.ggml.padding_token_id u32 = 151643 llama_model_loader: - kv 22: tokenizer.ggml.add_bos_token bool = true llama_model_loader: - kv 23: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 24: tokenizer.chat_template str = {% if not add_generation_prompt is de... llama_model_loader: - kv 25: general.quantization_version u32 = 2 llama_model_loader: - type f32: 141 tensors llama_model_loader: - type q4_K: 169 tensors llama_model_loader: - type q6_K: 29 tensors print_info: file format = GGUF V3 (latest) print_info: file type = Q4_K - Medium print_info: file size = 4.36 GiB (4.91 BPW) init_tokenizer: initializing tokenizer for type 2 load: control token: 151659 '<|fim_prefix|>' is not marked as EOG load: control token: 151656 '<|video_pad|>' is not marked as EOG load: control token: 151655 '<|image_pad|>' is not marked as EOG load: control token: 151653 '<|vision_end|>' is not marked as EOG load: control token: 151652 '<|vision_start|>' is not marked as EOG load: control token: 151651 '<|quad_end|>' is not marked as EOG load: control token: 151646 '<｜begin▁of▁sentence｜>' is not marked as EOG load: control token: 151644 '<｜User｜>' is not marked as EOG load: control token: 151661 '<|fim_suffix|>' is not marked as EOG load: control token: 151660 '<|fim_middle|>' is not marked as EOG load: control token: 151654 '<|vision_pad|>' is not marked as EOG load: control token: 151650 '<|quad_start|>' is not marked as EOG load: control token: 151647 '<|EOT|>' is not marked as EOG load: control token: 151643 '<｜end▁of▁sentence｜>' is not marked as EOG load: control token: 151645 '<｜Assistant｜>' is not marked as EOG load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect load: special tokens cache size = 22 load: token to piece cache size = 0.9310 MB print_info: arch = qwen2 print_info: vocab_only = 1 print_info: model type = ?B print_info: model params = 7.62 B print_info: general.name = DeepSeek R1 Distill Qwen 7B print_info: vocab type = BPE print_info: n_vocab = 152064 print_info: n_merges = 151387 print_info: BOS token = 151646 '<｜begin▁of▁sentence｜>' print_info: EOS token = 151643 '<｜end▁of▁sentence｜>' print_info: EOT token = 151643 '<｜end▁of▁sentence｜>' print_info: PAD token = 151643 '<｜end▁of▁sentence｜>' print_info: LF token = 198 'Ċ' print_info: FIM PRE token = 151659 '<|fim_prefix|>' print_info: FIM SUF token = 151661 '<|fim_suffix|>' print_info: FIM MID token = 151660 '<|fim_middle|>' print_info: FIM PAD token = 151662 '<|fim_pad|>' print_info: FIM REP token = 151663 '<|repo_name|>' print_info: FIM SEP token = 151664 '<|file_sep|>' print_info: EOG token = 151643 '<｜end▁of▁sentence｜>' print_info: EOG token = 151662 '<|fim_pad|>' print_info: EOG token = 151663 '<|repo_name|>' print_info: EOG token = 151664 '<|file_sep|>' print_info: max token length = 256 llama_model_load: vocab only - skipping tensors time=2025-04-25T12:53:00.453+02:00 level=DEBUG source=gpu.go:695 msg="no filter required for library cpu" time=2025-04-25T12:53:00.453+02:00 level=INFO source=server.go:405 msg="starting llama server" cmd="C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --model C:\\Users\\User\\.ollama\\models\\blobs\\sha256-96c415656d377afbff962f6cdb2394ab092ccbcbaab4b82525bc4ca800fe8a49 --ctx-size 8192 --batch-size 512 --verbose --threads 4 --no-mmap --parallel 4 --port 49765" time=2025-04-25T12:53:00.454+02:00 level=DEBUG source=server.go:423 msg=subprocess environment="[CUDA_ERROR_LEVEL=50 CUDA_PATH=C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.4 CUDA_PATH_V12_4=C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.4 CUDA_VISIBLE_DEVICES=GPU-0d1397db-6149-11b2-b139-80f1d9baaab7 PATH=C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cuda_v11;C:\\Windows\\system32;C:\\Windows;C:\\Windows\\System32\\Wbem;C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\;C:\\Windows\\System32\\OpenSSH\\;C:\\Program Files\\NVIDIA Corporation\\NVIDIA App\\NvDLISR;C:\\Program Files\\NVIDIA Corporation\\Nsight Compute 2024.1.0\\;C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common;C:\\Users\\User\\AppData\\Local\\Programs\\Ollama;C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cuda_v11;C:\\Users\\User\\AppData\\Local\\Microsoft\\WindowsApps;C:\\Users\\User\\AppData\\Local\\Programs\\Ollama;;C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\lib\\ollama]" time=2025-04-25T12:53:00.461+02:00 level=INFO source=sched.go:451 msg="loaded runners" count=1 time=2025-04-25T12:53:00.461+02:00 level=INFO source=server.go:580 msg="waiting for llama runner to start responding" time=2025-04-25T12:53:00.462+02:00 level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server error" time=2025-04-25T12:53:00.482+02:00 level=INFO source=runner.go:853 msg="starting go runner" time=2025-04-25T12:53:00.512+02:00 level=DEBUG source=ggml.go:99 msg="ggml backend load all from path" path=C:\Users\User\AppData\Local\Programs\Ollama\lib\ollama\cuda_v11 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA A16-16A, compute capability 8.6, VMM: no load_backend: loaded CUDA backend from C:\Users\User\AppData\Local\Programs\Ollama\lib\ollama\cuda_v11\ggml-cuda.dll time=2025-04-25T12:53:38.145+02:00 level=DEBUG source=ggml.go:93 msg="skipping path which is not part of ollama" path=C:\Windows\system32 time=2025-04-25T12:53:38.145+02:00 level=DEBUG source=ggml.go:93 msg="skipping path which is not part of ollama" path=C:\Windows time=2025-04-25T12:53:38.145+02:00 level=DEBUG source=ggml.go:93 msg="skipping path which is not part of ollama" path=C:\Windows\System32\Wbem time=2025-04-25T12:53:38.145+02:00 level=DEBUG source=ggml.go:93 msg="skipping path which is not part of ollama" path=C:\Windows\System32\WindowsPowerShell\v1.0 time=2025-04-25T12:53:38.145+02:00 level=DEBUG source=ggml.go:93 msg="skipping path which is not part of ollama" path=C:\Windows\System32\OpenSSH time=2025-04-25T12:53:38.145+02:00 level=DEBUG source=ggml.go:93 msg="skipping path which is not part of ollama" path="C:\\Program Files\\NVIDIA Corporation\\NVIDIA App\\NvDLISR" time=2025-04-25T12:53:38.145+02:00 level=DEBUG source=ggml.go:93 msg="skipping path which is not part of ollama" path="C:\\Program Files\\NVIDIA Corporation\\Nsight Compute 2024.1.0" time=2025-04-25T12:53:38.145+02:00 level=DEBUG source=ggml.go:93 msg="skipping path which is not part of ollama" path="C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common" time=2025-04-25T12:53:38.145+02:00 level=DEBUG source=ggml.go:99 msg="ggml backend load all from path" path=C:\Users\User\AppData\Local\Programs\Ollama time=2025-04-25T12:53:38.146+02:00 level=DEBUG source=ggml.go:93 msg="skipping path which is not part of ollama" path=C:\Users\User\AppData\Local\Microsoft\WindowsApps time=2025-04-25T12:53:38.146+02:00 level=DEBUG source=ggml.go:99 msg="ggml backend load all from path" path=C:\Users\User\AppData\Local\Programs\Ollama\lib\ollama load_backend: loaded CPU backend from C:\Users\User\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-icelake.dll time=2025-04-25T12:53:38.370+02:00 level=INFO source=ggml.go:109 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.AVX512=1 CPU.0.AVX512_VBMI=1 CPU.0.AVX512_VNNI=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) CUDA error: all CUDA-capable devices are busy or unavailable current device: 0, in function ggml_backend_cuda_device_get_memory at C:\a\ollama\ollama\ml\backend\ggml\ggml\src\ggml-cuda\ggml-cuda.cu:2959 cudaMemGetInfo(free, total) C:\a\ollama\ollama\ml\backend\ggml\ggml\src\ggml-cuda\ggml-cuda.cu:75: CUDA error time=2025-04-25T12:53:38.536+02:00 level=ERROR source=server.go:449 msg="llama runner terminated" error="exit status 0xc0000409" time=2025-04-25T12:53:38.579+02:00 level=ERROR source=sched.go:457 msg="error loading llama server" error="llama runner process has terminated: CUDA error" time=2025-04-25T12:53:38.579+02:00 level=DEBUG source=sched.go:460 msg="triggering expiration for failed load" model=C:\Users\User\.ollama\models\blobs\sha256-96c415656d377afbff962f6cdb2394ab092ccbcbaab4b82525bc4ca800fe8a49 [GIN] 2025/04/25 - 12:53:38 | 500 | 38.4837386s | 127.0.0.1 | POST "/api/generate" time=2025-04-25T12:53:38.579+02:00 level=DEBUG source=sched.go:362 msg="runner expired event received" modelPath=C:\Users\User\.ollama\models\blobs\sha256-96c415656d377afbff962f6cdb2394ab092ccbcbaab4b82525bc4ca800fe8a49 time=2025-04-25T12:53:38.581+02:00 level=DEBUG source=sched.go:377 msg="got lock to unload" modelPath=C:\Users\User\.ollama\models\blobs\sha256-96c415656d377afbff962f6cdb2394ab092ccbcbaab4b82525bc4ca800fe8a49 time=2025-04-25T12:53:38.628+02:00 level=DEBUG source=server.go:1001 msg="stopping llama server" time=2025-04-25T12:53:38.628+02:00 level=DEBUG source=sched.go:382 msg="runner released" modelPath=C:\Users\User\.ollama\models\blobs\sha256-96c415656d377afbff962f6cdb2394ab092ccbcbaab4b82525bc4ca800fe8a49 time=2025-04-25T12:53:38.628+02:00 level=DEBUG source=sched.go:386 msg="sending an unloaded event" modelPath=C:\Users\User\.ollama\models\blobs\sha256-96c415656d377afbff962f6cdb2394ab092ccbcbaab4b82525bc4ca800fe8a49 time=2025-04-25T12:53:38.628+02:00 level=DEBUG source=sched.go:310 msg="ignoring unload event with no pending requests" ``` ### OS Windows 11 ### GPU Nvidia A16 ### CPU Intel ### Ollama version 0.6.6
GiteaMirror added the bug label 2026-04-29 02:42:07 -05:00
GiteaMirror commented 2026-04-29 02:42:09 -05:00
Author
Owner
Copy Link

@Sven1403 commented on GitHub (Apr 25, 2025):

Changing the profile from A to Q did the Trick :)

Image

<!-- gh-comment-id:2830407583 --> @Sven1403 commented on GitHub (Apr 25, 2025): Changing the profile from A to Q did the Trick :) ![Image](https://github.com/user-attachments/assets/a2bd7216-0e29-437a-bc6a-7572884a1300)
GiteaMirror commented 2026-04-29 02:42:09 -05:00
Author
Owner
Copy Link

@sunisstar commented on GitHub (Jun 12, 2025):

I'm now in the same situation: error="cuda driver library failed to get device context 801"
How did you fix it?

<!-- gh-comment-id:2966121637 --> @sunisstar commented on GitHub (Jun 12, 2025): I'm now in the same situation: error="cuda driver library failed to get device context 801" How did you fix it?
GiteaMirror commented 2026-04-29 02:42:10 -05:00
Author
Owner
Copy Link

@Sven1403 commented on GitHub (Jun 13, 2025):

@sunisstar see my answer: https://github.com/ollama/ollama/issues/10409#issuecomment-2830407583 vGPU has different profiles. You need the Q Profile

<!-- gh-comment-id:2969221961 --> @Sven1403 commented on GitHub (Jun 13, 2025): @sunisstar see my answer: https://github.com/ollama/ollama/issues/10409#issuecomment-2830407583 vGPU has different profiles. You need the Q Profile
GiteaMirror commented 2026-04-29 02:42:11 -05:00
Author
Owner
Copy Link

@bingobob commented on GitHub (Aug 24, 2025):

@sunisstar see my answer: #10409 (comment) vGPU has different profiles. You need the Q Profile

You are my hero - hacking about and googling for a solution.
i had a tesla p4 as a vgpu on proxmox and was getting the 801 error.
Changing the profile in proxmox MDev type to nvidia-65 (a Q profile) fixed it. THANK YOU!

<!-- gh-comment-id:3217784901 --> @bingobob commented on GitHub (Aug 24, 2025): > [@sunisstar](https://github.com/sunisstar) see my answer: [#10409 (comment)](https://github.com/ollama/ollama/issues/10409#issuecomment-2830407583) vGPU has different profiles. You need the Q Profile You are my hero - hacking about and googling for a solution. i had a tesla p4 as a vgpu on proxmox and was getting the 801 error. Changing the profile in proxmox MDev type to nvidia-65 (a Q profile) fixed it. THANK YOU!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#53353
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?