github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-17 05:36:16 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #5689] System wide old version of cuda v11 used instead of bundled version - runner fails to start due to missing symbols #3544

New Issue
Closed
opened 2026-04-12 14:15:43 -05:00 by GiteaMirror · 23 comments
GiteaMirror commented 2026-04-12 14:15:43 -05:00
Owner
Copy Link

Originally created by @hljhyb on GitHub (Jul 14, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/5689

Originally assigned to: @dhiltgen on GitHub.

What is the issue?

Previously, all models could run very well, but after a recent upgrade, errors are occurring. I have enabled Debug mode. What could be the issue?
[GIN] 2024/07/14 - 16:05:11 | 200 | 0s | 127.0.0.1 | HEAD "/"
[GIN] 2024/07/14 - 16:05:20 | 200 | 69.6µs | 127.0.0.1 | HEAD "/"
[GIN] 2024/07/14 - 16:05:20 | 200 | 23.4842ms | 127.0.0.1 | POST "/api/show"
time=2024-07-14T16:05:21.026+08:00 level=INFO source=sched.go:701 msg="new model will fit in available VRAM in single GPU, loading" model=C:\Users\X170.ollama\models\blobs\sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa gpu=GPU-06932cd6-8249-ad0c-67b5-f2bcc68be311 parallel=4 available=14287372288 required="6.2 GiB"
time=2024-07-14T16:05:21.026+08:00 level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[13.3 GiB]" memory.required.full="6.2 GiB" memory.required.partial="6.2 GiB" memory.required.kv="1.0 GiB" memory.required.allocations="[6.2 GiB]" memory.weights.total="4.7 GiB" memory.weights.repeating="4.3 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="560.0 MiB" memory.graph.partial="677.5 MiB"
time=2024-07-14T16:05:21.046+08:00 level=INFO source=server.go:383 msg="starting llama server" cmd="C:\Users\X170\AppData\Local\Programs\Ollama\ollama_runners\cuda_v11.3\ollama_llama_server.exe --model C:\Users\X170\.ollama\models\blobs\sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa --ctx-size 8192 --batch-size 512 --embedding --log-disable --n-gpu-layers 33 --no-mmap --parallel 4 --port 14534"
time=2024-07-14T16:05:21.050+08:00 level=INFO source=sched.go:437 msg="loaded runners" count=1
time=2024-07-14T16:05:21.050+08:00 level=INFO source=server.go:571 msg="waiting for llama runner to start responding"
time=2024-07-14T16:05:21.051+08:00 level=INFO source=server.go:612 msg="waiting for server to become available" status="llm server error"
time=2024-07-14T16:05:21.314+08:00 level=ERROR source=sched.go:443 msg="error loading llama server" error="llama runner process has terminated: exit status 0xc0000139 "
[GIN] 2024/07/14 - 16:05:21 | 500 | 354.9161ms | 127.0.0.1 | POST "/api/chat"

server.log
app.log

OS

Windows

GPU

Intel

CPU

Intel

Ollama version

0.2.5

Originally created by @hljhyb on GitHub (Jul 14, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/5689 Originally assigned to: @dhiltgen on GitHub. ### What is the issue? Previously, all models could run very well, but after a recent upgrade, errors are occurring. I have enabled Debug mode. What could be the issue? [GIN] 2024/07/14 - 16:05:11 | 200 | 0s | 127.0.0.1 | HEAD "/" [GIN] 2024/07/14 - 16:05:20 | 200 | 69.6µs | 127.0.0.1 | HEAD "/" [GIN] 2024/07/14 - 16:05:20 | 200 | 23.4842ms | 127.0.0.1 | POST "/api/show" time=2024-07-14T16:05:21.026+08:00 level=INFO source=sched.go:701 msg="new model will fit in available VRAM in single GPU, loading" model=C:\Users\X170\.ollama\models\blobs\sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa gpu=GPU-06932cd6-8249-ad0c-67b5-f2bcc68be311 parallel=4 available=14287372288 required="6.2 GiB" time=2024-07-14T16:05:21.026+08:00 level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[13.3 GiB]" memory.required.full="6.2 GiB" memory.required.partial="6.2 GiB" memory.required.kv="1.0 GiB" memory.required.allocations="[6.2 GiB]" memory.weights.total="4.7 GiB" memory.weights.repeating="4.3 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="560.0 MiB" memory.graph.partial="677.5 MiB" time=2024-07-14T16:05:21.046+08:00 level=INFO source=server.go:383 msg="starting llama server" cmd="C:\\Users\\X170\\AppData\\Local\\Programs\\Ollama\\ollama_runners\\cuda_v11.3\\ollama_llama_server.exe --model C:\\Users\\X170\\.ollama\\models\\blobs\\sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa --ctx-size 8192 --batch-size 512 --embedding --log-disable --n-gpu-layers 33 --no-mmap --parallel 4 --port 14534" time=2024-07-14T16:05:21.050+08:00 level=INFO source=sched.go:437 msg="loaded runners" count=1 time=2024-07-14T16:05:21.050+08:00 level=INFO source=server.go:571 msg="waiting for llama runner to start responding" time=2024-07-14T16:05:21.051+08:00 level=INFO source=server.go:612 msg="waiting for server to become available" status="llm server error" time=2024-07-14T16:05:21.314+08:00 level=ERROR source=sched.go:443 msg="error loading llama server" error="llama runner process has terminated: exit status 0xc0000139 " [GIN] 2024/07/14 - 16:05:21 | 500 | 354.9161ms | 127.0.0.1 | POST "/api/chat" [server.log](https://github.com/user-attachments/files/16216040/server.log) [app.log](https://github.com/user-attachments/files/16216041/app.log) ### OS Windows ### GPU Intel ### CPU Intel ### Ollama version 0.2.5
GiteaMirror added the nvidiabugwindows labels 2026-04-12 14:15:43 -05:00
GiteaMirror commented 2026-04-12 14:15:45 -05:00
Author
Owner
Copy Link

@IAmTheNerdNextDoor commented on GitHub (Jul 14, 2024):

Have you tried reinstalling? This has fixed this issue for me. I'm assuming this is an issue with upgrading and installing manually seems to fix it.

You don't need to uninstall, you just download the latest Windows release and close Ollama and just let it overwrite. (This will not change Ollama environment tables or erase models, just the core of Ollama.)

<!-- gh-comment-id:2227357531 --> @IAmTheNerdNextDoor commented on GitHub (Jul 14, 2024): Have you tried reinstalling? This has fixed this issue for me. I'm assuming this is an issue with upgrading and installing manually seems to fix it. You don't need to uninstall, you just download the latest Windows release and close Ollama and just let it overwrite. (This will not change Ollama environment tables or erase models, just the core of Ollama.)
GiteaMirror commented 2026-04-12 14:15:47 -05:00
Author
Owner
Copy Link

@hljhyb commented on GitHub (Jul 14, 2024):

Have you tried reinstalling? This has fixed this issue for me. I'm assuming this is an issue with upgrading and installing manually seems to fix it.

You don't need to uninstall, you just download the latest Windows release and close Ollama and just let it overwrite. (This will not change Ollama environment tables or erase models, just the core of Ollama.)

I have reinstalled several times and upgraded from 0.2.1 to 0.2.4 to 0.2.5. The same problem occurs every time

<!-- gh-comment-id:2227430754 --> @hljhyb commented on GitHub (Jul 14, 2024): > Have you tried reinstalling? This has fixed this issue for me. I'm assuming this is an issue with upgrading and installing manually seems to fix it. > > You don't need to uninstall, you just download the latest Windows release and close Ollama and just let it overwrite. (This will not change Ollama environment tables or erase models, just the core of Ollama.) I have reinstalled several times and upgraded from 0.2.1 to 0.2.4 to 0.2.5. The same problem occurs every time
GiteaMirror commented 2026-04-12 14:15:48 -05:00
Author
Owner
Copy Link

@IAmTheNerdNextDoor commented on GitHub (Jul 14, 2024):

I have reinstalled several times and upgraded from 0.2.1 to 0.2.4 to 0.2.5. The same problem occurs every time

Have you updated your graphics drivers recently? It may of been an issue with that. If not, try using a version pre-0.2.x. If any Windows Updates were installed recently prior to this issue occurring, try removing them in case they caused something to go wrong.

<!-- gh-comment-id:2227488220 --> @IAmTheNerdNextDoor commented on GitHub (Jul 14, 2024): > I have reinstalled several times and upgraded from 0.2.1 to 0.2.4 to 0.2.5. The same problem occurs every time Have you updated your graphics drivers recently? It may of been an issue with that. If not, try using a version pre-0.2.x. If any Windows Updates were installed recently prior to this issue occurring, try removing them in case they caused something to go wrong.
GiteaMirror commented 2026-04-12 14:15:49 -05:00
Author
Owner
Copy Link

@jmorganca commented on GitHub (Jul 14, 2024):

Hi @hljhyb sorry this happened – will get this fixed.

Would it be possible for you to run in powershell:

$env:PATH="C:\Users\jmorgan\AppData\Local\Programs\Ollama\cuda;" +$env:PATH
explorer.exe C:\Users\X170\AppData\Local\Programs\Ollama\ollama_runners\cuda_v11.3\ollama_llama_server.exe

and let me know what error shows

<!-- gh-comment-id:2227507070 --> @jmorganca commented on GitHub (Jul 14, 2024): Hi @hljhyb sorry this happened – will get this fixed. Would it be possible for you to run in powershell: ``` $env:PATH="C:\Users\jmorgan\AppData\Local\Programs\Ollama\cuda;" +$env:PATH explorer.exe C:\Users\X170\AppData\Local\Programs\Ollama\ollama_runners\cuda_v11.3\ollama_llama_server.exe ``` and let me know what error shows
GiteaMirror commented 2026-04-12 14:15:50 -05:00
Author
Owner
Copy Link

@hljhyb commented on GitHub (Jul 15, 2024):

Hi @hljhyb sorry this happened – will get this fixed.

Would it be possible for you to run in powershell:

$env:PATH="C:\Users\jmorgan\AppData\Local\Programs\Ollama\cuda;" +$env:PATH
explorer.exe C:\Users\X170\AppData\Local\Programs\Ollama\ollama_runners\cuda_v11.3\ollama_llama_server.exe

and let me know what error shows

INFO [wmain] build info | build=3337 commit="a8db2a9c" tid="335452" timestamp=1721003942
INFO [wmain] system info | n_threads=10 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 0 | " tid="335452" timestamp=1721003942 total_threads=20
INFO [wmain] HTTP server listening | hostname="127.0.0.1" n_threads_http="19" port="8080" tid="335452" timestamp=1721003942
llama_model_load: error loading model: llama_model_loader: failed to load model from

llama_load_model_from_file: exception loading model

<!-- gh-comment-id:2227546789 --> @hljhyb commented on GitHub (Jul 15, 2024): > Hi @hljhyb sorry this happened – will get this fixed. > > Would it be possible for you to run in powershell: > > ``` > $env:PATH="C:\Users\jmorgan\AppData\Local\Programs\Ollama\cuda;" +$env:PATH > explorer.exe C:\Users\X170\AppData\Local\Programs\Ollama\ollama_runners\cuda_v11.3\ollama_llama_server.exe > ``` > > and let me know what error shows INFO [wmain] build info | build=3337 commit="a8db2a9c" tid="335452" timestamp=1721003942 INFO [wmain] system info | n_threads=10 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 0 | " tid="335452" timestamp=1721003942 total_threads=20 INFO [wmain] HTTP server listening | hostname="127.0.0.1" n_threads_http="19" port="8080" tid="335452" timestamp=1721003942 llama_model_load: error loading model: llama_model_loader: failed to load model from llama_load_model_from_file: exception loading model
GiteaMirror commented 2026-04-12 14:15:51 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Jul 23, 2024):

@hljhyb the attached logs don't seem to have debug enabled. Can you try updating to v0.2.8 and if that doesn't clear it up, try the following so we can try to see why it's not able to start up correctly. Quit the tray app, then run the following which will restart it with debugging enabled.

$env:OLLAMA_DEBUG="1"
& "ollama app"

Then try to load a model, and assuming it fails, share the server.log.

<!-- gh-comment-id:2246307281 --> @dhiltgen commented on GitHub (Jul 23, 2024): @hljhyb the attached logs don't seem to have debug enabled. Can you try updating to [v0.2.8](https://github.com/ollama/ollama/releases/tag/v0.2.8) and if that doesn't clear it up, try the following so we can try to see why it's not able to start up correctly. Quit the tray app, then run the following which will restart it with debugging enabled. ```powershell $env:OLLAMA_DEBUG="1" & "ollama app" ``` Then try to load a model, and assuming it fails, share the server.log.
GiteaMirror commented 2026-04-12 14:15:52 -05:00
Author
Owner
Copy Link

@hljhyb commented on GitHub (Jul 24, 2024):

@dhiltgen
I did it as you said, but it still doesn't work.
server.log

<!-- gh-comment-id:2246837617 --> @hljhyb commented on GitHub (Jul 24, 2024): @dhiltgen I did it as you said, but it still doesn't work. [server.log](https://github.com/user-attachments/files/16357034/server.log)
GiteaMirror commented 2026-04-12 14:15:53 -05:00
Author
Owner
Copy Link

@IAmTheNerdNextDoor commented on GitHub (Jul 24, 2024):

The error code 0xC0000139 does relate to an NTSTATUS error of NT_STATUS_ENTRYPOINT_NOT_FOUND which means “The specified procedure could not be found.”

This could mean their Windows install could potentially be holding corrupted files or missing files required for Ollama to work correctly.

Have you tried running cmd as admin and running "sfc /scannow" and "dism /online /cleanup-image /restorehealth" which helps to fix corrupted files and repair the system.

Let us know the results of that and if Ollama is working correctly after the commands!

The NTSTATUS code list I used is here by the way if you want to check yourself: https://gist.github.com/mbrownnycnyc/24cb49aa2cba35be4764b1fa37bf9063

<!-- gh-comment-id:2247764692 --> @IAmTheNerdNextDoor commented on GitHub (Jul 24, 2024): The error code 0xC0000139 does relate to an NTSTATUS error of NT_STATUS_ENTRYPOINT_NOT_FOUND which means “The specified procedure could not be found.” This could mean their Windows install could potentially be holding corrupted files or missing files required for Ollama to work correctly. Have you tried running cmd as admin and running "sfc /scannow" and "dism /online /cleanup-image /restorehealth" which helps to fix corrupted files and repair the system. Let us know the results of that and if Ollama is working correctly after the commands! The NTSTATUS code list I used is here by the way if you want to check yourself: https://gist.github.com/mbrownnycnyc/24cb49aa2cba35be4764b1fa37bf9063
GiteaMirror commented 2026-04-12 14:15:53 -05:00
Author
Owner
Copy Link

@hljhyb commented on GitHub (Jul 26, 2024):

I ran "sfc /scannow" and "dism /online /cleanup-image /restorehealth," and changed the CUDA version to 11.5.
But it's still the same problem. Maybe there's an issue with my operating system.
Still, thank you for your help.

<!-- gh-comment-id:2252088469 --> @hljhyb commented on GitHub (Jul 26, 2024): I ran "sfc /scannow" and "dism /online /cleanup-image /restorehealth," and changed the CUDA version to 11.5. But it's still the same problem. Maybe there's an issue with my operating system. Still, thank you for your help.
GiteaMirror commented 2026-04-12 14:15:54 -05:00
Author
Owner
Copy Link

@IAmTheNerdNextDoor commented on GitHub (Jul 26, 2024):

I ran "sfc /scannow" and "dism /online /cleanup-image /restorehealth," and changed the CUDA version to 11.5. But it's still the same problem. Maybe there's an issue with my operating system. Still, thank you for your help.

Ah, no problem. It may be an OS issue but I'm not sure now. Can you try the latest 0.3 update to Ollama and see if that works? If not, a Windows Update assisted OS reinstallation may be needed but that's only when we exhaust all options.

If you are sure about reinstalling, you can do it by heading to "Recovery" in Settings and clicking "Reinstall now" at the "Fix problems using Windows Update" option.

This will keep all data, settings and apps but I still recommend backing up important data for an edge case where something doesn't get kept. This also requires some patience as this could take an hour or two.

I know that the reinstall idea sounds like bad advice but it's the safest way to fix this when all options have been tried with failure as a result.

<!-- gh-comment-id:2252828004 --> @IAmTheNerdNextDoor commented on GitHub (Jul 26, 2024): > I ran "sfc /scannow" and "dism /online /cleanup-image /restorehealth," and changed the CUDA version to 11.5. But it's still the same problem. Maybe there's an issue with my operating system. Still, thank you for your help. Ah, no problem. It may be an OS issue but I'm not sure now. Can you try the latest 0.3 update to Ollama and see if that works? If not, a Windows Update assisted OS reinstallation may be needed but that's only when we exhaust all options. If you are sure about reinstalling, you can do it by heading to "Recovery" in Settings and clicking "Reinstall now" at the "Fix problems using Windows Update" option. This will keep all data, settings and apps but I still recommend backing up important data for an edge case where something doesn't get kept. This also requires some patience as this could take an hour or two. I know that the reinstall idea sounds like bad advice but it's the safest way to fix this when all options have been tried with failure as a result.
GiteaMirror commented 2026-04-12 14:15:56 -05:00
Author
Owner
Copy Link

@hljhyb commented on GitHub (Jul 29, 2024):

I ran "sfc /scannow" and "dism /online /cleanup-image /restorehealth," and changed the CUDA version to 11.5. But it's still the same problem. Maybe there's an issue with my operating system. Still, thank you for your help.

Ah, no problem. It may be an OS issue but I'm not sure now. Can you try the latest 0.3 update to Ollama and see if that works? If not, a Windows Update assisted OS reinstallation may be needed but that's only when we exhaust all options.

If you are sure about reinstalling, you can do it by heading to "Recovery" in Settings and clicking "Reinstall now" at the "Fix problems using Windows Update" option.

This will keep all data, settings and apps but I still recommend backing up important data for an edge case where something doesn't get kept. This also requires some patience as this could take an hour or two.

I know that the reinstall idea sounds like bad advice but it's the safest way to fix this when all options have been tried with failure as a result.

I did it , but it still doesn't work. I tried using "Recovery" in Settings as you said, but after reinstall, I still got the same error message.

<!-- gh-comment-id:2255243893 --> @hljhyb commented on GitHub (Jul 29, 2024): > > I ran "sfc /scannow" and "dism /online /cleanup-image /restorehealth," and changed the CUDA version to 11.5. But it's still the same problem. Maybe there's an issue with my operating system. Still, thank you for your help. > > Ah, no problem. It may be an OS issue but I'm not sure now. Can you try the latest 0.3 update to Ollama and see if that works? If not, a Windows Update assisted OS reinstallation may be needed but that's only when we exhaust all options. > > If you are sure about reinstalling, you can do it by heading to "Recovery" in Settings and clicking "Reinstall now" at the "Fix problems using Windows Update" option. > > This will keep all data, settings and apps but I still recommend backing up important data for an edge case where something doesn't get kept. This also requires some patience as this could take an hour or two. > > I know that the reinstall idea sounds like bad advice but it's the safest way to fix this when all options have been tried with failure as a result. I did it , but it still doesn't work. I tried using "Recovery" in Settings as you said, but after reinstall, I still got the same error message.
GiteaMirror commented 2026-04-12 14:15:57 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Jul 29, 2024):

@hljhyb we might get a little more insight if you try to run the subprocess directly. Try the following:

$env:PATH="C:\Users\X170\AppData\Local\Programs\Ollama\cuda;C:\Users\X170\AppData\Local\Programs\Ollama\ollama_runners\cuda_v11.3;C:\Users\X170\AppData\Local\Programs\Ollama\ollama_runners;$env:PATH"
C:\Users\X170\AppData\Local\Programs\Ollama\ollama_runners\cuda_v11.3\ollama_llama_server.exe --help

The most likely explanation for this status code is a missing or incorrect DLL, and starting it manually should hopefully cause an error dialog to pop up showing what DLL(s) are the problem.

<!-- gh-comment-id:2256383150 --> @dhiltgen commented on GitHub (Jul 29, 2024): @hljhyb we might get a little more insight if you try to run the subprocess directly. Try the following: ```powershell $env:PATH="C:\Users\X170\AppData\Local\Programs\Ollama\cuda;C:\Users\X170\AppData\Local\Programs\Ollama\ollama_runners\cuda_v11.3;C:\Users\X170\AppData\Local\Programs\Ollama\ollama_runners;$env:PATH" C:\Users\X170\AppData\Local\Programs\Ollama\ollama_runners\cuda_v11.3\ollama_llama_server.exe --help ``` The most likely explanation for this status code is a missing or incorrect DLL, and starting it manually should hopefully cause an error dialog to pop up showing what DLL(s) are the problem.
GiteaMirror commented 2026-04-12 14:15:58 -05:00
Author
Owner
Copy Link

@hljhyb commented on GitHub (Jul 30, 2024):

@hljhyb we might get a little more insight if you try to run the subprocess directly. Try the following:

$env:PATH="C:\Users\X170\AppData\Local\Programs\Ollama\cuda;C:\Users\X170\AppData\Local\Programs\Ollama\ollama_runners\cuda_v11.3;C:\Users\X170\AppData\Local\Programs\Ollama\ollama_runners;$env:PATH"
C:\Users\X170\AppData\Local\Programs\Ollama\ollama_runners\cuda_v11.3\ollama_llama_server.exe --help

The most likely explanation for this status code is a missing or incorrect DLL, and starting it manually should hopefully cause an error dialog to pop up showing what DLL(s) are the problem.

I may not have understood your method clearly. I did not receive any prompts after running these commands.
11222

<!-- gh-comment-id:2257328428 --> @hljhyb commented on GitHub (Jul 30, 2024): > @hljhyb we might get a little more insight if you try to run the subprocess directly. Try the following: > > ```powershell > $env:PATH="C:\Users\X170\AppData\Local\Programs\Ollama\cuda;C:\Users\X170\AppData\Local\Programs\Ollama\ollama_runners\cuda_v11.3;C:\Users\X170\AppData\Local\Programs\Ollama\ollama_runners;$env:PATH" > C:\Users\X170\AppData\Local\Programs\Ollama\ollama_runners\cuda_v11.3\ollama_llama_server.exe --help > ``` > > The most likely explanation for this status code is a missing or incorrect DLL, and starting it manually should hopefully cause an error dialog to pop up showing what DLL(s) are the problem. I may not have understood your method clearly. I did not receive any prompts after running these commands. ![11222](https://github.com/user-attachments/assets/e6a75066-9ca3-4861-a0c1-46e1ff101390)
GiteaMirror commented 2026-04-12 14:15:59 -05:00
Author
Owner
Copy Link

@IAmTheNerdNextDoor commented on GitHub (Jul 30, 2024):

I ran "sfc /scannow" and "dism /online /cleanup-image /restorehealth," and changed the CUDA version to 11.5. But it's still the same problem. Maybe there's an issue with my operating system. Still, thank you for your help.

Ah, no problem. It may be an OS issue but I'm not sure now. Can you try the latest 0.3 update to Ollama and see if that works? If not, a Windows Update assisted OS reinstallation may be needed but that's only when we exhaust all options.
If you are sure about reinstalling, you can do it by heading to "Recovery" in Settings and clicking "Reinstall now" at the "Fix problems using Windows Update" option.
This will keep all data, settings and apps but I still recommend backing up important data for an edge case where something doesn't get kept. This also requires some patience as this could take an hour or two.
I know that the reinstall idea sounds like bad advice but it's the safest way to fix this when all options have been tried with failure as a result.

I did it , but it still doesn't work. I tried using "Recovery" in Settings as you said, but after reinstall, I still got the same error message.

I'm completely at a loss now, I don't exactly know what's happened at this point. Sorry for not being able to help. I hope your issue gets fixed sooner rather than later.

Dropping out here. Good luck to you.

<!-- gh-comment-id:2258567562 --> @IAmTheNerdNextDoor commented on GitHub (Jul 30, 2024): > > > I ran "sfc /scannow" and "dism /online /cleanup-image /restorehealth," and changed the CUDA version to 11.5. But it's still the same problem. Maybe there's an issue with my operating system. Still, thank you for your help. > > > > > > Ah, no problem. It may be an OS issue but I'm not sure now. Can you try the latest 0.3 update to Ollama and see if that works? If not, a Windows Update assisted OS reinstallation may be needed but that's only when we exhaust all options. > > If you are sure about reinstalling, you can do it by heading to "Recovery" in Settings and clicking "Reinstall now" at the "Fix problems using Windows Update" option. > > This will keep all data, settings and apps but I still recommend backing up important data for an edge case where something doesn't get kept. This also requires some patience as this could take an hour or two. > > I know that the reinstall idea sounds like bad advice but it's the safest way to fix this when all options have been tried with failure as a result. > > I did it , but it still doesn't work. I tried using "Recovery" in Settings as you said, but after reinstall, I still got the same error message. I'm completely at a loss now, I don't exactly know what's happened at this point. Sorry for not being able to help. I hope your issue gets fixed sooner rather than later. Dropping out here. Good luck to you.
GiteaMirror commented 2026-04-12 14:16:00 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Jul 30, 2024):

@hljhyb sorry about that. It looks like powershell hides the error dialog I was hoping would pop up. We need to use a cmd terminal for this experiment.

set PATH=C:\Users\X170\AppData\Local\Programs\Ollama\cuda;C:\Users\X170\AppData\Local\Programs\Ollama\ollama_runners\cuda_v11.3;C:\Users\X170\AppData\Local\Programs\Ollama\ollama_runners;%PATH%
C:\Users\X170\AppData\Local\Programs\Ollama\ollama_runners\cuda_v11.3\ollama_llama_server.exe --help

You should get a dialog that pops up that looks something like this (most likely a different dll will be listed)

Screenshot 2024-07-30 at 8 57 41 AM
<!-- gh-comment-id:2258690339 --> @dhiltgen commented on GitHub (Jul 30, 2024): @hljhyb sorry about that. It looks like powershell hides the error dialog I was hoping would pop up. We need to use a `cmd` terminal for this experiment. ```cmd set PATH=C:\Users\X170\AppData\Local\Programs\Ollama\cuda;C:\Users\X170\AppData\Local\Programs\Ollama\ollama_runners\cuda_v11.3;C:\Users\X170\AppData\Local\Programs\Ollama\ollama_runners;%PATH% C:\Users\X170\AppData\Local\Programs\Ollama\ollama_runners\cuda_v11.3\ollama_llama_server.exe --help ``` You should get a dialog that pops up that looks something like this (most likely a different dll will be listed) <img width="490" alt="Screenshot 2024-07-30 at 8 57 41 AM" src="https://github.com/user-attachments/assets/577bc7c6-a6bf-438d-8e9f-83fb779d9c99">
GiteaMirror commented 2026-04-12 14:16:01 -05:00
Author
Owner
Copy Link

@hljhyb commented on GitHub (Jul 31, 2024):

@AFellowSpeedrunner

I ran "sfc /scannow" and "dism /online /cleanup-image /restorehealth," and changed the CUDA version to 11.5. But it's still the same problem. Maybe there's an issue with my operating system. Still, thank you for your help.

Ah, no problem. It may be an OS issue but I'm not sure now. Can you try the latest 0.3 update to Ollama and see if that works? If not, a Windows Update assisted OS reinstallation may be needed but that's only when we exhaust all options.
If you are sure about reinstalling, you can do it by heading to "Recovery" in Settings and clicking "Reinstall now" at the "Fix problems using Windows Update" option.
This will keep all data, settings and apps but I still recommend backing up important data for an edge case where something doesn't get kept. This also requires some patience as this could take an hour or two.
I know that the reinstall idea sounds like bad advice but it's the safest way to fix this when all options have been tried with failure as a result.

I did it , but it still doesn't work. I tried using "Recovery" in Settings as you said, but after reinstall, I still got the same error message.

I'm completely at a loss now, I don't exactly know what's happened at this point. Sorry for not being able to help. I hope your issue gets fixed sooner rather than later.

Dropping out here. Good luck to you.

Thank you so much for your kind help.

<!-- gh-comment-id:2259562166 --> @hljhyb commented on GitHub (Jul 31, 2024): @AFellowSpeedrunner > > > > I ran "sfc /scannow" and "dism /online /cleanup-image /restorehealth," and changed the CUDA version to 11.5. But it's still the same problem. Maybe there's an issue with my operating system. Still, thank you for your help. > > > > > > > > > Ah, no problem. It may be an OS issue but I'm not sure now. Can you try the latest 0.3 update to Ollama and see if that works? If not, a Windows Update assisted OS reinstallation may be needed but that's only when we exhaust all options. > > > If you are sure about reinstalling, you can do it by heading to "Recovery" in Settings and clicking "Reinstall now" at the "Fix problems using Windows Update" option. > > > This will keep all data, settings and apps but I still recommend backing up important data for an edge case where something doesn't get kept. This also requires some patience as this could take an hour or two. > > > I know that the reinstall idea sounds like bad advice but it's the safest way to fix this when all options have been tried with failure as a result. > > > > > > I did it , but it still doesn't work. I tried using "Recovery" in Settings as you said, but after reinstall, I still got the same error message. > > I'm completely at a loss now, I don't exactly know what's happened at this point. Sorry for not being able to help. I hope your issue gets fixed sooner rather than later. > > Dropping out here. Good luck to you. Thank you so much for your kind help.
GiteaMirror commented 2026-04-12 14:16:02 -05:00
Author
Owner
Copy Link

@hljhyb commented on GitHub (Jul 31, 2024):

@dhiltgen

@hljhyb sorry about that. It looks like powershell hides the error dialog I was hoping would pop up. We need to use a cmd terminal for this experiment.

set PATH=C:\Users\X170\AppData\Local\Programs\Ollama\cuda;C:\Users\X170\AppData\Local\Programs\Ollama\ollama_runners\cuda_v11.3;C:\Users\X170\AppData\Local\Programs\Ollama\ollama_runners;%PATH%
C:\Users\X170\AppData\Local\Programs\Ollama\ollama_runners\cuda_v11.3\ollama_llama_server.exe --help

You should get a dialog that pops up that looks something like this (most likely a different dll will be listed)

Screenshot 2024-07-30 at 8 57 41 AM

It shows a dialog like this.
The procedure entry point cublasLtGetStatusString could notbe located in the dynamic link library C:\WINDOWS\SYSTEM32\cublas64_11.dll
image

<!-- gh-comment-id:2259920145 --> @hljhyb commented on GitHub (Jul 31, 2024): @dhiltgen > @hljhyb sorry about that. It looks like powershell hides the error dialog I was hoping would pop up. We need to use a `cmd` terminal for this experiment. > > ```batchfile > set PATH=C:\Users\X170\AppData\Local\Programs\Ollama\cuda;C:\Users\X170\AppData\Local\Programs\Ollama\ollama_runners\cuda_v11.3;C:\Users\X170\AppData\Local\Programs\Ollama\ollama_runners;%PATH% > C:\Users\X170\AppData\Local\Programs\Ollama\ollama_runners\cuda_v11.3\ollama_llama_server.exe --help > ``` > > You should get a dialog that pops up that looks something like this (most likely a different dll will be listed) > > <img alt="Screenshot 2024-07-30 at 8 57 41 AM" width="490" src="https://private-user-images.githubusercontent.com/4033016/353530707-577bc7c6-a6bf-438d-8e9f-83fb779d9c99.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjIzOTU3NjUsIm5iZiI6MTcyMjM5NTQ2NSwicGF0aCI6Ii80MDMzMDE2LzM1MzUzMDcwNy01NzdiYzdjNi1hNmJmLTQzOGQtOGU5Zi04M2ZiNzc5ZDljOTkucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDczMSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA3MzFUMDMxMTA1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9NmQzN2EzMWNmZTg2ZGM2MDMwYWZjYzZhNGQ0ZjE5ZTcwMjY1ZTQ3MmQ5ODY5MDkwOTFkMGI3ZDM4MzdmMWVkYSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.65wHlhbJMcVSmU7fkCnyyB974Lkwnn5EsoDSGZG3rn8"> It shows a dialog like this. The procedure entry point cublasLtGetStatusString could notbe located in the dynamic link library C:\WINDOWS\SYSTEM32\cublas64_11.dll ![image](https://github.com/user-attachments/assets/4ef7d529-3e43-4897-9d2b-4a1342334f90)
GiteaMirror commented 2026-04-12 14:16:03 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Aug 1, 2024):

It should have picked up C:\Users\X170\AppData\Local\Programs\Ollama\cuda\cublas64_11.dll instead of the one in C:\Windows\System32 given the PATH we set above, so that makes me think somehow C:\Users\X170\AppData\Local\Programs\Ollama\cuda\ is missing the libraries. Can you share what is reported with

dir C:\Users\X170\AppData\Local\Programs\Ollama\cuda\
<!-- gh-comment-id:2264213543 --> @dhiltgen commented on GitHub (Aug 1, 2024): It should have picked up `C:\Users\X170\AppData\Local\Programs\Ollama\cuda\cublas64_11.dll` instead of the one in `C:\Windows\System32` given the PATH we set above, so that makes me think somehow `C:\Users\X170\AppData\Local\Programs\Ollama\cuda\` is missing the libraries. Can you share what is reported with ``` dir C:\Users\X170\AppData\Local\Programs\Ollama\cuda\ ```
GiteaMirror commented 2026-04-12 14:16:04 -05:00
Author
Owner
Copy Link

@hljhyb commented on GitHub (Aug 5, 2024):

112177632

<!-- gh-comment-id:2268024488 --> @hljhyb commented on GitHub (Aug 5, 2024): ![112177632](https://github.com/user-attachments/assets/2b443d65-1dee-4a63-a813-d81c3e9775ad)
GiteaMirror commented 2026-04-12 14:16:04 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Aug 5, 2024):

Thanks for that information. So our bundled DLL exists, but even though we specify our directory first in the PATH, windows is defaulting to the system installed version of CUDA v11, and your install is most likely older than 11.3.1. Until we can figure out how to fix the search order to force our copy, you can workaround this by upgrading cuda v11 to 11.3.1 or newer, or uninstall cuda v11 if you don't have other applications that need that version.

Update: looks like we'll need to inject a side by side assembly into our subprocesse executables so they add a probing privatePath to search for our packaged dependencies before the system directories are searched to prevent an incompatible v11 library from accidentally getting loaded.

<!-- gh-comment-id:2270042660 --> @dhiltgen commented on GitHub (Aug 5, 2024): Thanks for that information. So our bundled DLL exists, but even though we specify our directory first in the PATH, windows is defaulting to the system installed version of CUDA v11, and your install is most likely older than 11.3.1. Until we ~~can figure out how to~~ fix the search order to force our copy, you can workaround this by upgrading cuda v11 to 11.3.1 or newer, or uninstall cuda v11 if you don't have other applications that need that version. **Update:** looks like we'll need to inject a side by side assembly into our subprocesse executables so they add a probing privatePath to search for our packaged dependencies before the system directories are searched to prevent an incompatible v11 library from accidentally getting loaded.
GiteaMirror commented 2026-04-12 14:16:05 -05:00
Author
Owner
Copy Link

@hljhyb commented on GitHub (Aug 12, 2024):

Thanks for that information. So our bundled DLL exists, but even though we specify our directory first in the PATH, windows is defaulting to the system installed version of CUDA v11, and your install is most likely older than 11.3.1. Until we can figure out how to fix the search order to force our copy, you can workaround this by upgrading cuda v11 to 11.3.1 or newer, or uninstall cuda v11 if you don't have other applications that need that version.

Update: looks like we'll need to inject a side by side assembly into our subprocesse executables so they add a probing privatePath to search for our packaged dependencies before the system directories are searched to prevent an incompatible v11 library from accidentally getting loaded.

I've already uninstalled CUDA, but now a pop-up appears when I try to run the model, and it still doesn't work.
Snipaste_2024-08-12_11-26-55

<!-- gh-comment-id:2283053072 --> @hljhyb commented on GitHub (Aug 12, 2024): > Thanks for that information. So our bundled DLL exists, but even though we specify our directory first in the PATH, windows is defaulting to the system installed version of CUDA v11, and your install is most likely older than 11.3.1. Until we ~can figure out how to~ fix the search order to force our copy, you can workaround this by upgrading cuda v11 to 11.3.1 or newer, or uninstall cuda v11 if you don't have other applications that need that version. > > **Update:** looks like we'll need to inject a side by side assembly into our subprocesse executables so they add a probing privatePath to search for our packaged dependencies before the system directories are searched to prevent an incompatible v11 library from accidentally getting loaded. I've already uninstalled CUDA, but now a pop-up appears when I try to run the model, and it still doesn't work. ![Snipaste_2024-08-12_11-26-55](https://github.com/user-attachments/assets/125ccb5b-a9e2-4683-8cdf-a0bb07c6d5b9)
GiteaMirror commented 2026-04-12 14:16:06 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Aug 12, 2024):

I wouldn't expect that dll to exist if cuda v11 has been uninstalled. Maybe some other piece of software installed it as a dependency in the system location?

Until we get the issue fixed, you should be able to workaround this by installing cuda v11.3.1.

<!-- gh-comment-id:2284670164 --> @dhiltgen commented on GitHub (Aug 12, 2024): I wouldn't expect that dll to exist if cuda v11 has been uninstalled. Maybe some other piece of software installed it as a dependency in the system location? Until we get the issue fixed, you should be able to workaround this by installing cuda v11.3.1.
GiteaMirror commented 2026-04-12 14:16:07 -05:00
Author
Owner
Copy Link

@hljhyb commented on GitHub (Aug 13, 2024):

I wouldn't expect that dll to exist if cuda v11 has been uninstalled. Maybe some other piece of software installed it as a dependency in the system location?

Until we get the issue fixed, you should be able to workaround this by installing cuda v11.3.1.

Thank you so much! You helped me solve this problem. It seems that a previous version of CUDA copied the cublas64_11.dll version 11.10.1 file into the system directory. The earlier version of Ollama could run, but after updating, it stopped working. The CUDA version installed later didn't update this file, so every time it was called, it used the old version. I just replaced the file, and now it's working!

<!-- gh-comment-id:2285514266 --> @hljhyb commented on GitHub (Aug 13, 2024): > I wouldn't expect that dll to exist if cuda v11 has been uninstalled. Maybe some other piece of software installed it as a dependency in the system location? > > Until we get the issue fixed, you should be able to workaround this by installing cuda v11.3.1. Thank you so much! You helped me solve this problem. It seems that a previous version of CUDA copied the cublas64_11.dll version 11.10.1 file into the system directory. The earlier version of Ollama could run, but after updating, it stopped working. The CUDA version installed later didn't update this file, so every time it was called, it used the old version. I just replaced the file, and now it's working!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug nvidia windows
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#3544
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?