github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-08 17:49:24 -05:00
Code Issues 14.5k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #4982] SIGSEGV during ollama serve cgo execution CUDA - invalid cudart link to libelf-0.185.so #49661

New Issue
Closed
opened 2026-04-28 12:33:19 -05:00 by GiteaMirror · 4 comments
GiteaMirror commented 2026-04-28 12:33:19 -05:00
Owner
Copy Link

Originally created by @7910f6ba7ee4 on GitHub (Jun 11, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/4982

Originally assigned to: @dhiltgen on GitHub.

I'm running ollama using the binary provided in releases. When I try to run ollama serve, I get the error below. I have tried 0.1.43, 0.1.42, and older versions such as 0.1.34. I am using an AMD GPU, and do not have an NVIDIA, so I do not need the CUDA checks, but this is still happening with the OLLAMA_LLM_LIBRARY flag being set to something like cpu.

Appears similar to previous issue https://github.com/ollama/ollama/issues/2198

Debug info excerpt below with OLLAMA_DEBUG=1 (there are too many goroutines to copy paste, but I can provide more)

time=2024-06-11T11:24:34.121-04:00 level=DEBUG source=gpu.go:307 msg="discovered GPU libraries" paths="[/tmp/ollama479946545/runners/cuda_v11/libcudart.so.11.0 /usr/lib64/libelf-0.185.so]"
cudaSetDevice err: 35
time=2024-06-11T11:24:34.122-04:00 level=DEBUG source=gpu.go:319 msg="Unable to load cudart" library=/tmp/ollama479946545/runners/cuda_v11/libcudart.so.11.0 error="your nvidia driver is too old or missing.  If
you have a CUDA GPU please upgrade to run ollama"
SIGSEGV: segmentation violation
PC=0x0 m=18 sigcode=1 addr=0x0
signal arrived during cgo execution
 
goroutine 1 gp=0xc0000061c0 m=18 mp=0xc000800808 [syscall]:
runtime.cgocall(0xf241b0, 0xc00006b5a0)
        runtime/cgocall.go:157 +0x4b fp=0xc00006b578 sp=0xc00006b540 pc=0x42c7cb
github.com/ollama/ollama/gpu._Cfunc_cudart_init(0x7f4c54000ca0, 0xc000616c60)
        _cgo_gotypes.go:301 +0x3f fp=0xc00006b5a0 sp=0xc00006b578 pc=0x8b4bdf
github.com/ollama/ollama/gpu.LoadCUDARTMgmt.func2(0x7f4c54000ca0, 0xc000616c60)
        github.com/ollama/ollama/gpu/gpu.go:317 +0x4a fp=0xc00006b5d0 sp=0xc00006b5a0 pc=0x8b738a
github.com/ollama/ollama/gpu.LoadCUDARTMgmt({0xc000706460, 0x2, 0xc00025a5b0?})
        github.com/ollama/ollama/gpu/gpu.go:317 +0x23e fp=0xc00006b6c8 sp=0xc00006b5d0 pc=0x8b71de
github.com/ollama/ollama/gpu.initGPUHandles()
        github.com/ollama/ollama/gpu/gpu.go:136 +0x236 fp=0xc00006b850 sp=0xc00006b6c8 pc=0x8b5256
github.com/ollama/ollama/gpu.GetGPUInfo()
        github.com/ollama/ollama/gpu/gpu.go:153 +0xa7 fp=0xc00006bae8 sp=0xc00006b850 pc=0x8b55a7
github.com/ollama/ollama/server.Serve({0x1203b6a8, 0xc000707720})
        github.com/ollama/ollama/server/routes.go:1059 +0x771 fp=0xc00006bc70 sp=0xc00006bae8 pc=0xf00871
github.com/ollama/ollama/cmd.RunServer(0xc000309200?, {0x128d1f20?, 0x4?, 0x1215292?})
        github.com/ollama/ollama/cmd/cmd.go:901 +0x177 fp=0xc00006bd58 sp=0xc00006bc70 pc=0xf1ac77
github.com/spf13/cobra.(*Command).execute(0xc00054af08, {0x128d1f20, 0x0, 0x0})
        github.com/spf13/cobra@v1.7.0/command.go:940 +0x882 fp=0xc00006be78 sp=0xc00006bd58 pc=0x7bb942
github.com/spf13/cobra.(*Command).ExecuteC(0xc00054a308)
        github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3a5 fp=0xc00006bf30 sp=0xc00006be78 pc=0x7bc185
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/cobra@v1.7.0/command.go:992
github.com/spf13/cobra.(*Command).ExecuteContext(...)
        github.com/spf13/cobra@v1.7.0/command.go:985
main.main()
        github.com/ollama/ollama/main.go:11 +0x4d fp=0xc00006bf50 sp=0xc00006bf30 pc=0xf2334d
runtime.main()
        runtime/proc.go:271 +0x29d fp=0xc00006bfe0 sp=0xc00006bf50 pc=0x46313d
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0x496361
 
goroutine 18 gp=0xc000202380 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc0001807a8 sp=0xc000180788 pc=0x46356e
runtime.goparkunlock(...)
        runtime/proc.go:408
runtime.forcegchelper()
        runtime/proc.go:326 +0xb3 fp=0xc0001807e0 sp=0xc0001807a8 pc=0x4633f3
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0001807e8 sp=0xc0001807e0 pc=0x496361
created by runtime.init.6 in goroutine 1
        runtime/proc.go:314 +0x1a
 
goroutine 19 gp=0xc000202540 m=nil [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000180f80 sp=0xc000180f60 pc=0x46356e
runtime.goparkunlock(...)
        runtime/proc.go:408
runtime.bgsweep(0xc000210000)
        runtime/mgcsweep.go:318 +0xdf fp=0xc000180fc8 sp=0xc000180f80 pc=0x44eb1f
runtime.gcenable.gowrap1()
        runtime/mgc.go:203 +0x25 fp=0xc000180fe0 sp=0xc000180fc8 pc=0x443405
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000180fe8 sp=0xc000180fe0 pc=0x496361
created by runtime.gcenable in goroutine 1
        runtime/mgc.go:203 +0x66
 
goroutine 20 gp=0xc000202700 m=nil [GC scavenge wait]:
runtime.gopark(0x1486b6e?, 0x13b140f?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000181778 sp=0xc000181758 pc=0x46356e
runtime.goparkunlock(...)
        runtime/proc.go:408
runtime.(*scavengerState).park(0x127a4d20)
        runtime/mgcscavenge.go:425 +0x49 fp=0xc0001817a8 sp=0xc000181778 pc=0x44c4a9
runtime.bgscavenge(0xc000210000)
        runtime/mgcscavenge.go:658 +0x59 fp=0xc0001817c8 sp=0xc0001817a8 pc=0x44ca59
runtime.gcenable.gowrap2()
        runtime/mgc.go:204 +0x25 fp=0xc0001817e0 sp=0xc0001817c8 pc=0x4433a5
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0001817e8 sp=0xc0001817e0 pc=0x496361
created by runtime.gcenable in goroutine 1
        runtime/mgc.go:204 +0xa5
Originally created by @7910f6ba7ee4 on GitHub (Jun 11, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/4982 Originally assigned to: @dhiltgen on GitHub. I'm running ollama using the binary provided in releases. When I try to run ollama serve, I get the error below. I have tried 0.1.43, 0.1.42, and older versions such as 0.1.34. I am using an AMD GPU, and do not have an NVIDIA, so I do not need the CUDA checks, but this is still happening with the OLLAMA_LLM_LIBRARY flag being set to something like cpu. Appears similar to previous issue https://github.com/ollama/ollama/issues/2198 Debug info excerpt below with OLLAMA_DEBUG=1 (there are too many goroutines to copy paste, but I can provide more) ``` time=2024-06-11T11:24:34.121-04:00 level=DEBUG source=gpu.go:307 msg="discovered GPU libraries" paths="[/tmp/ollama479946545/runners/cuda_v11/libcudart.so.11.0 /usr/lib64/libelf-0.185.so]" cudaSetDevice err: 35 time=2024-06-11T11:24:34.122-04:00 level=DEBUG source=gpu.go:319 msg="Unable to load cudart" library=/tmp/ollama479946545/runners/cuda_v11/libcudart.so.11.0 error="your nvidia driver is too old or missing. If you have a CUDA GPU please upgrade to run ollama" SIGSEGV: segmentation violation PC=0x0 m=18 sigcode=1 addr=0x0 signal arrived during cgo execution goroutine 1 gp=0xc0000061c0 m=18 mp=0xc000800808 [syscall]: runtime.cgocall(0xf241b0, 0xc00006b5a0) runtime/cgocall.go:157 +0x4b fp=0xc00006b578 sp=0xc00006b540 pc=0x42c7cb github.com/ollama/ollama/gpu._Cfunc_cudart_init(0x7f4c54000ca0, 0xc000616c60) _cgo_gotypes.go:301 +0x3f fp=0xc00006b5a0 sp=0xc00006b578 pc=0x8b4bdf github.com/ollama/ollama/gpu.LoadCUDARTMgmt.func2(0x7f4c54000ca0, 0xc000616c60) github.com/ollama/ollama/gpu/gpu.go:317 +0x4a fp=0xc00006b5d0 sp=0xc00006b5a0 pc=0x8b738a github.com/ollama/ollama/gpu.LoadCUDARTMgmt({0xc000706460, 0x2, 0xc00025a5b0?}) github.com/ollama/ollama/gpu/gpu.go:317 +0x23e fp=0xc00006b6c8 sp=0xc00006b5d0 pc=0x8b71de github.com/ollama/ollama/gpu.initGPUHandles() github.com/ollama/ollama/gpu/gpu.go:136 +0x236 fp=0xc00006b850 sp=0xc00006b6c8 pc=0x8b5256 github.com/ollama/ollama/gpu.GetGPUInfo() github.com/ollama/ollama/gpu/gpu.go:153 +0xa7 fp=0xc00006bae8 sp=0xc00006b850 pc=0x8b55a7 github.com/ollama/ollama/server.Serve({0x1203b6a8, 0xc000707720}) github.com/ollama/ollama/server/routes.go:1059 +0x771 fp=0xc00006bc70 sp=0xc00006bae8 pc=0xf00871 github.com/ollama/ollama/cmd.RunServer(0xc000309200?, {0x128d1f20?, 0x4?, 0x1215292?}) github.com/ollama/ollama/cmd/cmd.go:901 +0x177 fp=0xc00006bd58 sp=0xc00006bc70 pc=0xf1ac77 github.com/spf13/cobra.(*Command).execute(0xc00054af08, {0x128d1f20, 0x0, 0x0}) github.com/spf13/cobra@v1.7.0/command.go:940 +0x882 fp=0xc00006be78 sp=0xc00006bd58 pc=0x7bb942 github.com/spf13/cobra.(*Command).ExecuteC(0xc00054a308) github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3a5 fp=0xc00006bf30 sp=0xc00006be78 pc=0x7bc185 github.com/spf13/cobra.(*Command).Execute(...) github.com/spf13/cobra@v1.7.0/command.go:992 github.com/spf13/cobra.(*Command).ExecuteContext(...) github.com/spf13/cobra@v1.7.0/command.go:985 main.main() github.com/ollama/ollama/main.go:11 +0x4d fp=0xc00006bf50 sp=0xc00006bf30 pc=0xf2334d runtime.main() runtime/proc.go:271 +0x29d fp=0xc00006bfe0 sp=0xc00006bf50 pc=0x46313d runtime.goexit({}) runtime/asm_amd64.s:1695 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0x496361 goroutine 18 gp=0xc000202380 m=nil [force gc (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:402 +0xce fp=0xc0001807a8 sp=0xc000180788 pc=0x46356e runtime.goparkunlock(...) runtime/proc.go:408 runtime.forcegchelper() runtime/proc.go:326 +0xb3 fp=0xc0001807e0 sp=0xc0001807a8 pc=0x4633f3 runtime.goexit({}) runtime/asm_amd64.s:1695 +0x1 fp=0xc0001807e8 sp=0xc0001807e0 pc=0x496361 created by runtime.init.6 in goroutine 1 runtime/proc.go:314 +0x1a goroutine 19 gp=0xc000202540 m=nil [GC sweep wait]: runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:402 +0xce fp=0xc000180f80 sp=0xc000180f60 pc=0x46356e runtime.goparkunlock(...) runtime/proc.go:408 runtime.bgsweep(0xc000210000) runtime/mgcsweep.go:318 +0xdf fp=0xc000180fc8 sp=0xc000180f80 pc=0x44eb1f runtime.gcenable.gowrap1() runtime/mgc.go:203 +0x25 fp=0xc000180fe0 sp=0xc000180fc8 pc=0x443405 runtime.goexit({}) runtime/asm_amd64.s:1695 +0x1 fp=0xc000180fe8 sp=0xc000180fe0 pc=0x496361 created by runtime.gcenable in goroutine 1 runtime/mgc.go:203 +0x66 goroutine 20 gp=0xc000202700 m=nil [GC scavenge wait]: runtime.gopark(0x1486b6e?, 0x13b140f?, 0x0?, 0x0?, 0x0?) runtime/proc.go:402 +0xce fp=0xc000181778 sp=0xc000181758 pc=0x46356e runtime.goparkunlock(...) runtime/proc.go:408 runtime.(*scavengerState).park(0x127a4d20) runtime/mgcscavenge.go:425 +0x49 fp=0xc0001817a8 sp=0xc000181778 pc=0x44c4a9 runtime.bgscavenge(0xc000210000) runtime/mgcscavenge.go:658 +0x59 fp=0xc0001817c8 sp=0xc0001817a8 pc=0x44ca59 runtime.gcenable.gowrap2() runtime/mgc.go:204 +0x25 fp=0xc0001817e0 sp=0xc0001817c8 pc=0x4433a5 runtime.goexit({}) runtime/asm_amd64.s:1695 +0x1 fp=0xc0001817e8 sp=0xc0001817e0 pc=0x496361 created by runtime.gcenable in goroutine 1 runtime/mgc.go:204 +0xa5 ```
GiteaMirror added the bugamd labels 2026-04-28 12:33:21 -05:00
GiteaMirror commented 2026-04-28 12:33:24 -05:00
Author
Owner
Copy Link

@jmorganca commented on GitHub (Jun 12, 2024):

Hi @7910f6ba7ee4 do you know if you have the nvidia cuda drivers installed by chance? It seems they are being picked up. Sorry this happened

<!-- gh-comment-id:2162047920 --> @jmorganca commented on GitHub (Jun 12, 2024): Hi @7910f6ba7ee4 do you know if you have the nvidia cuda drivers installed by chance? It seems they are being picked up. Sorry this happened
GiteaMirror commented 2026-04-28 12:33:27 -05:00
Author
Owner
Copy Link

@7910f6ba7ee4 commented on GitHub (Jun 12, 2024):

I don't think so, as nvidia-smi results in a command not found, while rocm-smi is fine. I am on a remote server though without a lot of access, so I am both not entirely sure if the CUDA drivers are installed and probably unable to do anything if they are.

Is this intended behavior if it is trying to pick up a CUDA driver then?

<!-- gh-comment-id:2162788706 --> @7910f6ba7ee4 commented on GitHub (Jun 12, 2024): I don't think so, as `nvidia-smi` results in a `command not found`, while `rocm-smi` is fine. I am on a remote server though without a lot of access, so I am both not entirely sure if the CUDA drivers are installed and probably unable to do anything if they are. Is this intended behavior if it is trying to pick up a CUDA driver then?
GiteaMirror commented 2026-04-28 12:33:29 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Jun 13, 2024):

Based on the log output it appears you have a libcudart.so library someplace on your system that is a symbolic link to /usr/lib64/libelf-0.185.so and when we try to load that library as if it were a libcudart library, it's causing a crash. If you look a little earlier in the logs you should see a line gpu library search which indicates where we're looking which may help you find this broken link.

We shouldn't crash, so that's a bug, but to get past this, remove the link (if it's not required for some other app)

<!-- gh-comment-id:2166432122 --> @dhiltgen commented on GitHub (Jun 13, 2024): Based on the log output it appears you have a libcudart.so library someplace on your system that is a symbolic link to `/usr/lib64/libelf-0.185.so` and when we try to load that library as if it were a libcudart library, it's causing a crash. If you look a little earlier in the logs you should see a line `gpu library search` which indicates where we're looking which may help you find this broken link. We shouldn't crash, so that's a bug, but to get past this, remove the link (if it's not required for some other app)
GiteaMirror commented 2026-04-28 12:33:31 -05:00
Author
Owner
Copy Link

@7910f6ba7ee4 commented on GitHub (Jun 14, 2024):

Unfortunately since I am using a remote server, I do not have write access to a lot of files, so I can't remove the link. For anyone else that has this issue though, I was able to get it working by removing everything from the LoadCUDARTMgmt function in gpu/gpu.go except return 0, nil, "" and compiling to a new binary.

<!-- gh-comment-id:2168570054 --> @7910f6ba7ee4 commented on GitHub (Jun 14, 2024): Unfortunately since I am using a remote server, I do not have write access to a lot of files, so I can't remove the link. For anyone else that has this issue though, I was able to get it working by removing everything from the `LoadCUDARTMgmt` function in `gpu/gpu.go` except `return 0, nil, ""` and compiling to a new binary.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label amd bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#49661
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?