github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-06 08:02:14 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #2899] SIGSEGV: segmentation violation with 0.1.27 and starcoder2:!5b #1774

New Issue
Closed
opened 2026-04-12 11:47:42 -05:00 by GiteaMirror · 5 comments
GiteaMirror commented 2026-04-12 11:47:42 -05:00
Owner
Copy Link

Originally created by @joseluisamado on GitHub (Mar 3, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/2899

Hello, I am having an issue while running the recently released starcoder2 model. Please find below the details.

How to reproduce the issue

Running ollama in a container with Docker Compose

  ollama:
    image: ollama/ollama:0.1.27
    container_name: ollama
    restart: unless-stopped
    networks:
      - common
    volumes:
      - /home/ollama:/root/.ollama
    ports:
      - "11434:11434"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

Pull and run the model

$ docker exec ollama ollama pull starcoder2:15b
pulling manifest
pulling dc5deb763c38... 100% ▕████████████████▏ 9.1 GB
pulling 4ec42cd966c9... 100% ▕████████████████▏  12 KB
pulling 5671842f8d52... 100% ▕████████████████▏  346 B
verifying sha256 digest
writing manifest
removing any unused layers
success
$ docker exec ollama ollama run starcoder2:15b
Error: Post "http://0.0.0.0:11434/api/generate": EOF

The container gets killed. Here are the logs

time=2024-03-03T18:20:36.119Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-03-03T18:20:36.119Z level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.9"
time=2024-03-03T18:20:36.119Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-03-03T18:20:36.119Z level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.9"
time=2024-03-03T18:20:36.119Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-03-03T18:20:36.123Z level=INFO source=dyn_ext_server.go:90 msg="Loading Dynamic llm server: /tmp/ollama3819308268/cuda_v11/libext_server.so"
time=2024-03-03T18:20:36.123Z level=INFO source=dyn_ext_server.go:150 msg="Initializing llama server"
ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   yes
ggml_init_cublas: CUDA_USE_TENSOR_CORES: no
ggml_init_cublas: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 4080 SUPER, compute capability 8.9, VMM: yes
SIGSEGV: segmentation violation
PC=0x7fcb8138e7fd m=12 sigcode=1
signal arrived during cgo execution

goroutine 50 [syscall]:
runtime.cgocall(0x9bcdd0, 0xc0001446c8)
	/usr/local/go/src/runtime/cgocall.go:157 +0x4b fp=0xc0001446a0 sp=0xc000144668 pc=0x409b0b
github.com/jmorganca/ollama/llm._Cfunc_dyn_llama_server_init({0x7fcb0c001e10, 0x7fcb05250250, 0x7fcb05240ca0, 0x7fcb05243f20, 0x7fcb05253c00, 0x7fcb0524d840, 0x7fcb05243ba0, 0x7fcb05240d20, 0x7fcb05254500, 0x7fcb052537a0, ...}, ...)
	_cgo_gotypes.go:282 +0x45 fp=0xc0001446c8 sp=0xc0001446a0 pc=0x7c5485
github.com/jmorganca/ollama/llm.newDynExtServer.func7(0xaf20c4?, 0xc?)
	/go/src/github.com/jmorganca/ollama/llm/dyn_ext_server.go:153 +0xef fp=0xc0001447b8 sp=0xc0001446c8 pc=0x7c69cf
github.com/jmorganca/ollama/llm.newDynExtServer({0xc00069e000, 0x2f}, {0xc000002150, _}, {_, _, _}, {0x0, 0x0, 0x0}, ...)
	/go/src/github.com/jmorganca/ollama/llm/dyn_ext_server.go:153 +0xa65 fp=0xc000144a58 sp=0xc0001447b8 pc=0x7c6665
github.com/jmorganca/ollama/llm.newLlmServer({{_, _, _}, {_, _}, {_, _}}, {_, _}, {0xc000002150, ...}, ...)
	/go/src/github.com/jmorganca/ollama/llm/llm.go:158 +0x425 fp=0xc000144c18 sp=0xc000144a58 pc=0x7c2dc5
github.com/jmorganca/ollama/llm.New({0xc000282a50, 0x15}, {0xc000002150, _}, {_, _, _}, {0x0, 0x0, 0x0}, ...)
	/go/src/github.com/jmorganca/ollama/llm/llm.go:123 +0x713 fp=0xc000144e98 sp=0xc000144c18 pc=0x7c2733
github.com/jmorganca/ollama/server.load(0xc0000fe900?, 0xc0000fe900, {{0x0, 0x800, 0x200, 0x1, 0xffffffffffffffff, 0x0, 0x0, 0x1, ...}, ...}, ...)
	/go/src/github.com/jmorganca/ollama/server/routes.go:85 +0x3a5 fp=0xc000145018 sp=0xc000144e98 pc=0x996945
github.com/jmorganca/ollama/server.ChatHandler(0xc000408600)
	/go/src/github.com/jmorganca/ollama/server/routes.go:1173 +0xa37 fp=0xc000145748 sp=0xc000145018 pc=0x9a1f77
github.com/gin-gonic/gin.(*Context).Next(...)
	/root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/jmorganca/ollama/server.(*Server).GenerateRoutes.func1(0xc000408600)
	/go/src/github.com/jmorganca/ollama/server/routes.go:943 +0x68 fp=0xc000145780 sp=0xc000145748 pc=0x9a07a8
github.com/gin-gonic/gin.(*Context).Next(...)
	/root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/gin-gonic/gin.CustomRecoveryWithWriter.func1(0xc000408600)
	/root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/recovery.go:102 +0x7a fp=0xc0001457d0 sp=0xc000145780 pc=0x97803a
github.com/gin-gonic/gin.(*Context).Next(...)
	/root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/gin-gonic/gin.LoggerWithConfig.func1(0xc000408600)
	/root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/logger.go:240 +0xde fp=0xc000145980 sp=0xc0001457d0 pc=0x9771de
github.com/gin-gonic/gin.(*Context).Next(...)
	/root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/gin-gonic/gin.(*Engine).handleHTTPRequest(0xc0003f64e0, 0xc000408600)
	/root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:620 +0x65b fp=0xc000145b08 sp=0xc000145980 pc=0x97629b
github.com/gin-gonic/gin.(*Engine).ServeHTTP(0xc0003f64e0, {0x11403a20?, 0xc00011c0e0}, 0xc000408500)
	/root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:576 +0x1dd fp=0xc000145b48 sp=0xc000145b08 pc=0x975a5d
net/http.serverHandler.ServeHTTP({0x11401d40?}, {0x11403a20?, 0xc00011c0e0?}, 0x6?)
	/usr/local/go/src/net/http/server.go:2938 +0x8e fp=0xc000145b78 sp=0xc000145b48 pc=0x6ced4e
net/http.(*conn).serve(0xc0000fc1b0, {0x11405088, 0xc0001542a0})
	/usr/local/go/src/net/http/server.go:2009 +0x5f4 fp=0xc000145fb8 sp=0xc000145b78 pc=0x6cac34
net/http.(*Server).Serve.func3()
	/usr/local/go/src/net/http/server.go:3086 +0x28 fp=0xc000145fe0 sp=0xc000145fb8 pc=0x6cf568
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000145fe8 sp=0xc000145fe0 pc=0x46e2c1
created by net/http.(*Server).Serve in goroutine 1
	/usr/local/go/src/net/http/server.go:3086 +0x5cb

goroutine 1 [IO wait]:
runtime.gopark(0x480f10?, 0xc0000e3850?, 0xa0?, 0x38?, 0x4f711d?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0000e3830 sp=0xc0000e3810 pc=0x43e7ee
runtime.netpollblock(0x0?, 0x4092a6?, 0x0?)
	/usr/local/go/src/runtime/netpoll.go:564 +0xf7 fp=0xc0000e3868 sp=0xc0000e3830 pc=0x437277
internal/poll.runtime_pollWait(0x7fcb38dfde28, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85 fp=0xc0000e3888 sp=0xc0000e3868 pc=0x468a05
internal/poll.(*pollDesc).wait(0xc00040a000?, 0x11846a68?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0000e38b0 sp=0xc0000e3888 pc=0x4efd67
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc00040a000)
	/usr/local/go/src/internal/poll/fd_unix.go:611 +0x2ac fp=0xc0000e3958 sp=0xc0000e38b0 pc=0x4f524c
net.(*netFD).accept(0xc00040a000)
	/usr/local/go/src/net/fd_unix.go:172 +0x29 fp=0xc0000e3a10 sp=0xc0000e3958 pc=0x56be29
net.(*TCPListener).accept(0xc0003e1560)
	/usr/local/go/src/net/tcpsock_posix.go:152 +0x1e fp=0xc0000e3a38 sp=0xc0000e3a10 pc=0x580c3e
net.(*TCPListener).Accept(0xc0003e1560)
	/usr/local/go/src/net/tcpsock.go:315 +0x30 fp=0xc0000e3a68 sp=0xc0000e3a38 pc=0x57fdf0
net/http.(*onceCloseListener).Accept(0xc0000fc1b0?)
	<autogenerated>:1 +0x24 fp=0xc0000e3a80 sp=0xc0000e3a68 pc=0x6f1ae4
net/http.(*Server).Serve(0xc000318ff0, {0x11403810, 0xc0003e1560})
	/usr/local/go/src/net/http/server.go:3056 +0x364 fp=0xc0000e3bb0 sp=0xc0000e3a80 pc=0x6cf1a4
github.com/jmorganca/ollama/server.Serve({0x11403810, 0xc0003e1560})
	/go/src/github.com/jmorganca/ollama/server/routes.go:1046 +0x454 fp=0xc0000e3c98 sp=0xc0000e3bb0 pc=0x9a0c54
github.com/jmorganca/ollama/cmd.RunServer(0xc000408300?, {0x1184c8c0?, 0x4?, 0xad9d6a?})
	/go/src/github.com/jmorganca/ollama/cmd/cmd.go:706 +0x1b9 fp=0xc0000e3d30 sp=0xc0000e3c98 pc=0x9b3d99
github.com/spf13/cobra.(*Command).execute(0xc0003a1800, {0x1184c8c0, 0x0, 0x0})
	/root/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940 +0x87c fp=0xc0000e3e68 sp=0xc0000e3d30 pc=0x764d9c
github.com/spf13/cobra.(*Command).ExecuteC(0xc0003a0c00)
	/root/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3a5 fp=0xc0000e3f20 sp=0xc0000e3e68 pc=0x7655c5
github.com/spf13/cobra.(*Command).Execute(...)
	/root/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992
github.com/spf13/cobra.(*Command).ExecuteContext(...)
	/root/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:985
main.main()
	/go/src/github.com/jmorganca/ollama/main.go:11 +0x4d fp=0xc0000e3f40 sp=0xc0000e3f20 pc=0x9bbeed
runtime.main()
	/usr/local/go/src/runtime/proc.go:267 +0x2bb fp=0xc0000e3fe0 sp=0xc0000e3f40 pc=0x43e39b
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000e3fe8 sp=0xc0000e3fe0 pc=0x46e2c1

goroutine 2 [force gc (idle), 2 minutes]:
runtime.gopark(0x16e8d765f29c?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000046fa8 sp=0xc000046f88 pc=0x43e7ee
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:404
runtime.forcegchelper()
	/usr/local/go/src/runtime/proc.go:322 +0xb3 fp=0xc000046fe0 sp=0xc000046fa8 pc=0x43e673
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000046fe8 sp=0xc000046fe0 pc=0x46e2c1
created by runtime.init.6 in goroutine 1
	/usr/local/go/src/runtime/proc.go:310 +0x1a

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000047778 sp=0xc000047758 pc=0x43e7ee
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:404
runtime.bgsweep(0x0?)
	/usr/local/go/src/runtime/mgcsweep.go:321 +0xdf fp=0xc0000477c8 sp=0xc000047778 pc=0x42a73f
runtime.gcenable.func1()
	/usr/local/go/src/runtime/mgc.go:200 +0x25 fp=0xc0000477e0 sp=0xc0000477c8 pc=0x41f865
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000477e8 sp=0xc0000477e0 pc=0x46e2c1
created by runtime.gcenable in goroutine 1
	/usr/local/go/src/runtime/mgc.go:200 +0x66

goroutine 4 [GC scavenge wait]:
runtime.gopark(0xe910d?, 0x16176886?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000047f70 sp=0xc000047f50 pc=0x43e7ee
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:404
runtime.(*scavengerState).park(0x1181cc40)
	/usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000047fa0 sp=0xc000047f70 pc=0x427f69
runtime.bgscavenge(0x0?)
	/usr/local/go/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc000047fc8 sp=0xc000047fa0 pc=0x428519
runtime.gcenable.func2()
	/usr/local/go/src/runtime/mgc.go:201 +0x25 fp=0xc000047fe0 sp=0xc000047fc8 pc=0x41f805
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000047fe8 sp=0xc000047fe0 pc=0x46e2c1
created by runtime.gcenable in goroutine 1
	/usr/local/go/src/runtime/mgc.go:201 +0xa5

goroutine 18 [finalizer wait, 6 minutes]:
runtime.gopark(0xad2d20?, 0x10043f901?, 0x0?, 0x0?, 0x4469a5?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000046628 sp=0xc000046608 pc=0x43e7ee
runtime.runfinq()
	/usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc0000467e0 sp=0xc000046628 pc=0x41e8e7
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000467e8 sp=0xc0000467e0 pc=0x46e2c1
created by runtime.createfing in goroutine 1
	/usr/local/go/src/runtime/mfinal.go:163 +0x3d

goroutine 19 [select, 6 minutes, locked to thread]:
runtime.gopark(0xc0000427a8?, 0x2?, 0x89?, 0xea?, 0xc0000427a4?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000042638 sp=0xc000042618 pc=0x43e7ee
runtime.selectgo(0xc0000427a8, 0xc0000427a0, 0x0?, 0x0, 0x0?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000042758 sp=0xc000042638 pc=0x44e325
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal_unix.go:1014 +0x19f fp=0xc0000427e0 sp=0xc000042758 pc=0x46535f
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000427e8 sp=0xc0000427e0 pc=0x46e2c1
created by runtime.ensureSigM in goroutine 1
	/usr/local/go/src/runtime/signal_unix.go:997 +0xc8

goroutine 5 [syscall, 6 minutes]:
runtime.notetsleepg(0x0?, 0x0?)
	/usr/local/go/src/runtime/lock_futex.go:236 +0x29 fp=0xc0000487a0 sp=0xc000048768 pc=0x411349
os/signal.signal_recv()
	/usr/local/go/src/runtime/sigqueue.go:152 +0x29 fp=0xc0000487c0 sp=0xc0000487a0 pc=0x46ac89
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x13 fp=0xc0000487e0 sp=0xc0000487c0 pc=0x6f4513
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000487e8 sp=0xc0000487e0 pc=0x46e2c1
created by os/signal.Notify.func1.1 in goroutine 1
	/usr/local/go/src/os/signal/signal.go:151 +0x1f

goroutine 20 [chan receive, 6 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000042f18 sp=0xc000042ef8 pc=0x43e7ee
runtime.chanrecv(0xc000436120, 0x0, 0x1)
	/usr/local/go/src/runtime/chan.go:583 +0x3cd fp=0xc000042f90 sp=0xc000042f18 pc=0x40beed
runtime.chanrecv1(0x0?, 0x0?)
	/usr/local/go/src/runtime/chan.go:442 +0x12 fp=0xc000042fb8 sp=0xc000042f90 pc=0x40baf2
github.com/jmorganca/ollama/server.Serve.func2()
	/go/src/github.com/jmorganca/ollama/server/routes.go:1028 +0x25 fp=0xc000042fe0 sp=0xc000042fb8 pc=0x9a0ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000042fe8 sp=0xc000042fe0 pc=0x46e2c1
created by github.com/jmorganca/ollama/server.Serve in goroutine 1
	/go/src/github.com/jmorganca/ollama/server/routes.go:1027 +0x3c7

goroutine 30 [GC worker (idle)]:
runtime.gopark(0x16f2d48eda1f?, 0x3?, 0x9d?, 0x72?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00043d750 sp=0xc00043d730 pc=0x43e7ee
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc00043d7e0 sp=0xc00043d750 pc=0x4213e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00043d7e8 sp=0xc00043d7e0 pc=0x46e2c1
created by runtime.gcBgMarkStartWorkers in goroutine 29
	/usr/local/go/src/runtime/mgc.go:1217 +0x1c

goroutine 31 [GC worker (idle)]:
runtime.gopark(0x16f2d6231cd4?, 0x3?, 0xf7?, 0x20?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00043df50 sp=0xc00043df30 pc=0x43e7ee
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc00043dfe0 sp=0xc00043df50 pc=0x4213e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00043dfe8 sp=0xc00043dfe0 pc=0x46e2c1
created by runtime.gcBgMarkStartWorkers in goroutine 29
	/usr/local/go/src/runtime/mgc.go:1217 +0x1c

goroutine 32 [GC worker (idle)]:
runtime.gopark(0x16f2d6231cb1?, 0x3?, 0x7?, 0xb?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00043e750 sp=0xc00043e730 pc=0x43e7ee
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc00043e7e0 sp=0xc00043e750 pc=0x4213e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00043e7e8 sp=0xc00043e7e0 pc=0x46e2c1
created by runtime.gcBgMarkStartWorkers in goroutine 29
	/usr/local/go/src/runtime/mgc.go:1217 +0x1c

goroutine 34 [GC worker (idle)]:
runtime.gopark(0x16f2d6b56164?, 0x2?, 0x55?, 0x12?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000438750 sp=0xc000438730 pc=0x43e7ee
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc0004387e0 sp=0xc000438750 pc=0x4213e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004387e8 sp=0xc0004387e0 pc=0x46e2c1
created by runtime.gcBgMarkStartWorkers in goroutine 29
	/usr/local/go/src/runtime/mgc.go:1217 +0x1c

goroutine 33 [GC worker (idle)]:
runtime.gopark(0x16f2d6b56114?, 0x2?, 0x17?, 0x8f?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00043ef50 sp=0xc00043ef30 pc=0x43e7ee
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc00043efe0 sp=0xc00043ef50 pc=0x4213e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00043efe8 sp=0xc00043efe0 pc=0x46e2c1
created by runtime.gcBgMarkStartWorkers in goroutine 29
	/usr/local/go/src/runtime/mgc.go:1217 +0x1c

goroutine 35 [GC worker (idle)]:
runtime.gopark(0x16f2d6b5645a?, 0x3?, 0xc1?, 0xd?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000438f50 sp=0xc000438f30 pc=0x43e7ee
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc000438fe0 sp=0xc000438f50 pc=0x4213e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000438fe8 sp=0xc000438fe0 pc=0x46e2c1
created by runtime.gcBgMarkStartWorkers in goroutine 29
	/usr/local/go/src/runtime/mgc.go:1217 +0x1c

goroutine 51 [IO wait]:
runtime.gopark(0x11410780?, 0xb?, 0x0?, 0x0?, 0xc?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00043cda0 sp=0xc00043cd80 pc=0x43e7ee
runtime.netpollblock(0x47f078?, 0x4092a6?, 0x0?)
	/usr/local/go/src/runtime/netpoll.go:564 +0xf7 fp=0xc00043cdd8 sp=0xc00043cda0 pc=0x437277
internal/poll.runtime_pollWait(0x7fcb38dfdd30, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85 fp=0xc00043cdf8 sp=0xc00043cdd8 pc=0x468a05
internal/poll.(*pollDesc).wait(0xc000126000?, 0xc000154521?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc00043ce20 sp=0xc00043cdf8 pc=0x4efd67
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000126000, {0xc000154521, 0x1, 0x1})
	/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a fp=0xc00043ceb8 sp=0xc00043ce20 pc=0x4f105a
net.(*netFD).Read(0xc000126000, {0xc000154521?, 0x0?, 0xc000282a50?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25 fp=0xc00043cf00 sp=0xc00043ceb8 pc=0x569e05
net.(*conn).Read(0xc0004aa008, {0xc000154521?, 0x0?, 0xc00043cee0?})
	/usr/local/go/src/net/net.go:179 +0x45 fp=0xc00043cf48 sp=0xc00043cf00 pc=0x5780a5
net.(*TCPConn).Read(0x0?, {0xc000154521?, 0x100000000000000?, 0xc0004342c0?})
	<autogenerated>:1 +0x25 fp=0xc00043cf78 sp=0xc00043cf48 pc=0x589fa5
net/http.(*connReader).backgroundRead(0xc000154510)
	/usr/local/go/src/net/http/server.go:683 +0x37 fp=0xc00043cfc8 sp=0xc00043cf78 pc=0x6c4ab7
net/http.(*connReader).startBackgroundRead.func2()
	/usr/local/go/src/net/http/server.go:679 +0x25 fp=0xc00043cfe0 sp=0xc00043cfc8 pc=0x6c49e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00043cfe8 sp=0xc00043cfe0 pc=0x46e2c1
created by net/http.(*connReader).startBackgroundRead in goroutine 50
	/usr/local/go/src/net/http/server.go:679 +0xba

rax    0x0
rbx    0x7fcb28ff8430
rcx    0x67
rdx    0x0
rdi    0x0
rsi    0x0
rbp    0x7fcb28ff83f0
rsp    0x7fcb28ff8258
r8     0x7fca482cfb00
r9     0x7fca482cfb28
r10    0x7fcb81200b40
r11    0x7fcb8138aa80
r12    0x0
r13    0x7fcb0c2026a8
r14    0x7fcb0c2044a0
r15    0x0
rip    0x7fcb8138e7fd
rflags 0x10283
cs     0x33
fs     0x0
gs     0x0

Thanks!

Originally created by @joseluisamado on GitHub (Mar 3, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/2899 Hello, I am having an issue while running the recently released `starcoder2` model. Please find below the details. **How to reproduce the issue** Running ollama in a container with Docker Compose ``` ollama: image: ollama/ollama:0.1.27 container_name: ollama restart: unless-stopped networks: - common volumes: - /home/ollama:/root/.ollama ports: - "11434:11434" deploy: resources: reservations: devices: - driver: nvidia count: 1 capabilities: [gpu] ``` Pull and run the model ``` $ docker exec ollama ollama pull starcoder2:15b pulling manifest pulling dc5deb763c38... 100% ▕████████████████▏ 9.1 GB pulling 4ec42cd966c9... 100% ▕████████████████▏ 12 KB pulling 5671842f8d52... 100% ▕████████████████▏ 346 B verifying sha256 digest writing manifest removing any unused layers success $ docker exec ollama ollama run starcoder2:15b Error: Post "http://0.0.0.0:11434/api/generate": EOF ``` The container gets killed. Here are the logs ``` time=2024-03-03T18:20:36.119Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2" time=2024-03-03T18:20:36.119Z level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.9" time=2024-03-03T18:20:36.119Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2" time=2024-03-03T18:20:36.119Z level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.9" time=2024-03-03T18:20:36.119Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2" time=2024-03-03T18:20:36.123Z level=INFO source=dyn_ext_server.go:90 msg="Loading Dynamic llm server: /tmp/ollama3819308268/cuda_v11/libext_server.so" time=2024-03-03T18:20:36.123Z level=INFO source=dyn_ext_server.go:150 msg="Initializing llama server" ggml_init_cublas: GGML_CUDA_FORCE_MMQ: yes ggml_init_cublas: CUDA_USE_TENSOR_CORES: no ggml_init_cublas: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 4080 SUPER, compute capability 8.9, VMM: yes SIGSEGV: segmentation violation PC=0x7fcb8138e7fd m=12 sigcode=1 signal arrived during cgo execution goroutine 50 [syscall]: runtime.cgocall(0x9bcdd0, 0xc0001446c8) /usr/local/go/src/runtime/cgocall.go:157 +0x4b fp=0xc0001446a0 sp=0xc000144668 pc=0x409b0b github.com/jmorganca/ollama/llm._Cfunc_dyn_llama_server_init({0x7fcb0c001e10, 0x7fcb05250250, 0x7fcb05240ca0, 0x7fcb05243f20, 0x7fcb05253c00, 0x7fcb0524d840, 0x7fcb05243ba0, 0x7fcb05240d20, 0x7fcb05254500, 0x7fcb052537a0, ...}, ...) _cgo_gotypes.go:282 +0x45 fp=0xc0001446c8 sp=0xc0001446a0 pc=0x7c5485 github.com/jmorganca/ollama/llm.newDynExtServer.func7(0xaf20c4?, 0xc?) /go/src/github.com/jmorganca/ollama/llm/dyn_ext_server.go:153 +0xef fp=0xc0001447b8 sp=0xc0001446c8 pc=0x7c69cf github.com/jmorganca/ollama/llm.newDynExtServer({0xc00069e000, 0x2f}, {0xc000002150, _}, {_, _, _}, {0x0, 0x0, 0x0}, ...) /go/src/github.com/jmorganca/ollama/llm/dyn_ext_server.go:153 +0xa65 fp=0xc000144a58 sp=0xc0001447b8 pc=0x7c6665 github.com/jmorganca/ollama/llm.newLlmServer({{_, _, _}, {_, _}, {_, _}}, {_, _}, {0xc000002150, ...}, ...) /go/src/github.com/jmorganca/ollama/llm/llm.go:158 +0x425 fp=0xc000144c18 sp=0xc000144a58 pc=0x7c2dc5 github.com/jmorganca/ollama/llm.New({0xc000282a50, 0x15}, {0xc000002150, _}, {_, _, _}, {0x0, 0x0, 0x0}, ...) /go/src/github.com/jmorganca/ollama/llm/llm.go:123 +0x713 fp=0xc000144e98 sp=0xc000144c18 pc=0x7c2733 github.com/jmorganca/ollama/server.load(0xc0000fe900?, 0xc0000fe900, {{0x0, 0x800, 0x200, 0x1, 0xffffffffffffffff, 0x0, 0x0, 0x1, ...}, ...}, ...) /go/src/github.com/jmorganca/ollama/server/routes.go:85 +0x3a5 fp=0xc000145018 sp=0xc000144e98 pc=0x996945 github.com/jmorganca/ollama/server.ChatHandler(0xc000408600) /go/src/github.com/jmorganca/ollama/server/routes.go:1173 +0xa37 fp=0xc000145748 sp=0xc000145018 pc=0x9a1f77 github.com/gin-gonic/gin.(*Context).Next(...) /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 github.com/jmorganca/ollama/server.(*Server).GenerateRoutes.func1(0xc000408600) /go/src/github.com/jmorganca/ollama/server/routes.go:943 +0x68 fp=0xc000145780 sp=0xc000145748 pc=0x9a07a8 github.com/gin-gonic/gin.(*Context).Next(...) /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 github.com/gin-gonic/gin.CustomRecoveryWithWriter.func1(0xc000408600) /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/recovery.go:102 +0x7a fp=0xc0001457d0 sp=0xc000145780 pc=0x97803a github.com/gin-gonic/gin.(*Context).Next(...) /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 github.com/gin-gonic/gin.LoggerWithConfig.func1(0xc000408600) /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/logger.go:240 +0xde fp=0xc000145980 sp=0xc0001457d0 pc=0x9771de github.com/gin-gonic/gin.(*Context).Next(...) /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 github.com/gin-gonic/gin.(*Engine).handleHTTPRequest(0xc0003f64e0, 0xc000408600) /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:620 +0x65b fp=0xc000145b08 sp=0xc000145980 pc=0x97629b github.com/gin-gonic/gin.(*Engine).ServeHTTP(0xc0003f64e0, {0x11403a20?, 0xc00011c0e0}, 0xc000408500) /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:576 +0x1dd fp=0xc000145b48 sp=0xc000145b08 pc=0x975a5d net/http.serverHandler.ServeHTTP({0x11401d40?}, {0x11403a20?, 0xc00011c0e0?}, 0x6?) /usr/local/go/src/net/http/server.go:2938 +0x8e fp=0xc000145b78 sp=0xc000145b48 pc=0x6ced4e net/http.(*conn).serve(0xc0000fc1b0, {0x11405088, 0xc0001542a0}) /usr/local/go/src/net/http/server.go:2009 +0x5f4 fp=0xc000145fb8 sp=0xc000145b78 pc=0x6cac34 net/http.(*Server).Serve.func3() /usr/local/go/src/net/http/server.go:3086 +0x28 fp=0xc000145fe0 sp=0xc000145fb8 pc=0x6cf568 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000145fe8 sp=0xc000145fe0 pc=0x46e2c1 created by net/http.(*Server).Serve in goroutine 1 /usr/local/go/src/net/http/server.go:3086 +0x5cb goroutine 1 [IO wait]: runtime.gopark(0x480f10?, 0xc0000e3850?, 0xa0?, 0x38?, 0x4f711d?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0000e3830 sp=0xc0000e3810 pc=0x43e7ee runtime.netpollblock(0x0?, 0x4092a6?, 0x0?) /usr/local/go/src/runtime/netpoll.go:564 +0xf7 fp=0xc0000e3868 sp=0xc0000e3830 pc=0x437277 internal/poll.runtime_pollWait(0x7fcb38dfde28, 0x72) /usr/local/go/src/runtime/netpoll.go:343 +0x85 fp=0xc0000e3888 sp=0xc0000e3868 pc=0x468a05 internal/poll.(*pollDesc).wait(0xc00040a000?, 0x11846a68?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0000e38b0 sp=0xc0000e3888 pc=0x4efd67 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Accept(0xc00040a000) /usr/local/go/src/internal/poll/fd_unix.go:611 +0x2ac fp=0xc0000e3958 sp=0xc0000e38b0 pc=0x4f524c net.(*netFD).accept(0xc00040a000) /usr/local/go/src/net/fd_unix.go:172 +0x29 fp=0xc0000e3a10 sp=0xc0000e3958 pc=0x56be29 net.(*TCPListener).accept(0xc0003e1560) /usr/local/go/src/net/tcpsock_posix.go:152 +0x1e fp=0xc0000e3a38 sp=0xc0000e3a10 pc=0x580c3e net.(*TCPListener).Accept(0xc0003e1560) /usr/local/go/src/net/tcpsock.go:315 +0x30 fp=0xc0000e3a68 sp=0xc0000e3a38 pc=0x57fdf0 net/http.(*onceCloseListener).Accept(0xc0000fc1b0?) <autogenerated>:1 +0x24 fp=0xc0000e3a80 sp=0xc0000e3a68 pc=0x6f1ae4 net/http.(*Server).Serve(0xc000318ff0, {0x11403810, 0xc0003e1560}) /usr/local/go/src/net/http/server.go:3056 +0x364 fp=0xc0000e3bb0 sp=0xc0000e3a80 pc=0x6cf1a4 github.com/jmorganca/ollama/server.Serve({0x11403810, 0xc0003e1560}) /go/src/github.com/jmorganca/ollama/server/routes.go:1046 +0x454 fp=0xc0000e3c98 sp=0xc0000e3bb0 pc=0x9a0c54 github.com/jmorganca/ollama/cmd.RunServer(0xc000408300?, {0x1184c8c0?, 0x4?, 0xad9d6a?}) /go/src/github.com/jmorganca/ollama/cmd/cmd.go:706 +0x1b9 fp=0xc0000e3d30 sp=0xc0000e3c98 pc=0x9b3d99 github.com/spf13/cobra.(*Command).execute(0xc0003a1800, {0x1184c8c0, 0x0, 0x0}) /root/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940 +0x87c fp=0xc0000e3e68 sp=0xc0000e3d30 pc=0x764d9c github.com/spf13/cobra.(*Command).ExecuteC(0xc0003a0c00) /root/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3a5 fp=0xc0000e3f20 sp=0xc0000e3e68 pc=0x7655c5 github.com/spf13/cobra.(*Command).Execute(...) /root/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992 github.com/spf13/cobra.(*Command).ExecuteContext(...) /root/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:985 main.main() /go/src/github.com/jmorganca/ollama/main.go:11 +0x4d fp=0xc0000e3f40 sp=0xc0000e3f20 pc=0x9bbeed runtime.main() /usr/local/go/src/runtime/proc.go:267 +0x2bb fp=0xc0000e3fe0 sp=0xc0000e3f40 pc=0x43e39b runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000e3fe8 sp=0xc0000e3fe0 pc=0x46e2c1 goroutine 2 [force gc (idle), 2 minutes]: runtime.gopark(0x16e8d765f29c?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000046fa8 sp=0xc000046f88 pc=0x43e7ee runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:404 runtime.forcegchelper() /usr/local/go/src/runtime/proc.go:322 +0xb3 fp=0xc000046fe0 sp=0xc000046fa8 pc=0x43e673 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000046fe8 sp=0xc000046fe0 pc=0x46e2c1 created by runtime.init.6 in goroutine 1 /usr/local/go/src/runtime/proc.go:310 +0x1a goroutine 3 [GC sweep wait]: runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000047778 sp=0xc000047758 pc=0x43e7ee runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:404 runtime.bgsweep(0x0?) /usr/local/go/src/runtime/mgcsweep.go:321 +0xdf fp=0xc0000477c8 sp=0xc000047778 pc=0x42a73f runtime.gcenable.func1() /usr/local/go/src/runtime/mgc.go:200 +0x25 fp=0xc0000477e0 sp=0xc0000477c8 pc=0x41f865 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000477e8 sp=0xc0000477e0 pc=0x46e2c1 created by runtime.gcenable in goroutine 1 /usr/local/go/src/runtime/mgc.go:200 +0x66 goroutine 4 [GC scavenge wait]: runtime.gopark(0xe910d?, 0x16176886?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000047f70 sp=0xc000047f50 pc=0x43e7ee runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:404 runtime.(*scavengerState).park(0x1181cc40) /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000047fa0 sp=0xc000047f70 pc=0x427f69 runtime.bgscavenge(0x0?) /usr/local/go/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc000047fc8 sp=0xc000047fa0 pc=0x428519 runtime.gcenable.func2() /usr/local/go/src/runtime/mgc.go:201 +0x25 fp=0xc000047fe0 sp=0xc000047fc8 pc=0x41f805 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000047fe8 sp=0xc000047fe0 pc=0x46e2c1 created by runtime.gcenable in goroutine 1 /usr/local/go/src/runtime/mgc.go:201 +0xa5 goroutine 18 [finalizer wait, 6 minutes]: runtime.gopark(0xad2d20?, 0x10043f901?, 0x0?, 0x0?, 0x4469a5?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000046628 sp=0xc000046608 pc=0x43e7ee runtime.runfinq() /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc0000467e0 sp=0xc000046628 pc=0x41e8e7 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000467e8 sp=0xc0000467e0 pc=0x46e2c1 created by runtime.createfing in goroutine 1 /usr/local/go/src/runtime/mfinal.go:163 +0x3d goroutine 19 [select, 6 minutes, locked to thread]: runtime.gopark(0xc0000427a8?, 0x2?, 0x89?, 0xea?, 0xc0000427a4?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000042638 sp=0xc000042618 pc=0x43e7ee runtime.selectgo(0xc0000427a8, 0xc0000427a0, 0x0?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000042758 sp=0xc000042638 pc=0x44e325 runtime.ensureSigM.func1() /usr/local/go/src/runtime/signal_unix.go:1014 +0x19f fp=0xc0000427e0 sp=0xc000042758 pc=0x46535f runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000427e8 sp=0xc0000427e0 pc=0x46e2c1 created by runtime.ensureSigM in goroutine 1 /usr/local/go/src/runtime/signal_unix.go:997 +0xc8 goroutine 5 [syscall, 6 minutes]: runtime.notetsleepg(0x0?, 0x0?) /usr/local/go/src/runtime/lock_futex.go:236 +0x29 fp=0xc0000487a0 sp=0xc000048768 pc=0x411349 os/signal.signal_recv() /usr/local/go/src/runtime/sigqueue.go:152 +0x29 fp=0xc0000487c0 sp=0xc0000487a0 pc=0x46ac89 os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:23 +0x13 fp=0xc0000487e0 sp=0xc0000487c0 pc=0x6f4513 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000487e8 sp=0xc0000487e0 pc=0x46e2c1 created by os/signal.Notify.func1.1 in goroutine 1 /usr/local/go/src/os/signal/signal.go:151 +0x1f goroutine 20 [chan receive, 6 minutes]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000042f18 sp=0xc000042ef8 pc=0x43e7ee runtime.chanrecv(0xc000436120, 0x0, 0x1) /usr/local/go/src/runtime/chan.go:583 +0x3cd fp=0xc000042f90 sp=0xc000042f18 pc=0x40beed runtime.chanrecv1(0x0?, 0x0?) /usr/local/go/src/runtime/chan.go:442 +0x12 fp=0xc000042fb8 sp=0xc000042f90 pc=0x40baf2 github.com/jmorganca/ollama/server.Serve.func2() /go/src/github.com/jmorganca/ollama/server/routes.go:1028 +0x25 fp=0xc000042fe0 sp=0xc000042fb8 pc=0x9a0ce5 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000042fe8 sp=0xc000042fe0 pc=0x46e2c1 created by github.com/jmorganca/ollama/server.Serve in goroutine 1 /go/src/github.com/jmorganca/ollama/server/routes.go:1027 +0x3c7 goroutine 30 [GC worker (idle)]: runtime.gopark(0x16f2d48eda1f?, 0x3?, 0x9d?, 0x72?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00043d750 sp=0xc00043d730 pc=0x43e7ee runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc00043d7e0 sp=0xc00043d750 pc=0x4213e5 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00043d7e8 sp=0xc00043d7e0 pc=0x46e2c1 created by runtime.gcBgMarkStartWorkers in goroutine 29 /usr/local/go/src/runtime/mgc.go:1217 +0x1c goroutine 31 [GC worker (idle)]: runtime.gopark(0x16f2d6231cd4?, 0x3?, 0xf7?, 0x20?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00043df50 sp=0xc00043df30 pc=0x43e7ee runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc00043dfe0 sp=0xc00043df50 pc=0x4213e5 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00043dfe8 sp=0xc00043dfe0 pc=0x46e2c1 created by runtime.gcBgMarkStartWorkers in goroutine 29 /usr/local/go/src/runtime/mgc.go:1217 +0x1c goroutine 32 [GC worker (idle)]: runtime.gopark(0x16f2d6231cb1?, 0x3?, 0x7?, 0xb?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00043e750 sp=0xc00043e730 pc=0x43e7ee runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc00043e7e0 sp=0xc00043e750 pc=0x4213e5 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00043e7e8 sp=0xc00043e7e0 pc=0x46e2c1 created by runtime.gcBgMarkStartWorkers in goroutine 29 /usr/local/go/src/runtime/mgc.go:1217 +0x1c goroutine 34 [GC worker (idle)]: runtime.gopark(0x16f2d6b56164?, 0x2?, 0x55?, 0x12?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000438750 sp=0xc000438730 pc=0x43e7ee runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc0004387e0 sp=0xc000438750 pc=0x4213e5 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004387e8 sp=0xc0004387e0 pc=0x46e2c1 created by runtime.gcBgMarkStartWorkers in goroutine 29 /usr/local/go/src/runtime/mgc.go:1217 +0x1c goroutine 33 [GC worker (idle)]: runtime.gopark(0x16f2d6b56114?, 0x2?, 0x17?, 0x8f?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00043ef50 sp=0xc00043ef30 pc=0x43e7ee runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc00043efe0 sp=0xc00043ef50 pc=0x4213e5 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00043efe8 sp=0xc00043efe0 pc=0x46e2c1 created by runtime.gcBgMarkStartWorkers in goroutine 29 /usr/local/go/src/runtime/mgc.go:1217 +0x1c goroutine 35 [GC worker (idle)]: runtime.gopark(0x16f2d6b5645a?, 0x3?, 0xc1?, 0xd?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000438f50 sp=0xc000438f30 pc=0x43e7ee runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc000438fe0 sp=0xc000438f50 pc=0x4213e5 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000438fe8 sp=0xc000438fe0 pc=0x46e2c1 created by runtime.gcBgMarkStartWorkers in goroutine 29 /usr/local/go/src/runtime/mgc.go:1217 +0x1c goroutine 51 [IO wait]: runtime.gopark(0x11410780?, 0xb?, 0x0?, 0x0?, 0xc?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00043cda0 sp=0xc00043cd80 pc=0x43e7ee runtime.netpollblock(0x47f078?, 0x4092a6?, 0x0?) /usr/local/go/src/runtime/netpoll.go:564 +0xf7 fp=0xc00043cdd8 sp=0xc00043cda0 pc=0x437277 internal/poll.runtime_pollWait(0x7fcb38dfdd30, 0x72) /usr/local/go/src/runtime/netpoll.go:343 +0x85 fp=0xc00043cdf8 sp=0xc00043cdd8 pc=0x468a05 internal/poll.(*pollDesc).wait(0xc000126000?, 0xc000154521?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc00043ce20 sp=0xc00043cdf8 pc=0x4efd67 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc000126000, {0xc000154521, 0x1, 0x1}) /usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a fp=0xc00043ceb8 sp=0xc00043ce20 pc=0x4f105a net.(*netFD).Read(0xc000126000, {0xc000154521?, 0x0?, 0xc000282a50?}) /usr/local/go/src/net/fd_posix.go:55 +0x25 fp=0xc00043cf00 sp=0xc00043ceb8 pc=0x569e05 net.(*conn).Read(0xc0004aa008, {0xc000154521?, 0x0?, 0xc00043cee0?}) /usr/local/go/src/net/net.go:179 +0x45 fp=0xc00043cf48 sp=0xc00043cf00 pc=0x5780a5 net.(*TCPConn).Read(0x0?, {0xc000154521?, 0x100000000000000?, 0xc0004342c0?}) <autogenerated>:1 +0x25 fp=0xc00043cf78 sp=0xc00043cf48 pc=0x589fa5 net/http.(*connReader).backgroundRead(0xc000154510) /usr/local/go/src/net/http/server.go:683 +0x37 fp=0xc00043cfc8 sp=0xc00043cf78 pc=0x6c4ab7 net/http.(*connReader).startBackgroundRead.func2() /usr/local/go/src/net/http/server.go:679 +0x25 fp=0xc00043cfe0 sp=0xc00043cfc8 pc=0x6c49e5 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00043cfe8 sp=0xc00043cfe0 pc=0x46e2c1 created by net/http.(*connReader).startBackgroundRead in goroutine 50 /usr/local/go/src/net/http/server.go:679 +0xba rax 0x0 rbx 0x7fcb28ff8430 rcx 0x67 rdx 0x0 rdi 0x0 rsi 0x0 rbp 0x7fcb28ff83f0 rsp 0x7fcb28ff8258 r8 0x7fca482cfb00 r9 0x7fca482cfb28 r10 0x7fcb81200b40 r11 0x7fcb8138aa80 r12 0x0 r13 0x7fcb0c2026a8 r14 0x7fcb0c2044a0 r15 0x0 rip 0x7fcb8138e7fd rflags 0x10283 cs 0x33 fs 0x0 gs 0x0 ``` Thanks!
GiteaMirror commented 2026-04-12 11:47:45 -05:00
Author
Owner
Copy Link

@jmorganca commented on GitHub (Mar 3, 2024):

Hi there, would it be possible to try re-downloading the version 0.1.28 pre-release? https://github.com/ollama/ollama/releases/tag/v0.1.28 this version should support it. Sorry about that!

<!-- gh-comment-id:1975268979 --> @jmorganca commented on GitHub (Mar 3, 2024): Hi there, would it be possible to try re-downloading the version 0.1.28 pre-release? https://github.com/ollama/ollama/releases/tag/v0.1.28 this version should support it. Sorry about that!
GiteaMirror commented 2026-04-12 11:47:46 -05:00
Author
Owner
Copy Link

@joseluisamado commented on GitHub (Mar 3, 2024):

It is working with 0.1.28. Thanks for the quick answer!

<!-- gh-comment-id:1975272287 --> @joseluisamado commented on GitHub (Mar 3, 2024): It is working with 0.1.28. Thanks for the quick answer!
GiteaMirror commented 2026-04-12 11:47:47 -05:00
Author
Owner
Copy Link

@d-goldin commented on GitHub (Mar 7, 2024):

Sorry for piling onto a closed issue, but maybe better than opening a new one.

For me 0.1.28 still segfaults for starcoder2:7b and starcoder2:15b alike:

[...]
ollama        | time=2024-03-07T12:47:28.652Z level=INFO source=images.go:710 msg="total blobs: 35"
ollama        | time=2024-03-07T12:47:28.654Z level=INFO source=images.go:717 msg="total unused blobs removed: 0"
ollama        | time=2024-03-07T12:47:28.654Z level=INFO source=routes.go:1021 msg="Listening on 0.0.0.0:11434 (version 0.1.28)"
ollama        | time=2024-03-07T12:47:28.655Z level=INFO source=payload_common.go:107 msg="Extracting dynamic libraries..."
ollama        | time=2024-03-07T12:47:31.262Z level=INFO source=payload_common.go:146 msg="Dynamic LLM libraries [rocm_v5 rocm_v6 cpu cpu_avx cpu_avx2 cuda_v11]"
ollama        | time=2024-03-07T12:47:31.262Z level=INFO source=gpu.go:94 msg="Detecting GPU type"
ollama        | time=2024-03-07T12:47:31.262Z level=INFO source=gpu.go:265 msg="Searching for GPU management library libnvidia-ml.so"
ollama        | time=2024-03-07T12:47:31.263Z level=INFO source=gpu.go:311 msg="Discovered GPU libraries: [/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.545.29.02]"
ollama        | time=2024-03-07T12:47:31.270Z level=INFO source=gpu.go:99 msg="Nvidia GPU detected"
ollama        | time=2024-03-07T12:47:31.270Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
ollama        | time=2024-03-07T12:47:31.275Z level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.6"
ollama        | [GIN] 2024/03/07 - 12:48:05 | 200 |     1.38027ms |      172.19.0.3 | GET      "/api/tags"
ollama        | time=2024-03-07T12:48:06.159Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
ollama        | time=2024-03-07T12:48:06.159Z level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.6"
ollama        | time=2024-03-07T12:48:06.159Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
ollama        | time=2024-03-07T12:48:06.159Z level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.6"
ollama        | time=2024-03-07T12:48:06.159Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
ollama        | loading library /tmp/ollama641164978/cuda_v11/libext_server.so
ollama        | time=2024-03-07T12:48:06.166Z level=INFO source=dyn_ext_server.go:90 msg="Loading Dynamic llm server: /tmp/ollama641164978/cuda_v11/libext_server.so"
ollama        | time=2024-03-07T12:48:06.166Z level=INFO source=dyn_ext_server.go:150 msg="Initializing llama server"
ollama        | ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   yes
ollama        | ggml_init_cublas: CUDA_USE_TENSOR_CORES: no
ollama        | ggml_init_cublas: found 1 CUDA devices:
ollama        |   Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
ollama        | SIGSEGV: segmentation violation
ollama        | PC=0x7fe846e2b7fd m=20 sigcode=1
ollama        | signal arrived during cgo execution
[...]
<!-- gh-comment-id:1983444744 --> @d-goldin commented on GitHub (Mar 7, 2024): Sorry for piling onto a closed issue, but maybe better than opening a new one. For me 0.1.28 still segfaults for starcoder2:7b and starcoder2:15b alike: ``` [...] ollama | time=2024-03-07T12:47:28.652Z level=INFO source=images.go:710 msg="total blobs: 35" ollama | time=2024-03-07T12:47:28.654Z level=INFO source=images.go:717 msg="total unused blobs removed: 0" ollama | time=2024-03-07T12:47:28.654Z level=INFO source=routes.go:1021 msg="Listening on 0.0.0.0:11434 (version 0.1.28)" ollama | time=2024-03-07T12:47:28.655Z level=INFO source=payload_common.go:107 msg="Extracting dynamic libraries..." ollama | time=2024-03-07T12:47:31.262Z level=INFO source=payload_common.go:146 msg="Dynamic LLM libraries [rocm_v5 rocm_v6 cpu cpu_avx cpu_avx2 cuda_v11]" ollama | time=2024-03-07T12:47:31.262Z level=INFO source=gpu.go:94 msg="Detecting GPU type" ollama | time=2024-03-07T12:47:31.262Z level=INFO source=gpu.go:265 msg="Searching for GPU management library libnvidia-ml.so" ollama | time=2024-03-07T12:47:31.263Z level=INFO source=gpu.go:311 msg="Discovered GPU libraries: [/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.545.29.02]" ollama | time=2024-03-07T12:47:31.270Z level=INFO source=gpu.go:99 msg="Nvidia GPU detected" ollama | time=2024-03-07T12:47:31.270Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2" ollama | time=2024-03-07T12:47:31.275Z level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.6" ollama | [GIN] 2024/03/07 - 12:48:05 | 200 | 1.38027ms | 172.19.0.3 | GET "/api/tags" ollama | time=2024-03-07T12:48:06.159Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2" ollama | time=2024-03-07T12:48:06.159Z level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.6" ollama | time=2024-03-07T12:48:06.159Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2" ollama | time=2024-03-07T12:48:06.159Z level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.6" ollama | time=2024-03-07T12:48:06.159Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2" ollama | loading library /tmp/ollama641164978/cuda_v11/libext_server.so ollama | time=2024-03-07T12:48:06.166Z level=INFO source=dyn_ext_server.go:90 msg="Loading Dynamic llm server: /tmp/ollama641164978/cuda_v11/libext_server.so" ollama | time=2024-03-07T12:48:06.166Z level=INFO source=dyn_ext_server.go:150 msg="Initializing llama server" ollama | ggml_init_cublas: GGML_CUDA_FORCE_MMQ: yes ollama | ggml_init_cublas: CUDA_USE_TENSOR_CORES: no ollama | ggml_init_cublas: found 1 CUDA devices: ollama | Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes ollama | SIGSEGV: segmentation violation ollama | PC=0x7fe846e2b7fd m=20 sigcode=1 ollama | signal arrived during cgo execution [...] ```
GiteaMirror commented 2026-04-12 11:47:48 -05:00
Author
Owner
Copy Link

@KPHIBYE commented on GitHub (Mar 8, 2024):

The issue #2953 now tracks this problem for version 0.1.28

<!-- gh-comment-id:1986337216 --> @KPHIBYE commented on GitHub (Mar 8, 2024): The issue #2953 now tracks this problem for version 0.1.28
GiteaMirror commented 2026-04-12 11:47:49 -05:00
Author
Owner
Copy Link

@d-goldin commented on GitHub (Mar 9, 2024):

The issue #2953 now tracks this problem for version 0.1.28

Thanks!

<!-- gh-comment-id:1986664505 --> @d-goldin commented on GitHub (Mar 9, 2024): > The issue #2953 now tracks this problem for version 0.1.28 Thanks!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#1774
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?