github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-17 13:47:08 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #9665] SIGSEGV: segmentation violation #68363

New Issue
Open
opened 2026-05-04 13:34:56 -05:00 by GiteaMirror · 3 comments
GiteaMirror commented 2026-05-04 13:34:56 -05:00
Owner
Copy Link

Originally created by @sanchez314c on GitHub (Mar 12, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/9665

I've modified source prior to building for legacy AMD and nVidia model cards. I get no errors when compiling however whenever I run Ollama with any model I am getting this SIGSEGV: segmentation violation error.

Any ideas fellow Llamas?

Clean install of uBuntu 24.0.1, ROCm 6.3, nVidia 470 Drivers, CUDA Toolkit, Standard Ollama Repo, RocBlas for GFX803, AMD RX580 8GB, nVidia Tesla K80 24GB

###########################################################################
[GIN] 2025/03/11 - 21:07:32 | 200 | 62.292µs | 127.0.0.1 | HEAD "/"
[GIN] 2025/03/11 - 21:07:32 | 200 | 7.152064ms | 127.0.0.1 | POST "/api/show"
time=2025-03-11T21:07:32.230-04:00 level=WARN source=ggml.go:132 msg="key not found" key=llama.attention.key_length default=64
time=2025-03-11T21:07:32.230-04:00 level=WARN source=ggml.go:132 msg="key not found" key=llama.attention.value_length default=64
time=2025-03-11T21:07:32.230-04:00 level=INFO source=sched.go:715 msg="new model will fit in available VRAM in single GPU, loading" model=/home/heathen-admin/.ollama/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 gpu=0 parallel=4 available=7647727616 required="1.7 GiB"
time=2025-03-11T21:07:32.230-04:00 level=INFO source=server.go:97 msg="system memory" total="125.5 GiB" free="119.8 GiB" free_swap="8.0 GiB"
time=2025-03-11T21:07:32.231-04:00 level=WARN source=ggml.go:132 msg="key not found" key=llama.attention.key_length default=64
time=2025-03-11T21:07:32.231-04:00 level=WARN source=ggml.go:132 msg="key not found" key=llama.attention.value_length default=64
time=2025-03-11T21:07:32.231-04:00 level=INFO source=server.go:130 msg=offload library=rocm layers.requested=-1 layers.model=23 layers.offload=23 layers.split="" memory.available="[7.1 GiB]" memory.gpu_overhead="0 B" memory.required.full="1.7 GiB" memory.required.partial="1.7 GiB" memory.required.kv="176.0 MiB" memory.required.allocations="[1.7 GiB]" memory.weights.total="696.1 MiB" memory.weights.repeating="644.8 MiB" memory.weights.nonrepeating="51.3 MiB" memory.graph.full="544.0 MiB" memory.graph.partial="546.3 MiB"
time=2025-03-11T21:07:32.231-04:00 level=INFO source=server.go:380 msg="starting llama server" cmd="/home/heathen-admin/ollama/ollama runner --model /home/heathen-admin/.ollama/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 --ctx-size 8192 --batch-size 512 --n-gpu-layers 23 --threads 10 --parallel 4 --port 37087"
time=2025-03-11T21:07:32.231-04:00 level=INFO source=sched.go:450 msg="loaded runners" count=1
time=2025-03-11T21:07:32.231-04:00 level=INFO source=server.go:557 msg="waiting for llama runner to start responding"
time=2025-03-11T21:07:32.232-04:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server error"
time=2025-03-11T21:07:32.243-04:00 level=INFO source=runner.go:932 msg="starting go runner"
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
Device 0: Tesla K80, compute capability 3.7, VMM: yes
Device 1: Tesla K80, compute capability 3.7, VMM: yes
load_backend: loaded CUDA backend from /home/heathen-admin/ollama/build/lib/ollama/libggml-cuda.so
⠏ ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
Device 0: Radeon RX 580 Series, compute capability 8.0, VMM: no
load_backend: loaded ROCm backend from /home/heathen-admin/ollama/build/lib/ollama/libggml-hip.so
load_backend: loaded CPU backend from /home/heathen-admin/ollama/build/lib/ollama/libggml-cpu-skylakex.so
time=2025-03-11T21:07:33.178-04:00 level=INFO source=runner.go:935 msg=system info="CPU : LLAMAFILE = 1 | CPU : LLAMAFILE = 1 | CUDA : ARCHS = 370 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | ROCm : PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | AVX512 = 1 | LLAMAFILE = 1 | cgo(gcc)" threads=10
time=2025-03-11T21:07:33.178-04:00 level=INFO source=runner.go:993 msg="Server listening on 127.0.0.1:37087"
⠏ time=2025-03-11T21:07:33.237-04:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server loading model"
llama_load_model_from_file: using device CUDA0 (Tesla K80) - 11354 MiB free
⠙ llama_load_model_from_file: using device CUDA1 (Tesla K80) - 11354 MiB free
llama_load_model_from_file: using device ROCm0 (Radeon RX 580 Series) - 8148 MiB free
llama_model_loader: loaded meta data with 23 key-value pairs and 201 tensors from /home/heathen-admin/.ollama/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.name str = TinyLlama
llama_model_loader: - kv 2: llama.context_length u32 = 2048
llama_model_loader: - kv 3: llama.embedding_length u32 = 2048
llama_model_loader: - kv 4: llama.block_count u32 = 22
llama_model_loader: - kv 5: llama.feed_forward_length u32 = 5632
llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 64
llama_model_loader: - kv 7: llama.attention.head_count u32 = 32
llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 4
llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 10: llama.rope.freq_base f32 = 10000.000000
llama_model_loader: - kv 11: general.file_type u32 = 2
llama_model_loader: - kv 12: tokenizer.ggml.model str = llama
llama_model_loader: - kv 13: tokenizer.ggml.tokens arr[str,32000] = ["", "", "", "<0x00>", "<...
llama_model_loader: - kv 14: tokenizer.ggml.scores arr[f32,32000] = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv 15: tokenizer.ggml.token_type arr[i32,32000] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv 16: tokenizer.ggml.merges arr[str,61249] = ["▁ t", "e r", "i n", "▁ a", "e n...
llama_model_loader: - kv 17: tokenizer.ggml.bos_token_id u32 = 1
llama_model_loader: - kv 18: tokenizer.ggml.eos_token_id u32 = 2
llama_model_loader: - kv 19: tokenizer.ggml.unknown_token_id u32 = 0
llama_model_loader: - kv 20: tokenizer.ggml.padding_token_id u32 = 2
llama_model_loader: - kv 21: tokenizer.chat_template str = {% for message in messages %}\n{% if m...
llama_model_loader: - kv 22: general.quantization_version u32 = 2
llama_model_loader: - type f32: 45 tensors
llama_model_loader: - type q4_0: 155 tensors
llama_model_loader: - type q6_K: 1 tensors
⠹ llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
llm_load_vocab: special tokens cache size = 3
llm_load_vocab: token to piece cache size = 0.1684 MB
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = SPM
llm_load_print_meta: n_vocab = 32000
llm_load_print_meta: n_merges = 0
llm_load_print_meta: vocab_only = 0
llm_load_print_meta: n_ctx_train = 2048
llm_load_print_meta: n_embd = 2048
llm_load_print_meta: n_layer = 22
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 4
llm_load_print_meta: n_rot = 64
llm_load_print_meta: n_swa = 0
llm_load_print_meta: n_embd_head_k = 64
llm_load_print_meta: n_embd_head_v = 64
llm_load_print_meta: n_gqa = 8
llm_load_print_meta: n_embd_k_gqa = 256
llm_load_print_meta: n_embd_v_gqa = 256
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 5632
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn = 2048
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: ssm_dt_b_c_rms = 0
llm_load_print_meta: model type = 1B
llm_load_print_meta: model ftype = Q4_0
llm_load_print_meta: model params = 1.10 B
llm_load_print_meta: model size = 606.53 MiB (4.63 BPW)
llm_load_print_meta: general.name = TinyLlama
llm_load_print_meta: BOS token = 1 ''
llm_load_print_meta: EOS token = 2 ''
llm_load_print_meta: UNK token = 0 ''
llm_load_print_meta: PAD token = 2 ''
llm_load_print_meta: LF token = 13 '<0x0A>'
llm_load_print_meta: EOG token = 2 ''
llm_load_print_meta: max token length = 48
llm_load_tensors: offloading 22 repeating layers to GPU
llm_load_tensors: offloading output layer to GPU
llm_load_tensors: offloaded 23/23 layers to GPU
llm_load_tensors: ROCm0 model buffer size = 169.48 MiB
llm_load_tensors: CUDA0 model buffer size = 212.77 MiB
llm_load_tensors: CUDA1 model buffer size = 189.12 MiB
llm_load_tensors: CPU_Mapped model buffer size = 35.16 MiB
SIGSEGV: segmentation violation
PC=0x704aa8dac935 m=5 sigcode=1 addr=0x18
signal arrived during cgo execution

goroutine 50 gp=0xc0004a4a80 m=5 mp=0xc000100008 [syscall]:
runtime.cgocall(0x109e360, 0xc0000abb78)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/cgocall.go:167 +0x4b fp=0xc0000abb50 sp=0xc0000abb18 pc=0x48a96b
github.com/ollama/ollama/llama._Cfunc_llama_load_model_from_file(0x704abc000b70, {0x0, 0x17, 0x1, 0x0, 0x0, 0x0, 0x109dc80, 0xc000342110, 0x0, ...})
_cgo_gotypes.go:697 +0x4c fp=0xc0000abb78 sp=0xc0000abb50 pc=0x8407ac
github.com/ollama/ollama/llama.LoadModelFromFile.func1({0x7ffd98b6dd5b?, 0x0?}, {0x0, 0x17, 0x1, 0x0, 0x0, 0x0, 0x109dc80, 0xc000342110, ...})
/home/heathen-admin/ollama/llama/llama.go:271 +0x127 fp=0xc0000abc78 sp=0xc0000abb78 pc=0x843e07
github.com/ollama/ollama/llama.LoadModelFromFile({0x7ffd98b6dd5b, 0x70}, {0x17, 0x0, 0x1, 0x0, {0x0, 0x0, 0x0}, 0xc000502060, ...})
/home/heathen-admin/ollama/llama/llama.go:271 +0x2d2 fp=0xc0000abdc8 sp=0xc0000abc78 pc=0x843af2
github.com/ollama/ollama/runner/llamarunner.(*Server).loadModel(0xc00011a000, {0x17, 0x0, 0x1, 0x0, {0x0, 0x0, 0x0}, 0xc000502060, 0x0}, ...)
/home/heathen-admin/ollama/runner/llamarunner/runner.go:849 +0x9b fp=0xc0000abf10 sp=0xc0000abdc8 pc=0x86091b
github.com/ollama/ollama/runner/llamarunner.Execute.gowrap1()
/home/heathen-admin/ollama/runner/llamarunner/runner.go:968 +0xda fp=0xc0000abfe0 sp=0xc0000abf10 pc=0x86223a
runtime.goexit({})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000abfe8 sp=0xc0000abfe0 pc=0x499341
created by github.com/ollama/ollama/runner/llamarunner.Execute in goroutine 1
/home/heathen-admin/ollama/runner/llamarunner/runner.go:968 +0xcd5

goroutine 1 gp=0xc0000061c0 m=nil [IO wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc0005155c0 sp=0xc0005155a0 pc=0x49106e
runtime.netpollblock(0xc000515610?, 0x428186?, 0x0?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/netpoll.go:575 +0xf7 fp=0xc0005155f8 sp=0xc0005155c0 pc=0x454ef7
internal/poll.runtime_pollWait(0x704b1f43e680, 0x72)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/netpoll.go:351 +0x85 fp=0xc000515618 sp=0xc0005155f8 pc=0x490365
internal/poll.(*pollDesc).wait(0xc000022080?, 0x2c?, 0x0)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000515640 sp=0xc000515618 pc=0x518487
internal/poll.(*pollDesc).waitRead(...)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000022080)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/internal/poll/fd_unix.go:620 +0x295 fp=0xc0005156e8 sp=0xc000515640 pc=0x51d855
net.(*netFD).accept(0xc000022080)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/net/fd_unix.go:172 +0x29 fp=0xc0005157a0 sp=0xc0005156e8 pc=0x586909
net.(*TCPListener).accept(0xc00011e040)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/net/tcpsock_posix.go:159 +0x1e fp=0xc0005157f0 sp=0xc0005157a0 pc=0x59c53e
net.(*TCPListener).Accept(0xc00011e040)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/net/tcpsock.go:372 +0x30 fp=0xc000515820 sp=0xc0005157f0 pc=0x59b3f0
net/http.(*onceCloseListener).Accept(0xc0001d3560?)
:1 +0x24 fp=0xc000515838 sp=0xc000515820 pc=0x7e5324
net/http.(*Server).Serve(0xc00001e3c0, {0x1609618, 0xc00011e040})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/net/http/server.go:3330 +0x30c fp=0xc000515968 sp=0xc000515838 pc=0x7bd2ac
github.com/ollama/ollama/runner/llamarunner.Execute({0xc000036120, 0xe, 0xe})
/home/heathen-admin/ollama/runner/llamarunner/runner.go:994 +0x1174 fp=0xc000515d08 sp=0xc000515968 pc=0x861e14
github.com/ollama/ollama/runner.Execute({0xc000036110?, 0x0?, 0x0?})
/home/heathen-admin/ollama/runner/runner.go:22 +0xd4 fp=0xc000515d30 sp=0xc000515d08 pc=0xa91614
github.com/ollama/ollama/cmd.NewCLI.func2(0xc000118f00?, {0x1440478?, 0x4?, 0x144047c?})
/home/heathen-admin/ollama/cmd/cmd.go:1280 +0x45 fp=0xc000515d58 sp=0xc000515d30 pc=0x109d605
github.com/spf13/cobra.(*Command).execute(0xc0004e9b08, {0xc0004b55e0, 0xe, 0xe})
/home/heathen-admin/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940 +0x862 fp=0xc000515e78 sp=0xc000515d58 pc=0x5ff5c2
github.com/spf13/cobra.(*Command).ExecuteC(0xc000497b08)
/home/heathen-admin/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3a5 fp=0xc000515f30 sp=0xc000515e78 pc=0x5ffe05
github.com/spf13/cobra.(*Command).Execute(...)
/home/heathen-admin/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992
github.com/spf13/cobra.(*Command).ExecuteContext(...)
/home/heathen-admin/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:985
main.main()
/home/heathen-admin/ollama/main.go:12 +0x4d fp=0xc000515f50 sp=0xc000515f30 pc=0x109d98d
runtime.main()
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:272 +0x28b fp=0xc000515fe0 sp=0xc000515f50 pc=0x45c54b
runtime.goexit({})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000515fe8 sp=0xc000515fe0 pc=0x499341

goroutine 2 gp=0xc000006c40 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000096fa8 sp=0xc000096f88 pc=0x49106e
runtime.goparkunlock(...)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:430
runtime.forcegchelper()
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:337 +0xb3 fp=0xc000096fe0 sp=0xc000096fa8 pc=0x45c893
runtime.goexit({})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000096fe8 sp=0xc000096fe0 pc=0x499341
created by runtime.init.7 in goroutine 1
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:325 +0x1a

goroutine 3 gp=0xc000007180 m=nil [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000097780 sp=0xc000097760 pc=0x49106e
runtime.goparkunlock(...)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:430
runtime.bgsweep(0xc000044100)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgcsweep.go:317 +0xdf fp=0xc0000977c8 sp=0xc000097780 pc=0x446f9f
runtime.gcenable.gowrap1()
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:204 +0x25 fp=0xc0000977e0 sp=0xc0000977c8 pc=0x43b645
runtime.goexit({})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000977e8 sp=0xc0000977e0 pc=0x499341
created by runtime.gcenable in goroutine 1
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:204 +0x66

goroutine 4 gp=0xc000007340 m=nil [GC scavenge wait]:
runtime.gopark(0x10000?, 0x15f7fc8?, 0x0?, 0x0?, 0x0?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000097f78 sp=0xc000097f58 pc=0x49106e
runtime.goparkunlock(...)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:430
runtime.(*scavengerState).park(0x1eacf40)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000097fa8 sp=0xc000097f78 pc=0x444969
runtime.bgscavenge(0xc000044100)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc000097fc8 sp=0xc000097fa8 pc=0x444ef9
runtime.gcenable.gowrap2()
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:205 +0x25 fp=0xc000097fe0 sp=0xc000097fc8 pc=0x43b5e5
runtime.goexit({})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000097fe8 sp=0xc000097fe0 pc=0x499341
created by runtime.gcenable in goroutine 1
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:205 +0xa5

goroutine 5 gp=0xc000007c00 m=nil [finalizer wait]:
runtime.gopark(0xc000096648?, 0x431b85?, 0xb0?, 0x1?, 0xc0000061c0?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000096620 sp=0xc000096600 pc=0x49106e
runtime.runfinq()
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mfinal.go:193 +0x107 fp=0xc0000967e0 sp=0xc000096620 pc=0x43a6c7
runtime.goexit({})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000967e8 sp=0xc0000967e0 pc=0x499341
created by runtime.createfing in goroutine 1
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mfinal.go:163 +0x3d

goroutine 6 gp=0xc0001f1500 m=nil [chan receive]:
runtime.gopark(0xc000098760?, 0x56dfa5?, 0x60?, 0x9?, 0x1620e80?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000098718 sp=0xc0000986f8 pc=0x49106e
runtime.chanrecv(0xc00004a380, 0x0, 0x1)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/chan.go:639 +0x41c fp=0xc000098790 sp=0xc000098718 pc=0x42ad7c
runtime.chanrecv1(0x0?, 0x0?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/chan.go:489 +0x12 fp=0xc0000987b8 sp=0xc000098790 pc=0x42a932
runtime.unique_runtime_registerUniqueMapCleanup.func1(...)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1781
runtime.unique_runtime_registerUniqueMapCleanup.gowrap1()
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1784 +0x2f fp=0xc0000987e0 sp=0xc0000987b8 pc=0x43e66f
runtime.goexit({})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000987e8 sp=0xc0000987e0 pc=0x499341
created by unique.runtime_registerUniqueMapCleanup in goroutine 1
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1779 +0x96

goroutine 7 gp=0xc0001f1880 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000098f38 sp=0xc000098f18 pc=0x49106e
runtime.gcBgMarkWorker(0xc00004b7a0)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc000098fc8 sp=0xc000098f38 pc=0x43d989
runtime.gcBgMarkStartWorkers.gowrap1()
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc000098fe0 sp=0xc000098fc8 pc=0x43d865
runtime.goexit({})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000098fe8 sp=0xc000098fe0 pc=0x499341
created by runtime.gcBgMarkStartWorkers in goroutine 1
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105

goroutine 8 gp=0xc0001f1a40 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000099738 sp=0xc000099718 pc=0x49106e
runtime.gcBgMarkWorker(0xc00004b7a0)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc0000997c8 sp=0xc000099738 pc=0x43d989
runtime.gcBgMarkStartWorkers.gowrap1()
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc0000997e0 sp=0xc0000997c8 pc=0x43d865
runtime.goexit({})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000997e8 sp=0xc0000997e0 pc=0x499341
created by runtime.gcBgMarkStartWorkers in goroutine 1
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105

goroutine 18 gp=0xc000104380 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000092738 sp=0xc000092718 pc=0x49106e
runtime.gcBgMarkWorker(0xc00004b7a0)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc0000927c8 sp=0xc000092738 pc=0x43d989
runtime.gcBgMarkStartWorkers.gowrap1()
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc0000927e0 sp=0xc0000927c8 pc=0x43d865
runtime.goexit({})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000927e8 sp=0xc0000927e0 pc=0x499341
created by runtime.gcBgMarkStartWorkers in goroutine 1
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105

goroutine 34 gp=0xc000504000 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc00050a738 sp=0xc00050a718 pc=0x49106e
runtime.gcBgMarkWorker(0xc00004b7a0)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc00050a7c8 sp=0xc00050a738 pc=0x43d989
runtime.gcBgMarkStartWorkers.gowrap1()
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc00050a7e0 sp=0xc00050a7c8 pc=0x43d865
runtime.goexit({})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00050a7e8 sp=0xc00050a7e0 pc=0x499341
created by runtime.gcBgMarkStartWorkers in goroutine 1
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105

goroutine 35 gp=0xc0005041c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc00050af38 sp=0xc00050af18 pc=0x49106e
runtime.gcBgMarkWorker(0xc00004b7a0)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc00050afc8 sp=0xc00050af38 pc=0x43d989
runtime.gcBgMarkStartWorkers.gowrap1()
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc00050afe0 sp=0xc00050afc8 pc=0x43d865
runtime.goexit({})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00050afe8 sp=0xc00050afe0 pc=0x499341
created by runtime.gcBgMarkStartWorkers in goroutine 1
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105

goroutine 9 gp=0xc0001f1c00 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000099f38 sp=0xc000099f18 pc=0x49106e
runtime.gcBgMarkWorker(0xc00004b7a0)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc000099fc8 sp=0xc000099f38 pc=0x43d989
runtime.gcBgMarkStartWorkers.gowrap1()
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc000099fe0 sp=0xc000099fc8 pc=0x43d865
runtime.goexit({})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000099fe8 sp=0xc000099fe0 pc=0x499341
created by runtime.gcBgMarkStartWorkers in goroutine 1
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105

goroutine 19 gp=0xc000104540 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000092f38 sp=0xc000092f18 pc=0x49106e
runtime.gcBgMarkWorker(0xc00004b7a0)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc000092fc8 sp=0xc000092f38 pc=0x43d989
runtime.gcBgMarkStartWorkers.gowrap1()
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc000092fe0 sp=0xc000092fc8 pc=0x43d865
runtime.goexit({})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000092fe8 sp=0xc000092fe0 pc=0x499341
created by runtime.gcBgMarkStartWorkers in goroutine 1
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105

goroutine 20 gp=0xc000104700 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000093738 sp=0xc000093718 pc=0x49106e
runtime.gcBgMarkWorker(0xc00004b7a0)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc0000937c8 sp=0xc000093738 pc=0x43d989
runtime.gcBgMarkStartWorkers.gowrap1()
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc0000937e0 sp=0xc0000937c8 pc=0x43d865
runtime.goexit({})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000937e8 sp=0xc0000937e0 pc=0x499341
created by runtime.gcBgMarkStartWorkers in goroutine 1
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105

goroutine 21 gp=0xc0001048c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000093f38 sp=0xc000093f18 pc=0x49106e
runtime.gcBgMarkWorker(0xc00004b7a0)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc000093fc8 sp=0xc000093f38 pc=0x43d989
runtime.gcBgMarkStartWorkers.gowrap1()
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc000093fe0 sp=0xc000093fc8 pc=0x43d865
runtime.goexit({})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000093fe8 sp=0xc000093fe0 pc=0x499341
created by runtime.gcBgMarkStartWorkers in goroutine 1
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105

goroutine 22 gp=0xc000104a80 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000094738 sp=0xc000094718 pc=0x49106e
runtime.gcBgMarkWorker(0xc00004b7a0)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc0000947c8 sp=0xc000094738 pc=0x43d989
runtime.gcBgMarkStartWorkers.gowrap1()
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc0000947e0 sp=0xc0000947c8 pc=0x43d865
runtime.goexit({})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000947e8 sp=0xc0000947e0 pc=0x499341
created by runtime.gcBgMarkStartWorkers in goroutine 1
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105

goroutine 10 gp=0xc0001f1dc0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000506738 sp=0xc000506718 pc=0x49106e
runtime.gcBgMarkWorker(0xc00004b7a0)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc0005067c8 sp=0xc000506738 pc=0x43d989
runtime.gcBgMarkStartWorkers.gowrap1()
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc0005067e0 sp=0xc0005067c8 pc=0x43d865
runtime.goexit({})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0005067e8 sp=0xc0005067e0 pc=0x499341
created by runtime.gcBgMarkStartWorkers in goroutine 1
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105

goroutine 11 gp=0xc0004a4000 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000506f38 sp=0xc000506f18 pc=0x49106e
runtime.gcBgMarkWorker(0xc00004b7a0)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc000506fc8 sp=0xc000506f38 pc=0x43d989
runtime.gcBgMarkStartWorkers.gowrap1()
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc000506fe0 sp=0xc000506fc8 pc=0x43d865
runtime.goexit({})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000506fe8 sp=0xc000506fe0 pc=0x499341
created by runtime.gcBgMarkStartWorkers in goroutine 1
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105

goroutine 12 gp=0xc0004a41c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000507738 sp=0xc000507718 pc=0x49106e
runtime.gcBgMarkWorker(0xc00004b7a0)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc0005077c8 sp=0xc000507738 pc=0x43d989
runtime.gcBgMarkStartWorkers.gowrap1()
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc0005077e0 sp=0xc0005077c8 pc=0x43d865
runtime.goexit({})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0005077e8 sp=0xc0005077e0 pc=0x499341
created by runtime.gcBgMarkStartWorkers in goroutine 1
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105

goroutine 13 gp=0xc0004a4380 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000507f38 sp=0xc000507f18 pc=0x49106e
runtime.gcBgMarkWorker(0xc00004b7a0)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc000507fc8 sp=0xc000507f38 pc=0x43d989
runtime.gcBgMarkStartWorkers.gowrap1()
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc000507fe0 sp=0xc000507fc8 pc=0x43d865
runtime.goexit({})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000507fe8 sp=0xc000507fe0 pc=0x499341
created by runtime.gcBgMarkStartWorkers in goroutine 1
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105

goroutine 14 gp=0xc0004a4540 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000508738 sp=0xc000508718 pc=0x49106e
runtime.gcBgMarkWorker(0xc00004b7a0)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc0005087c8 sp=0xc000508738 pc=0x43d989
runtime.gcBgMarkStartWorkers.gowrap1()
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc0005087e0 sp=0xc0005087c8 pc=0x43d865
runtime.goexit({})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0005087e8 sp=0xc0005087e0 pc=0x499341
created by runtime.gcBgMarkStartWorkers in goroutine 1
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105

goroutine 15 gp=0xc0004a4700 m=nil [GC worker (idle)]:
runtime.gopark(0x92948a8028a?, 0x0?, 0x0?, 0x0?, 0x0?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000508f38 sp=0xc000508f18 pc=0x49106e
runtime.gcBgMarkWorker(0xc00004b7a0)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc000508fc8 sp=0xc000508f38 pc=0x43d989
runtime.gcBgMarkStartWorkers.gowrap1()
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc000508fe0 sp=0xc000508fc8 pc=0x43d865
runtime.goexit({})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000508fe8 sp=0xc000508fe0 pc=0x499341
created by runtime.gcBgMarkStartWorkers in goroutine 1
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105

goroutine 23 gp=0xc000104c40 m=nil [GC worker (idle)]:
runtime.gopark(0x92948a7f660?, 0x1?, 0x67?, 0x86?, 0x0?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000094f38 sp=0xc000094f18 pc=0x49106e
runtime.gcBgMarkWorker(0xc00004b7a0)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc000094fc8 sp=0xc000094f38 pc=0x43d989
runtime.gcBgMarkStartWorkers.gowrap1()
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc000094fe0 sp=0xc000094fc8 pc=0x43d865
runtime.goexit({})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000094fe8 sp=0xc000094fe0 pc=0x499341
created by runtime.gcBgMarkStartWorkers in goroutine 1
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105

goroutine 36 gp=0xc000504380 m=nil [GC worker (idle)]:
runtime.gopark(0x1f5b5c0?, 0x1?, 0x37?, 0xb?, 0x0?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc00050b738 sp=0xc00050b718 pc=0x49106e
runtime.gcBgMarkWorker(0xc00004b7a0)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc00050b7c8 sp=0xc00050b738 pc=0x43d989
runtime.gcBgMarkStartWorkers.gowrap1()
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc00050b7e0 sp=0xc00050b7c8 pc=0x43d865
runtime.goexit({})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00050b7e8 sp=0xc00050b7e0 pc=0x499341
created by runtime.gcBgMarkStartWorkers in goroutine 1
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105

goroutine 24 gp=0xc000104e00 m=nil [GC worker (idle)]:
runtime.gopark(0x1f5b5c0?, 0x1?, 0x4b?, 0x8c?, 0x0?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000095738 sp=0xc000095718 pc=0x49106e
runtime.gcBgMarkWorker(0xc00004b7a0)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc0000957c8 sp=0xc000095738 pc=0x43d989
runtime.gcBgMarkStartWorkers.gowrap1()
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc0000957e0 sp=0xc0000957c8 pc=0x43d865
runtime.goexit({})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000957e8 sp=0xc0000957e0 pc=0x499341
created by runtime.gcBgMarkStartWorkers in goroutine 1
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105

goroutine 16 gp=0xc0004a48c0 m=nil [GC worker (idle)]:
runtime.gopark(0x1f5b5c0?, 0x1?, 0xb4?, 0xbb?, 0x0?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000509738 sp=0xc000509718 pc=0x49106e
runtime.gcBgMarkWorker(0xc00004b7a0)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc0005097c8 sp=0xc000509738 pc=0x43d989
runtime.gcBgMarkStartWorkers.gowrap1()
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc0005097e0 sp=0xc0005097c8 pc=0x43d865
runtime.goexit({})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0005097e8 sp=0xc0005097e0 pc=0x499341
created by runtime.gcBgMarkStartWorkers in goroutine 1
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105

goroutine 51 gp=0xc0004a4c40 m=nil [semacquire]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x60?, 0x0?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000095e18 sp=0xc000095df8 pc=0x49106e
runtime.goparkunlock(...)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:430
runtime.semacquire1(0xc00011a008, 0x0, 0x1, 0x0, 0x12)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/sema.go:178 +0x225 fp=0xc000095e80 sp=0xc000095e18 pc=0x46f4c5
sync.runtime_Semacquire(0x0?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/sema.go:71 +0x25 fp=0xc000095eb8 sp=0xc000095e80 pc=0x492865
sync.(*WaitGroup).Wait(0x0?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/sync/waitgroup.go:118 +0x48 fp=0xc000095ee0 sp=0xc000095eb8 pc=0x4a7c88
github.com/ollama/ollama/runner/llamarunner.(*Server).run(0xc00011a000, {0x160ba40, 0xc00018ab40})
/home/heathen-admin/ollama/runner/llamarunner/runner.go:316 +0x47 fp=0xc000095fb8 sp=0xc000095ee0 pc=0x85d127
github.com/ollama/ollama/runner/llamarunner.Execute.gowrap2()
/home/heathen-admin/ollama/runner/llamarunner/runner.go:973 +0x28 fp=0xc000095fe0 sp=0xc000095fb8 pc=0x862128
runtime.goexit({})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000095fe8 sp=0xc000095fe0 pc=0x499341
created by github.com/ollama/ollama/runner/llamarunner.Execute in goroutine 1
/home/heathen-admin/ollama/runner/llamarunner/runner.go:973 +0xdb5

goroutine 66 gp=0xc000504540 m=nil [IO wait]:
runtime.gopark(0xc000112720?, 0xc0001dfe00?, 0x10?, 0x3a?, 0xb?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000243918 sp=0xc0002438f8 pc=0x49106e
runtime.netpollblock(0x4b4458?, 0x428186?, 0x0?)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/netpoll.go:575 +0xf7 fp=0xc000243950 sp=0xc000243918 pc=0x454ef7
internal/poll.runtime_pollWait(0x704b1f43e568, 0x72)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/netpoll.go:351 +0x85 fp=0xc000243970 sp=0xc000243950 pc=0x490365
internal/poll.(*pollDesc).wait(0xc0001dfe00?, 0xc0004fd000?, 0x0)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000243998 sp=0xc000243970 pc=0x518487
internal/poll.(*pollDesc).waitRead(...)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0001dfe00, {0xc0004fd000, 0x1000, 0x1000})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/internal/poll/fd_unix.go:165 +0x27a fp=0xc000243a30 sp=0xc000243998 pc=0x51977a
net.(*netFD).Read(0xc0001dfe00, {0xc0004fd000?, 0xc000243aa0?, 0x518945?})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/net/fd_posix.go:55 +0x25 fp=0xc000243a78 sp=0xc000243a30 pc=0x584945
net.(*conn).Read(0xc00009aab0, {0xc0004fd000?, 0x0?, 0xc0004e6ab8?})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/net/net.go:189 +0x45 fp=0xc000243ac0 sp=0xc000243a78 pc=0x592f45
net.(*TCPConn).Read(0xc0004e6ab0?, {0xc0004fd000?, 0xc0001dfe00?, 0xc000243af8?})
:1 +0x25 fp=0xc000243af0 sp=0xc000243ac0 pc=0x5a60c5
net/http.(*connReader).Read(0xc0004e6ab0, {0xc0004fd000, 0x1000, 0x1000})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/net/http/server.go:798 +0x14b fp=0xc000243b40 sp=0xc000243af0 pc=0x7b306b
bufio.(*Reader).fill(0xc0001126c0)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/bufio/bufio.go:110 +0x103 fp=0xc000243b78 sp=0xc000243b40 pc=0x5aa7c3
bufio.(*Reader).Peek(0xc0001126c0, 0x4)
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/bufio/bufio.go:148 +0x53 fp=0xc000243b98 sp=0xc000243b78 pc=0x5aa8f3
net/http.(*conn).serve(0xc0001d3560, {0x160ba08, 0xc000020300})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/net/http/server.go:2127 +0x738 fp=0xc000243fb8 sp=0xc000243b98 pc=0x7b83b8
net/http.(*Server).Serve.gowrap3()
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/net/http/server.go:3360 +0x28 fp=0xc000243fe0 sp=0xc000243fb8 pc=0x7bd6a8
runtime.goexit({})
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000243fe8 sp=0xc000243fe0 pc=0x499341
created by net/http.(*Server).Serve in goroutine 1
/home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/net/http/server.go:3360 +0x485

rax 0x704ad6fbc5e0
rbx 0x7049d59b2750
rcx 0x3
rdx 0x7049d4f47a30
rdi 0x704aa8f73d78
rsi 0x3
rbp 0x704ad6fbc5d8
rsp 0x704ad6fbae40
r8 0x0
r9 0x0
r10 0x7049d59b2c40
r11 0x290
r12 0x0
r13 0x0
r14 0x1
r15 0x18
rip 0x704aa8dac935
rflags 0x10202
cs 0x33
fs 0x0
gs 0x0
⠹ time=2025-03-11T21:07:33.548-04:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server error"
⠴ time=2025-03-11T21:07:33.799-04:00 level=ERROR source=sched.go:456 msg="error loading llama server" error="llama runner process has terminated: exit status 2"
[GIN] 2025/03/11 - 21:07:33 | 500 | 1.589999219s | 127.0.0.1 | POST "/api/generate"
Error: llama runner process has terminated: exit status 2
heathen-admin@LLMServer:~/ollama$

Originally created by @sanchez314c on GitHub (Mar 12, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/9665 I've modified source prior to building for legacy AMD and nVidia model cards. I get no errors when compiling however whenever I run Ollama with any model I am getting this SIGSEGV: segmentation violation error. Any ideas fellow Llamas? Clean install of uBuntu 24.0.1, ROCm 6.3, nVidia 470 Drivers, CUDA Toolkit, Standard Ollama Repo, RocBlas for GFX803, AMD RX580 8GB, nVidia Tesla K80 24GB ########################################################################### [GIN] 2025/03/11 - 21:07:32 | 200 | 62.292µs | 127.0.0.1 | HEAD "/" [GIN] 2025/03/11 - 21:07:32 | 200 | 7.152064ms | 127.0.0.1 | POST "/api/show" time=2025-03-11T21:07:32.230-04:00 level=WARN source=ggml.go:132 msg="key not found" key=llama.attention.key_length default=64 time=2025-03-11T21:07:32.230-04:00 level=WARN source=ggml.go:132 msg="key not found" key=llama.attention.value_length default=64 time=2025-03-11T21:07:32.230-04:00 level=INFO source=sched.go:715 msg="new model will fit in available VRAM in single GPU, loading" model=/home/heathen-admin/.ollama/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 gpu=0 parallel=4 available=7647727616 required="1.7 GiB" time=2025-03-11T21:07:32.230-04:00 level=INFO source=server.go:97 msg="system memory" total="125.5 GiB" free="119.8 GiB" free_swap="8.0 GiB" time=2025-03-11T21:07:32.231-04:00 level=WARN source=ggml.go:132 msg="key not found" key=llama.attention.key_length default=64 time=2025-03-11T21:07:32.231-04:00 level=WARN source=ggml.go:132 msg="key not found" key=llama.attention.value_length default=64 time=2025-03-11T21:07:32.231-04:00 level=INFO source=server.go:130 msg=offload library=rocm layers.requested=-1 layers.model=23 layers.offload=23 layers.split="" memory.available="[7.1 GiB]" memory.gpu_overhead="0 B" memory.required.full="1.7 GiB" memory.required.partial="1.7 GiB" memory.required.kv="176.0 MiB" memory.required.allocations="[1.7 GiB]" memory.weights.total="696.1 MiB" memory.weights.repeating="644.8 MiB" memory.weights.nonrepeating="51.3 MiB" memory.graph.full="544.0 MiB" memory.graph.partial="546.3 MiB" time=2025-03-11T21:07:32.231-04:00 level=INFO source=server.go:380 msg="starting llama server" cmd="/home/heathen-admin/ollama/ollama runner --model /home/heathen-admin/.ollama/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 --ctx-size 8192 --batch-size 512 --n-gpu-layers 23 --threads 10 --parallel 4 --port 37087" time=2025-03-11T21:07:32.231-04:00 level=INFO source=sched.go:450 msg="loaded runners" count=1 time=2025-03-11T21:07:32.231-04:00 level=INFO source=server.go:557 msg="waiting for llama runner to start responding" time=2025-03-11T21:07:32.232-04:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server error" time=2025-03-11T21:07:32.243-04:00 level=INFO source=runner.go:932 msg="starting go runner" ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 2 CUDA devices: Device 0: Tesla K80, compute capability 3.7, VMM: yes Device 1: Tesla K80, compute capability 3.7, VMM: yes load_backend: loaded CUDA backend from /home/heathen-admin/ollama/build/lib/ollama/libggml-cuda.so ⠏ ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 ROCm devices: Device 0: Radeon RX 580 Series, compute capability 8.0, VMM: no load_backend: loaded ROCm backend from /home/heathen-admin/ollama/build/lib/ollama/libggml-hip.so load_backend: loaded CPU backend from /home/heathen-admin/ollama/build/lib/ollama/libggml-cpu-skylakex.so time=2025-03-11T21:07:33.178-04:00 level=INFO source=runner.go:935 msg=system info="CPU : LLAMAFILE = 1 | CPU : LLAMAFILE = 1 | CUDA : ARCHS = 370 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | ROCm : PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | AVX512 = 1 | LLAMAFILE = 1 | cgo(gcc)" threads=10 time=2025-03-11T21:07:33.178-04:00 level=INFO source=runner.go:993 msg="Server listening on 127.0.0.1:37087" ⠏ time=2025-03-11T21:07:33.237-04:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server loading model" llama_load_model_from_file: using device CUDA0 (Tesla K80) - 11354 MiB free ⠙ llama_load_model_from_file: using device CUDA1 (Tesla K80) - 11354 MiB free llama_load_model_from_file: using device ROCm0 (Radeon RX 580 Series) - 8148 MiB free llama_model_loader: loaded meta data with 23 key-value pairs and 201 tensors from /home/heathen-admin/.ollama/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.name str = TinyLlama llama_model_loader: - kv 2: llama.context_length u32 = 2048 llama_model_loader: - kv 3: llama.embedding_length u32 = 2048 llama_model_loader: - kv 4: llama.block_count u32 = 22 llama_model_loader: - kv 5: llama.feed_forward_length u32 = 5632 llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 64 llama_model_loader: - kv 7: llama.attention.head_count u32 = 32 llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 4 llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 10: llama.rope.freq_base f32 = 10000.000000 llama_model_loader: - kv 11: general.file_type u32 = 2 llama_model_loader: - kv 12: tokenizer.ggml.model str = llama llama_model_loader: - kv 13: tokenizer.ggml.tokens arr[str,32000] = ["<unk>", "<s>", "</s>", "<0x00>", "<... llama_model_loader: - kv 14: tokenizer.ggml.scores arr[f32,32000] = [0.000000, 0.000000, 0.000000, 0.0000... llama_model_loader: - kv 15: tokenizer.ggml.token_type arr[i32,32000] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ... llama_model_loader: - kv 16: tokenizer.ggml.merges arr[str,61249] = ["▁ t", "e r", "i n", "▁ a", "e n... llama_model_loader: - kv 17: tokenizer.ggml.bos_token_id u32 = 1 llama_model_loader: - kv 18: tokenizer.ggml.eos_token_id u32 = 2 llama_model_loader: - kv 19: tokenizer.ggml.unknown_token_id u32 = 0 llama_model_loader: - kv 20: tokenizer.ggml.padding_token_id u32 = 2 llama_model_loader: - kv 21: tokenizer.chat_template str = {% for message in messages %}\n{% if m... llama_model_loader: - kv 22: general.quantization_version u32 = 2 llama_model_loader: - type f32: 45 tensors llama_model_loader: - type q4_0: 155 tensors llama_model_loader: - type q6_K: 1 tensors ⠹ llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect llm_load_vocab: special tokens cache size = 3 llm_load_vocab: token to piece cache size = 0.1684 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = llama llm_load_print_meta: vocab type = SPM llm_load_print_meta: n_vocab = 32000 llm_load_print_meta: n_merges = 0 llm_load_print_meta: vocab_only = 0 llm_load_print_meta: n_ctx_train = 2048 llm_load_print_meta: n_embd = 2048 llm_load_print_meta: n_layer = 22 llm_load_print_meta: n_head = 32 llm_load_print_meta: n_head_kv = 4 llm_load_print_meta: n_rot = 64 llm_load_print_meta: n_swa = 0 llm_load_print_meta: n_embd_head_k = 64 llm_load_print_meta: n_embd_head_v = 64 llm_load_print_meta: n_gqa = 8 llm_load_print_meta: n_embd_k_gqa = 256 llm_load_print_meta: n_embd_v_gqa = 256 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 5632 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 0 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 10000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_ctx_orig_yarn = 2048 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: ssm_dt_b_c_rms = 0 llm_load_print_meta: model type = 1B llm_load_print_meta: model ftype = Q4_0 llm_load_print_meta: model params = 1.10 B llm_load_print_meta: model size = 606.53 MiB (4.63 BPW) llm_load_print_meta: general.name = TinyLlama llm_load_print_meta: BOS token = 1 '<s>' llm_load_print_meta: EOS token = 2 '</s>' llm_load_print_meta: UNK token = 0 '<unk>' llm_load_print_meta: PAD token = 2 '</s>' llm_load_print_meta: LF token = 13 '<0x0A>' llm_load_print_meta: EOG token = 2 '</s>' llm_load_print_meta: max token length = 48 llm_load_tensors: offloading 22 repeating layers to GPU llm_load_tensors: offloading output layer to GPU llm_load_tensors: offloaded 23/23 layers to GPU llm_load_tensors: ROCm0 model buffer size = 169.48 MiB llm_load_tensors: CUDA0 model buffer size = 212.77 MiB llm_load_tensors: CUDA1 model buffer size = 189.12 MiB llm_load_tensors: CPU_Mapped model buffer size = 35.16 MiB SIGSEGV: segmentation violation PC=0x704aa8dac935 m=5 sigcode=1 addr=0x18 signal arrived during cgo execution goroutine 50 gp=0xc0004a4a80 m=5 mp=0xc000100008 [syscall]: runtime.cgocall(0x109e360, 0xc0000abb78) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/cgocall.go:167 +0x4b fp=0xc0000abb50 sp=0xc0000abb18 pc=0x48a96b github.com/ollama/ollama/llama._Cfunc_llama_load_model_from_file(0x704abc000b70, {0x0, 0x17, 0x1, 0x0, 0x0, 0x0, 0x109dc80, 0xc000342110, 0x0, ...}) _cgo_gotypes.go:697 +0x4c fp=0xc0000abb78 sp=0xc0000abb50 pc=0x8407ac github.com/ollama/ollama/llama.LoadModelFromFile.func1({0x7ffd98b6dd5b?, 0x0?}, {0x0, 0x17, 0x1, 0x0, 0x0, 0x0, 0x109dc80, 0xc000342110, ...}) /home/heathen-admin/ollama/llama/llama.go:271 +0x127 fp=0xc0000abc78 sp=0xc0000abb78 pc=0x843e07 github.com/ollama/ollama/llama.LoadModelFromFile({0x7ffd98b6dd5b, 0x70}, {0x17, 0x0, 0x1, 0x0, {0x0, 0x0, 0x0}, 0xc000502060, ...}) /home/heathen-admin/ollama/llama/llama.go:271 +0x2d2 fp=0xc0000abdc8 sp=0xc0000abc78 pc=0x843af2 github.com/ollama/ollama/runner/llamarunner.(*Server).loadModel(0xc00011a000, {0x17, 0x0, 0x1, 0x0, {0x0, 0x0, 0x0}, 0xc000502060, 0x0}, ...) /home/heathen-admin/ollama/runner/llamarunner/runner.go:849 +0x9b fp=0xc0000abf10 sp=0xc0000abdc8 pc=0x86091b github.com/ollama/ollama/runner/llamarunner.Execute.gowrap1() /home/heathen-admin/ollama/runner/llamarunner/runner.go:968 +0xda fp=0xc0000abfe0 sp=0xc0000abf10 pc=0x86223a runtime.goexit({}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000abfe8 sp=0xc0000abfe0 pc=0x499341 created by github.com/ollama/ollama/runner/llamarunner.Execute in goroutine 1 /home/heathen-admin/ollama/runner/llamarunner/runner.go:968 +0xcd5 goroutine 1 gp=0xc0000061c0 m=nil [IO wait]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc0005155c0 sp=0xc0005155a0 pc=0x49106e runtime.netpollblock(0xc000515610?, 0x428186?, 0x0?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/netpoll.go:575 +0xf7 fp=0xc0005155f8 sp=0xc0005155c0 pc=0x454ef7 internal/poll.runtime_pollWait(0x704b1f43e680, 0x72) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/netpoll.go:351 +0x85 fp=0xc000515618 sp=0xc0005155f8 pc=0x490365 internal/poll.(*pollDesc).wait(0xc000022080?, 0x2c?, 0x0) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000515640 sp=0xc000515618 pc=0x518487 internal/poll.(*pollDesc).waitRead(...) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Accept(0xc000022080) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/internal/poll/fd_unix.go:620 +0x295 fp=0xc0005156e8 sp=0xc000515640 pc=0x51d855 net.(*netFD).accept(0xc000022080) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/net/fd_unix.go:172 +0x29 fp=0xc0005157a0 sp=0xc0005156e8 pc=0x586909 net.(*TCPListener).accept(0xc00011e040) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/net/tcpsock_posix.go:159 +0x1e fp=0xc0005157f0 sp=0xc0005157a0 pc=0x59c53e net.(*TCPListener).Accept(0xc00011e040) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/net/tcpsock.go:372 +0x30 fp=0xc000515820 sp=0xc0005157f0 pc=0x59b3f0 net/http.(*onceCloseListener).Accept(0xc0001d3560?) <autogenerated>:1 +0x24 fp=0xc000515838 sp=0xc000515820 pc=0x7e5324 net/http.(*Server).Serve(0xc00001e3c0, {0x1609618, 0xc00011e040}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/net/http/server.go:3330 +0x30c fp=0xc000515968 sp=0xc000515838 pc=0x7bd2ac github.com/ollama/ollama/runner/llamarunner.Execute({0xc000036120, 0xe, 0xe}) /home/heathen-admin/ollama/runner/llamarunner/runner.go:994 +0x1174 fp=0xc000515d08 sp=0xc000515968 pc=0x861e14 github.com/ollama/ollama/runner.Execute({0xc000036110?, 0x0?, 0x0?}) /home/heathen-admin/ollama/runner/runner.go:22 +0xd4 fp=0xc000515d30 sp=0xc000515d08 pc=0xa91614 github.com/ollama/ollama/cmd.NewCLI.func2(0xc000118f00?, {0x1440478?, 0x4?, 0x144047c?}) /home/heathen-admin/ollama/cmd/cmd.go:1280 +0x45 fp=0xc000515d58 sp=0xc000515d30 pc=0x109d605 github.com/spf13/cobra.(*Command).execute(0xc0004e9b08, {0xc0004b55e0, 0xe, 0xe}) /home/heathen-admin/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940 +0x862 fp=0xc000515e78 sp=0xc000515d58 pc=0x5ff5c2 github.com/spf13/cobra.(*Command).ExecuteC(0xc000497b08) /home/heathen-admin/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3a5 fp=0xc000515f30 sp=0xc000515e78 pc=0x5ffe05 github.com/spf13/cobra.(*Command).Execute(...) /home/heathen-admin/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992 github.com/spf13/cobra.(*Command).ExecuteContext(...) /home/heathen-admin/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:985 main.main() /home/heathen-admin/ollama/main.go:12 +0x4d fp=0xc000515f50 sp=0xc000515f30 pc=0x109d98d runtime.main() /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:272 +0x28b fp=0xc000515fe0 sp=0xc000515f50 pc=0x45c54b runtime.goexit({}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000515fe8 sp=0xc000515fe0 pc=0x499341 goroutine 2 gp=0xc000006c40 m=nil [force gc (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000096fa8 sp=0xc000096f88 pc=0x49106e runtime.goparkunlock(...) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:430 runtime.forcegchelper() /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:337 +0xb3 fp=0xc000096fe0 sp=0xc000096fa8 pc=0x45c893 runtime.goexit({}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000096fe8 sp=0xc000096fe0 pc=0x499341 created by runtime.init.7 in goroutine 1 /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:325 +0x1a goroutine 3 gp=0xc000007180 m=nil [GC sweep wait]: runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000097780 sp=0xc000097760 pc=0x49106e runtime.goparkunlock(...) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:430 runtime.bgsweep(0xc000044100) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgcsweep.go:317 +0xdf fp=0xc0000977c8 sp=0xc000097780 pc=0x446f9f runtime.gcenable.gowrap1() /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:204 +0x25 fp=0xc0000977e0 sp=0xc0000977c8 pc=0x43b645 runtime.goexit({}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000977e8 sp=0xc0000977e0 pc=0x499341 created by runtime.gcenable in goroutine 1 /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:204 +0x66 goroutine 4 gp=0xc000007340 m=nil [GC scavenge wait]: runtime.gopark(0x10000?, 0x15f7fc8?, 0x0?, 0x0?, 0x0?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000097f78 sp=0xc000097f58 pc=0x49106e runtime.goparkunlock(...) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:430 runtime.(*scavengerState).park(0x1eacf40) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000097fa8 sp=0xc000097f78 pc=0x444969 runtime.bgscavenge(0xc000044100) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc000097fc8 sp=0xc000097fa8 pc=0x444ef9 runtime.gcenable.gowrap2() /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:205 +0x25 fp=0xc000097fe0 sp=0xc000097fc8 pc=0x43b5e5 runtime.goexit({}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000097fe8 sp=0xc000097fe0 pc=0x499341 created by runtime.gcenable in goroutine 1 /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:205 +0xa5 goroutine 5 gp=0xc000007c00 m=nil [finalizer wait]: runtime.gopark(0xc000096648?, 0x431b85?, 0xb0?, 0x1?, 0xc0000061c0?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000096620 sp=0xc000096600 pc=0x49106e runtime.runfinq() /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mfinal.go:193 +0x107 fp=0xc0000967e0 sp=0xc000096620 pc=0x43a6c7 runtime.goexit({}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000967e8 sp=0xc0000967e0 pc=0x499341 created by runtime.createfing in goroutine 1 /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mfinal.go:163 +0x3d goroutine 6 gp=0xc0001f1500 m=nil [chan receive]: runtime.gopark(0xc000098760?, 0x56dfa5?, 0x60?, 0x9?, 0x1620e80?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000098718 sp=0xc0000986f8 pc=0x49106e runtime.chanrecv(0xc00004a380, 0x0, 0x1) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/chan.go:639 +0x41c fp=0xc000098790 sp=0xc000098718 pc=0x42ad7c runtime.chanrecv1(0x0?, 0x0?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/chan.go:489 +0x12 fp=0xc0000987b8 sp=0xc000098790 pc=0x42a932 runtime.unique_runtime_registerUniqueMapCleanup.func1(...) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1781 runtime.unique_runtime_registerUniqueMapCleanup.gowrap1() /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1784 +0x2f fp=0xc0000987e0 sp=0xc0000987b8 pc=0x43e66f runtime.goexit({}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000987e8 sp=0xc0000987e0 pc=0x499341 created by unique.runtime_registerUniqueMapCleanup in goroutine 1 /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1779 +0x96 goroutine 7 gp=0xc0001f1880 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000098f38 sp=0xc000098f18 pc=0x49106e runtime.gcBgMarkWorker(0xc00004b7a0) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc000098fc8 sp=0xc000098f38 pc=0x43d989 runtime.gcBgMarkStartWorkers.gowrap1() /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc000098fe0 sp=0xc000098fc8 pc=0x43d865 runtime.goexit({}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000098fe8 sp=0xc000098fe0 pc=0x499341 created by runtime.gcBgMarkStartWorkers in goroutine 1 /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105 goroutine 8 gp=0xc0001f1a40 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000099738 sp=0xc000099718 pc=0x49106e runtime.gcBgMarkWorker(0xc00004b7a0) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc0000997c8 sp=0xc000099738 pc=0x43d989 runtime.gcBgMarkStartWorkers.gowrap1() /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc0000997e0 sp=0xc0000997c8 pc=0x43d865 runtime.goexit({}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000997e8 sp=0xc0000997e0 pc=0x499341 created by runtime.gcBgMarkStartWorkers in goroutine 1 /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105 goroutine 18 gp=0xc000104380 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000092738 sp=0xc000092718 pc=0x49106e runtime.gcBgMarkWorker(0xc00004b7a0) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc0000927c8 sp=0xc000092738 pc=0x43d989 runtime.gcBgMarkStartWorkers.gowrap1() /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc0000927e0 sp=0xc0000927c8 pc=0x43d865 runtime.goexit({}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000927e8 sp=0xc0000927e0 pc=0x499341 created by runtime.gcBgMarkStartWorkers in goroutine 1 /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105 goroutine 34 gp=0xc000504000 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc00050a738 sp=0xc00050a718 pc=0x49106e runtime.gcBgMarkWorker(0xc00004b7a0) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc00050a7c8 sp=0xc00050a738 pc=0x43d989 runtime.gcBgMarkStartWorkers.gowrap1() /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc00050a7e0 sp=0xc00050a7c8 pc=0x43d865 runtime.goexit({}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00050a7e8 sp=0xc00050a7e0 pc=0x499341 created by runtime.gcBgMarkStartWorkers in goroutine 1 /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105 goroutine 35 gp=0xc0005041c0 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc00050af38 sp=0xc00050af18 pc=0x49106e runtime.gcBgMarkWorker(0xc00004b7a0) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc00050afc8 sp=0xc00050af38 pc=0x43d989 runtime.gcBgMarkStartWorkers.gowrap1() /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc00050afe0 sp=0xc00050afc8 pc=0x43d865 runtime.goexit({}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00050afe8 sp=0xc00050afe0 pc=0x499341 created by runtime.gcBgMarkStartWorkers in goroutine 1 /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105 goroutine 9 gp=0xc0001f1c00 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000099f38 sp=0xc000099f18 pc=0x49106e runtime.gcBgMarkWorker(0xc00004b7a0) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc000099fc8 sp=0xc000099f38 pc=0x43d989 runtime.gcBgMarkStartWorkers.gowrap1() /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc000099fe0 sp=0xc000099fc8 pc=0x43d865 runtime.goexit({}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000099fe8 sp=0xc000099fe0 pc=0x499341 created by runtime.gcBgMarkStartWorkers in goroutine 1 /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105 goroutine 19 gp=0xc000104540 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000092f38 sp=0xc000092f18 pc=0x49106e runtime.gcBgMarkWorker(0xc00004b7a0) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc000092fc8 sp=0xc000092f38 pc=0x43d989 runtime.gcBgMarkStartWorkers.gowrap1() /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc000092fe0 sp=0xc000092fc8 pc=0x43d865 runtime.goexit({}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000092fe8 sp=0xc000092fe0 pc=0x499341 created by runtime.gcBgMarkStartWorkers in goroutine 1 /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105 goroutine 20 gp=0xc000104700 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000093738 sp=0xc000093718 pc=0x49106e runtime.gcBgMarkWorker(0xc00004b7a0) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc0000937c8 sp=0xc000093738 pc=0x43d989 runtime.gcBgMarkStartWorkers.gowrap1() /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc0000937e0 sp=0xc0000937c8 pc=0x43d865 runtime.goexit({}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000937e8 sp=0xc0000937e0 pc=0x499341 created by runtime.gcBgMarkStartWorkers in goroutine 1 /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105 goroutine 21 gp=0xc0001048c0 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000093f38 sp=0xc000093f18 pc=0x49106e runtime.gcBgMarkWorker(0xc00004b7a0) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc000093fc8 sp=0xc000093f38 pc=0x43d989 runtime.gcBgMarkStartWorkers.gowrap1() /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc000093fe0 sp=0xc000093fc8 pc=0x43d865 runtime.goexit({}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000093fe8 sp=0xc000093fe0 pc=0x499341 created by runtime.gcBgMarkStartWorkers in goroutine 1 /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105 goroutine 22 gp=0xc000104a80 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000094738 sp=0xc000094718 pc=0x49106e runtime.gcBgMarkWorker(0xc00004b7a0) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc0000947c8 sp=0xc000094738 pc=0x43d989 runtime.gcBgMarkStartWorkers.gowrap1() /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc0000947e0 sp=0xc0000947c8 pc=0x43d865 runtime.goexit({}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000947e8 sp=0xc0000947e0 pc=0x499341 created by runtime.gcBgMarkStartWorkers in goroutine 1 /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105 goroutine 10 gp=0xc0001f1dc0 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000506738 sp=0xc000506718 pc=0x49106e runtime.gcBgMarkWorker(0xc00004b7a0) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc0005067c8 sp=0xc000506738 pc=0x43d989 runtime.gcBgMarkStartWorkers.gowrap1() /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc0005067e0 sp=0xc0005067c8 pc=0x43d865 runtime.goexit({}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0005067e8 sp=0xc0005067e0 pc=0x499341 created by runtime.gcBgMarkStartWorkers in goroutine 1 /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105 goroutine 11 gp=0xc0004a4000 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000506f38 sp=0xc000506f18 pc=0x49106e runtime.gcBgMarkWorker(0xc00004b7a0) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc000506fc8 sp=0xc000506f38 pc=0x43d989 runtime.gcBgMarkStartWorkers.gowrap1() /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc000506fe0 sp=0xc000506fc8 pc=0x43d865 runtime.goexit({}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000506fe8 sp=0xc000506fe0 pc=0x499341 created by runtime.gcBgMarkStartWorkers in goroutine 1 /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105 goroutine 12 gp=0xc0004a41c0 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000507738 sp=0xc000507718 pc=0x49106e runtime.gcBgMarkWorker(0xc00004b7a0) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc0005077c8 sp=0xc000507738 pc=0x43d989 runtime.gcBgMarkStartWorkers.gowrap1() /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc0005077e0 sp=0xc0005077c8 pc=0x43d865 runtime.goexit({}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0005077e8 sp=0xc0005077e0 pc=0x499341 created by runtime.gcBgMarkStartWorkers in goroutine 1 /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105 goroutine 13 gp=0xc0004a4380 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000507f38 sp=0xc000507f18 pc=0x49106e runtime.gcBgMarkWorker(0xc00004b7a0) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc000507fc8 sp=0xc000507f38 pc=0x43d989 runtime.gcBgMarkStartWorkers.gowrap1() /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc000507fe0 sp=0xc000507fc8 pc=0x43d865 runtime.goexit({}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000507fe8 sp=0xc000507fe0 pc=0x499341 created by runtime.gcBgMarkStartWorkers in goroutine 1 /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105 goroutine 14 gp=0xc0004a4540 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000508738 sp=0xc000508718 pc=0x49106e runtime.gcBgMarkWorker(0xc00004b7a0) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc0005087c8 sp=0xc000508738 pc=0x43d989 runtime.gcBgMarkStartWorkers.gowrap1() /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc0005087e0 sp=0xc0005087c8 pc=0x43d865 runtime.goexit({}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0005087e8 sp=0xc0005087e0 pc=0x499341 created by runtime.gcBgMarkStartWorkers in goroutine 1 /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105 goroutine 15 gp=0xc0004a4700 m=nil [GC worker (idle)]: runtime.gopark(0x92948a8028a?, 0x0?, 0x0?, 0x0?, 0x0?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000508f38 sp=0xc000508f18 pc=0x49106e runtime.gcBgMarkWorker(0xc00004b7a0) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc000508fc8 sp=0xc000508f38 pc=0x43d989 runtime.gcBgMarkStartWorkers.gowrap1() /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc000508fe0 sp=0xc000508fc8 pc=0x43d865 runtime.goexit({}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000508fe8 sp=0xc000508fe0 pc=0x499341 created by runtime.gcBgMarkStartWorkers in goroutine 1 /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105 goroutine 23 gp=0xc000104c40 m=nil [GC worker (idle)]: runtime.gopark(0x92948a7f660?, 0x1?, 0x67?, 0x86?, 0x0?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000094f38 sp=0xc000094f18 pc=0x49106e runtime.gcBgMarkWorker(0xc00004b7a0) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc000094fc8 sp=0xc000094f38 pc=0x43d989 runtime.gcBgMarkStartWorkers.gowrap1() /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc000094fe0 sp=0xc000094fc8 pc=0x43d865 runtime.goexit({}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000094fe8 sp=0xc000094fe0 pc=0x499341 created by runtime.gcBgMarkStartWorkers in goroutine 1 /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105 goroutine 36 gp=0xc000504380 m=nil [GC worker (idle)]: runtime.gopark(0x1f5b5c0?, 0x1?, 0x37?, 0xb?, 0x0?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc00050b738 sp=0xc00050b718 pc=0x49106e runtime.gcBgMarkWorker(0xc00004b7a0) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc00050b7c8 sp=0xc00050b738 pc=0x43d989 runtime.gcBgMarkStartWorkers.gowrap1() /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc00050b7e0 sp=0xc00050b7c8 pc=0x43d865 runtime.goexit({}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00050b7e8 sp=0xc00050b7e0 pc=0x499341 created by runtime.gcBgMarkStartWorkers in goroutine 1 /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105 goroutine 24 gp=0xc000104e00 m=nil [GC worker (idle)]: runtime.gopark(0x1f5b5c0?, 0x1?, 0x4b?, 0x8c?, 0x0?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000095738 sp=0xc000095718 pc=0x49106e runtime.gcBgMarkWorker(0xc00004b7a0) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc0000957c8 sp=0xc000095738 pc=0x43d989 runtime.gcBgMarkStartWorkers.gowrap1() /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc0000957e0 sp=0xc0000957c8 pc=0x43d865 runtime.goexit({}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000957e8 sp=0xc0000957e0 pc=0x499341 created by runtime.gcBgMarkStartWorkers in goroutine 1 /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105 goroutine 16 gp=0xc0004a48c0 m=nil [GC worker (idle)]: runtime.gopark(0x1f5b5c0?, 0x1?, 0xb4?, 0xbb?, 0x0?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000509738 sp=0xc000509718 pc=0x49106e runtime.gcBgMarkWorker(0xc00004b7a0) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1412 +0xe9 fp=0xc0005097c8 sp=0xc000509738 pc=0x43d989 runtime.gcBgMarkStartWorkers.gowrap1() /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x25 fp=0xc0005097e0 sp=0xc0005097c8 pc=0x43d865 runtime.goexit({}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0005097e8 sp=0xc0005097e0 pc=0x499341 created by runtime.gcBgMarkStartWorkers in goroutine 1 /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/mgc.go:1328 +0x105 goroutine 51 gp=0xc0004a4c40 m=nil [semacquire]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x60?, 0x0?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000095e18 sp=0xc000095df8 pc=0x49106e runtime.goparkunlock(...) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:430 runtime.semacquire1(0xc00011a008, 0x0, 0x1, 0x0, 0x12) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/sema.go:178 +0x225 fp=0xc000095e80 sp=0xc000095e18 pc=0x46f4c5 sync.runtime_Semacquire(0x0?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/sema.go:71 +0x25 fp=0xc000095eb8 sp=0xc000095e80 pc=0x492865 sync.(*WaitGroup).Wait(0x0?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/sync/waitgroup.go:118 +0x48 fp=0xc000095ee0 sp=0xc000095eb8 pc=0x4a7c88 github.com/ollama/ollama/runner/llamarunner.(*Server).run(0xc00011a000, {0x160ba40, 0xc00018ab40}) /home/heathen-admin/ollama/runner/llamarunner/runner.go:316 +0x47 fp=0xc000095fb8 sp=0xc000095ee0 pc=0x85d127 github.com/ollama/ollama/runner/llamarunner.Execute.gowrap2() /home/heathen-admin/ollama/runner/llamarunner/runner.go:973 +0x28 fp=0xc000095fe0 sp=0xc000095fb8 pc=0x862128 runtime.goexit({}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000095fe8 sp=0xc000095fe0 pc=0x499341 created by github.com/ollama/ollama/runner/llamarunner.Execute in goroutine 1 /home/heathen-admin/ollama/runner/llamarunner/runner.go:973 +0xdb5 goroutine 66 gp=0xc000504540 m=nil [IO wait]: runtime.gopark(0xc000112720?, 0xc0001dfe00?, 0x10?, 0x3a?, 0xb?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/proc.go:424 +0xce fp=0xc000243918 sp=0xc0002438f8 pc=0x49106e runtime.netpollblock(0x4b4458?, 0x428186?, 0x0?) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/netpoll.go:575 +0xf7 fp=0xc000243950 sp=0xc000243918 pc=0x454ef7 internal/poll.runtime_pollWait(0x704b1f43e568, 0x72) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/netpoll.go:351 +0x85 fp=0xc000243970 sp=0xc000243950 pc=0x490365 internal/poll.(*pollDesc).wait(0xc0001dfe00?, 0xc0004fd000?, 0x0) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000243998 sp=0xc000243970 pc=0x518487 internal/poll.(*pollDesc).waitRead(...) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc0001dfe00, {0xc0004fd000, 0x1000, 0x1000}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/internal/poll/fd_unix.go:165 +0x27a fp=0xc000243a30 sp=0xc000243998 pc=0x51977a net.(*netFD).Read(0xc0001dfe00, {0xc0004fd000?, 0xc000243aa0?, 0x518945?}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/net/fd_posix.go:55 +0x25 fp=0xc000243a78 sp=0xc000243a30 pc=0x584945 net.(*conn).Read(0xc00009aab0, {0xc0004fd000?, 0x0?, 0xc0004e6ab8?}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/net/net.go:189 +0x45 fp=0xc000243ac0 sp=0xc000243a78 pc=0x592f45 net.(*TCPConn).Read(0xc0004e6ab0?, {0xc0004fd000?, 0xc0001dfe00?, 0xc000243af8?}) <autogenerated>:1 +0x25 fp=0xc000243af0 sp=0xc000243ac0 pc=0x5a60c5 net/http.(*connReader).Read(0xc0004e6ab0, {0xc0004fd000, 0x1000, 0x1000}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/net/http/server.go:798 +0x14b fp=0xc000243b40 sp=0xc000243af0 pc=0x7b306b bufio.(*Reader).fill(0xc0001126c0) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/bufio/bufio.go:110 +0x103 fp=0xc000243b78 sp=0xc000243b40 pc=0x5aa7c3 bufio.(*Reader).Peek(0xc0001126c0, 0x4) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/bufio/bufio.go:148 +0x53 fp=0xc000243b98 sp=0xc000243b78 pc=0x5aa8f3 net/http.(*conn).serve(0xc0001d3560, {0x160ba08, 0xc000020300}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/net/http/server.go:2127 +0x738 fp=0xc000243fb8 sp=0xc000243b98 pc=0x7b83b8 net/http.(*Server).Serve.gowrap3() /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/net/http/server.go:3360 +0x28 fp=0xc000243fe0 sp=0xc000243fb8 pc=0x7bd6a8 runtime.goexit({}) /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000243fe8 sp=0xc000243fe0 pc=0x499341 created by net/http.(*Server).Serve in goroutine 1 /home/heathen-admin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/net/http/server.go:3360 +0x485 rax 0x704ad6fbc5e0 rbx 0x7049d59b2750 rcx 0x3 rdx 0x7049d4f47a30 rdi 0x704aa8f73d78 rsi 0x3 rbp 0x704ad6fbc5d8 rsp 0x704ad6fbae40 r8 0x0 r9 0x0 r10 0x7049d59b2c40 r11 0x290 r12 0x0 r13 0x0 r14 0x1 r15 0x18 rip 0x704aa8dac935 rflags 0x10202 cs 0x33 fs 0x0 gs 0x0 ⠹ time=2025-03-11T21:07:33.548-04:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server error" ⠴ time=2025-03-11T21:07:33.799-04:00 level=ERROR source=sched.go:456 msg="error loading llama server" error="llama runner process has terminated: exit status 2" [GIN] 2025/03/11 - 21:07:33 | 500 | 1.589999219s | 127.0.0.1 | POST "/api/generate" Error: llama runner process has terminated: exit status 2 heathen-admin@LLMServer:~/ollama$
GiteaMirror commented 2026-05-04 13:35:00 -05:00
Author
Owner
Copy Link

@cli-ish commented on GitHub (Apr 7, 2025):

Had the same issue.

Check your config files i had this configured (in my case ~/.zshrc):

# OLLAMA STUFF
export HSA_OVERRIDE_GFX_VERSION=10.3.0

The HSA_OVERRIDE_GFX_VERSION is wrong and needs to be commented out like this:

# OLLAMA STUFF
# export HSA_OVERRIDE_GFX_VERSION=10.3.0

Just close and reopen your terminal and it should work.

<!-- gh-comment-id:2783498341 --> @cli-ish commented on GitHub (Apr 7, 2025): Had the same issue. Check your config files i had this configured (in my case `~/.zshrc`): ```bash # OLLAMA STUFF export HSA_OVERRIDE_GFX_VERSION=10.3.0 ``` The HSA_OVERRIDE_GFX_VERSION is wrong and needs to be commented out like this: ```bash # OLLAMA STUFF # export HSA_OVERRIDE_GFX_VERSION=10.3.0 ``` Just close and reopen your terminal and it should work.
GiteaMirror commented 2026-05-04 13:35:03 -05:00
Author
Owner
Copy Link

@robertrosenbusch commented on GitHub (Apr 9, 2025):

@cli-ish : i guess you are running into the false direction.Let me know if this hint was successful or not. it could be save lifetime -.^

Caution

Prevent ROCm SegFaults on your Linux Distro

After some feedback/research from Users who are using the Dockercontainer on GFX803 from my GIT in Ollama and PyTorch/ComfyUI, cause the devices /dev/dri and /dev/kfd crashed with SegFaults. Please proofe your used Linux-Kernel Version and switch up or down to a well known working Kernel-Version. Fedora 41, Arch and Debian 13 using (in April 2015) suspicious Linux-Kernel-Versions as default.

Kernel Version 5.19 6.2 6.8 6.9 6.10 6.11 6.12 6.13 6.14
working on ROCm 6.3 for Ollama/PyTorch ✅ ✅ ✅ ✅ ✅ ✅ 🟥 🟥 ✅

a simple "uname -a" will be proofe the prob ^·^

Robert Rosenbusch

<!-- gh-comment-id:2791054902 --> @robertrosenbusch commented on GitHub (Apr 9, 2025): @cli-ish : i guess you are running into the false direction.Let me know if this hint was successful or not. it could be save lifetime -.^ > [!CAUTION] > #### Prevent ROCm SegFaults on your Linux Distro > After some feedback/research from Users who are using the Dockercontainer on GFX803 from my GIT in [Ollama](https://github.com/robertrosenbusch/gfx803_rocm/issues/8#issue-2919996555) and [PyTorch/ComfyUI](https://github.com/robertrosenbusch/gfx803_rocm/issues/13#issuecomment-2754796999), cause the devices `/dev/dri` and `/dev/kfd` crashed with SegFaults. Please proofe your used Linux-Kernel Version and switch up or down to a well known working Kernel-Version. Fedora 41, Arch and Debian 13 using (in April 2015) suspicious Linux-Kernel-Versions as default. > |Kernel Version|5.19|6.2|6.8|6.9|6.10|6.11|6.12|6.13|6.14| > |--------------|-----|-----|------|-----|------|-----|-----|-----|-----| > |working on ROCm 6.3 for Ollama/PyTorch|✅|✅|✅|✅|✅|✅|🟥|🟥|✅| --- a simple `"uname -a"` will be proofe the prob ^·^ Robert Rosenbusch
GiteaMirror commented 2026-05-04 13:35:04 -05:00
Author
Owner
Copy Link

@cli-ish commented on GitHub (Apr 10, 2025):

@robertrosenbusch I am not exactly sure what you mean, the problem is now solved since i removed the config. if i add it again i have the same issue.

Just in case if this gives you any more info's:

uname -r

6.12.20-2-MANJARO

I had issue with the snap build too since i did not know where to move the rocm libs but since i used the AUR package everything works.

Ps: if this was not about my own problems solution then this may be the case for the author. Just seen this seems to be unique for gfx803 any my error most likely seems to be a different sigsegv.

<!-- gh-comment-id:2791663440 --> @cli-ish commented on GitHub (Apr 10, 2025): @robertrosenbusch I am not exactly sure what you mean, the problem is now solved since i removed the config. if i add it again i have the same issue. Just in case if this gives you any more info's: #### uname -r `6.12.20-2-MANJARO` I had issue with the snap build too since i did not know where to move the rocm libs but since i used the AUR package everything works. Ps: if this was not about my own problems solution then this may be the case for the author. Just seen this seems to be unique for `gfx803` any my error most likely seems to be a different sigsegv.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#68363
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?