github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-06 16:11:34 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #11358] Cannot load a single model, had to revert! SEGFAULTS from GO, GPU alloc OOM, all sorts of strange errors. #54006

New Issue
Closed
opened 2026-04-29 05:05:08 -05:00 by GiteaMirror · 5 comments
GiteaMirror commented 2026-04-29 05:05:08 -05:00
Owner
Copy Link

Originally created by @chhu on GitHub (Jul 10, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/11358

What is the issue?

I am on Linux w/ 8 RTX3090 that run on RedHat9 CUDA 12.6.
Revert to 0.9.4 and all was good!!

Relevant log output

OS

Linux

GPU

Nvidia

CPU

AMD

Ollama version

0.9.6

Originally created by @chhu on GitHub (Jul 10, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/11358 ### What is the issue? I am on Linux w/ 8 RTX3090 that run on RedHat9 CUDA 12.6. Revert to 0.9.4 and all was good!! ### Relevant log output ```shell ``` ### OS Linux ### GPU Nvidia ### CPU AMD ### Ollama version 0.9.6
GiteaMirror added the bug label 2026-04-29 05:05:08 -05:00
GiteaMirror commented 2026-04-29 05:05:10 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jul 10, 2025):

Server logs would aid in debugging.

<!-- gh-comment-id:3057597948 --> @rick-github commented on GitHub (Jul 10, 2025): [Server logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues) would aid in debugging.
GiteaMirror commented 2026-04-29 05:05:10 -05:00
Author
Owner
Copy Link

@chhu commented on GitHub (Jul 10, 2025):

Sorry, can't do, production env. The error was different depending on the size of the model, < 1GPU lead to segfaults, >1 lead to cuda malloc errors. A wild guess is that memory calculations are off, it tried to alloc 28G from a card it knows has 23G.
...
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 25937.08 MiB on device 1: cudaMalloc failed: out of memory
alloc_tensor_range: failed to allocate CUDA1 buffer of size 27197001728
...

... earlier...
CUDA driver version: 12.8
calling cuDeviceGetCount
device count 8
time=2025-07-10T15:46:35.438+02:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-1068f375-e3fe-422f-f71b-73a2394e701f name="NVIDIA GeForce RTX 3090" overhead="0 B" before.total="23.6 GiB" before.free="23.3 GiB" now.total="23.6 GiB" now.free="23.3 GiB" now.used="263.3 MiB"
time=2025-07-10T15:46:35.668+02:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-615262d3-7dfe-666a-cf37-566e760e4ed1 name="NVIDIA GeForce RTX 3090" overhead="0 B" before.total="23.6 GiB" before.free="23.3 GiB" now.total="23.6 GiB" now.free="23.3 GiB" now.used="263.3 MiB"
time=2025-07-10T15:46:35.902+02:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-f22fdea8-5405-3129-d32a-7f916d1f8b7c name="NVIDIA GeForce RTX 3090" overhead="0 B" before.total="23.6 GiB" before.free="23.3 GiB" now.total="23.6 GiB" now.free="23.3 GiB" now.used="263.3 MiB"
time=2025-07-10T15:46:36.125+02:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-7f0f16bf-bb19-c951-3cab-78c75e8f1036 name="NVIDIA GeForce RTX 3090" overhead="0 B" before.total="23.6 GiB" before.free="23.3 GiB" now.total="23.6 GiB" now.free="23.3 GiB" now.used="263.3 MiB"
time=2025-07-10T15:46:36.351+02:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-02de372e-3119-89d2-b10e-16ce6064e6e0 name="NVIDIA GeForce RTX 3090" overhead="0 B" before.total="23.6 GiB" before.free="23.3 GiB" now.total="23.6 GiB" now.free="23.3 GiB" now.used="263.3 MiB"
time=2025-07-10T15:46:36.580+02:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-80159554-dd91-f6ab-b8a5-bbf5a34c59c5 name="NVIDIA GeForce RTX 3090" overhead="0 B" before.total="23.6 GiB" before.free="23.3 GiB" now.total="23.6 GiB" now.free="23.3 GiB" now.used="263.3 MiB"
time=2025-07-10T15:46:36.805+02:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-9190655b-d4a4-131c-ae18-7b58cce3b7f3 name="NVIDIA GeForce RTX 3090" overhead="0 B" before.total="23.6 GiB" before.free="23.3 GiB" now.total="23.6 GiB" now.free="23.3 GiB" now.used="263.3 MiB"
time=2025-07-10T15:46:37.039+02:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-b355efbb-1628-3337-ee30-11313755b901 name="NVIDIA GeForce RTX 3090" overhead="0 B" before.total="23.6 GiB" before.free="23.3 GiB" now.total="23.6 GiB" now.free="23.3 GiB" now.used="263.3 MiB"
releasing cuda driver library
time=2025-07-10T15:46:37.039+02:00 level=INFO source=server.go:135 msg="system memory" total="503.2 GiB" free="490.6 GiB" free_swap="0 B"
time=2025-07-10T15:46:37.039+02:00 level=DEBUG source=memory.go:111 msg=evaluating library=cuda gpu_count=8 available="[23.3 GiB 23.3 GiB 23.3 GiB 23.3 GiB 23.3 GiB 23.3 GiB 23.3 GiB 23.3 GiB]"
time=2025-07-10T15:46:37.041+02:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.2 GiB" before.free="490.6 GiB" before.free_swap="0 B" now.total="503.2 GiB" now.free="490.6 GiB" now.free_swap="0 B"
initializing /usr/lib64/libcuda.so.570.124.06
... repeating itself a few times ?

Some hints from the log >1 GPU :

print_info: file format = GGUF V3 (latest)
print_info: file type = Q8_0
print_info: file size = 32.71 GiB (8.58 BPW)
init_tokenizer: initializing tokenizer for type 2
load: control token: 151659 '<|fim_prefix|>' is not marked as EOG
load: control token: 151656 '<|video_pad|>' is not marked as EOG
load: control token: 151655 '<|image_pad|>' is not marked as EOG
load: control token: 151653 '<|vision_end|>' is not marked as EOG
load: control token: 151651 '<|quad_end|>' is not marked as EOG
load: control token: 151650 '<|quad_start|>' is not marked as EOG
load: control token: 151649 '<|box_end|>' is not marked as EOG
load: control token: 151647 '<|object_ref_end|>' is not marked as EOG
load: control token: 151648 '<|box_start|>' is not marked as EOG
load: control token: 151661 '<|fim_suffix|>' is not marked as EOG
load: control token: 151646 '<|object_ref_start|>' is not marked as EOG
load: control token: 151644 '<|im_start|>' is not marked as EOG
load: control token: 151660 '<|fim_middle|>' is not marked as EOG
load: control token: 151652 '<|vision_start|>' is not marked as EOG
load: control token: 151654 '<|vision_pad|>' is not marked as EOG
load: special tokens cache size = 26
load: token to piece cache size = 0.9311 MB
print_info: arch = qwen3
print_info: vocab_only = 0
print_info: n_ctx_train = 40960
print_info: n_embd = 5120
print_info: n_layer = 64
print_info: n_head = 64
print_info: n_head_kv = 8
print_info: n_rot = 128
print_info: n_swa = 0
print_info: n_swa_pattern = 1
print_info: n_embd_head_k = 128
print_info: n_embd_head_v = 128
print_info: n_gqa = 8
print_info: n_embd_k_gqa = 1024
print_info: n_embd_v_gqa = 1024
print_info: f_norm_eps = 0.0e+00
print_info: f_norm_rms_eps = 1.0e-06
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 0.0e+00
print_info: f_attn_scale = 0.0e+00
print_info: n_ff = 25600
print_info: n_expert = 0
print_info: n_expert_used = 0
print_info: causal attn = 1
print_info: pooling type = 0
print_info: rope type = 2
print_info: rope scaling = linear
print_info: freq_base_train = 1000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn = 40960
print_info: rope_finetuned = unknown
print_info: ssm_d_conv = 0
print_info: ssm_d_inner = 0
print_info: ssm_d_state = 0
print_info: ssm_dt_rank = 0
print_info: ssm_dt_b_c_rms = 0
print_info: model type = 32B
print_info: model params = 32.76 B
print_info: general.name = Qwen3 32B
print_info: vocab type = BPE
print_info: n_vocab = 151936
print_info: n_merges = 151387
print_info: BOS token = 151643 '<|endoftext|>'
print_info: EOS token = 151645 '<|im_end|>'
print_info: EOT token = 151643 '<|endoftext|>'
print_info: PAD token = 151643 '<|endoftext|>'
print_info: LF token = 198 'Ċ'
print_info: FIM PRE token = 151659 '<|fim_prefix|>'
print_info: FIM SUF token = 151661 '<|fim_suffix|>'
print_info: FIM MID token = 151660 '<|fim_middle|>'
print_info: FIM PAD token = 151662 '<|fim_pad|>'
print_info: FIM REP token = 151663 '<|repo_name|>'
print_info: FIM SEP token = 151664 '<|file_sep|>'
print_info: EOG token = 151643 '<|endoftext|>'
print_info: EOG token = 151645 '<|im_end|>'
print_info: EOG token = 151662 '<|fim_pad|>'
print_info: EOG token = 151663 '<|repo_name|>'
print_info: EOG token = 151664 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: layer 0 assigned to device CUDA0, is_swa = 0
load_tensors: layer 1 assigned to device CUDA1, is_swa = 0
load_tensors: layer 2 assigned to device CUDA1, is_swa = 0
load_tensors: layer 3 assigned to device CUDA1, is_swa = 0
load_tensors: layer 4 assigned to device CUDA1, is_swa = 0
load_tensors: layer 5 assigned to device CUDA1, is_swa = 0
load_tensors: layer 6 assigned to device CUDA1, is_swa = 0
load_tensors: layer 7 assigned to device CUDA1, is_swa = 0
load_tensors: layer 8 assigned to device CUDA1, is_swa = 0
load_tensors: layer 9 assigned to device CUDA1, is_swa = 0
load_tensors: layer 10 assigned to device CUDA1, is_swa = 0
load_tensors: layer 11 assigned to device CUDA1, is_swa = 0
load_tensors: layer 12 assigned to device CUDA1, is_swa = 0
load_tensors: layer 13 assigned to device CUDA1, is_swa = 0
load_tensors: layer 14 assigned to device CUDA1, is_swa = 0
load_tensors: layer 15 assigned to device CUDA1, is_swa = 0
load_tensors: layer 16 assigned to device CUDA1, is_swa = 0
load_tensors: layer 17 assigned to device CUDA1, is_swa = 0
load_tensors: layer 18 assigned to device CUDA1, is_swa = 0
load_tensors: layer 19 assigned to device CUDA1, is_swa = 0
load_tensors: layer 20 assigned to device CUDA1, is_swa = 0
load_tensors: layer 21 assigned to device CUDA1, is_swa = 0
load_tensors: layer 22 assigned to device CUDA1, is_swa = 0
load_tensors: layer 23 assigned to device CUDA1, is_swa = 0
load_tensors: layer 24 assigned to device CUDA1, is_swa = 0
load_tensors: layer 25 assigned to device CUDA1, is_swa = 0
load_tensors: layer 26 assigned to device CUDA1, is_swa = 0
load_tensors: layer 27 assigned to device CUDA1, is_swa = 0
load_tensors: layer 28 assigned to device CUDA1, is_swa = 0
load_tensors: layer 29 assigned to device CUDA1, is_swa = 0
load_tensors: layer 30 assigned to device CUDA1, is_swa = 0
load_tensors: layer 31 assigned to device CUDA1, is_swa = 0
load_tensors: layer 32 assigned to device CUDA1, is_swa = 0
load_tensors: layer 33 assigned to device CUDA1, is_swa = 0
load_tensors: layer 34 assigned to device CUDA1, is_swa = 0
load_tensors: layer 35 assigned to device CUDA1, is_swa = 0
load_tensors: layer 36 assigned to device CUDA1, is_swa = 0
load_tensors: layer 37 assigned to device CUDA1, is_swa = 0
load_tensors: layer 38 assigned to device CUDA1, is_swa = 0
load_tensors: layer 39 assigned to device CUDA1, is_swa = 0
load_tensors: layer 40 assigned to device CUDA1, is_swa = 0
load_tensors: layer 41 assigned to device CUDA1, is_swa = 0
load_tensors: layer 42 assigned to device CUDA1, is_swa = 0
load_tensors: layer 43 assigned to device CUDA1, is_swa = 0
load_tensors: layer 44 assigned to device CUDA1, is_swa = 0
load_tensors: layer 45 assigned to device CUDA1, is_swa = 0
load_tensors: layer 46 assigned to device CUDA1, is_swa = 0
load_tensors: layer 47 assigned to device CUDA1, is_swa = 0
load_tensors: layer 48 assigned to device CUDA1, is_swa = 0
load_tensors: layer 49 assigned to device CUDA1, is_swa = 0
load_tensors: layer 50 assigned to device CUDA1, is_swa = 0
load_tensors: layer 51 assigned to device CUDA1, is_swa = 0
load_tensors: layer 52 assigned to device CUDA1, is_swa = 0
load_tensors: layer 53 assigned to device CUDA5, is_swa = 0
load_tensors: layer 54 assigned to device CUDA5, is_swa = 0
load_tensors: layer 55 assigned to device CUDA5, is_swa = 0
load_tensors: layer 56 assigned to device CUDA5, is_swa = 0
load_tensors: layer 57 assigned to device CUDA5, is_swa = 0
load_tensors: layer 58 assigned to device CUDA5, is_swa = 0
load_tensors: layer 59 assigned to device CUDA5, is_swa = 0
load_tensors: layer 60 assigned to device CUDA5, is_swa = 0
load_tensors: layer 61 assigned to device CUDA5, is_swa = 0
load_tensors: layer 62 assigned to device CUDA5, is_swa = 0
load_tensors: layer 63 assigned to device CUDA5, is_swa = 0
load_tensors: layer 64 assigned to device CUDA5, is_swa = 0
load_tensors: tensor 'token_embd.weight' (q8_0) (and 0 others) cannot be used with preferred buffer type CUDA_Host, using CPU instead
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 25937.08 MiB on device 1: cudaMalloc failed: out of memory
alloc_tensor_range: failed to allocate CUDA1 buffer of size 27197001728
llama_model_load: error loading model: unable to allocate CUDA1 buffer
llama_model_load_from_file_impl: failed to load model
panic: unable to load model: /scr2/new/ollama/models/models/blobs/sha256-de447d788da3df6b4ea340408b13fc2c3a2043a2dfc19178b12d501a4bd96484

goroutine 125 [running]:
github.com/ollama/ollama/runner/llamarunner.(*Server).loadModel(0xc0003d66c0, {0x41, 0x0, 0x1, {0xc000350e20, 0x8, 0x8}, 0xc0005a9050, 0x0}, {0x7ffdf5e1d97b, ...}, ...)
github.com/ollama/ollama/runner/llamarunner/runner.go:751 +0x395
created by github.com/ollama/ollama/runner/llamarunner.Execute in goroutine 1
github.com/ollama/ollama/runner/llamarunner/runner.go:848 +0xb57
time=2025-07-10T15:44:57.893+02:00 level=INFO source=server.go:632 msg="waiting for server to become available" status="llm server not responding"
time=2025-07-10T15:44:58.502+02:00 level=ERROR source=server.go:464 msg="llama runner terminated" error="exit status 2"
time=2025-07-10T15:44:58.645+02:00 level=ERROR source=sched.go:489 msg="error loading llama server" error="llama runner process has terminated: error loading model: unable to allocate CUDA1 buffer"
time=2025-07-10T15:44:58.645+02:00 level=DEBUG source=sched.go:491 msg="triggering expiration for failed load" runner.name=registry.ollama.ai/library/qwen3:32b-q8_0 runner.inference=cuda runner.devices=8 runner.size="51.2 GiB" runner.vram="51.2 GiB" runner.parallel=1 runner.pid=764615

<1 GPU:
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 8 CUDA devices:
Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Device 3: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Device 4: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Device 5: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Device 6: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Device 7: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
load_backend: loaded CUDA backend from /scr2/new/ollama/lib/ollama/libggml-cuda.so
load_backend: loaded CPU backend from /scr2/new/ollama/lib/ollama/libggml-cpu-haswell.so
time=2025-07-10T15:46:40.303+02:00 level=DEBUG source=ggml.go:94 msg="ggml backend load all from path" path=/scr2/new/ollama/lib/ollama/cuda_v12
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 8 CUDA devices:
Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Device 3: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Device 4: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Device 5: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Device 6: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Device 7: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
load_backend: loaded CUDA backend from /scr2/new/ollama/lib/ollama/cuda_v12/libggml-cuda.so
time=2025-07-10T15:46:40.493+02:00 level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 CUDA.1.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.1.USE_GRAPHS=1 CUDA.1.PEER_MAX_BATCH_SIZE=128 CUDA.2.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.2.USE_GRAPHS=1 CUDA.2.PEER_MAX_BATCH_SIZE=128 CUDA.3.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.3.USE_GRAPHS=1 CUDA.3.PEER_MAX_BATCH_SIZE=128 CUDA.4.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.4.USE_GRAPHS=1 CUDA.4.PEER_MAX_BATCH_SIZE=128 CUDA.5.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.5.USE_GRAPHS=1 CUDA.5.PEER_MAX_BATCH_SIZE=128 CUDA.6.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.6.USE_GRAPHS=1 CUDA.6.PEER_MAX_BATCH_SIZE=128 CUDA.7.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.7.USE_GRAPHS=1 CUDA.7.PEER_MAX_BATCH_SIZE=128 CUDA.8.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.8.USE_GRAPHS=1 CUDA.8.PEER_MAX_BATCH_SIZE=128 CUDA.9.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.9.USE_GRAPHS=1 CUDA.9.PEER_MAX_BATCH_SIZE=128 CUDA.10.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.10.USE_GRAPHS=1 CUDA.10.PEER_MAX_BATCH_SIZE=128 CUDA.11.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.11.USE_GRAPHS=1 CUDA.11.PEER_MAX_BATCH_SIZE=128 CUDA.12.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.12.USE_GRAPHS=1 CUDA.12.PEER_MAX_BATCH_SIZE=128 CUDA.13.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.13.USE_GRAPHS=1 CUDA.13.PEER_MAX_BATCH_SIZE=128 CUDA.14.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.14.USE_GRAPHS=1 CUDA.14.PEER_MAX_BATCH_SIZE=128 CUDA.15.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.15.USE_GRAPHS=1 CUDA.15.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc)
unexpected fault address 0x5d3fe0000
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x2 addr=0x5d3fe0000 pc=0x559255224780]

goroutine 116 gp=0xc000522700 m=4 mp=0xc00019d808 [running]:
runtime.throw({0x55925613d479?, 0xc000522700?})
runtime/panic.go:1096 +0x4a fp=0xc0018930b8 sp=0xc001893088 pc=0x559255294c2a
runtime.sigpanic()
runtime/signal_unix.go:939 +0x26c fp=0xc001893118 sp=0xc0018930b8 pc=0x5592552970ac
indexbytebody()
internal/bytealg/indexbyte_amd64.s:131 +0xe0 fp=0xc001893120 sp=0xc001893118 pc=0x559255224780
runtime.findnull(0xc0018931a0?)
runtime/string.go:577 +0x79 fp=0xc001893178 sp=0xc001893120 pc=0x55925527c8b9
runtime.gostring(0x5d3fe0000)
runtime/string.go:363 +0x1c fp=0xc0018931b0 sp=0xc001893178 pc=0x559255297f1c
github.com/ollama/ollama/ml/backend/ggml._Cfunc_GoString(...)
_cgo_gotypes.go:311
github.com/ollama/ollama/ml/backend/ggml.New({0x7ffc98d8b962, 0x6c}, {0x40, 0x0, 0x31, {0xc0005bede0, 0x8, 0x8}, 0x1})
github.com/ollama/ollama/ml/backend/ggml/ggml.go:158 +0x1336 fp=0xc001893c18 sp=0xc0018931b0 pc=0x5592556d1376
github.com/ollama/ollama/ml.NewBackend({0x7ffc98d8b962, 0x6c}, {0x40, 0x0, 0x31, {0xc0005bede0, 0x8, 0x8}, 0x1})
github.com/ollama/ollama/ml/backend.go:209 +0xb1 fp=0xc001893c70 sp=0xc001893c18 pc=0x5592556c2e11
github.com/ollama/ollama/model.New({0x7ffc98d8b962?, 0x0?}, {0x40, 0x0, 0x31, {0xc0005bede0, 0x8, 0x8}, 0x1})
github.com/ollama/ollama/model/model.go:102 +0x8f fp=0xc001893d68 sp=0xc001893c70 pc=0x5592556e0d2f
github.com/ollama/ollama/runner/ollamarunner.(*Server).initModel(0xc0005f4ea0, {0x7ffc98d8b962?, 0x0?}, {0x40, 0x0, 0x31, {0xc0005bede0, 0x8, 0x8}, 0x1}, ...)
github.com/ollama/ollama/runner/ollamarunner/runner.go:841 +0x8d fp=0xc001893dc8 sp=0xc001893d68 pc=0x559255782e6d
github.com/ollama/ollama/runner/ollamarunner.(*Server).load(0xc0005f4ea0, {0x5592565f5700, 0xc000597db0}, {0x7ffc98d8b962?, 0x0?}, {0x40, 0x0, 0x31, {0xc0005bede0, 0x8, ...}, ...}, ...)
github.com/ollama/ollama/runner/ollamarunner/runner.go:878 +0xb8 fp=0xc001893f20 sp=0xc001893dc8 pc=0x5592557831d8
github.com/ollama/ollama/runner/ollamarunner.Execute.gowrap1()
github.com/ollama/ollama/runner/ollamarunner/runner.go:959 +0xc7 fp=0xc001893fe0 sp=0xc001893f20 pc=0x559255784607
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc001893fe8 sp=0xc001893fe0 pc=0x55925529c481
created by github.com/ollama/ollama/runner/ollamarunner.Execute in goroutine 1
github.com/ollama/ollama/runner/ollamarunner/runner.go:959 +0xa11

goroutine 1 gp=0xc000002380 m=nil [IO wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000055650 sp=0xc000055630 pc=0x559255294d4e
runtime.netpollblock(0xc0000556a0?, 0x5522db46?, 0x92?)
runtime/netpoll.go:575 +0xf7 fp=0xc000055688 sp=0xc000055650 pc=0x559255259837
internal/poll.runtime_pollWait(0x7effac51ce70, 0x72)
runtime/netpoll.go:351 +0x85 fp=0xc0000556a8 sp=0xc000055688 pc=0x559255293f65
internal/poll.(*pollDesc).wait(0xc0005d5900?, 0x900000036?, 0x0)
internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0000556d0 sp=0xc0000556a8 pc=0x55925531b3a7
internal/poll.(*pollDesc).waitRead(...)
internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0005d5900)
internal/poll/fd_unix.go:620 +0x295 fp=0xc000055778 sp=0xc0000556d0 pc=0x559255320775
net.(*netFD).accept(0xc0005d5900)
net/fd_unix.go:172 +0x29 fp=0xc000055830 sp=0xc000055778 pc=0x559255392c89
net.(*TCPListener).accept(0xc0005a7d80)
net/tcpsock_posix.go:159 +0x1b fp=0xc000055880 sp=0xc000055830 pc=0x5592553a863b
net.(*TCPListener).Accept(0xc0005a7d80)
net/tcpsock.go:380 +0x30 fp=0xc0000558b0 sp=0xc000055880 pc=0x5592553a74f0
net/http.(*onceCloseListener).Accept(0xc0003d6240?)
:1 +0x24 fp=0xc0000558c8 sp=0xc0000558b0 pc=0x5592555bec44
net/http.(*Server).Serve(0xc000fb0000, {0x5592565f3288, 0xc0005a7d80})
net/http/server.go:3424 +0x30c fp=0xc0000559f8 sp=0xc0000558c8 pc=0x55925559650c
github.com/ollama/ollama/runner/ollamarunner.Execute({0xc000034170, 0x11, 0x11})
github.com/ollama/ollama/runner/ollamarunner/runner.go:984 +0xe09 fp=0xc000055d08 sp=0xc0000559f8 pc=0x559255784269
github.com/ollama/ollama/runner.Execute({0xc000034150?, 0x0?, 0x0?})
github.com/ollama/ollama/runner/runner.go:20 +0xc9 fp=0xc000055d30 sp=0xc000055d08 pc=0x559255784b69
github.com/ollama/ollama/cmd.NewCLI.func2(0xc000311800?, {0x55925613c075?, 0x4?, 0x55925613c079?})
github.com/ollama/ollama/cmd/cmd.go:1529 +0x45 fp=0xc000055d58 sp=0xc000055d30 pc=0x559255ee18c5
github.com/spf13/cobra.(*Command).execute(0xc00061cf08, {0xc0005f4b40, 0x12, 0x12})
github.com/spf13/cobra@v1.7.0/command.go:940 +0x85c fp=0xc000055e78 sp=0xc000055d58 pc=0x55925540c2dc
github.com/spf13/cobra.(*Command).ExecuteC(0xc0005b4908)
github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3a5 fp=0xc000055f30 sp=0xc000055e78 pc=0x55925540cb25
github.com/spf13/cobra.(*Command).Execute(...)
github.com/spf13/cobra@v1.7.0/command.go:992
github.com/spf13/cobra.(*Command).ExecuteContext(...)
github.com/spf13/cobra@v1.7.0/command.go:985
main.main()
github.com/ollama/ollama/main.go:12 +0x4d fp=0xc000055f50 sp=0xc000055f30 pc=0x559255ee234d
runtime.main()
runtime/proc.go:283 +0x29d fp=0xc000055fe0 sp=0xc000055f50 pc=0x559255260ebd
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc000055fe8 sp=0xc000055fe0 pc=0x55925529c481

goroutine 2 gp=0xc000002e00 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000196fa8 sp=0xc000196f88 pc=0x559255294d4e
runtime.goparkunlock(...)
runtime/proc.go:441
runtime.forcegchelper()
runtime/proc.go:348 +0xb8 fp=0xc000196fe0 sp=0xc000196fa8 pc=0x5592552611f8
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc000196fe8 sp=0xc000196fe0 pc=0x55925529c481
created by runtime.init.7 in goroutine 1
runtime/proc.go:336 +0x1a

goroutine 3 gp=0xc000003340 m=nil [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000197780 sp=0xc000197760 pc=0x559255294d4e
runtime.goparkunlock(...)
runtime/proc.go:441
runtime.bgsweep(0xc000042100)
runtime/mgcsweep.go:316 +0xdf fp=0xc0001977c8 sp=0xc000197780 pc=0x55925524b99f
runtime.gcenable.gowrap1()
runtime/mgc.go:204 +0x25 fp=0xc0001977e0 sp=0xc0001977c8 pc=0x55925523fd85
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0001977e8 sp=0xc0001977e0 pc=0x55925529c481
created by runtime.gcenable in goroutine 1
runtime/mgc.go:204 +0x66

goroutine 4 gp=0xc000003500 m=nil [GC scavenge wait]:
runtime.gopark(0x10000?, 0x5592562fbf10?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000197f78 sp=0xc000197f58 pc=0x559255294d4e
runtime.goparkunlock(...)
runtime/proc.go:441
runtime.(*scavengerState).park(0x559256e838e0)
runtime/mgcscavenge.go:425 +0x49 fp=0xc000197fa8 sp=0xc000197f78 pc=0x5592552493e9
runtime.bgscavenge(0xc000042100)
runtime/mgcscavenge.go:658 +0x59 fp=0xc000197fc8 sp=0xc000197fa8 pc=0x559255249979
runtime.gcenable.gowrap2()
runtime/mgc.go:205 +0x25 fp=0xc000197fe0 sp=0xc000197fc8 pc=0x55925523fd25
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc000197fe8 sp=0xc000197fe0 pc=0x55925529c481
created by runtime.gcenable in goroutine 1
runtime/mgc.go:205 +0xa5

goroutine 5 gp=0xc000003dc0 m=nil [finalizer wait]:
runtime.gopark(0x1b8?, 0xc000002380?, 0x1?, 0x23?, 0xc000196688?)
runtime/proc.go:435 +0xce fp=0xc000196630 sp=0xc000196610 pc=0x559255294d4e
runtime.runfinq()
runtime/mfinal.go:196 +0x107 fp=0xc0001967e0 sp=0xc000196630 pc=0x55925523ed47
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0001967e8 sp=0xc0001967e0 pc=0x55925529c481
created by runtime.createfing in goroutine 1
runtime/mfinal.go:166 +0x3d

goroutine 6 gp=0xc0002f28c0 m=nil [chan receive]:
runtime.gopark(0xc000343860?, 0xc001080048?, 0x60?, 0x87?, 0x5592553799c8?)
runtime/proc.go:435 +0xce fp=0xc000198718 sp=0xc0001986f8 pc=0x559255294d4e
runtime.chanrecv(0xc0001ce310, 0x0, 0x1)
runtime/chan.go:664 +0x445 fp=0xc000198790 sp=0xc000198718 pc=0x559255230725
runtime.chanrecv1(0x0?, 0x0?)
runtime/chan.go:506 +0x12 fp=0xc0001987b8 sp=0xc000198790 pc=0x5592552302b2
runtime.unique_runtime_registerUniqueMapCleanup.func2(...)
runtime/mgc.go:1796
runtime.unique_runtime_registerUniqueMapCleanup.gowrap1()
runtime/mgc.go:1799 +0x2f fp=0xc0001987e0 sp=0xc0001987b8 pc=0x559255242f2f
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0001987e8 sp=0xc0001987e0 pc=0x55925529c481
created by unique.runtime_registerUniqueMapCleanup in goroutine 1
runtime/mgc.go:1794 +0x85

goroutine 7 gp=0xc0002f2a80 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000198f38 sp=0xc000198f18 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc000198fc8 sp=0xc000198f38 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc000198fe0 sp=0xc000198fc8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc000198fe8 sp=0xc000198fe0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 8 gp=0xc0002f2c40 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000199738 sp=0xc000199718 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc0001997c8 sp=0xc000199738 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc0001997e0 sp=0xc0001997c8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0001997e8 sp=0xc0001997e0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 9 gp=0xc0002f2e00 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000199f38 sp=0xc000199f18 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc000199fc8 sp=0xc000199f38 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc000199fe0 sp=0xc000199fc8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc000199fe8 sp=0xc000199fe0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 10 gp=0xc0002f2fc0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000192738 sp=0xc000192718 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc0001927c8 sp=0xc000192738 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc0001927e0 sp=0xc0001927c8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0001927e8 sp=0xc0001927e0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 11 gp=0xc0002f3180 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000192f38 sp=0xc000192f18 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc000192fc8 sp=0xc000192f38 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc000192fe0 sp=0xc000192fc8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc000192fe8 sp=0xc000192fe0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 18 gp=0xc000202380 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc00021a738 sp=0xc00021a718 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc00021a7c8 sp=0xc00021a738 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc00021a7e0 sp=0xc00021a7c8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc00021a7e8 sp=0xc00021a7e0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 19 gp=0xc000202540 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc00021af38 sp=0xc00021af18 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc00021afc8 sp=0xc00021af38 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc00021afe0 sp=0xc00021afc8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc00021afe8 sp=0xc00021afe0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 20 gp=0xc000202700 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc00021b738 sp=0xc00021b718 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc00021b7c8 sp=0xc00021b738 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc00021b7e0 sp=0xc00021b7c8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc00021b7e8 sp=0xc00021b7e0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 21 gp=0xc0002028c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc00021bf38 sp=0xc00021bf18 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc00021bfc8 sp=0xc00021bf38 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc00021bfe0 sp=0xc00021bfc8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc00021bfe8 sp=0xc00021bfe0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 22 gp=0xc000202a80 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc00021c738 sp=0xc00021c718 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc00021c7c8 sp=0xc00021c738 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc00021c7e0 sp=0xc00021c7c8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc00021c7e8 sp=0xc00021c7e0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 23 gp=0xc000202c40 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc00021cf38 sp=0xc00021cf18 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc00021cfc8 sp=0xc00021cf38 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc00021cfe0 sp=0xc00021cfc8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc00021cfe8 sp=0xc00021cfe0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 24 gp=0xc000202e00 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc00021d738 sp=0xc00021d718 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc00021d7c8 sp=0xc00021d738 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc00021d7e0 sp=0xc00021d7c8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc00021d7e8 sp=0xc00021d7e0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 25 gp=0xc000202fc0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc00021df38 sp=0xc00021df18 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc00021dfc8 sp=0xc00021df38 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc00021dfe0 sp=0xc00021dfc8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc00021dfe8 sp=0xc00021dfe0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 26 gp=0xc000203180 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000216738 sp=0xc000216718 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc0002167c8 sp=0xc000216738 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc0002167e0 sp=0xc0002167c8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0002167e8 sp=0xc0002167e0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 27 gp=0xc000203340 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000216f38 sp=0xc000216f18 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc000216fc8 sp=0xc000216f38 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc000216fe0 sp=0xc000216fc8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc000216fe8 sp=0xc000216fe0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 28 gp=0xc000203500 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000217738 sp=0xc000217718 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc0002177c8 sp=0xc000217738 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc0002177e0 sp=0xc0002177c8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0002177e8 sp=0xc0002177e0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 29 gp=0xc0002036c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000217f38 sp=0xc000217f18 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc000217fc8 sp=0xc000217f38 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc000217fe0 sp=0xc000217fc8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc000217fe8 sp=0xc000217fe0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 30 gp=0xc000203880 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000218738 sp=0xc000218718 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc0002187c8 sp=0xc000218738 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc0002187e0 sp=0xc0002187c8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0002187e8 sp=0xc0002187e0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 31 gp=0xc000203a40 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000218f38 sp=0xc000218f18 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc000218fc8 sp=0xc000218f38 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc000218fe0 sp=0xc000218fc8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc000218fe8 sp=0xc000218fe0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 32 gp=0xc000203c00 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000219738 sp=0xc000219718 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc0002197c8 sp=0xc000219738 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc0002197e0 sp=0xc0002197c8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0002197e8 sp=0xc0002197e0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 33 gp=0xc000203dc0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000219f38 sp=0xc000219f18 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc000219fc8 sp=0xc000219f38 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc000219fe0 sp=0xc000219fc8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc000219fe8 sp=0xc000219fe0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 34 gp=0xc000220000 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000226738 sp=0xc000226718 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc0002267c8 sp=0xc000226738 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc0002267e0 sp=0xc0002267c8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0002267e8 sp=0xc0002267e0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 35 gp=0xc0002201c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000226f38 sp=0xc000226f18 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc000226fc8 sp=0xc000226f38 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc000226fe0 sp=0xc000226fc8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc000226fe8 sp=0xc000226fe0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 36 gp=0xc000220380 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000227738 sp=0xc000227718 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc0002277c8 sp=0xc000227738 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc0002277e0 sp=0xc0002277c8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0002277e8 sp=0xc0002277e0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 37 gp=0xc000220540 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000227f38 sp=0xc000227f18 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc000227fc8 sp=0xc000227f38 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc000227fe0 sp=0xc000227fc8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc000227fe8 sp=0xc000227fe0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 12 gp=0xc0002f3340 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000193738 sp=0xc000193718 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc0001937c8 sp=0xc000193738 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc0001937e0 sp=0xc0001937c8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0001937e8 sp=0xc0001937e0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 13 gp=0xc0002f3500 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000193f38 sp=0xc000193f18 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc000193fc8 sp=0xc000193f38 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc000193fe0 sp=0xc000193fc8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc000193fe8 sp=0xc000193fe0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 14 gp=0xc0002f36c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000194738 sp=0xc000194718 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc0001947c8 sp=0xc000194738 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc0001947e0 sp=0xc0001947c8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0001947e8 sp=0xc0001947e0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 15 gp=0xc0002f3880 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000194f38 sp=0xc000194f18 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc000194fc8 sp=0xc000194f38 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc000194fe0 sp=0xc000194fc8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc000194fe8 sp=0xc000194fe0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 16 gp=0xc0002f3a40 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000195738 sp=0xc000195718 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc0001957c8 sp=0xc000195738 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc0001957e0 sp=0xc0001957c8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0001957e8 sp=0xc0001957e0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 50 gp=0xc000504000 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000222738 sp=0xc000222718 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc0002227c8 sp=0xc000222738 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc0002227e0 sp=0xc0002227c8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0002227e8 sp=0xc0002227e0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 66 gp=0xc0002f3c00 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000195f38 sp=0xc000195f18 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc000195fc8 sp=0xc000195f38 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc000195fe0 sp=0xc000195fc8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc000195fe8 sp=0xc000195fe0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 67 gp=0xc0002f3dc0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc00049a738 sp=0xc00049a718 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc00049a7c8 sp=0xc00049a738 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc00049a7e0 sp=0xc00049a7c8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc00049a7e8 sp=0xc00049a7e0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 51 gp=0xc0005041c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000222f38 sp=0xc000222f18 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc000222fc8 sp=0xc000222f38 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc000222fe0 sp=0xc000222fc8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc000222fe8 sp=0xc000222fe0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 52 gp=0xc000504380 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000223738 sp=0xc000223718 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc0002237c8 sp=0xc000223738 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc0002237e0 sp=0xc0002237c8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0002237e8 sp=0xc0002237e0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 53 gp=0xc000504540 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000223f38 sp=0xc000223f18 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc000223fc8 sp=0xc000223f38 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc000223fe0 sp=0xc000223fc8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc000223fe8 sp=0xc000223fe0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 54 gp=0xc000504700 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000224738 sp=0xc000224718 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc0002247c8 sp=0xc000224738 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc0002247e0 sp=0xc0002247c8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0002247e8 sp=0xc0002247e0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 55 gp=0xc0005048c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000224f38 sp=0xc000224f18 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc000224fc8 sp=0xc000224f38 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc000224fe0 sp=0xc000224fc8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc000224fe8 sp=0xc000224fe0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 56 gp=0xc000504a80 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000225738 sp=0xc000225718 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc0002257c8 sp=0xc000225738 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc0002257e0 sp=0xc0002257c8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0002257e8 sp=0xc0002257e0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 38 gp=0xc000220700 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000228738 sp=0xc000228718 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc0002287c8 sp=0xc000228738 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc0002287e0 sp=0xc0002287c8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0002287e8 sp=0xc0002287e0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 39 gp=0xc0002208c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000228f38 sp=0xc000228f18 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc000228fc8 sp=0xc000228f38 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc000228fe0 sp=0xc000228fc8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc000228fe8 sp=0xc000228fe0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 68 gp=0xc00049e000 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc00049af38 sp=0xc00049af18 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc00049afc8 sp=0xc00049af38 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc00049afe0 sp=0xc00049afc8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc00049afe8 sp=0xc00049afe0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 69 gp=0xc00049e1c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc00049b738 sp=0xc00049b718 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc00049b7c8 sp=0xc00049b738 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc00049b7e0 sp=0xc00049b7c8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc00049b7e8 sp=0xc00049b7e0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 70 gp=0xc00049e380 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc00049bf38 sp=0xc00049bf18 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc00049bfc8 sp=0xc00049bf38 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc00049bfe0 sp=0xc00049bfc8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc00049bfe8 sp=0xc00049bfe0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 71 gp=0xc00049e540 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc00049c738 sp=0xc00049c718 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc00049c7c8 sp=0xc00049c738 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc00049c7e0 sp=0xc00049c7c8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc00049c7e8 sp=0xc00049c7e0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 57 gp=0xc000504c40 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000225f38 sp=0xc000225f18 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc000225fc8 sp=0xc000225f38 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc000225fe0 sp=0xc000225fc8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc000225fe8 sp=0xc000225fe0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 58 gp=0xc000504e00 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000496738 sp=0xc000496718 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc0004967c8 sp=0xc000496738 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc0004967e0 sp=0xc0004967c8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0004967e8 sp=0xc0004967e0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 59 gp=0xc000504fc0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000496f38 sp=0xc000496f18 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc000496fc8 sp=0xc000496f38 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc000496fe0 sp=0xc000496fc8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc000496fe8 sp=0xc000496fe0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 60 gp=0xc000505180 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000497738 sp=0xc000497718 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc0004977c8 sp=0xc000497738 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc0004977e0 sp=0xc0004977c8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0004977e8 sp=0xc0004977e0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 61 gp=0xc000505340 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000497f38 sp=0xc000497f18 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc000497fc8 sp=0xc000497f38 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc000497fe0 sp=0xc000497fc8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc000497fe8 sp=0xc000497fe0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 62 gp=0xc000505500 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000498738 sp=0xc000498718 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc0004987c8 sp=0xc000498738 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc0004987e0 sp=0xc0004987c8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0004987e8 sp=0xc0004987e0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

goroutine 40 gp=0xc000220a80 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:435 +0xce fp=0xc000229738 sp=0xc000229718 pc=0x559255294d4e
runtime.gcBgMarkWorker(0xc0001cf3b0)
runtime/mgc.go:1423 +0xe9 fp=0xc0002297c8 sp=0xc000229738 pc=0x559255242249
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1339 +0x25 fp=0xc0002297e0 sp=0xc0002297c8 pc=0x559255242125
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0002297e8 sp=0xc0002297e0 pc=0x55925529c481
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1339 +0x105

...

<!-- gh-comment-id:3057693793 --> @chhu commented on GitHub (Jul 10, 2025): Sorry, can't do, production env. The error was different depending on the size of the model, < 1GPU lead to segfaults, >1 lead to cuda malloc errors. A wild guess is that memory calculations are off, it tried to alloc 28G from a card it knows has 23G. ... ggml_backend_cuda_buffer_type_alloc_buffer: allocating 25937.08 MiB on device 1: cudaMalloc failed: out of memory alloc_tensor_range: failed to allocate CUDA1 buffer of size 27197001728 ... ... earlier... CUDA driver version: 12.8 calling cuDeviceGetCount device count 8 time=2025-07-10T15:46:35.438+02:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-1068f375-e3fe-422f-f71b-73a2394e701f name="NVIDIA GeForce RTX 3090" overhead="0 B" before.total="23.6 GiB" before.free="23.3 GiB" now.total="23.6 GiB" now.free="23.3 GiB" now.used="263.3 MiB" time=2025-07-10T15:46:35.668+02:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-615262d3-7dfe-666a-cf37-566e760e4ed1 name="NVIDIA GeForce RTX 3090" overhead="0 B" before.total="23.6 GiB" before.free="23.3 GiB" now.total="23.6 GiB" now.free="23.3 GiB" now.used="263.3 MiB" time=2025-07-10T15:46:35.902+02:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-f22fdea8-5405-3129-d32a-7f916d1f8b7c name="NVIDIA GeForce RTX 3090" overhead="0 B" before.total="23.6 GiB" before.free="23.3 GiB" now.total="23.6 GiB" now.free="23.3 GiB" now.used="263.3 MiB" time=2025-07-10T15:46:36.125+02:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-7f0f16bf-bb19-c951-3cab-78c75e8f1036 name="NVIDIA GeForce RTX 3090" overhead="0 B" before.total="23.6 GiB" before.free="23.3 GiB" now.total="23.6 GiB" now.free="23.3 GiB" now.used="263.3 MiB" time=2025-07-10T15:46:36.351+02:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-02de372e-3119-89d2-b10e-16ce6064e6e0 name="NVIDIA GeForce RTX 3090" overhead="0 B" before.total="23.6 GiB" before.free="23.3 GiB" now.total="23.6 GiB" now.free="23.3 GiB" now.used="263.3 MiB" time=2025-07-10T15:46:36.580+02:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-80159554-dd91-f6ab-b8a5-bbf5a34c59c5 name="NVIDIA GeForce RTX 3090" overhead="0 B" before.total="23.6 GiB" before.free="23.3 GiB" now.total="23.6 GiB" now.free="23.3 GiB" now.used="263.3 MiB" time=2025-07-10T15:46:36.805+02:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-9190655b-d4a4-131c-ae18-7b58cce3b7f3 name="NVIDIA GeForce RTX 3090" overhead="0 B" before.total="23.6 GiB" before.free="23.3 GiB" now.total="23.6 GiB" now.free="23.3 GiB" now.used="263.3 MiB" time=2025-07-10T15:46:37.039+02:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-b355efbb-1628-3337-ee30-11313755b901 name="NVIDIA GeForce RTX 3090" overhead="0 B" before.total="23.6 GiB" before.free="23.3 GiB" now.total="23.6 GiB" now.free="23.3 GiB" now.used="263.3 MiB" releasing cuda driver library time=2025-07-10T15:46:37.039+02:00 level=INFO source=server.go:135 msg="system memory" total="503.2 GiB" free="490.6 GiB" free_swap="0 B" time=2025-07-10T15:46:37.039+02:00 level=DEBUG source=memory.go:111 msg=evaluating library=cuda gpu_count=8 available="[23.3 GiB 23.3 GiB 23.3 GiB 23.3 GiB 23.3 GiB 23.3 GiB 23.3 GiB 23.3 GiB]" time=2025-07-10T15:46:37.041+02:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.2 GiB" before.free="490.6 GiB" before.free_swap="0 B" now.total="503.2 GiB" now.free="490.6 GiB" now.free_swap="0 B" initializing /usr/lib64/libcuda.so.570.124.06 ... repeating itself a few times ? Some hints from the log >1 GPU : print_info: file format = GGUF V3 (latest) print_info: file type = Q8_0 print_info: file size = 32.71 GiB (8.58 BPW) init_tokenizer: initializing tokenizer for type 2 load: control token: 151659 '<|fim_prefix|>' is not marked as EOG load: control token: 151656 '<|video_pad|>' is not marked as EOG load: control token: 151655 '<|image_pad|>' is not marked as EOG load: control token: 151653 '<|vision_end|>' is not marked as EOG load: control token: 151651 '<|quad_end|>' is not marked as EOG load: control token: 151650 '<|quad_start|>' is not marked as EOG load: control token: 151649 '<|box_end|>' is not marked as EOG load: control token: 151647 '<|object_ref_end|>' is not marked as EOG load: control token: 151648 '<|box_start|>' is not marked as EOG load: control token: 151661 '<|fim_suffix|>' is not marked as EOG load: control token: 151646 '<|object_ref_start|>' is not marked as EOG load: control token: 151644 '<|im_start|>' is not marked as EOG load: control token: 151660 '<|fim_middle|>' is not marked as EOG load: control token: 151652 '<|vision_start|>' is not marked as EOG load: control token: 151654 '<|vision_pad|>' is not marked as EOG load: special tokens cache size = 26 load: token to piece cache size = 0.9311 MB print_info: arch = qwen3 print_info: vocab_only = 0 print_info: n_ctx_train = 40960 print_info: n_embd = 5120 print_info: n_layer = 64 print_info: n_head = 64 print_info: n_head_kv = 8 print_info: n_rot = 128 print_info: n_swa = 0 print_info: n_swa_pattern = 1 print_info: n_embd_head_k = 128 print_info: n_embd_head_v = 128 print_info: n_gqa = 8 print_info: n_embd_k_gqa = 1024 print_info: n_embd_v_gqa = 1024 print_info: f_norm_eps = 0.0e+00 print_info: f_norm_rms_eps = 1.0e-06 print_info: f_clamp_kqv = 0.0e+00 print_info: f_max_alibi_bias = 0.0e+00 print_info: f_logit_scale = 0.0e+00 print_info: f_attn_scale = 0.0e+00 print_info: n_ff = 25600 print_info: n_expert = 0 print_info: n_expert_used = 0 print_info: causal attn = 1 print_info: pooling type = 0 print_info: rope type = 2 print_info: rope scaling = linear print_info: freq_base_train = 1000000.0 print_info: freq_scale_train = 1 print_info: n_ctx_orig_yarn = 40960 print_info: rope_finetuned = unknown print_info: ssm_d_conv = 0 print_info: ssm_d_inner = 0 print_info: ssm_d_state = 0 print_info: ssm_dt_rank = 0 print_info: ssm_dt_b_c_rms = 0 print_info: model type = 32B print_info: model params = 32.76 B print_info: general.name = Qwen3 32B print_info: vocab type = BPE print_info: n_vocab = 151936 print_info: n_merges = 151387 print_info: BOS token = 151643 '<|endoftext|>' print_info: EOS token = 151645 '<|im_end|>' print_info: EOT token = 151643 '<|endoftext|>' print_info: PAD token = 151643 '<|endoftext|>' print_info: LF token = 198 'Ċ' print_info: FIM PRE token = 151659 '<|fim_prefix|>' print_info: FIM SUF token = 151661 '<|fim_suffix|>' print_info: FIM MID token = 151660 '<|fim_middle|>' print_info: FIM PAD token = 151662 '<|fim_pad|>' print_info: FIM REP token = 151663 '<|repo_name|>' print_info: FIM SEP token = 151664 '<|file_sep|>' print_info: EOG token = 151643 '<|endoftext|>' print_info: EOG token = 151645 '<|im_end|>' print_info: EOG token = 151662 '<|fim_pad|>' print_info: EOG token = 151663 '<|repo_name|>' print_info: EOG token = 151664 '<|file_sep|>' print_info: max token length = 256 load_tensors: loading model tensors, this can take a while... (mmap = true) load_tensors: layer 0 assigned to device CUDA0, is_swa = 0 load_tensors: layer 1 assigned to device CUDA1, is_swa = 0 load_tensors: layer 2 assigned to device CUDA1, is_swa = 0 load_tensors: layer 3 assigned to device CUDA1, is_swa = 0 load_tensors: layer 4 assigned to device CUDA1, is_swa = 0 load_tensors: layer 5 assigned to device CUDA1, is_swa = 0 load_tensors: layer 6 assigned to device CUDA1, is_swa = 0 load_tensors: layer 7 assigned to device CUDA1, is_swa = 0 load_tensors: layer 8 assigned to device CUDA1, is_swa = 0 load_tensors: layer 9 assigned to device CUDA1, is_swa = 0 load_tensors: layer 10 assigned to device CUDA1, is_swa = 0 load_tensors: layer 11 assigned to device CUDA1, is_swa = 0 load_tensors: layer 12 assigned to device CUDA1, is_swa = 0 load_tensors: layer 13 assigned to device CUDA1, is_swa = 0 load_tensors: layer 14 assigned to device CUDA1, is_swa = 0 load_tensors: layer 15 assigned to device CUDA1, is_swa = 0 load_tensors: layer 16 assigned to device CUDA1, is_swa = 0 load_tensors: layer 17 assigned to device CUDA1, is_swa = 0 load_tensors: layer 18 assigned to device CUDA1, is_swa = 0 load_tensors: layer 19 assigned to device CUDA1, is_swa = 0 load_tensors: layer 20 assigned to device CUDA1, is_swa = 0 load_tensors: layer 21 assigned to device CUDA1, is_swa = 0 load_tensors: layer 22 assigned to device CUDA1, is_swa = 0 load_tensors: layer 23 assigned to device CUDA1, is_swa = 0 load_tensors: layer 24 assigned to device CUDA1, is_swa = 0 load_tensors: layer 25 assigned to device CUDA1, is_swa = 0 load_tensors: layer 26 assigned to device CUDA1, is_swa = 0 load_tensors: layer 27 assigned to device CUDA1, is_swa = 0 load_tensors: layer 28 assigned to device CUDA1, is_swa = 0 load_tensors: layer 29 assigned to device CUDA1, is_swa = 0 load_tensors: layer 30 assigned to device CUDA1, is_swa = 0 load_tensors: layer 31 assigned to device CUDA1, is_swa = 0 load_tensors: layer 32 assigned to device CUDA1, is_swa = 0 load_tensors: layer 33 assigned to device CUDA1, is_swa = 0 load_tensors: layer 34 assigned to device CUDA1, is_swa = 0 load_tensors: layer 35 assigned to device CUDA1, is_swa = 0 load_tensors: layer 36 assigned to device CUDA1, is_swa = 0 load_tensors: layer 37 assigned to device CUDA1, is_swa = 0 load_tensors: layer 38 assigned to device CUDA1, is_swa = 0 load_tensors: layer 39 assigned to device CUDA1, is_swa = 0 load_tensors: layer 40 assigned to device CUDA1, is_swa = 0 load_tensors: layer 41 assigned to device CUDA1, is_swa = 0 load_tensors: layer 42 assigned to device CUDA1, is_swa = 0 load_tensors: layer 43 assigned to device CUDA1, is_swa = 0 load_tensors: layer 44 assigned to device CUDA1, is_swa = 0 load_tensors: layer 45 assigned to device CUDA1, is_swa = 0 load_tensors: layer 46 assigned to device CUDA1, is_swa = 0 load_tensors: layer 47 assigned to device CUDA1, is_swa = 0 load_tensors: layer 48 assigned to device CUDA1, is_swa = 0 load_tensors: layer 49 assigned to device CUDA1, is_swa = 0 load_tensors: layer 50 assigned to device CUDA1, is_swa = 0 load_tensors: layer 51 assigned to device CUDA1, is_swa = 0 load_tensors: layer 52 assigned to device CUDA1, is_swa = 0 load_tensors: layer 53 assigned to device CUDA5, is_swa = 0 load_tensors: layer 54 assigned to device CUDA5, is_swa = 0 load_tensors: layer 55 assigned to device CUDA5, is_swa = 0 load_tensors: layer 56 assigned to device CUDA5, is_swa = 0 load_tensors: layer 57 assigned to device CUDA5, is_swa = 0 load_tensors: layer 58 assigned to device CUDA5, is_swa = 0 load_tensors: layer 59 assigned to device CUDA5, is_swa = 0 load_tensors: layer 60 assigned to device CUDA5, is_swa = 0 load_tensors: layer 61 assigned to device CUDA5, is_swa = 0 load_tensors: layer 62 assigned to device CUDA5, is_swa = 0 load_tensors: layer 63 assigned to device CUDA5, is_swa = 0 load_tensors: layer 64 assigned to device CUDA5, is_swa = 0 load_tensors: tensor 'token_embd.weight' (q8_0) (and 0 others) cannot be used with preferred buffer type CUDA_Host, using CPU instead ggml_backend_cuda_buffer_type_alloc_buffer: allocating 25937.08 MiB on device 1: cudaMalloc failed: out of memory alloc_tensor_range: failed to allocate CUDA1 buffer of size 27197001728 llama_model_load: error loading model: unable to allocate CUDA1 buffer llama_model_load_from_file_impl: failed to load model panic: unable to load model: /scr2/new/ollama/models/models/blobs/sha256-de447d788da3df6b4ea340408b13fc2c3a2043a2dfc19178b12d501a4bd96484 goroutine 125 [running]: github.com/ollama/ollama/runner/llamarunner.(*Server).loadModel(0xc0003d66c0, {0x41, 0x0, 0x1, {0xc000350e20, 0x8, 0x8}, 0xc0005a9050, 0x0}, {0x7ffdf5e1d97b, ...}, ...) github.com/ollama/ollama/runner/llamarunner/runner.go:751 +0x395 created by github.com/ollama/ollama/runner/llamarunner.Execute in goroutine 1 github.com/ollama/ollama/runner/llamarunner/runner.go:848 +0xb57 time=2025-07-10T15:44:57.893+02:00 level=INFO source=server.go:632 msg="waiting for server to become available" status="llm server not responding" time=2025-07-10T15:44:58.502+02:00 level=ERROR source=server.go:464 msg="llama runner terminated" error="exit status 2" time=2025-07-10T15:44:58.645+02:00 level=ERROR source=sched.go:489 msg="error loading llama server" error="llama runner process has terminated: error loading model: unable to allocate CUDA1 buffer" time=2025-07-10T15:44:58.645+02:00 level=DEBUG source=sched.go:491 msg="triggering expiration for failed load" runner.name=registry.ollama.ai/library/qwen3:32b-q8_0 runner.inference=cuda runner.devices=8 runner.size="51.2 GiB" runner.vram="51.2 GiB" runner.parallel=1 runner.pid=764615 <1 GPU: ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 8 CUDA devices: Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Device 3: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Device 4: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Device 5: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Device 6: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Device 7: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes load_backend: loaded CUDA backend from /scr2/new/ollama/lib/ollama/libggml-cuda.so load_backend: loaded CPU backend from /scr2/new/ollama/lib/ollama/libggml-cpu-haswell.so time=2025-07-10T15:46:40.303+02:00 level=DEBUG source=ggml.go:94 msg="ggml backend load all from path" path=/scr2/new/ollama/lib/ollama/cuda_v12 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 8 CUDA devices: Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Device 3: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Device 4: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Device 5: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Device 6: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Device 7: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes load_backend: loaded CUDA backend from /scr2/new/ollama/lib/ollama/cuda_v12/libggml-cuda.so time=2025-07-10T15:46:40.493+02:00 level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 CUDA.1.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.1.USE_GRAPHS=1 CUDA.1.PEER_MAX_BATCH_SIZE=128 CUDA.2.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.2.USE_GRAPHS=1 CUDA.2.PEER_MAX_BATCH_SIZE=128 CUDA.3.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.3.USE_GRAPHS=1 CUDA.3.PEER_MAX_BATCH_SIZE=128 CUDA.4.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.4.USE_GRAPHS=1 CUDA.4.PEER_MAX_BATCH_SIZE=128 CUDA.5.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.5.USE_GRAPHS=1 CUDA.5.PEER_MAX_BATCH_SIZE=128 CUDA.6.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.6.USE_GRAPHS=1 CUDA.6.PEER_MAX_BATCH_SIZE=128 CUDA.7.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.7.USE_GRAPHS=1 CUDA.7.PEER_MAX_BATCH_SIZE=128 CUDA.8.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.8.USE_GRAPHS=1 CUDA.8.PEER_MAX_BATCH_SIZE=128 CUDA.9.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.9.USE_GRAPHS=1 CUDA.9.PEER_MAX_BATCH_SIZE=128 CUDA.10.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.10.USE_GRAPHS=1 CUDA.10.PEER_MAX_BATCH_SIZE=128 CUDA.11.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.11.USE_GRAPHS=1 CUDA.11.PEER_MAX_BATCH_SIZE=128 CUDA.12.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.12.USE_GRAPHS=1 CUDA.12.PEER_MAX_BATCH_SIZE=128 CUDA.13.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.13.USE_GRAPHS=1 CUDA.13.PEER_MAX_BATCH_SIZE=128 CUDA.14.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.14.USE_GRAPHS=1 CUDA.14.PEER_MAX_BATCH_SIZE=128 CUDA.15.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.15.USE_GRAPHS=1 CUDA.15.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc) unexpected fault address 0x5d3fe0000 fatal error: fault [signal SIGSEGV: segmentation violation code=0x2 addr=0x5d3fe0000 pc=0x559255224780] goroutine 116 gp=0xc000522700 m=4 mp=0xc00019d808 [running]: runtime.throw({0x55925613d479?, 0xc000522700?}) runtime/panic.go:1096 +0x4a fp=0xc0018930b8 sp=0xc001893088 pc=0x559255294c2a runtime.sigpanic() runtime/signal_unix.go:939 +0x26c fp=0xc001893118 sp=0xc0018930b8 pc=0x5592552970ac indexbytebody() internal/bytealg/indexbyte_amd64.s:131 +0xe0 fp=0xc001893120 sp=0xc001893118 pc=0x559255224780 runtime.findnull(0xc0018931a0?) runtime/string.go:577 +0x79 fp=0xc001893178 sp=0xc001893120 pc=0x55925527c8b9 runtime.gostring(0x5d3fe0000) runtime/string.go:363 +0x1c fp=0xc0018931b0 sp=0xc001893178 pc=0x559255297f1c github.com/ollama/ollama/ml/backend/ggml._Cfunc_GoString(...) _cgo_gotypes.go:311 github.com/ollama/ollama/ml/backend/ggml.New({0x7ffc98d8b962, 0x6c}, {0x40, 0x0, 0x31, {0xc0005bede0, 0x8, 0x8}, 0x1}) github.com/ollama/ollama/ml/backend/ggml/ggml.go:158 +0x1336 fp=0xc001893c18 sp=0xc0018931b0 pc=0x5592556d1376 github.com/ollama/ollama/ml.NewBackend({0x7ffc98d8b962, 0x6c}, {0x40, 0x0, 0x31, {0xc0005bede0, 0x8, 0x8}, 0x1}) github.com/ollama/ollama/ml/backend.go:209 +0xb1 fp=0xc001893c70 sp=0xc001893c18 pc=0x5592556c2e11 github.com/ollama/ollama/model.New({0x7ffc98d8b962?, 0x0?}, {0x40, 0x0, 0x31, {0xc0005bede0, 0x8, 0x8}, 0x1}) github.com/ollama/ollama/model/model.go:102 +0x8f fp=0xc001893d68 sp=0xc001893c70 pc=0x5592556e0d2f github.com/ollama/ollama/runner/ollamarunner.(*Server).initModel(0xc0005f4ea0, {0x7ffc98d8b962?, 0x0?}, {0x40, 0x0, 0x31, {0xc0005bede0, 0x8, 0x8}, 0x1}, ...) github.com/ollama/ollama/runner/ollamarunner/runner.go:841 +0x8d fp=0xc001893dc8 sp=0xc001893d68 pc=0x559255782e6d github.com/ollama/ollama/runner/ollamarunner.(*Server).load(0xc0005f4ea0, {0x5592565f5700, 0xc000597db0}, {0x7ffc98d8b962?, 0x0?}, {0x40, 0x0, 0x31, {0xc0005bede0, 0x8, ...}, ...}, ...) github.com/ollama/ollama/runner/ollamarunner/runner.go:878 +0xb8 fp=0xc001893f20 sp=0xc001893dc8 pc=0x5592557831d8 github.com/ollama/ollama/runner/ollamarunner.Execute.gowrap1() github.com/ollama/ollama/runner/ollamarunner/runner.go:959 +0xc7 fp=0xc001893fe0 sp=0xc001893f20 pc=0x559255784607 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc001893fe8 sp=0xc001893fe0 pc=0x55925529c481 created by github.com/ollama/ollama/runner/ollamarunner.Execute in goroutine 1 github.com/ollama/ollama/runner/ollamarunner/runner.go:959 +0xa11 goroutine 1 gp=0xc000002380 m=nil [IO wait]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000055650 sp=0xc000055630 pc=0x559255294d4e runtime.netpollblock(0xc0000556a0?, 0x5522db46?, 0x92?) runtime/netpoll.go:575 +0xf7 fp=0xc000055688 sp=0xc000055650 pc=0x559255259837 internal/poll.runtime_pollWait(0x7effac51ce70, 0x72) runtime/netpoll.go:351 +0x85 fp=0xc0000556a8 sp=0xc000055688 pc=0x559255293f65 internal/poll.(*pollDesc).wait(0xc0005d5900?, 0x900000036?, 0x0) internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0000556d0 sp=0xc0000556a8 pc=0x55925531b3a7 internal/poll.(*pollDesc).waitRead(...) internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Accept(0xc0005d5900) internal/poll/fd_unix.go:620 +0x295 fp=0xc000055778 sp=0xc0000556d0 pc=0x559255320775 net.(*netFD).accept(0xc0005d5900) net/fd_unix.go:172 +0x29 fp=0xc000055830 sp=0xc000055778 pc=0x559255392c89 net.(*TCPListener).accept(0xc0005a7d80) net/tcpsock_posix.go:159 +0x1b fp=0xc000055880 sp=0xc000055830 pc=0x5592553a863b net.(*TCPListener).Accept(0xc0005a7d80) net/tcpsock.go:380 +0x30 fp=0xc0000558b0 sp=0xc000055880 pc=0x5592553a74f0 net/http.(*onceCloseListener).Accept(0xc0003d6240?) <autogenerated>:1 +0x24 fp=0xc0000558c8 sp=0xc0000558b0 pc=0x5592555bec44 net/http.(*Server).Serve(0xc000fb0000, {0x5592565f3288, 0xc0005a7d80}) net/http/server.go:3424 +0x30c fp=0xc0000559f8 sp=0xc0000558c8 pc=0x55925559650c github.com/ollama/ollama/runner/ollamarunner.Execute({0xc000034170, 0x11, 0x11}) github.com/ollama/ollama/runner/ollamarunner/runner.go:984 +0xe09 fp=0xc000055d08 sp=0xc0000559f8 pc=0x559255784269 github.com/ollama/ollama/runner.Execute({0xc000034150?, 0x0?, 0x0?}) github.com/ollama/ollama/runner/runner.go:20 +0xc9 fp=0xc000055d30 sp=0xc000055d08 pc=0x559255784b69 github.com/ollama/ollama/cmd.NewCLI.func2(0xc000311800?, {0x55925613c075?, 0x4?, 0x55925613c079?}) github.com/ollama/ollama/cmd/cmd.go:1529 +0x45 fp=0xc000055d58 sp=0xc000055d30 pc=0x559255ee18c5 github.com/spf13/cobra.(*Command).execute(0xc00061cf08, {0xc0005f4b40, 0x12, 0x12}) github.com/spf13/cobra@v1.7.0/command.go:940 +0x85c fp=0xc000055e78 sp=0xc000055d58 pc=0x55925540c2dc github.com/spf13/cobra.(*Command).ExecuteC(0xc0005b4908) github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3a5 fp=0xc000055f30 sp=0xc000055e78 pc=0x55925540cb25 github.com/spf13/cobra.(*Command).Execute(...) github.com/spf13/cobra@v1.7.0/command.go:992 github.com/spf13/cobra.(*Command).ExecuteContext(...) github.com/spf13/cobra@v1.7.0/command.go:985 main.main() github.com/ollama/ollama/main.go:12 +0x4d fp=0xc000055f50 sp=0xc000055f30 pc=0x559255ee234d runtime.main() runtime/proc.go:283 +0x29d fp=0xc000055fe0 sp=0xc000055f50 pc=0x559255260ebd runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000055fe8 sp=0xc000055fe0 pc=0x55925529c481 goroutine 2 gp=0xc000002e00 m=nil [force gc (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000196fa8 sp=0xc000196f88 pc=0x559255294d4e runtime.goparkunlock(...) runtime/proc.go:441 runtime.forcegchelper() runtime/proc.go:348 +0xb8 fp=0xc000196fe0 sp=0xc000196fa8 pc=0x5592552611f8 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000196fe8 sp=0xc000196fe0 pc=0x55925529c481 created by runtime.init.7 in goroutine 1 runtime/proc.go:336 +0x1a goroutine 3 gp=0xc000003340 m=nil [GC sweep wait]: runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000197780 sp=0xc000197760 pc=0x559255294d4e runtime.goparkunlock(...) runtime/proc.go:441 runtime.bgsweep(0xc000042100) runtime/mgcsweep.go:316 +0xdf fp=0xc0001977c8 sp=0xc000197780 pc=0x55925524b99f runtime.gcenable.gowrap1() runtime/mgc.go:204 +0x25 fp=0xc0001977e0 sp=0xc0001977c8 pc=0x55925523fd85 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0001977e8 sp=0xc0001977e0 pc=0x55925529c481 created by runtime.gcenable in goroutine 1 runtime/mgc.go:204 +0x66 goroutine 4 gp=0xc000003500 m=nil [GC scavenge wait]: runtime.gopark(0x10000?, 0x5592562fbf10?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000197f78 sp=0xc000197f58 pc=0x559255294d4e runtime.goparkunlock(...) runtime/proc.go:441 runtime.(*scavengerState).park(0x559256e838e0) runtime/mgcscavenge.go:425 +0x49 fp=0xc000197fa8 sp=0xc000197f78 pc=0x5592552493e9 runtime.bgscavenge(0xc000042100) runtime/mgcscavenge.go:658 +0x59 fp=0xc000197fc8 sp=0xc000197fa8 pc=0x559255249979 runtime.gcenable.gowrap2() runtime/mgc.go:205 +0x25 fp=0xc000197fe0 sp=0xc000197fc8 pc=0x55925523fd25 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000197fe8 sp=0xc000197fe0 pc=0x55925529c481 created by runtime.gcenable in goroutine 1 runtime/mgc.go:205 +0xa5 goroutine 5 gp=0xc000003dc0 m=nil [finalizer wait]: runtime.gopark(0x1b8?, 0xc000002380?, 0x1?, 0x23?, 0xc000196688?) runtime/proc.go:435 +0xce fp=0xc000196630 sp=0xc000196610 pc=0x559255294d4e runtime.runfinq() runtime/mfinal.go:196 +0x107 fp=0xc0001967e0 sp=0xc000196630 pc=0x55925523ed47 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0001967e8 sp=0xc0001967e0 pc=0x55925529c481 created by runtime.createfing in goroutine 1 runtime/mfinal.go:166 +0x3d goroutine 6 gp=0xc0002f28c0 m=nil [chan receive]: runtime.gopark(0xc000343860?, 0xc001080048?, 0x60?, 0x87?, 0x5592553799c8?) runtime/proc.go:435 +0xce fp=0xc000198718 sp=0xc0001986f8 pc=0x559255294d4e runtime.chanrecv(0xc0001ce310, 0x0, 0x1) runtime/chan.go:664 +0x445 fp=0xc000198790 sp=0xc000198718 pc=0x559255230725 runtime.chanrecv1(0x0?, 0x0?) runtime/chan.go:506 +0x12 fp=0xc0001987b8 sp=0xc000198790 pc=0x5592552302b2 runtime.unique_runtime_registerUniqueMapCleanup.func2(...) runtime/mgc.go:1796 runtime.unique_runtime_registerUniqueMapCleanup.gowrap1() runtime/mgc.go:1799 +0x2f fp=0xc0001987e0 sp=0xc0001987b8 pc=0x559255242f2f runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0001987e8 sp=0xc0001987e0 pc=0x55925529c481 created by unique.runtime_registerUniqueMapCleanup in goroutine 1 runtime/mgc.go:1794 +0x85 goroutine 7 gp=0xc0002f2a80 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000198f38 sp=0xc000198f18 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc000198fc8 sp=0xc000198f38 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc000198fe0 sp=0xc000198fc8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000198fe8 sp=0xc000198fe0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 8 gp=0xc0002f2c40 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000199738 sp=0xc000199718 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc0001997c8 sp=0xc000199738 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc0001997e0 sp=0xc0001997c8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0001997e8 sp=0xc0001997e0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 9 gp=0xc0002f2e00 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000199f38 sp=0xc000199f18 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc000199fc8 sp=0xc000199f38 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc000199fe0 sp=0xc000199fc8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000199fe8 sp=0xc000199fe0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 10 gp=0xc0002f2fc0 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000192738 sp=0xc000192718 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc0001927c8 sp=0xc000192738 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc0001927e0 sp=0xc0001927c8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0001927e8 sp=0xc0001927e0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 11 gp=0xc0002f3180 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000192f38 sp=0xc000192f18 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc000192fc8 sp=0xc000192f38 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc000192fe0 sp=0xc000192fc8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000192fe8 sp=0xc000192fe0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 18 gp=0xc000202380 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc00021a738 sp=0xc00021a718 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc00021a7c8 sp=0xc00021a738 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc00021a7e0 sp=0xc00021a7c8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00021a7e8 sp=0xc00021a7e0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 19 gp=0xc000202540 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc00021af38 sp=0xc00021af18 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc00021afc8 sp=0xc00021af38 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc00021afe0 sp=0xc00021afc8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00021afe8 sp=0xc00021afe0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 20 gp=0xc000202700 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc00021b738 sp=0xc00021b718 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc00021b7c8 sp=0xc00021b738 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc00021b7e0 sp=0xc00021b7c8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00021b7e8 sp=0xc00021b7e0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 21 gp=0xc0002028c0 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc00021bf38 sp=0xc00021bf18 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc00021bfc8 sp=0xc00021bf38 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc00021bfe0 sp=0xc00021bfc8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00021bfe8 sp=0xc00021bfe0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 22 gp=0xc000202a80 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc00021c738 sp=0xc00021c718 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc00021c7c8 sp=0xc00021c738 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc00021c7e0 sp=0xc00021c7c8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00021c7e8 sp=0xc00021c7e0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 23 gp=0xc000202c40 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc00021cf38 sp=0xc00021cf18 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc00021cfc8 sp=0xc00021cf38 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc00021cfe0 sp=0xc00021cfc8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00021cfe8 sp=0xc00021cfe0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 24 gp=0xc000202e00 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc00021d738 sp=0xc00021d718 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc00021d7c8 sp=0xc00021d738 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc00021d7e0 sp=0xc00021d7c8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00021d7e8 sp=0xc00021d7e0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 25 gp=0xc000202fc0 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc00021df38 sp=0xc00021df18 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc00021dfc8 sp=0xc00021df38 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc00021dfe0 sp=0xc00021dfc8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00021dfe8 sp=0xc00021dfe0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 26 gp=0xc000203180 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000216738 sp=0xc000216718 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc0002167c8 sp=0xc000216738 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc0002167e0 sp=0xc0002167c8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0002167e8 sp=0xc0002167e0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 27 gp=0xc000203340 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000216f38 sp=0xc000216f18 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc000216fc8 sp=0xc000216f38 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc000216fe0 sp=0xc000216fc8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000216fe8 sp=0xc000216fe0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 28 gp=0xc000203500 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000217738 sp=0xc000217718 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc0002177c8 sp=0xc000217738 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc0002177e0 sp=0xc0002177c8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0002177e8 sp=0xc0002177e0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 29 gp=0xc0002036c0 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000217f38 sp=0xc000217f18 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc000217fc8 sp=0xc000217f38 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc000217fe0 sp=0xc000217fc8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000217fe8 sp=0xc000217fe0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 30 gp=0xc000203880 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000218738 sp=0xc000218718 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc0002187c8 sp=0xc000218738 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc0002187e0 sp=0xc0002187c8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0002187e8 sp=0xc0002187e0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 31 gp=0xc000203a40 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000218f38 sp=0xc000218f18 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc000218fc8 sp=0xc000218f38 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc000218fe0 sp=0xc000218fc8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000218fe8 sp=0xc000218fe0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 32 gp=0xc000203c00 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000219738 sp=0xc000219718 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc0002197c8 sp=0xc000219738 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc0002197e0 sp=0xc0002197c8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0002197e8 sp=0xc0002197e0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 33 gp=0xc000203dc0 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000219f38 sp=0xc000219f18 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc000219fc8 sp=0xc000219f38 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc000219fe0 sp=0xc000219fc8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000219fe8 sp=0xc000219fe0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 34 gp=0xc000220000 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000226738 sp=0xc000226718 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc0002267c8 sp=0xc000226738 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc0002267e0 sp=0xc0002267c8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0002267e8 sp=0xc0002267e0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 35 gp=0xc0002201c0 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000226f38 sp=0xc000226f18 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc000226fc8 sp=0xc000226f38 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc000226fe0 sp=0xc000226fc8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000226fe8 sp=0xc000226fe0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 36 gp=0xc000220380 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000227738 sp=0xc000227718 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc0002277c8 sp=0xc000227738 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc0002277e0 sp=0xc0002277c8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0002277e8 sp=0xc0002277e0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 37 gp=0xc000220540 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000227f38 sp=0xc000227f18 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc000227fc8 sp=0xc000227f38 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc000227fe0 sp=0xc000227fc8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000227fe8 sp=0xc000227fe0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 12 gp=0xc0002f3340 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000193738 sp=0xc000193718 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc0001937c8 sp=0xc000193738 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc0001937e0 sp=0xc0001937c8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0001937e8 sp=0xc0001937e0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 13 gp=0xc0002f3500 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000193f38 sp=0xc000193f18 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc000193fc8 sp=0xc000193f38 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc000193fe0 sp=0xc000193fc8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000193fe8 sp=0xc000193fe0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 14 gp=0xc0002f36c0 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000194738 sp=0xc000194718 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc0001947c8 sp=0xc000194738 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc0001947e0 sp=0xc0001947c8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0001947e8 sp=0xc0001947e0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 15 gp=0xc0002f3880 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000194f38 sp=0xc000194f18 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc000194fc8 sp=0xc000194f38 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc000194fe0 sp=0xc000194fc8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000194fe8 sp=0xc000194fe0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 16 gp=0xc0002f3a40 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000195738 sp=0xc000195718 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc0001957c8 sp=0xc000195738 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc0001957e0 sp=0xc0001957c8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0001957e8 sp=0xc0001957e0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 50 gp=0xc000504000 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000222738 sp=0xc000222718 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc0002227c8 sp=0xc000222738 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc0002227e0 sp=0xc0002227c8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0002227e8 sp=0xc0002227e0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 66 gp=0xc0002f3c00 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000195f38 sp=0xc000195f18 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc000195fc8 sp=0xc000195f38 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc000195fe0 sp=0xc000195fc8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000195fe8 sp=0xc000195fe0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 67 gp=0xc0002f3dc0 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc00049a738 sp=0xc00049a718 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc00049a7c8 sp=0xc00049a738 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc00049a7e0 sp=0xc00049a7c8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00049a7e8 sp=0xc00049a7e0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 51 gp=0xc0005041c0 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000222f38 sp=0xc000222f18 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc000222fc8 sp=0xc000222f38 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc000222fe0 sp=0xc000222fc8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000222fe8 sp=0xc000222fe0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 52 gp=0xc000504380 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000223738 sp=0xc000223718 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc0002237c8 sp=0xc000223738 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc0002237e0 sp=0xc0002237c8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0002237e8 sp=0xc0002237e0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 53 gp=0xc000504540 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000223f38 sp=0xc000223f18 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc000223fc8 sp=0xc000223f38 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc000223fe0 sp=0xc000223fc8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000223fe8 sp=0xc000223fe0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 54 gp=0xc000504700 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000224738 sp=0xc000224718 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc0002247c8 sp=0xc000224738 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc0002247e0 sp=0xc0002247c8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0002247e8 sp=0xc0002247e0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 55 gp=0xc0005048c0 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000224f38 sp=0xc000224f18 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc000224fc8 sp=0xc000224f38 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc000224fe0 sp=0xc000224fc8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000224fe8 sp=0xc000224fe0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 56 gp=0xc000504a80 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000225738 sp=0xc000225718 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc0002257c8 sp=0xc000225738 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc0002257e0 sp=0xc0002257c8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0002257e8 sp=0xc0002257e0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 38 gp=0xc000220700 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000228738 sp=0xc000228718 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc0002287c8 sp=0xc000228738 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc0002287e0 sp=0xc0002287c8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0002287e8 sp=0xc0002287e0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 39 gp=0xc0002208c0 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000228f38 sp=0xc000228f18 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc000228fc8 sp=0xc000228f38 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc000228fe0 sp=0xc000228fc8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000228fe8 sp=0xc000228fe0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 68 gp=0xc00049e000 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc00049af38 sp=0xc00049af18 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc00049afc8 sp=0xc00049af38 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc00049afe0 sp=0xc00049afc8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00049afe8 sp=0xc00049afe0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 69 gp=0xc00049e1c0 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc00049b738 sp=0xc00049b718 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc00049b7c8 sp=0xc00049b738 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc00049b7e0 sp=0xc00049b7c8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00049b7e8 sp=0xc00049b7e0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 70 gp=0xc00049e380 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc00049bf38 sp=0xc00049bf18 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc00049bfc8 sp=0xc00049bf38 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc00049bfe0 sp=0xc00049bfc8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00049bfe8 sp=0xc00049bfe0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 71 gp=0xc00049e540 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc00049c738 sp=0xc00049c718 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc00049c7c8 sp=0xc00049c738 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc00049c7e0 sp=0xc00049c7c8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00049c7e8 sp=0xc00049c7e0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 57 gp=0xc000504c40 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000225f38 sp=0xc000225f18 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc000225fc8 sp=0xc000225f38 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc000225fe0 sp=0xc000225fc8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000225fe8 sp=0xc000225fe0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 58 gp=0xc000504e00 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000496738 sp=0xc000496718 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc0004967c8 sp=0xc000496738 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc0004967e0 sp=0xc0004967c8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0004967e8 sp=0xc0004967e0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 59 gp=0xc000504fc0 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000496f38 sp=0xc000496f18 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc000496fc8 sp=0xc000496f38 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc000496fe0 sp=0xc000496fc8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000496fe8 sp=0xc000496fe0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 60 gp=0xc000505180 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000497738 sp=0xc000497718 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc0004977c8 sp=0xc000497738 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc0004977e0 sp=0xc0004977c8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0004977e8 sp=0xc0004977e0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 61 gp=0xc000505340 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000497f38 sp=0xc000497f18 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc000497fc8 sp=0xc000497f38 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc000497fe0 sp=0xc000497fc8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000497fe8 sp=0xc000497fe0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 62 gp=0xc000505500 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000498738 sp=0xc000498718 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc0004987c8 sp=0xc000498738 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc0004987e0 sp=0xc0004987c8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0004987e8 sp=0xc0004987e0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 40 gp=0xc000220a80 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000229738 sp=0xc000229718 pc=0x559255294d4e runtime.gcBgMarkWorker(0xc0001cf3b0) runtime/mgc.go:1423 +0xe9 fp=0xc0002297c8 sp=0xc000229738 pc=0x559255242249 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc0002297e0 sp=0xc0002297c8 pc=0x559255242125 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0002297e8 sp=0xc0002297e0 pc=0x55925529c481 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 ...
GiteaMirror commented 2026-04-29 05:05:10 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jul 10, 2025):

Sorry, can't do, production env

OK. For generic ways to mitigate OOMs, see here.

<!-- gh-comment-id:3057741952 --> @rick-github commented on GitHub (Jul 10, 2025): > Sorry, can't do, production env OK. For generic ways to mitigate OOMs, see [here](https://github.com/ollama/ollama/issues/8597#issuecomment-2614533288).
GiteaMirror commented 2026-04-29 05:05:11 -05:00
Author
Owner
Copy Link

@chhu commented on GitHub (Jul 10, 2025):

The test ran w/ flash attn = 1, parallel =1. Besides, a gemma3 12B q4 w/ default context should fit comfy on a single 3090.

<!-- gh-comment-id:3057919574 --> @chhu commented on GitHub (Jul 10, 2025): The test ran w/ flash attn = 1, parallel =1. Besides, a gemma3 12B q4 w/ default context should fit comfy on a single 3090.
GiteaMirror commented 2026-04-29 05:05:11 -05:00
Author
Owner
Copy Link

@jessegross commented on GitHub (Jul 10, 2025):

One of the cases looks like https://github.com/ollama/ollama/issues/11211

The other does look like an OOM but I would start by addressing the above issue since it sounds like the problems started happening at the same time for you.

<!-- gh-comment-id:3058341559 --> @jessegross commented on GitHub (Jul 10, 2025): One of the cases looks like https://github.com/ollama/ollama/issues/11211 The other does look like an OOM but I would start by addressing the above issue since it sounds like the problems started happening at the same time for you.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#54006
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?