[GH-ISSUE #1019] Error: llama runner exited #498

New Issue

GiteaMirror · 2026-04-12T10:11:50-05:00

GiteaMirror commented

2026-04-12 10:11:50 -05:00

Originally created by @krenax on GitHub (Nov 6, 2023).
Original GitHub issue: https://github.com/ollama/ollama/issues/1019

Originally assigned to: @BruceMacD on GitHub.

Using mistral and llama2 with ollama, I received the following error message: Error: llama runner exited, you may not have enough available memory to run this model?.

The README.md states that at least 16GB of RAM is required to run 7B models, which is met by my workstation specifications.

Originally created by @krenax on GitHub (Nov 6, 2023). Original GitHub issue: https://github.com/ollama/ollama/issues/1019 Originally assigned to: @BruceMacD on GitHub. Using mistral and llama2 with ollama, I received the following error message: `Error: llama runner exited, you may not have enough available memory to run this model?`. The `README.md` states that at least 16GB of RAM is required to run 7B models, which is met by my workstation specifications.

GiteaMirror closed this issue

2026-04-12 10:11:51 -05:00

GiteaMirror commented

2026-04-12 10:11:51 -05:00

@igorschlum commented on GitHub (Nov 6, 2023):

@kraemi. 16GB or RAM to run 7B models, means 16 GB for Ollama, not 16 GB for the system + some apps + Ollama.try to restart your mac (if you are running ollama on a mac). Quite all applications and start Ollama. This way you may be able to run the models but with very slow answers.

I have a 32Gb powerbook and I can run falcon:40b the answers are very slow to show, but it works this way.

@igorschlum commented on GitHub (Nov 6, 2023): @kraemi. 16GB or RAM to run 7B models, means 16 GB for Ollama, not 16 GB for the system + some apps + Ollama.try to restart your mac (if you are running ollama on a mac). Quite all applications and start Ollama. This way you may be able to run the models but with very slow answers. I have a 32Gb powerbook and I can run falcon:40b the answers are very slow to show, but it works this way.

GiteaMirror commented

2026-04-12 10:11:51 -05:00

@BruceMacD commented on GitHub (Nov 7, 2023):

@kraemi do you have an nvidia gpu? Im wondering if this could be from the VRAM getting filled up.

@BruceMacD commented on GitHub (Nov 7, 2023): @kraemi do you have an nvidia gpu? Im wondering if this could be from the VRAM getting filled up.

GiteaMirror commented

2026-04-12 10:11:52 -05:00

@krenax commented on GitHub (Nov 7, 2023):

@BruceMacD, I am using the NVIDIA GeForce GT 1030 with 2GB VRAM.

@krenax commented on GitHub (Nov 7, 2023): @BruceMacD, I am using the NVIDIA GeForce GT 1030 with 2GB VRAM.

GiteaMirror commented

2026-04-12 10:11:52 -05:00

@ahaslam commented on GitHub (Nov 8, 2023):

ollama version 0.1.8

I too am getting this error message - but only when I attempt to use samantha-mistral:latest or zephyr:latest.

When I use mistral:latest, it runs fine. Since they are all sourced from the same 7B model, it sounds like there's something else other than a memory constraint going on.

In case it was possibly being caused by having multiple concurrent runners at the same time exhausting memory, I also tried killing the ollama server and any other ollama related processes, restarting and running one of the problematic models and the problem still persisted.

I am running ollama on an older Macbook Pro 2015, 2.8GHz, 16Gb RAM, no GPU

When I run "ollama serve" followed by "ollama run samantha-mistral:latest" then prompting it, I get the following logging output - the error message about possibly not having enough memory correlates with the segmentation fault error message at the bottom of the log output.

I would have expected some other error message if it had insufficient memory available.

_
(base) test@test-MBP-6> ~/Desktop/dev/ollama-ui
2023/11/08 13:26:45 images.go:824: 2023/11/08 13:26:45 images.go:831: 2023/11/08 13:26:45 routes.go:680: [GIN] 2023/11/08 - 13:26:50 | 200 | [GIN] 2023/11/08 - 13:26:50 | 200 | 2023/11/08 13:26:51 llama.go:384: 2023/11/08 13:26:51 llama.go:386: 2023/11/08 13:26:51 llama.go:384: 2023/11/08 13:26:51 llama.go:442: {"timestamp":1699403211,&# {"timestamp":1699403211,&# {"timestamp":1699403211,&# llama_model_loader: loaded llama_model_loader: - tensor 0: llama_model_loader: - tensor 1: llama_model_loader: - tensor 2: llama_model_loader: - tensor 3: llama_model_loader: - tensor 4: llama_model_loader: - tensor 5: llama_model_loader: - tensor 6: llama_model_loader: - tensor 7: llama_model_loader: - tensor 8: llama_model_loader: - tensor 9: llama_model_loader: - tensor 10: llama_model_loader: - tensor 11: llama_model_loader: - tensor 12: llama_model_loader: - tensor 13: llama_model_loader: - tensor 14: llama_model_loader: - tensor 15: llama_model_loader: - tensor 16: llama_model_loader: - tensor 17: llama_model_loader: - tensor 18: llama_model_loader: - tensor 19: llama_model_loader: - tensor 20: llama_model_loader: - tensor 21: llama_model_loader: - tensor 22: llama_model_loader: - tensor 23: llama_model_loader: - tensor 24: llama_model_loader: - tensor 25: llama_model_loader: - tensor 26: llama_model_loader: - tensor 27: llama_model_loader: - tensor 28: llama_model_loader: - tensor 29: llama_model_loader: - tensor 30: llama_model_loader: - tensor 31: llama_model_loader: - tensor 32: llama_model_loader: - tensor 33: llama_model_loader: - tensor 34: llama_model_loader: - tensor 35: llama_model_loader: - tensor 36: llama_model_loader: - tensor 37: llama_model_loader: - tensor 38: llama_model_loader: - tensor 39: llama_model_loader: - tensor 40: llama_model_loader: - tensor 41: llama_model_loader: - tensor 42: llama_model_loader: - tensor 43: llama_model_loader: - tensor 44: llama_model_loader: - tensor 45: llama_model_loader: - tensor 46: llama_model_loader: - tensor 47: llama_model_loader: - tensor 48: llama_model_loader: - tensor 49: llama_model_loader: - tensor 50: llama_model_loader: - tensor 51: llama_model_loader: - tensor 52: llama_model_loader: - tensor 53: llama_model_loader: - tensor 54: llama_model_loader: - tensor 55: llama_model_loader: - tensor 56: llama_model_loader: - tensor 57: llama_model_loader: - tensor 58: llama_model_loader: - tensor 59: llama_model_loader: - tensor 60: llama_model_loader: - tensor 61: llama_model_loader: - tensor 62: llama_model_loader: - tensor 63: llama_model_loader: - tensor 64: llama_model_loader: - tensor 65: llama_model_loader: - tensor 66: llama_model_loader: - tensor 67: llama_model_loader: - tensor 68: llama_model_loader: - tensor 69: llama_model_loader: - tensor 70: llama_model_loader: - tensor 71: llama_model_loader: - tensor 72: llama_model_loader: - tensor 73: llama_model_loader: - tensor 74: llama_model_loader: - tensor 75: llama_model_loader: - tensor 76: llama_model_loader: - tensor 77: llama_model_loader: - tensor 78: llama_model_loader: - tensor 79: llama_model_loader: - tensor 80: llama_model_loader: - tensor 81: llama_model_loader: - tensor 82: llama_model_loader: - tensor 83: llama_model_loader: - tensor 84: llama_model_loader: - tensor 85: llama_model_loader: - tensor 86: llama_model_loader: - tensor 87: llama_model_loader: - tensor 88: llama_model_loader: - tensor 89: llama_model_loader: - tensor 90: llama_model_loader: - tensor 91: llama_model_loader: - tensor 92: llama_model_loader: - tensor 93: llama_model_loader: - tensor 94: llama_model_loader: - tensor 95: llama_model_loader: - tensor 96: llama_model_loader: - tensor 97: llama_model_loader: - tensor 98: llama_model_loader: - tensor 99: llama_model_loader: - tensor 100: llama_model_loader: - tensor 101: llama_model_loader: - tensor 102: llama_model_loader: - tensor 103: llama_model_loader: - tensor 104: llama_model_loader: - tensor 105: llama_model_loader: - tensor 106: llama_model_loader: - tensor 107: llama_model_loader: - tensor 108: llama_model_loader: - tensor 109: llama_model_loader: - tensor 110: llama_model_loader: - tensor 111: llama_model_loader: - tensor 112: llama_model_loader: - tensor 113: llama_model_loader: - tensor 114: llama_model_loader: - tensor 115: llama_model_loader: - tensor 116: llama_model_loader: - tensor 117: llama_model_loader: - tensor 118: llama_model_loader: - tensor 119: llama_model_loader: - tensor 120: llama_model_loader: - tensor 121: llama_model_loader: - tensor 122: llama_model_loader: - tensor 123: llama_model_loader: - tensor 124: llama_model_loader: - tensor 125: llama_model_loader: - tensor 126: llama_model_loader: - tensor 127: llama_model_loader: - tensor 128: llama_model_loader: - tensor 129: llama_model_loader: - tensor 130: llama_model_loader: - tensor 131: llama_model_loader: - tensor 132: llama_model_loader: - tensor 133: llama_model_loader: - tensor 134: llama_model_loader: - tensor 135: llama_model_loader: - tensor 136: llama_model_loader: - tensor 137: llama_model_loader: - tensor 138: llama_model_loader: - tensor 139: llama_model_loader: - tensor 140: llama_model_loader: - tensor 141: llama_model_loader: - tensor 142: llama_model_loader: - tensor 143: llama_model_loader: - tensor 144: llama_model_loader: - tensor 145: llama_model_loader: - tensor 146: llama_model_loader: - tensor 147: llama_model_loader: - tensor 148: llama_model_loader: - tensor 149: llama_model_loader: - tensor 150: llama_model_loader: - tensor 151: llama_model_loader: - tensor 152: llama_model_loader: - tensor 153: llama_model_loader: - tensor 154: llama_model_loader: - tensor 155: llama_model_loader: - tensor 156: llama_model_loader: - tensor 157: llama_model_loader: - tensor 158: llama_model_loader: - tensor 159: llama_model_loader: - tensor 160: llama_model_loader: - tensor 161: llama_model_loader: - tensor 162: llama_model_loader: - tensor 163: llama_model_loader: - tensor 164: llama_model_loader: - tensor 165: llama_model_loader: - tensor 166: llama_model_loader: - tensor 167: llama_model_loader: - tensor 168: llama_model_loader: - tensor 169: llama_model_loader: - tensor 170: llama_model_loader: - tensor 171: llama_model_loader: - tensor 172: llama_model_loader: - tensor 173: llama_model_loader: - tensor 174: llama_model_loader: - tensor 175: llama_model_loader: - tensor 176: llama_model_loader: - tensor 177: llama_model_loader: - tensor 178: llama_model_loader: - tensor 179: llama_model_loader: - tensor 180: llama_model_loader: - tensor 181: llama_model_loader: - tensor 182: llama_model_loader: - tensor 183: llama_model_loader: - tensor 184: llama_model_loader: - tensor 185: llama_model_loader: - tensor 186: llama_model_loader: - tensor 187: llama_model_loader: - tensor 188: llama_model_loader: - tensor 189: llama_model_loader: - tensor 190: llama_model_loader: - tensor 191: llama_model_loader: - tensor 192: llama_model_loader: - tensor 193: llama_model_loader: - tensor 194: llama_model_loader: - tensor 195: llama_model_loader: - tensor 196: llama_model_loader: - tensor 197: llama_model_loader: - tensor 198: llama_model_loader: - tensor 199: llama_model_loader: - tensor 200: llama_model_loader: - tensor 201: llama_model_loader: - tensor 202: llama_model_loader: - tensor 203: llama_model_loader: - tensor 204: llama_model_loader: - tensor 205: llama_model_loader: - tensor 206: llama_model_loader: - tensor 207: llama_model_loader: - tensor 208: llama_model_loader: - tensor 209: llama_model_loader: - tensor 210: llama_model_loader: - tensor 211: llama_model_loader: - tensor 212: llama_model_loader: - tensor 213: llama_model_loader: - tensor 214: llama_model_loader: - tensor 215: llama_model_loader: - tensor 216: llama_model_loader: - tensor 217: llama_model_loader: - tensor 218: llama_model_loader: - tensor 219: llama_model_loader: - tensor 220: llama_model_loader: - tensor 221: llama_model_loader: - tensor 222: llama_model_loader: - tensor 223: llama_model_loader: - tensor 224: llama_model_loader: - tensor 225: llama_model_loader: - tensor 226: llama_model_loader: - tensor 227: llama_model_loader: - tensor 228: llama_model_loader: - tensor 229: llama_model_loader: - tensor 230: llama_model_loader: - tensor 231: llama_model_loader: - tensor 232: llama_model_loader: - tensor 233: llama_model_loader: - tensor 234: llama_model_loader: - tensor 235: llama_model_loader: - tensor 236: llama_model_loader: - tensor 237: llama_model_loader: - tensor 238: llama_model_loader: - tensor 239: llama_model_loader: - tensor 240: llama_model_loader: - tensor 241: llama_model_loader: - tensor 242: llama_model_loader: - tensor 243: llama_model_loader: - tensor 244: llama_model_loader: - tensor 245: llama_model_loader: - tensor 246: llama_model_loader: - tensor 247: llama_model_loader: - tensor 248: llama_model_loader: - tensor 249: llama_model_loader: - tensor 250: llama_model_loader: - tensor 251: llama_model_loader: - tensor 252: llama_model_loader: - tensor 253: llama_model_loader: - tensor 254: llama_model_loader: - tensor 255: llama_model_loader: - tensor 256: llama_model_loader: - tensor 257: llama_model_loader: - tensor 258: llama_model_loader: - tensor 259: llama_model_loader: - tensor 260: llama_model_loader: - tensor 261: llama_model_loader: - tensor 262: llama_model_loader: - tensor 263: llama_model_loader: - tensor 264: llama_model_loader: - tensor 265: llama_model_loader: - tensor 266: llama_model_loader: - tensor 267: llama_model_loader: - tensor 268: llama_model_loader: - tensor 269: llama_model_loader: - tensor 270: llama_model_loader: - tensor 271: llama_model_loader: - tensor 272: llama_model_loader: - tensor 273: llama_model_loader: - tensor 274: llama_model_loader: - tensor 275: llama_model_loader: - tensor 276: llama_model_loader: - tensor 277: llama_model_loader: - tensor 278: llama_model_loader: - tensor 279: llama_model_loader: - tensor 280: llama_model_loader: - tensor 281: llama_model_loader: - tensor 282: llama_model_loader: - tensor 283: llama_model_loader: - tensor 284: llama_model_loader: - tensor 285: llama_model_loader: - tensor 286: llama_model_loader: - tensor 287: llama_model_loader: - tensor 288: llama_model_loader: - tensor 289: llama_model_loader: - tensor 290: llama_model_loader: - kv 0: llama_model_loader: - kv 1: llama_model_loader: - kv 2: llama_model_loader: - kv 3: llama_model_loader: - kv 4: llama_model_loader: - kv 5: llama_model_loader: - kv 6: llama_model_loader: - kv 7: llama_model_loader: - kv 8: llama_model_loader: - kv 9: llama_model_loader: - kv 10: llama_model_loader: - kv 11: llama_model_loader: - kv 12: llama_model_loader: - kv 13: llama_model_loader: - kv 14: llama_model_loader: - kv 15: llama_model_loader: - kv 16: llama_model_loader: - kv 17: llama_model_loader: - kv 18: llama_model_loader: - type f32: llama_model_loader: - type q4_0: llama_model_loader: - type q6_K: llm_load_vocab: special tokens llm_load_print_meta: format llm_load_print_meta: arch llm_load_print_meta: vocab type llm_load_print_meta: n_vocab llm_load_print_meta: n_merges llm_load_print_meta: n_ctx_train llm_load_print_meta: n_embd llm_load_print_meta: n_head llm_load_print_meta: n_head_kv llm_load_print_meta: n_layer llm_load_print_meta: n_rot llm_load_print_meta: n_gqa llm_load_print_meta: f_norm_eps llm_load_print_meta: f_norm_rms_eps llm_load_print_meta: f_clamp_kqv llm_load_print_meta: f_max_alibi_bias llm_load_print_meta: n_ff llm_load_print_meta: freq_base_train llm_load_print_meta: freq_scale_train llm_load_print_meta: model type llm_load_print_meta: model ftype llm_load_print_meta: model params llm_load_print_meta: model size llm_load_print_meta: general.name llm_load_print_meta: BOS llm_load_print_meta: EOS llm_load_print_meta: UNK llm_load_print_meta: LF token llm_load_tensors: ggml ctx size = llm_load_tensors: mem required .................................. llama_new_context_with_model: n_ctx llama_new_context_with_model: freq_base llama_new_context_with_model: llama_new_context_with_model: kv self size llama_new_context_with_model: ollama serve
total blobs: 12
total unused blobs removed: 0
Listening on 127.0.0.1:11434 (version 0.1.8)
74.118µs | 127.0.0.1 | HEAD "/"
573.601µs | 127.0.0.1 | POST "/api/show"
starting llama runner
error starting the external llama runner: fork/exec /var/folders/2h/n8rpr45d2hngdbvnjhvyfypm0000gp/T/ollama1345459382/llama.cpp/gguf/build/metal/bin/ollama-runner: bad CPU type in executable
starting llama runner
waiting for llama runner to start responding
34;level":"WARNING","function":"server_params_parse","line":873,"message":"Not compiled with GPU offload support, --n-gpu-layers option will be ignored. See main README.md for information on enabling GPU BLAS support","n_gpu_layers":-1}
34;level":"INFO","function":"main","line":1324,"message":"build info","build":219,"commit":"9e70cc0"}
34;level":"INFO","function":"main","line":1330,"message":"system info","n_threads":4,"n_threads_batch":-1,"total_threads":8,"system_info":"AVX = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | "}
meta data with 19 key-value pairs and 291 tensors from /Users/andrew/.ollama/models/blobs/sha256:4a3019290402c9eadf89a3bf793102a52a2a44dd76ea7b07fca53f9cbb789a63 (version GGUF V2 (latest))
token_embd.weight q4_0 [ 4096, 32002, 1, 1 ]
blk.0.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.0.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.0.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.0.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.0.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.0.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.0.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ]
blk.0.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.0.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.1.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.1.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.1.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.1.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.1.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.1.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.1.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ]
blk.1.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.1.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.2.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.2.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.2.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.2.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.2.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.2.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.2.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ]
blk.2.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.2.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.3.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.3.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.3.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.3.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.3.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.3.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.3.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ]
blk.3.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.3.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.4.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.4.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.4.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.4.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.4.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.4.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.4.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ]
blk.4.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.4.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.5.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.5.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.5.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.5.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.5.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.5.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.5.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ]
blk.5.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.5.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.6.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.6.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.6.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.6.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.6.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.6.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.6.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ]
blk.6.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.6.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.7.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.7.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.7.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.7.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.7.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.7.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.7.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ]
blk.7.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.7.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.8.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.8.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.8.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.8.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.8.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.8.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.8.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ]
blk.8.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.8.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.9.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.9.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.9.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.9.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.9.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.9.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.9.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ]
blk.9.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.9.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.10.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.10.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.10.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.10.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.10.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.10.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.10.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ]
blk.10.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.10.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.11.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.11.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.11.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.11.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.11.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.11.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.11.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ]
blk.11.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.11.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.12.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.12.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.12.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.12.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.12.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.12.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.12.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ]
blk.12.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.12.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.13.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.13.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.13.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.13.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.13.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.13.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.13.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ]
blk.13.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.13.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.14.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.14.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.14.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.14.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.14.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.14.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.14.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ]
blk.14.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.14.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.15.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.15.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.15.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.15.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.15.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.15.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.15.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ]
blk.15.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.15.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.16.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.16.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.16.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.16.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.16.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.16.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.16.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ]
blk.16.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.16.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.17.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.17.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.17.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.17.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.17.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.17.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.17.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ]
blk.17.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.17.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.18.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.18.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.18.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.18.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.18.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.18.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.18.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ]
blk.18.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.18.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.19.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.19.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.19.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.19.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.19.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.19.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.19.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ]
blk.19.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.19.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.20.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.20.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.20.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.20.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.20.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.20.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.20.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ]
blk.20.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.20.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.21.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.21.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.21.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.21.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.21.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.21.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.21.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ]
blk.21.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.21.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.22.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.22.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.22.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.22.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.22.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.22.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.22.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ]
blk.22.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.22.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.23.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.23.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.23.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.23.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.23.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.23.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.23.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ]
blk.23.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.23.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.24.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.24.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.24.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.24.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.24.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.24.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.24.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ]
blk.24.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.24.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.25.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.25.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.25.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.25.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.25.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.25.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.25.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ]
blk.25.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.25.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.26.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.26.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.26.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.26.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.26.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.26.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.26.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ]
blk.26.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.26.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.27.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.27.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.27.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.27.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.27.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.27.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.27.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ]
blk.27.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.27.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.28.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.28.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.28.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.28.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.28.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.28.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.28.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ]
blk.28.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.28.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.29.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.29.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.29.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.29.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.29.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.29.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.29.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ]
blk.29.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.29.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.30.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.30.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.30.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.30.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.30.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.30.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.30.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ]
blk.30.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.30.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.31.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.31.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.31.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ]
blk.31.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.31.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.31.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ]
blk.31.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ]
blk.31.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.31.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
output_norm.weight f32 [ 4096, 1, 1, 1 ]
output.weight q6_K [ 4096, 32002, 1, 1 ]
general.architecture str
general.name str
llama.context_length u32
llama.embedding_length u32
llama.block_count u32
llama.feed_forward_length u32
llama.rope.dimension_count u32
llama.attention.head_count u32
llama.attention.head_count_kv u32
llama.attention.layer_norm_rms_epsilon f32
llama.rope.freq_base f32
general.file_type u32
tokenizer.ggml.model str
tokenizer.ggml.tokens arr
tokenizer.ggml.scores arr
tokenizer.ggml.token_type arr
tokenizer.ggml.bos_token_id u32
tokenizer.ggml.eos_token_id u32
general.quantization_version u32
65 tensors
225 tensors
1 tensors
definition check successful ( 261/32002 ).
= GGUF V2 (latest)
= llama
= SPM
= 32002
= 0
= 32768
= 4096
= 32
= 8
= 32
= 128
= 4
= 0.0e+00
= 1.0e-05
= 0.0e+00
= 0.0e+00
= 14336
= 10000.0
= 1
= 7B
= mostly Q4_0
= 7.24 B
= 3.83 GiB (4.54 BPW)
= ehartford
token = 1
token = 32000 <|im_end|>
token = 0
= 13 <0x0A>
0.10 MB
= 3917.98 MB
................................................................
= 2048
= 10000.0
freq_scale = 1
= 256.00 MB
compute buffer total size = 162.13 MB

llama server listening at http://127.0.0.1:54339

{"timestamp":1699403212,"level":"INFO","function":"main","line":1749,"message":"HTTP server listening","hostname":"127.0.0.1","port":54339}
{"timestamp":1699403212,"level":"INFO","function":"log_server_request","line":1240,"message":"request","remote_addr":"127.0.0.1","remote_port":61211,"status":200,"method":"HEAD","path":"/","params":{}}
2023/11/08 13:26:52 llama.go:456: llama runner started in 1.402360 seconds
[GIN] 2023/11/08 - 13:26:52 | 200 | 1.571060041s | 127.0.0.1 | POST "/api/generate"
{"timestamp":1699403219,"level":"INFO","function":"log_server_request","line":1240,"message":"request","remote_addr":"127.0.0.1","remote_port":61214,"status":200,"method":"HEAD","path":"/","params":{}}
2023/11/08 13:26:59 llama.go:399: signal: segmentation fault
2023/11/08 13:26:59 llama.go:473: llama runner stopped successfully
[GIN] 2023/11/08 - 13:26:59 | 200 | 324.606211ms | 127.0.0.1 | POST "/api/generate"
^C2023/11/08 13:27:43 llama.go:473: llama runner stopped successfully
(base) test@test-MBP-6> ollama serve ~/Desktop/dev/ollama-ui
_

@ahaslam commented on GitHub (Nov 8, 2023): ollama version 0.1.8 I too am getting this error message - but only when I attempt to use **samantha-mistral:latest** or **zephyr:latest**. When I use **mistral:latest**, it runs fine. Since they are all sourced from the same 7B model, it sounds like there's something else other than a memory constraint going on. In case it was possibly being caused by having multiple concurrent runners at the same time exhausting memory, I also tried killing the ollama server and any other ollama related processes, restarting and running one of the problematic models and the problem still persisted. I am running ollama on an older Macbook Pro 2015, 2.8GHz, 16Gb RAM, no GPU When I run "ollama serve" followed by "ollama run samantha-mistral:latest" then prompting it, I get the following logging output - the error message about possibly not having enough memory correlates with the segmentation fault error message at the bottom of the log output. I would have expected some other error message if it had insufficient memory available. _ (base) test@test-MBP-6> ollama serve ~/Desktop/dev/ollama-ui 2023/11/08 13:26:45 images.go:824: total blobs: 12 2023/11/08 13:26:45 images.go:831: total unused blobs removed: 0 2023/11/08 13:26:45 routes.go:680: Listening on 127.0.0.1:11434 (version 0.1.8) [GIN] 2023/11/08 - 13:26:50 | 200 | 74.118µs | 127.0.0.1 | HEAD "/" [GIN] 2023/11/08 - 13:26:50 | 200 | 573.601µs | 127.0.0.1 | POST "/api/show" 2023/11/08 13:26:51 llama.go:384: starting llama runner 2023/11/08 13:26:51 llama.go:386: error starting the external llama runner: fork/exec /var/folders/2h/n8rpr45d2hngdbvnjhvyfypm0000gp/T/ollama1345459382/llama.cpp/gguf/build/metal/bin/ollama-runner: bad CPU type in executable 2023/11/08 13:26:51 llama.go:384: starting llama runner 2023/11/08 13:26:51 llama.go:442: waiting for llama runner to start responding {"timestamp":1699403211,"level":"WARNING","function":"server_params_parse","line":873,"message":"Not compiled with GPU offload support, --n-gpu-layers option will be ignored. See main README.md for information on enabling GPU BLAS support","n_gpu_layers":-1} {"timestamp":1699403211,"level":"INFO","function":"main","line":1324,"message":"build info","build":219,"commit":"9e70cc0"} {"timestamp":1699403211,"level":"INFO","function":"main","line":1330,"message":"system info","n_threads":4,"n_threads_batch":-1,"total_threads":8,"system_info":"AVX = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | "} llama_model_loader: loaded meta data with 19 key-value pairs and 291 tensors from /Users/andrew/.ollama/models/blobs/sha256:4a3019290402c9eadf89a3bf793102a52a2a44dd76ea7b07fca53f9cbb789a63 (version GGUF V2 (latest)) llama_model_loader: - tensor 0: token_embd.weight q4_0 [ 4096, 32002, 1, 1 ] llama_model_loader: - tensor 1: blk.0.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 2: blk.0.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 3: blk.0.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 4: blk.0.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 5: blk.0.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 6: blk.0.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 7: blk.0.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 8: blk.0.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 9: blk.0.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 10: blk.1.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 11: blk.1.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 12: blk.1.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 13: blk.1.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 14: blk.1.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 15: blk.1.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 16: blk.1.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 17: blk.1.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 18: blk.1.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 19: blk.2.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 20: blk.2.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 21: blk.2.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 22: blk.2.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 23: blk.2.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 24: blk.2.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 25: blk.2.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 26: blk.2.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 27: blk.2.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 28: blk.3.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 29: blk.3.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 30: blk.3.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 31: blk.3.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 32: blk.3.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 33: blk.3.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 34: blk.3.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 35: blk.3.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 36: blk.3.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 37: blk.4.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 38: blk.4.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 39: blk.4.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 40: blk.4.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 41: blk.4.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 42: blk.4.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 43: blk.4.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 44: blk.4.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 45: blk.4.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 46: blk.5.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 47: blk.5.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 48: blk.5.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 49: blk.5.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 50: blk.5.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 51: blk.5.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 52: blk.5.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 53: blk.5.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 54: blk.5.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 55: blk.6.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 56: blk.6.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 57: blk.6.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 58: blk.6.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 59: blk.6.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 60: blk.6.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 61: blk.6.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 62: blk.6.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 63: blk.6.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 64: blk.7.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 65: blk.7.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 66: blk.7.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 67: blk.7.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 68: blk.7.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 69: blk.7.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 70: blk.7.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 71: blk.7.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 72: blk.7.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 73: blk.8.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 74: blk.8.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 75: blk.8.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 76: blk.8.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 77: blk.8.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 78: blk.8.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 79: blk.8.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 80: blk.8.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 81: blk.8.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 82: blk.9.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 83: blk.9.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 84: blk.9.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 85: blk.9.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 86: blk.9.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 87: blk.9.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 88: blk.9.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 89: blk.9.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 90: blk.9.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 91: blk.10.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 92: blk.10.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 93: blk.10.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 94: blk.10.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 95: blk.10.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 96: blk.10.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 97: blk.10.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 98: blk.10.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 99: blk.10.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 100: blk.11.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 101: blk.11.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 102: blk.11.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 103: blk.11.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 104: blk.11.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 105: blk.11.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 106: blk.11.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 107: blk.11.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 108: blk.11.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 109: blk.12.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 110: blk.12.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 111: blk.12.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 112: blk.12.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 113: blk.12.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 114: blk.12.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 115: blk.12.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 116: blk.12.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 117: blk.12.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 118: blk.13.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 119: blk.13.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 120: blk.13.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 121: blk.13.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 122: blk.13.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 123: blk.13.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 124: blk.13.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 125: blk.13.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 126: blk.13.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 127: blk.14.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 128: blk.14.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 129: blk.14.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 130: blk.14.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 131: blk.14.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 132: blk.14.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 133: blk.14.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 134: blk.14.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 135: blk.14.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 136: blk.15.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 137: blk.15.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 138: blk.15.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 139: blk.15.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 140: blk.15.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 141: blk.15.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 142: blk.15.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 143: blk.15.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 144: blk.15.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 145: blk.16.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 146: blk.16.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 147: blk.16.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 148: blk.16.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 149: blk.16.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 150: blk.16.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 151: blk.16.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 152: blk.16.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 153: blk.16.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 154: blk.17.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 155: blk.17.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 156: blk.17.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 157: blk.17.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 158: blk.17.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 159: blk.17.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 160: blk.17.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 161: blk.17.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 162: blk.17.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 163: blk.18.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 164: blk.18.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 165: blk.18.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 166: blk.18.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 167: blk.18.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 168: blk.18.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 169: blk.18.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 170: blk.18.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 171: blk.18.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 172: blk.19.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 173: blk.19.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 174: blk.19.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 175: blk.19.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 176: blk.19.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 177: blk.19.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 178: blk.19.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 179: blk.19.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 180: blk.19.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 181: blk.20.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 182: blk.20.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 183: blk.20.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 184: blk.20.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 185: blk.20.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 186: blk.20.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 187: blk.20.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 188: blk.20.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 189: blk.20.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 190: blk.21.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 191: blk.21.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 192: blk.21.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 193: blk.21.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 194: blk.21.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 195: blk.21.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 196: blk.21.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 197: blk.21.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 198: blk.21.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 199: blk.22.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 200: blk.22.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 201: blk.22.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 202: blk.22.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 203: blk.22.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 204: blk.22.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 205: blk.22.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 206: blk.22.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 207: blk.22.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 208: blk.23.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 209: blk.23.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 210: blk.23.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 211: blk.23.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 212: blk.23.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 213: blk.23.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 214: blk.23.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 215: blk.23.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 216: blk.23.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 217: blk.24.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 218: blk.24.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 219: blk.24.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 220: blk.24.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 221: blk.24.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 222: blk.24.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 223: blk.24.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 224: blk.24.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 225: blk.24.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 226: blk.25.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 227: blk.25.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 228: blk.25.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 229: blk.25.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 230: blk.25.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 231: blk.25.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 232: blk.25.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 233: blk.25.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 234: blk.25.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 235: blk.26.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 236: blk.26.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 237: blk.26.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 238: blk.26.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 239: blk.26.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 240: blk.26.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 241: blk.26.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 242: blk.26.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 243: blk.26.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 244: blk.27.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 245: blk.27.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 246: blk.27.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 247: blk.27.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 248: blk.27.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 249: blk.27.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 250: blk.27.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 251: blk.27.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 252: blk.27.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 253: blk.28.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 254: blk.28.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 255: blk.28.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 256: blk.28.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 257: blk.28.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 258: blk.28.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 259: blk.28.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 260: blk.28.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 261: blk.28.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 262: blk.29.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 263: blk.29.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 264: blk.29.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 265: blk.29.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 266: blk.29.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 267: blk.29.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 268: blk.29.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 269: blk.29.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 270: blk.29.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 271: blk.30.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 272: blk.30.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 273: blk.30.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 274: blk.30.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 275: blk.30.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 276: blk.30.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 277: blk.30.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 278: blk.30.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 279: blk.30.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 280: blk.31.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 281: blk.31.attn_k.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 282: blk.31.attn_v.weight q4_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 283: blk.31.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 284: blk.31.ffn_gate.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 285: blk.31.ffn_up.weight q4_0 [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 286: blk.31.ffn_down.weight q4_0 [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 287: blk.31.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 288: blk.31.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 289: output_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 290: output.weight q6_K [ 4096, 32002, 1, 1 ] llama_model_loader: - kv 0: general.architecture str llama_model_loader: - kv 1: general.name str llama_model_loader: - kv 2: llama.context_length u32 llama_model_loader: - kv 3: llama.embedding_length u32 llama_model_loader: - kv 4: llama.block_count u32 llama_model_loader: - kv 5: llama.feed_forward_length u32 llama_model_loader: - kv 6: llama.rope.dimension_count u32 llama_model_loader: - kv 7: llama.attention.head_count u32 llama_model_loader: - kv 8: llama.attention.head_count_kv u32 llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 llama_model_loader: - kv 10: llama.rope.freq_base f32 llama_model_loader: - kv 11: general.file_type u32 llama_model_loader: - kv 12: tokenizer.ggml.model str llama_model_loader: - kv 13: tokenizer.ggml.tokens arr llama_model_loader: - kv 14: tokenizer.ggml.scores arr llama_model_loader: - kv 15: tokenizer.ggml.token_type arr llama_model_loader: - kv 16: tokenizer.ggml.bos_token_id u32 llama_model_loader: - kv 17: tokenizer.ggml.eos_token_id u32 llama_model_loader: - kv 18: general.quantization_version u32 llama_model_loader: - type f32: 65 tensors llama_model_loader: - type q4_0: 225 tensors llama_model_loader: - type q6_K: 1 tensors llm_load_vocab: special tokens definition check successful ( 261/32002 ). llm_load_print_meta: format = GGUF V2 (latest) llm_load_print_meta: arch = llama llm_load_print_meta: vocab type = SPM llm_load_print_meta: n_vocab = 32002 llm_load_print_meta: n_merges = 0 llm_load_print_meta: n_ctx_train = 32768 llm_load_print_meta: n_embd = 4096 llm_load_print_meta: n_head = 32 llm_load_print_meta: n_head_kv = 8 llm_load_print_meta: n_layer = 32 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_gqa = 4 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: n_ff = 14336 llm_load_print_meta: freq_base_train = 10000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: model type = 7B llm_load_print_meta: model ftype = mostly Q4_0 llm_load_print_meta: model params = 7.24 B llm_load_print_meta: model size = 3.83 GiB (4.54 BPW) llm_load_print_meta: general.name = ehartford llm_load_print_meta: BOS token = 1 <s> llm_load_print_meta: EOS token = 32000 <|im_end|> llm_load_print_meta: UNK token = 0 <unk> llm_load_print_meta: LF token = 13 <0x0A> llm_load_tensors: ggml ctx size = 0.10 MB llm_load_tensors: mem required = 3917.98 MB .................................................................................................. llama_new_context_with_model: n_ctx = 2048 llama_new_context_with_model: freq_base = 10000.0 llama_new_context_with_model: freq_scale = 1 llama_new_context_with_model: kv self size = 256.00 MB llama_new_context_with_model: compute buffer total size = 162.13 MB llama server listening at http://127.0.0.1:54339 {"timestamp":1699403212,"level":"INFO","function":"main","line":1749,"message":"HTTP server listening","hostname":"127.0.0.1","port":54339} {"timestamp":1699403212,"level":"INFO","function":"log_server_request","line":1240,"message":"request","remote_addr":"127.0.0.1","remote_port":61211,"status":200,"method":"HEAD","path":"/","params":{}} 2023/11/08 13:26:52 llama.go:456: llama runner started in 1.402360 seconds [GIN] 2023/11/08 - 13:26:52 | 200 | 1.571060041s | 127.0.0.1 | POST "/api/generate" {"timestamp":1699403219,"level":"INFO","function":"log_server_request","line":1240,"message":"request","remote_addr":"127.0.0.1","remote_port":61214,"status":200,"method":"HEAD","path":"/","params":{}} 2023/11/08 13:26:59 llama.go:399: signal: segmentation fault 2023/11/08 13:26:59 llama.go:473: llama runner stopped successfully [GIN] 2023/11/08 - 13:26:59 | 200 | 324.606211ms | 127.0.0.1 | POST "/api/generate" ^C2023/11/08 13:27:43 llama.go:473: llama runner stopped successfully (base) test@test-MBP-6> ollama serve ~/Desktop/dev/ollama-ui _

GiteaMirror commented

2026-04-12 10:11:52 -05:00

@ahaslam commented on GitHub (Nov 8, 2023):

Further to the above.

When I build Ollama myself and explicitly disable GPUs by setting NumGPU in the opts to zero (instead of -1), then everthing works fine.

Based on the dump below - for the exception from the logs in the console, it appears that llama.cpp is trying to use a GPU and I'm wondering whether Metal on an Intel Mac does not support the GPU ops required by llama.cpp?

FYI, an earlier version of Ollama did not appear to try and use the graphics card (the CPU was maxed out on all cores when generating). That, plus the fact that there have been fairly recent changes in the code (llm.go) that disabled GPU support for some quantiization/platform combinations might be the cause? With respect to a fix that would disable GPU support for the Intel Mac, I'm not entirely sure whether I should be expecting GPU support for such an old machine or whether it's just the quantization at fault etc - so it's hard to know how to adjust that code appropriately to disable GPU support. Here's my crack at it anyway:

`
if runtime.GOOS == "darwin" {
switch ggml.FileType() {
case "Q8_0":
if ggml.Name() != "gguf" && opts.NumGPU != 0 {
// GGML Q8_0 do not support Metal API and will
// cause the runner to segmentation fault so disable GPU
log.Printf("WARNING: GPU disabled for F32, Q5_0, Q5_1, and Q8_0")
opts.NumGPU = 0
}
case "F32", "Q5_0", "Q5_1":
if opts.NumGPU != 0 {
// F32, Q5_0, Q5_1, and Q8_0 do not support Metal API and will
// cause the runner to segmentation fault so disable GPU
log.Printf("WARNING: GPU disabled for F32, Q5_0, Q5_1, and Q8_0")
opts.NumGPU = 0
}
case "Q4_0":
if runtime.GOARCH == "amd64" {
log.Printf("WARNING: GPU disabled for Q4_0 on Intel Macs")
opts.NumGPU = 0
}
}

`

Translated Report (Full Report Below)

Process: ollama-runner [49666]
Path: /private/var/folders/*/ollama-runner
Identifier: ollama-runner
Version: ???
Code Type: X86-64 (Native)
Parent Process: ollama [49659]
Responsible: Ollama [49655]
User ID: 502

Date/Time: 2023-11-09 10:08:25.3352 +1300
OS Version: macOS 12.6.8 (21G725)
Report Version: 12
Anonymous UUID: DD178B7D-68D3-713E-6B40-1296DCDDF8E4

Sleep/Wake UUID: 474ADD4B-F40E-42A0-8E0B-1AD3E035CCBC

Time Awake Since Boot: 170000 seconds
Time Since Wake: 55596 seconds

System Integrity Protection: enabled

Crashed Thread: 2

Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000000
Exception Codes: 0x0000000000000001, 0x0000000000000000
Exception Note: EXC_CORPSE_NOTIFY

Termination Reason: Namespace SIGNAL, Code 11 Segmentation fault: 11
Terminating Process: exc handler [49666]

VM Region Info: 0 is not in any region. Bytes before following region: 4396072960
REGION TYPE START - END [ VSIZE] PRT/MAX SHRMOD REGION DETAIL
UNUSED SPACE AT START
--->
__TEXT 10606c000-1061b0000 [ 1296K] r-x/r-x SM=COW ...ollama-runner

Thread 0:: Dispatch queue: com.apple.main-thread
0 libsystem_kernel.dylib 0x7ff80dc87082 __accept + 10
1 ollama-runner 0x106087f5d httplib::Server::listen_internal() + 461
2 ollama-runner 0x10607182c main + 11068
3 dyld 0x11143852e start + 462

Thread 1:
0 libsystem_kernel.dylib 0x7ff80dc833da __psynch_cvwait + 10
1 libsystem_pthread.dylib 0x7ff80dcbda6f _pthread_cond_wait + 1249
2 libc++.1.dylib 0x7ff80dc1bd22 std::__1::condition_variable::wait(std::__1::unique_lockstd::__1::mutex&) + 18
3 ollama-runner 0x10607ceb1 httplib::ThreadPool::worker::operator()() + 145
4 ollama-runner 0x10607cd42 void* std::__1::__thread_proxy[abi:v15006]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_deletestd::__1::__thread_struct >, httplib::ThreadPool::worker> >(void*) + 50
5 libsystem_pthread.dylib 0x7ff80dcbd4e1 _pthread_start + 125
6 libsystem_pthread.dylib 0x7ff80dcb8f6b thread_start + 15

Thread 2 Crashed:
0 ??? 0x0 ???
1 ollama-runner 0x10615b69b ggml_compute_forward_mul_mat + 1579
2 ollama-runner 0x10613b673 ggml_graph_compute_thread + 435
3 ollama-runner 0x10613b3ff ggml_graph_compute + 319
4 ollama-runner 0x1060fd724 ggml_graph_compute_helper(std::__1::vector<unsigned char, std::__1::allocator >&, ggml_cgraph*, int) + 292
5 ollama-runner 0x1060fe6d5 llama_decode_internal(llama_context&, llama_batch) + 2453
6 ollama-runner 0x1060fedea llama_decode + 58
7 ollama-runner 0x1060c15f0 llama_server_context::nextToken() + 1424
8 ollama-runner 0x1060abb7d llama_server_context::doCompletion() + 45
9 ollama-runner 0x1060c5cdc std::__1::__function::__func<main::$_4::operator()(httplib::Request const&, httplib::Response&) const::'lambda'(unsigned long, httplib::DataSink&), std::__1::allocator<main::$_4::operator()(httplib::Request const&, httplib::Response&) const::'lambda'(unsigned long, httplib::DataSink&)>, bool (unsigned long, httplib::DataSink&)>::operator()(unsigned long&&, httplib::DataSink&) + 92
10 ollama-runner 0x1060c5935 std::__1::__function::__func<httplib::detail::ContentProviderAdapter, std::__1::allocatorhttplib::detail::ContentProviderAdapter, bool (unsigned long, unsigned long, httplib::DataSink&)>::operator()(unsigned long&&, unsigned long&&, httplib::DataSink&) + 37
11 ollama-runner 0x1060933c1 bool httplib::detail::write_content_chunked<httplib::Server::write_content_with_provider(httplib::Stream&, httplib::Request const&, httplib::Response&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&)::'lambda'(), httplib::detail::compressor>(httplib::Stream&, std::__1::function<bool (unsigned long, unsigned long, httplib::DataSink&)> const&, httplib::Server::write_content_with_provider(httplib::Stream&, httplib::Request const&, httplib::Response&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&)::'lambda'() const&, httplib::detail::compressor&, httplib::Error&) + 545
12 ollama-runner 0x10608f62d httplib::Server::write_content_with_provider(httplib::Stream&, httplib::Request const&, httplib::Response&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&) + 317
13 ollama-runner 0x10608e8ac httplib::Server::write_response_core(httplib::Stream&, bool, httplib::Request const&, httplib::Response&, bool) + 3484
14 ollama-runner 0x106089e4a httplib::Server::process_request(httplib::Stream&, bool, bool&, std::__1::function<void (httplib::Request&)> const&) + 4314
15 ollama-runner 0x106088c7c bool httplib::detail::process_server_socket<httplib::Server::process_and_close_socket(int)::'lambda'(httplib::Stream&, bool, bool&)>(std::__1::atomic const&, int, unsigned long, long, long, long, long, long, httplib::Server::process_and_close_socket(int)::'lambda'(httplib::Stream&, bool, bool&))::'lambda'(bool, bool&)::operator()(bool, bool&) const + 220
16 ollama-runner 0x106074acd httplib::Server::process_and_close_socket(int) + 189
17 ollama-runner 0x10607cfe2 httplib::ThreadPool::worker::operator()() + 450
18 ollama-runner 0x10607cd42 void* std::__1::__thread_proxy[abi:v15006]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_deletestd::__1::__thread_struct >, httplib::ThreadPool::worker> >(void*) + 50
19 libsystem_pthread.dylib 0x7ff80dcbd4e1 _pthread_start + 125
20 libsystem_pthread.dylib 0x7ff80dcb8f6b thread_start + 15

@ahaslam commented on GitHub (Nov 8, 2023): Further to the above. When I build Ollama myself and explicitly disable GPUs by setting NumGPU in the opts to zero (instead of -1), then everthing works fine. Based on the dump below - for the exception from the logs in the console, it appears that llama.cpp is trying to use a GPU and I'm wondering whether Metal on an Intel Mac does not support the GPU ops required by llama.cpp? FYI, an earlier version of Ollama did not appear to try and use the graphics card (the CPU was maxed out on all cores when generating). That, plus the fact that there have been fairly recent changes in the code (llm.go) that disabled GPU support for some quantiization/platform combinations might be the cause? With respect to a fix that would disable GPU support for the Intel Mac, I'm not entirely sure whether I should be expecting GPU support for such an old machine or whether it's just the quantization at fault etc - so it's hard to know how to adjust that code appropriately to disable GPU support. Here's my crack at it anyway: ` if runtime.GOOS == "darwin" { switch ggml.FileType() { case "Q8_0": if ggml.Name() != "gguf" && opts.NumGPU != 0 { // GGML Q8_0 do not support Metal API and will // cause the runner to segmentation fault so disable GPU log.Printf("WARNING: GPU disabled for F32, Q5_0, Q5_1, and Q8_0") opts.NumGPU = 0 } case "F32", "Q5_0", "Q5_1": if opts.NumGPU != 0 { // F32, Q5_0, Q5_1, and Q8_0 do not support Metal API and will // cause the runner to segmentation fault so disable GPU log.Printf("WARNING: GPU disabled for F32, Q5_0, Q5_1, and Q8_0") opts.NumGPU = 0 } case "Q4_0": if runtime.GOARCH == "amd64" { log.Printf("WARNING: GPU disabled for Q4_0 on Intel Macs") opts.NumGPU = 0 } } ` ------------------------------------- Translated Report (Full Report Below) ------------------------------------- Process: ollama-runner [49666] Path: /private/var/folders/*/ollama-runner Identifier: ollama-runner Version: ??? Code Type: X86-64 (Native) Parent Process: ollama [49659] Responsible: Ollama [49655] User ID: 502 Date/Time: 2023-11-09 10:08:25.3352 +1300 OS Version: macOS 12.6.8 (21G725) Report Version: 12 Anonymous UUID: DD178B7D-68D3-713E-6B40-1296DCDDF8E4 Sleep/Wake UUID: 474ADD4B-F40E-42A0-8E0B-1AD3E035CCBC Time Awake Since Boot: 170000 seconds Time Since Wake: 55596 seconds System Integrity Protection: enabled Crashed Thread: 2 Exception Type: EXC_BAD_ACCESS (SIGSEGV) Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000000 Exception Codes: 0x0000000000000001, 0x0000000000000000 Exception Note: EXC_CORPSE_NOTIFY Termination Reason: Namespace SIGNAL, Code 11 Segmentation fault: 11 Terminating Process: exc handler [49666] VM Region Info: 0 is not in any region. Bytes before following region: 4396072960 REGION TYPE START - END [ VSIZE] PRT/MAX SHRMOD REGION DETAIL UNUSED SPACE AT START ---> __TEXT 10606c000-1061b0000 [ 1296K] r-x/r-x SM=COW ...ollama-runner Thread 0:: Dispatch queue: com.apple.main-thread 0 libsystem_kernel.dylib 0x7ff80dc87082 __accept + 10 1 ollama-runner 0x106087f5d httplib::Server::listen_internal() + 461 2 ollama-runner 0x10607182c main + 11068 3 dyld 0x11143852e start + 462 Thread 1: 0 libsystem_kernel.dylib 0x7ff80dc833da __psynch_cvwait + 10 1 libsystem_pthread.dylib 0x7ff80dcbda6f _pthread_cond_wait + 1249 2 libc++.1.dylib 0x7ff80dc1bd22 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18 3 ollama-runner 0x10607ceb1 httplib::ThreadPool::worker::operator()() + 145 4 ollama-runner 0x10607cd42 void* std::__1::__thread_proxy[abi:v15006]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, httplib::ThreadPool::worker> >(void*) + 50 5 libsystem_pthread.dylib 0x7ff80dcbd4e1 _pthread_start + 125 6 libsystem_pthread.dylib 0x7ff80dcb8f6b thread_start + 15 Thread 2 Crashed: 0 ??? 0x0 ??? 1 ollama-runner 0x10615b69b ggml_compute_forward_mul_mat + 1579 2 ollama-runner 0x10613b673 ggml_graph_compute_thread + 435 3 ollama-runner 0x10613b3ff ggml_graph_compute + 319 4 ollama-runner 0x1060fd724 ggml_graph_compute_helper(std::__1::vector<unsigned char, std::__1::allocator<unsigned char> >&, ggml_cgraph*, int) + 292 5 ollama-runner 0x1060fe6d5 llama_decode_internal(llama_context&, llama_batch) + 2453 6 ollama-runner 0x1060fedea llama_decode + 58 7 ollama-runner 0x1060c15f0 llama_server_context::nextToken() + 1424 8 ollama-runner 0x1060abb7d llama_server_context::doCompletion() + 45 9 ollama-runner 0x1060c5cdc std::__1::__function::__func<main::$_4::operator()(httplib::Request const&, httplib::Response&) const::'lambda'(unsigned long, httplib::DataSink&), std::__1::allocator<main::$_4::operator()(httplib::Request const&, httplib::Response&) const::'lambda'(unsigned long, httplib::DataSink&)>, bool (unsigned long, httplib::DataSink&)>::operator()(unsigned long&&, httplib::DataSink&) + 92 10 ollama-runner 0x1060c5935 std::__1::__function::__func<httplib::detail::ContentProviderAdapter, std::__1::allocator<httplib::detail::ContentProviderAdapter>, bool (unsigned long, unsigned long, httplib::DataSink&)>::operator()(unsigned long&&, unsigned long&&, httplib::DataSink&) + 37 11 ollama-runner 0x1060933c1 bool httplib::detail::write_content_chunked<httplib::Server::write_content_with_provider(httplib::Stream&, httplib::Request const&, httplib::Response&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::'lambda'(), httplib::detail::compressor>(httplib::Stream&, std::__1::function<bool (unsigned long, unsigned long, httplib::DataSink&)> const&, httplib::Server::write_content_with_provider(httplib::Stream&, httplib::Request const&, httplib::Response&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::'lambda'() const&, httplib::detail::compressor&, httplib::Error&) + 545 12 ollama-runner 0x10608f62d httplib::Server::write_content_with_provider(httplib::Stream&, httplib::Request const&, httplib::Response&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) + 317 13 ollama-runner 0x10608e8ac httplib::Server::write_response_core(httplib::Stream&, bool, httplib::Request const&, httplib::Response&, bool) + 3484 14 ollama-runner 0x106089e4a httplib::Server::process_request(httplib::Stream&, bool, bool&, std::__1::function<void (httplib::Request&)> const&) + 4314 15 ollama-runner 0x106088c7c bool httplib::detail::process_server_socket<httplib::Server::process_and_close_socket(int)::'lambda'(httplib::Stream&, bool, bool&)>(std::__1::atomic<int> const&, int, unsigned long, long, long, long, long, long, httplib::Server::process_and_close_socket(int)::'lambda'(httplib::Stream&, bool, bool&))::'lambda'(bool, bool&)::operator()(bool, bool&) const + 220 16 ollama-runner 0x106074acd httplib::Server::process_and_close_socket(int) + 189 17 ollama-runner 0x10607cfe2 httplib::ThreadPool::worker::operator()() + 450 18 ollama-runner 0x10607cd42 void* std::__1::__thread_proxy[abi:v15006]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, httplib::ThreadPool::worker> >(void*) + 50 19 libsystem_pthread.dylib 0x7ff80dcbd4e1 _pthread_start + 125 20 libsystem_pthread.dylib 0x7ff80dcb8f6b thread_start + 15

GiteaMirror commented

2026-04-12 10:11:53 -05:00

@ahaslam commented on GitHub (Nov 8, 2023):

@BruceMacD - sorry, should have mentioned you in the above post ^^^

@ahaslam commented on GitHub (Nov 8, 2023): @BruceMacD - sorry, should have mentioned you in the above post ^^^

GiteaMirror commented

2026-04-12 10:11:53 -05:00

@ahaslam commented on GitHub (Nov 9, 2023):

@BruceMacD Many thanks for looking at this. I just checked out your branch to test it - but no joy. Took a look at your changes and realised that they will only ever be run on Linux anyway (every instance of a call to CheckVRAM() is conditioned on whether runtime.GOOS == "linux"). That code also relies on nvidia-smi to query the state of a the GPU on a Mac so not sure it will work to determine GPU ram even if it was called.

@ahaslam commented on GitHub (Nov 9, 2023): @BruceMacD Many thanks for looking at this. I just checked out your branch to test it - but no joy. Took a look at your changes and realised that they will only ever be run on Linux anyway (every instance of a call to CheckVRAM() is conditioned on whether runtime.GOOS == "linux"). That code also relies on nvidia-smi to query the state of a the GPU on a Mac so not sure it will work to determine GPU ram even if it was called.

GiteaMirror commented

2026-04-12 10:11:53 -05:00

@BruceMacD commented on GitHub (Nov 10, 2023):

@ahaslam your issue seems slightly different than the original report. When running on Mac the GPU will either be set to 0 or 1 depending on if the Metal API is supported or not. We have had a couple reports of Metal causing issues on Mac, we are looking into it now, I'll let you know if we find anything.

In the meantime as a workaround you could add parameter to force CPU mode if a model doesn't work. Here is how you could do that.

Create a modelfile which sets num_gpu to 0.

FROM zephyr:latest
PARAMETER num_gpu 0

Create the modified model.

ollama create zephyr:no-gpu -f ./path/to/modelfile

Run with no gpu:

ollama run  zephyr:no-gpu

Edit: fixed typo, thanks jaffee

@BruceMacD commented on GitHub (Nov 10, 2023): @ahaslam your issue seems slightly different than the original report. When running on Mac the GPU will either be set to 0 or 1 depending on if the Metal API is supported or not. We have had a couple reports of Metal causing issues on Mac, we are looking into it now, I'll let you know if we find anything. In the meantime as a workaround you could add parameter to force CPU mode if a model doesn't work. Here is how you could do that. 1. Create a modelfile which sets num_gpu to 0. ``` FROM zephyr:latest PARAMETER num_gpu 0 ``` 2. Create the modified model. ``` ollama create zephyr:no-gpu -f ./path/to/modelfile ``` 3. Run with no gpu: ``` ollama run zephyr:no-gpu ``` Edit: fixed typo, thanks jaffee

GiteaMirror commented

2026-04-12 10:11:54 -05:00

@jaffee commented on GitHub (Nov 16, 2023):

2019 Intel Mac here with plenty of memory. Experiencing a similar issue where I would see something like2023/11/16 15:44:45 llama.go:430: signal: segmentation fault in the server logs.

@BruceMacD's workaround almost worked, but it needs to be PARAMETER num_gpu 0 (rather than PARAM)

This was running on the latest as of this morning SHA: 30141b42e91019e3219dd2d

@jaffee commented on GitHub (Nov 16, 2023): 2019 Intel Mac here with plenty of memory. Experiencing a similar issue where I would see something like`2023/11/16 15:44:45 llama.go:430: signal: segmentation fault` in the server logs. @BruceMacD's workaround almost worked, but it needs to be `PARAMETER num_gpu 0` (rather than PARAM) This was running on the latest as of this morning SHA: `30141b42e91019e3219dd2d`

GiteaMirror commented

2026-04-12 10:11:54 -05:00

@krenax commented on GitHub (Nov 23, 2023):

@kraemi. 16GB or RAM to run 7B models, means 16 GB for Ollama, not 16 GB for the system + some apps + Ollama.try to restart your mac (if you are running ollama on a mac). Quite all applications and start Ollama. This way you may be able to run the models but with very slow answers.

I have a 32Gb powerbook and I can run falcon:40b the answers are very slow to show, but it works this way.

Seems to be the most reasonable answer for my case.

@krenax commented on GitHub (Nov 23, 2023): > @kraemi. 16GB or RAM to run 7B models, means 16 GB for Ollama, not 16 GB for the system + some apps + Ollama.try to restart your mac (if you are running ollama on a mac). Quite all applications and start Ollama. This way you may be able to run the models but with very slow answers. > > I have a 32Gb powerbook and I can run falcon:40b the answers are very slow to show, but it works this way. Seems to be the most reasonable answer for my case.

GiteaMirror commented

2026-04-12 10:11:54 -05:00

@ToddHoff commented on GitHub (Jan 8, 2024):

Create a modelfile which sets num_gpu to 0.

Thanks, that did the trick for dolphin-mixtral as well.

@ToddHoff commented on GitHub (Jan 8, 2024): > 1. Create a modelfile which sets num_gpu to 0. Thanks, that did the trick for dolphin-mixtral as well.

GiteaMirror referenced this issue

2026-04-22 01:52:22 -05:00

[GH-ISSUE #498] SSL certificate error. #25989

GiteaMirror referenced this issue

2026-04-27 23:47:41 -05:00

[GH-ISSUE #498] SSL certificate error. #46738

GiteaMirror referenced this issue

2026-05-03 08:02:51 -05:00

[GH-ISSUE #498] SSL certificate error. #62265

Sign in to join this conversation.

Branches Tags

main

dhiltgen/ci

parth-launch-plan-gating

hoyyeva/anthropic-reference-images-path

parth-anthropic-reference-images-path

brucemacd/download-before-remove

hoyyeva/editor-config-repair

parth-mlx-decode-checkpoints

parth-launch-codex-app

hoyyeva/fix-codex-model-metadata-warning

hoyyeva/qwen

parth/hide-claude-desktop-till-release

hoyyeva/opencode-image-modality

parth-add-claude-code-autoinstall

release_v0.22.0

pdevine/manifest-list

codex/fix-codex-model-metadata-warning

pdevine/addressable-manifest

brucemacd/launch-fetch-reccomended

jmorganca/llama-compat

launch-copilot-cli

hoyyeva/opencode-thinking

release_v0.20.7

parth-auto-save-backup

parth-test

jmorganca/gemma4-audio-replacements

fix-manifest-digest-on-pull

hoyyeva/vscode-improve

brucemacd/install-server-wait

parth/update-claude-docs

brucemac/start-ap-install

pdevine/mlx-update

pdevine/qwen35_vision

drifkin/api-show-fallback

mintlify/image-generation-1773352582

hoyyeva/server-context-length-local-config

jmorganca/faster-reptition-penalties

jmorganca/convert-nemotron

parth-pi-thinking

pdevine/sampling-penalties

jmorganca/fix-create-quantization-memory

dongchen/resumable_transfer_fix

pdevine/sampling-cache-error

jessegross/mlx-usage

hoyyeva/openclaw-config

hoyyeva/app-html

pdevine/qwen3next

brucemacd/sign-sh-install

brucemacd/tui-update

brucemacd/usage-api

jmorganca/launch-empty

fix-app-dist-embed

mxyng/mlx-compile

mxyng/mlx-quant

mxyng/mlx-glm4.7

mxyng/mlx

brucemacd/simplify-model-picker

jmorganca/qwen3-concurrent

fix-glm-4.7-flash-mla-config

drifkin/qwen3-coder-opening-tag

brucemacd/usage-cli

fix-cuda12-fattn-shmem

ollama-imagegen-docs

parth/fix-multiline-inputs

brucemacd/config-docs

mxyng/model-files

mxyng/simple-execute

fix-imagegen-ollama-models

mxyng/async-upload

jmorganca/lazy-no-dtype-changes

imagegen-auto-detect-create

parth/decrease-concurrent-download-hf

fix-mlx-quantize-init

jmorganca/x-cleanup

usage

imagegen-readme

jmorganca/glm-image

mlx-gpu-cd

jmorganca/imagegen-modelfile

parth/agent-skills

parth/agent-allowlist

parth/signed-in-offline

parth/agents

parth/fix-context-chopping

improve-cloud-flow

parth/add-models-websearch

parth/prompt-renderer-mcp

jmorganca/native-settings

jmorganca/download-stream-hash

jmorganca/client2-rebased

brucemacd/oai-chat-req-multipart

jessegross/multi_chunk_reserve

grace/additional-omit-empty

grace/mistral-3-large

mxyng/tokenizer2

mxyng/tokenizer

jessegross/flash

hoyyeva/windows-nacked-app

mxyng/cleanup-attention

grace/deepseek-parser

hoyyeva/remember-unsent-prompt

parth/add-lfs-pointer-error-conversion

parth/olmo2-test2

hoyyeva/ollama-launchagent-plist

nicole/olmo-model

parth/olmo-test

mxyng/remove-embedded

parth/render-template

jmorganca/intellect-3

parth/remove-prealloc-linter

jmorganca/cmd-eval

nicole/nomic-embed-text-fix

mxyng/lint-2

hoyyeva/add-gemini-3-pro-preview

hoyyeva/load-model-list

mxyng/expand-path

mxyng/environ-2

hoyyeva/deeplink-json-encoding

parth/improve-tool-calling-tests

hoyyeva/conversation

hoyyeva/assistant-edit-response

hoyyeva/thinking

origin/brucemacd/invalid-char-i-err

parth/improve-tool-calling

jmorganca/required-omitempty

grace/qwen3-vl-tests

mxyng/iter-client

parth/docs-readme

nicole/embed-test

pdevine/integration-benchstat

parth/remove-generate-cmd

parth/add-toolcall-id

mxyng/server-tests

jmorganca/glm-4.6

jmorganca/gin-h-compat

drifkin/stable-tool-args

pdevine/qwen3-more-thinking

parth/add-websearch-client

nicole/websearch_local

jmorganca/qwen3-coder-updates

grace/deepseek-v3-migration-tests

mxyng/fix-create

jmorganca/cloud-errors

pdevine/parser-tidy

revert-12233-parth/simplify-entrypoints-runner

parth/enable-so-gpt-oss

brucemacd/qwen3vl

jmorganca/readme-simplify

parth/gpt-oss-structured-outputs

revert-12039-jmorganca/tools-braces

mxyng/embeddings

mxyng/gguf

mxyng/benchmark

mxyng/types-null

parth/move-parsing

mxyng/gemma2

jmorganca/docs

mxyng/16-bit

mxyng/create-stdin

pdevine/authorizedkeys

mxyng/quant

parth/opt-in-error-context-window

brucemacd/cache-models

brucemacd/runner-completion

jmorganca/llama-update-6

brucemacd/benchmark-list

brucemacd/partial-read-caps

parth/deepseek-r1-tools

mxyng/omit-array

parth/tool-prefix-temp

brucemacd/runner-test

jmorganca/qwen25vl

brucemacd/model-forward-test-ext

parth/python-function-parsing

jmorganca/cuda-compression-none

drifkin/num-parallel

drifkin/chat-truncation-fix

jmorganca/sync

parth/python-tools-calling

drifkin/array-head-count

brucemacd/create-no-loop

parth/server-enable-content-stream-with-tools

qwen25omni

mxyng/v3

brucemacd/ropeconfig

jmorganca/silence-tokenizer

parth/sample-so-test

parth/sampling-structured-outputs

brucemacd/doc-go-engine

parth/constrained-sampling-json

jmorganca/mistral-wip

brucemacd/mistral-small-convert

parth/sample-unmarshal-json-for-params

brucemacd/jomorganca/mistral

pdevine/bfloat16

jmorganca/mistral

brucemacd/mistral

pdevine/logging

parth/sample-correctness-fix

parth/sample-fix-sorting

jmorgan/sample-fix-sorting-extras

jmorganca/temp-0-images

brucemacd/parallel-embed-models

brucemacd/shim-grammar

jmorganca/fix-gguf-error

bmizerany/nameswork

jmorganca/faster-releases

bmizerany/validatenames

brucemacd/err-no-vocab

brucemacd/rope-config

brucemacd/err-hint

brucemacd/qwen2_5

brucemacd/logprobs

brucemacd/new_runner_graph_bench

progress-flicker

brucemacd/forward-test

brucemacd/go_qwen2

pdevine/gemma2

jmorganca/add-missing-symlink-eval

mxyng/next-debug

parth/set-context-size-openai

brucemacd/next-bpe-bench

brucemacd/next-bpe-test

brucemacd/new_runner_e2e

brucemacd/new_runner_qwen2

pdevine/convert-cohere2

brucemacd/convert-cli

parth/log-probs

mxyng/next-mlx

mxyng/cmd-history

parth/templating

parth/tokenize-detokenize

brucemacd/check-key-register

bmizerany/grammar

jmorganca/vendor-081b29bd

mxyng/func-checks

jmorganca/fix-null-format

parth/fix-default-to-warn-json

jmorganca/qwen2vl

jmorganca/no-concat

parth/cmd-cleanup-SO

brucemacd/check-key-register-structured-err

parth/openai-stream-usage

parth/fix-referencing-so

stream-tools-stop

jmorganca/degin-1

brucemacd/install-path-clean

brucemacd/push-name-validation

brucemacd/browser-key-register

jmorganca/openai-fix-first-message

jmorganca/fix-proxy

jessegross/sample

parth/disallow-streaming-tools

dhiltgen/remove_submodule

jmorganca/ga

jmorganca/mllama

pdevine/newlines

pdevine/geems-2b

jmorganca/llama-bump

mxyng/modelname-7

mxyng/gin-slog

mxyng/modelname-6

jyan/convert-prog

jyan/quant5

paligemma-support

pdevine/import-docs

jmorganca/openai-context

jyan/paligemma

jyan/p2

jyan/palitest

bmizerany/embedspeedup

jmorganca/llama-vit

brucemacd/allow-ollama

royh/ep-methods

royh/whisper

mxyng/api-models

mxyng/fix-memory

jyan/q4_4/8

jyan/ollama-v

royh/stream-tools

roy-embed-parallel

bmizerany/hrm

revert-5963-revert-5924-mxyng/llama3.1-rope

royh/embed-viz

jyan/local2

jyan/auth

jyan/local

jyan/parse-temp

jmorganca/template-mistral

jyan/reord-g

royh-openai-suffixdocs

royh-imgembed

royh-embed-parallel

jyan/quant4

royh-precision

jyan/progress

pdevine/fix-template

jyan/quant3

pdevine/ggla

mxyng/update-registry-domain

jmorganca/ggml-static

mxyng/create-context

jyan/v0.146

mxyng/layers-from-files

build_dist

bmizerany/noseek

royh-ls

royh-name

timeout

mxyng/server-timestamp

bmizerany/nosillyggufslurps

royh-params

jmorganca/llama-cpp-7c26775

royh-openai-delete

royh-show-rigid

jmorganca/enable-fa

jmorganca/no-error-template

jyan/format

royh-testdelete

bmizerany/fastverify

language_support

pdevine/ps-glitches

brucemacd/tokenize

bruce/iq-quants

bmizerany/filepathwithcoloninhost

mxyng/split-bin

bmizerany/client-registry

jmorganca/if-none-match

native

jmorganca/native

jmorganca/batch-embeddings

jmorganca/initcmake

jmorganca/mm

pdevine/showggmlinfo

modenameenforcealphanum

bmizerany/modenameenforcealphanum

jmorganca/done-reason

jmorganca/llama-cpp-8960fe8

ollama.com

bmizerany/filepathnobuild

bmizerany/types/model/defaultfix

rmdisplaylong

nogogen

bmizerany/x

modelfile-readme

bmizerany/replacecolon

jmorganca/limit

jmorganca/execstack

jmorganca/replace-assets

mxyng/tune-concurrency

jmorganca/testing

whitespace-detection

jmorganca/options

upgrade-all

scratch

cuda-search

mattw/airenamer

mattw/allmodelsonhuggingface

mattw/quantcontext

mattw/whatneedstorun

brucemacd/llama-mem-calc

mattw/faq-context

mattw/communitylinks

mattw/noprune

mattw/python-functioncalling

rename

mxyng/install

pulse

remove-first

editor

mattw/selfqueryingretrieval

cgo

mattw/howtoquant

api

matt/streamingapi

format-config

mxyng/extra-args

shell

update-nous-hermes

cp-model

upload-progress

fix-unknown-model

fix-model-names

delete-fix

insecure-registry

ls

deletemodels

progressbar

readme-updates

license-layers

skip-list

list-models

modelpath

matt/examplemodelfiles

distribution

go-opts

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/ollama#498