github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 21:25:46 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #9761] Ollama 0.6.0 api not running gemma3:27b #68436

New Issue
Closed
opened 2026-05-04 13:56:42 -05:00 by GiteaMirror · 6 comments
GiteaMirror commented 2026-05-04 13:56:42 -05:00
Owner
Copy Link

Originally created by @mkarobi on GitHub (Mar 14, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/9761

What is the issue?

I run the model correctly using the command ollama run gemma3:27b on the command-line in terminal. But when I call it as an API, the error "litellm.APIConnectionError: Ollama_chatException - {"error":"POST predict: Post "http://127.0.0.1:65428/completion": read tcp 127.0.0.1:65430-\u003e127.0.0.1:65428: wsarecv: An existing connection was forcibly closed by the remote host."}"

Image

is returned.
What is the problem and what should I do?

Relevant log output

time=2025-03-14T11:30:28.370+03:30 level=INFO source=server.go:619 msg="waiting for server to become available" status="llm server loading model"
time=2025-03-14T11:30:28.740+03:30 level=INFO source=ggml.go:289 msg="model weights" buffer=CUDA0 size="3.8 GiB"
time=2025-03-14T11:30:28.740+03:30 level=INFO source=ggml.go:289 msg="model weights" buffer=CPU size="13.5 GiB"
time=2025-03-14T11:30:39.787+03:30 level=INFO source=ggml.go:356 msg="compute graph" backend=CUDA0 buffer_type=CUDA0
time=2025-03-14T11:30:39.789+03:30 level=INFO source=ggml.go:356 msg="compute graph" backend=CPU buffer_type=CUDA_Host
time=2025-03-14T11:30:39.791+03:30 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+"
time=2025-03-14T11:30:39.794+03:30 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.add_eot_token default=false
time=2025-03-14T11:30:39.796+03:30 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+"
time=2025-03-14T11:30:39.800+03:30 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.attention.layer_norm_rms_epsilon default=9.999999974752427e-07
time=2025-03-14T11:30:39.800+03:30 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.local.freq_base default=10000
time=2025-03-14T11:30:39.800+03:30 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.global.freq_base default=1e+06
time=2025-03-14T11:30:39.800+03:30 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.freq_scale default=1
time=2025-03-14T11:30:39.800+03:30 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.final_logit_softcapping default=30
time=2025-03-14T11:30:39.800+03:30 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.mm_tokens_per_image default=256
time=2025-03-14T11:30:40.031+03:30 level=INFO source=server.go:624 msg="llama runner started in 11.91 seconds"
llama_model_loader: loaded meta data with 35 key-value pairs and 1247 tensors from C:\Users\Foren2\.ollama\models\blobs\sha256-afa0ea2ef463c87a1eebb9af070e76a353107493b5d9a62e5e66f65a65409541 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                gemma3.attention.head_count u32              = 32
llama_model_loader: - kv   1:             gemma3.attention.head_count_kv u32              = 16
llama_model_loader: - kv   2:                gemma3.attention.key_length u32              = 128
llama_model_loader: - kv   3:            gemma3.attention.sliding_window u32              = 1024
llama_model_loader: - kv   4:              gemma3.attention.value_length u32              = 128
llama_model_loader: - kv   5:                         gemma3.block_count u32              = 62
llama_model_loader: - kv   6:                      gemma3.context_length u32              = 8192
llama_model_loader: - kv   7:                    gemma3.embedding_length u32              = 5376
llama_model_loader: - kv   8:                 gemma3.feed_forward_length u32              = 21504
llama_model_loader: - kv   9:         gemma3.vision.attention.head_count u32              = 16
llama_model_loader: - kv  10: gemma3.vision.attention.layer_norm_epsilon f32              = 0.000001
llama_model_loader: - kv  11:                  gemma3.vision.block_count u32              = 27
llama_model_loader: - kv  12:             gemma3.vision.embedding_length u32              = 1152
llama_model_loader: - kv  13:          gemma3.vision.feed_forward_length u32              = 4304
llama_model_loader: - kv  14:                   gemma3.vision.image_size u32              = 896
llama_model_loader: - kv  15:                 gemma3.vision.num_channels u32              = 3
llama_model_loader: - kv  16:                   gemma3.vision.patch_size u32              = 14
llama_model_loader: - kv  17:                       general.architecture str              = gemma3
llama_model_loader: - kv  18:                    tokenizer.chat_template str              = {{ bos_token }}\n{%- if messages[0]['r...
llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  21:           tokenizer.ggml.add_padding_token bool             = false
llama_model_loader: - kv  22:           tokenizer.ggml.add_unknown_token bool             = false
llama_model_loader: - kv  23:                tokenizer.ggml.bos_token_id u32              = 2
llama_model_loader: - kv  24:                tokenizer.ggml.eos_token_id u32              = 1
llama_model_loader: - kv  25:                      tokenizer.ggml.merges arr[str,514906]  = ["\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n", ...
llama_model_loader: - kv  26:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  27:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  28:                         tokenizer.ggml.pre str              = default
llama_model_loader: - kv  29:                      tokenizer.ggml.scores arr[f32,262145]  = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  30:                  tokenizer.ggml.token_type arr[i32,262145]  = [3, 3, 3, 2, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  31:                      tokenizer.ggml.tokens arr[str,262145]  = ["<pad>", "<eos>", "<bos>", "<unk>", ...
llama_model_loader: - kv  32:            tokenizer.ggml.unknown_token_id u32              = 3
llama_model_loader: - kv  33:               general.quantization_version u32              = 2
llama_model_loader: - kv  34:                          general.file_type u32              = 15
llama_model_loader: - type  f32:  647 tensors
llama_model_loader: - type  f16:  165 tensors
llama_model_loader: - type q4_K:  376 tensors
llama_model_loader: - type q6_K:   59 tensors
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: special tokens cache size = 7
load: token to piece cache size = 1.9446 MB
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 44183.76 MiB on device 0: cudaMalloc failed: out of memory
ggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 46330025984
Exception 0xc0000005 0x0 0x58 0x7ff60e0da844
PC=0x7ff60e0da844
signal arrived during external code execution
-------------------------------------------------------------------
goroutine 1464 gp=0xc000586700 m=nil [IO wait]:
runtime.gopark(0x0?, 0xc00017ef20?, 0xc8?, 0xef?, 0xc00017efcc?)
	C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc003183d58 sp=0xc003183d38 pc=0x7ff60d2a57ce
runtime.netpollblock(0x278?, 0xd2403c6?, 0xf6?)
	C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/netpoll.go:575 +0xf7 fp=0xc003183d90 sp=0xc003183d58 pc=0x7ff60d26b697
internal/poll.runtime_pollWait(0x27ee9c40d28, 0x72)
	C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/netpoll.go:351 +0x85 fp=0xc003183db0 sp=0xc003183d90 pc=0x7ff60d2a4965
internal/poll.(*pollDesc).wait(0xb15680?, 0xb1568000000002?, 0x0)
	C:/hostedtoolcache/windows/go/1.24.0/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc003183dd8 sp=0xc003183db0 pc=0x7ff60d33ad87
internal/poll.execIO(0xc00017ef20, 0x7ff60e470f98)
	C:/hostedtoolcache/windows/go/1.24.0/x64/src/internal/poll/fd_windows.go:177 +0x105 fp=0xc003183e50 sp=0xc003183dd8 pc=0x7ff60d33c1e5
internal/poll.(*FD).Read(0xc00017ef08, {0xc00021c1c1, 0x1, 0x1})
	C:/hostedtoolcache/windows/go/1.24.0/x64/src/internal/poll/fd_windows.go:438 +0x29b fp=0xc003183ef0 sp=0xc003183e50 pc=0x7ff60d33cebb
net.(*netFD).Read(0xc00017ef08, {0xc00021c1c1?, 0xc0004aa158?, 0xc003183f70?})
	C:/hostedtoolcache/windows/go/1.24.0/x64/src/net/fd_posix.go:55 +0x25 fp=0xc003183f38 sp=0xc003183ef0 pc=0x7ff60d3affc5
net.(*conn).Read(0xc000148300, {0xc00021c1c1?, 0xc0030cd200?, 0x7ff60d6a3320?})
	C:/hostedtoolcache/windows/go/1.24.0/x64/src/net/net.go:194 +0x45 fp=0xc003183f80 sp=0xc003183f38 pc=0x7ff60d3bf4a5
net/http.(*connReader).backgroundRead(0xc00021c1b0)
	C:/hostedtoolcache/windows/go/1.24.0/x64/src/net/http/server.go:690 +0x37 fp=0xc003183fc8 sp=0xc003183f80 pc=0x7ff60d5abd97
net/http.(*connReader).startBackgroundRead.gowrap2()
	C:/hostedtoolcache/windows/go/1.24.0/x64/src/net/http/server.go:686 +0x25 fp=0xc003183fe0 sp=0xc003183fc8 pc=0x7ff60d5abcc5
runtime.goexit({})
	C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc003183fe8 sp=0xc003183fe0 pc=0x7ff60d2acfc1
created by net/http.(*connReader).startBackgroundRead in goroutine 54
	C:/hostedtoolcache/windows/go/1.24.0/x64/src/net/http/server.go:686 +0xb6
rax     0x27ef39901a0
rbx     0x28331a6d060
rcx     0x0
rdx     0x0
rdi     0x27e804aec48
rsi     0x27e809fb280
rbp     0xe3188ffe30
rsp     0xe3188ffc00
r8      0x0
r9      0x5400
r10     0x1
r11     0x44
r12     0x0
r13     0x27e8028fec0
r14     0x1
r15     0x0
rip     0x7ff60e0da844
rflags  0x10206
cs      0x33
fs      0x53
gs      0x2b
[GIN] 2025/03/14 - 11:31:03 | 500 |   36.7943608s |       127.0.0.1 | POST     "/api/chat"
time=2025-03-14T11:31:05.149+03:30 level=ERROR source=server.go:449 msg="llama runner terminated" error="exit status 2"
time=2025-03-14T11:31:09.001+03:30 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.0154224 model=C:\Users\Foren2\.ollama\models\blobs\sha256-afa0ea2ef463c87a1eebb9af070e76a353107493b5d9a62e5e66f65a65409541
time=2025-03-14T11:31:09.251+03:30 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.2650615 model=C:\Users\Foren2\.ollama\models\blobs\sha256-afa0ea2ef463c87a1eebb9af070e76a353107493b5d9a62e5e66f65a65409541
time=2025-03-14T11:31:09.287+03:30 level=INFO source=server.go:105 msg="system memory" total="63.6 GiB" free="35.6 GiB" free_swap="42.6 GiB"
time=2025-03-14T11:31:09.300+03:30 level=INFO source=server.go:138 msg=offload library=cuda layers.requested=-1 layers.model=63 layers.offload=16 layers.split="" memory.available="[6.9 GiB]" memory.gpu_overhead="512.0 MiB" memory.required.full="19.5 GiB" memory.required.partial="6.3 GiB" memory.required.kv="992.0 MiB" memory.required.allocations="[6.3 GiB]" memory.weights.total="15.3 GiB" memory.weights.repeating="14.2 GiB" memory.weights.nonrepeating="1.1 GiB" memory.graph.full="522.5 MiB" memory.graph.partial="1.6 GiB"
time=2025-03-14T11:31:09.301+03:30 level=INFO source=server.go:185 msg="enabling flash attention"
time=2025-03-14T11:31:09.301+03:30 level=WARN source=server.go:193 msg="kv cache type not supported by model" type=""
time=2025-03-14T11:31:09.501+03:30 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.514924 model=C:\Users\Foren2\.ollama\models\blobs\sha256-afa0ea2ef463c87a1eebb9af070e76a353107493b5d9a62e5e66f65a65409541
time=2025-03-14T11:31:09.511+03:30 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+"
time=2025-03-14T11:31:09.515+03:30 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.add_eot_token default=false
time=2025-03-14T11:31:09.519+03:30 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+"
time=2025-03-14T11:31:09.525+03:30 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.attention.layer_norm_rms_epsilon default=9.999999974752427e-07
time=2025-03-14T11:31:09.525+03:30 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.local.freq_base default=10000
time=2025-03-14T11:31:09.525+03:30 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.global.freq_base default=1e+06
time=2025-03-14T11:31:09.525+03:30 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.freq_scale default=1
time=2025-03-14T11:31:09.525+03:30 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.final_logit_softcapping default=30
time=2025-03-14T11:31:09.525+03:30 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.mm_tokens_per_image default=256
time=2025-03-14T11:31:09.537+03:30 level=INFO source=server.go:405 msg="starting llama server" cmd="C:\\Users\\Foren2\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model C:\\Users\\Foren2\\.ollama\\models\\blobs\\sha256-afa0ea2ef463c87a1eebb9af070e76a353107493b5d9a62e5e66f65a65409541 --ctx-size 2048 --batch-size 512 --n-gpu-layers 16 --threads 8 --flash-attn --no-mmap --parallel 1 --port 65447"
time=2025-03-14T11:31:09.541+03:30 level=INFO source=sched.go:450 msg="loaded runners" count=1
time=2025-03-14T11:31:09.541+03:30 level=INFO source=server.go:585 msg="waiting for llama runner to start responding"
time=2025-03-14T11:31:09.544+03:30 level=INFO source=server.go:619 msg="waiting for server to become available" status="llm server error"
time=2025-03-14T11:31:09.570+03:30 level=INFO source=runner.go:882 msg="starting ollama engine"
time=2025-03-14T11:31:09.572+03:30 level=INFO source=runner.go:938 msg="Server listening on 127.0.0.1:65447"
time=2025-03-14T11:31:09.629+03:30 level=WARN source=ggml.go:149 msg="key not found" key=general.name default=""
time=2025-03-14T11:31:09.629+03:30 level=WARN source=ggml.go:149 msg="key not found" key=general.description default=""
time=2025-03-14T11:31:09.629+03:30 level=INFO source=ggml.go:67 msg="" architecture=gemma3 file_type=Q4_K_M name="" description="" num_tensors=1247 num_key_values=36
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 4070 Laptop GPU, compute capability 8.9, VMM: yes
load_backend: loaded CUDA backend from C:\Users\Foren2\AppData\Local\Programs\Ollama\lib\ollama\cuda_v12\ggml-cuda.dll
load_backend: loaded CPU backend from C:\Users\Foren2\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-alderlake.dll
time=2025-03-14T11:31:09.732+03:30 level=INFO source=ggml.go:109 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX_VNNI=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2025-03-14T11:31:09.795+03:30 level=INFO source=server.go:619 msg="waiting for server to become available" status="llm server loading model"
time=2025-03-14T11:31:10.244+03:30 level=INFO source=ggml.go:289 msg="model weights" buffer=CPU size="13.5 GiB"
time=2025-03-14T11:31:10.244+03:30 level=INFO source=ggml.go:289 msg="model weights" buffer=CUDA0 size="3.8 GiB"

OS

Windows

GPU

Nvidia

CPU

Intel

Ollama version

0.6.0

Originally created by @mkarobi on GitHub (Mar 14, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/9761 ### What is the issue? I run the model correctly using the command ollama run gemma3:27b on the command-line in terminal. But when I call it as an API, the error "litellm.APIConnectionError: Ollama_chatException - {"error":"POST predict: Post \"http://127.0.0.1:65428/completion\": read tcp 127.0.0.1:65430-\u003e127.0.0.1:65428: wsarecv: An existing connection was forcibly closed by the remote host."}" <img width="422" alt="Image" src="https://github.com/user-attachments/assets/74bea5db-4b50-49d4-828f-be9eda5a9300" /> is returned. What is the problem and what should I do? ### Relevant log output ```shell time=2025-03-14T11:30:28.370+03:30 level=INFO source=server.go:619 msg="waiting for server to become available" status="llm server loading model" time=2025-03-14T11:30:28.740+03:30 level=INFO source=ggml.go:289 msg="model weights" buffer=CUDA0 size="3.8 GiB" time=2025-03-14T11:30:28.740+03:30 level=INFO source=ggml.go:289 msg="model weights" buffer=CPU size="13.5 GiB" time=2025-03-14T11:30:39.787+03:30 level=INFO source=ggml.go:356 msg="compute graph" backend=CUDA0 buffer_type=CUDA0 time=2025-03-14T11:30:39.789+03:30 level=INFO source=ggml.go:356 msg="compute graph" backend=CPU buffer_type=CUDA_Host time=2025-03-14T11:30:39.791+03:30 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+" time=2025-03-14T11:30:39.794+03:30 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.add_eot_token default=false time=2025-03-14T11:30:39.796+03:30 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+" time=2025-03-14T11:30:39.800+03:30 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.attention.layer_norm_rms_epsilon default=9.999999974752427e-07 time=2025-03-14T11:30:39.800+03:30 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.local.freq_base default=10000 time=2025-03-14T11:30:39.800+03:30 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.global.freq_base default=1e+06 time=2025-03-14T11:30:39.800+03:30 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.freq_scale default=1 time=2025-03-14T11:30:39.800+03:30 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.final_logit_softcapping default=30 time=2025-03-14T11:30:39.800+03:30 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.mm_tokens_per_image default=256 time=2025-03-14T11:30:40.031+03:30 level=INFO source=server.go:624 msg="llama runner started in 11.91 seconds" llama_model_loader: loaded meta data with 35 key-value pairs and 1247 tensors from C:\Users\Foren2\.ollama\models\blobs\sha256-afa0ea2ef463c87a1eebb9af070e76a353107493b5d9a62e5e66f65a65409541 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: gemma3.attention.head_count u32 = 32 llama_model_loader: - kv 1: gemma3.attention.head_count_kv u32 = 16 llama_model_loader: - kv 2: gemma3.attention.key_length u32 = 128 llama_model_loader: - kv 3: gemma3.attention.sliding_window u32 = 1024 llama_model_loader: - kv 4: gemma3.attention.value_length u32 = 128 llama_model_loader: - kv 5: gemma3.block_count u32 = 62 llama_model_loader: - kv 6: gemma3.context_length u32 = 8192 llama_model_loader: - kv 7: gemma3.embedding_length u32 = 5376 llama_model_loader: - kv 8: gemma3.feed_forward_length u32 = 21504 llama_model_loader: - kv 9: gemma3.vision.attention.head_count u32 = 16 llama_model_loader: - kv 10: gemma3.vision.attention.layer_norm_epsilon f32 = 0.000001 llama_model_loader: - kv 11: gemma3.vision.block_count u32 = 27 llama_model_loader: - kv 12: gemma3.vision.embedding_length u32 = 1152 llama_model_loader: - kv 13: gemma3.vision.feed_forward_length u32 = 4304 llama_model_loader: - kv 14: gemma3.vision.image_size u32 = 896 llama_model_loader: - kv 15: gemma3.vision.num_channels u32 = 3 llama_model_loader: - kv 16: gemma3.vision.patch_size u32 = 14 llama_model_loader: - kv 17: general.architecture str = gemma3 llama_model_loader: - kv 18: tokenizer.chat_template str = {{ bos_token }}\n{%- if messages[0]['r... llama_model_loader: - kv 19: tokenizer.ggml.add_bos_token bool = true llama_model_loader: - kv 20: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 21: tokenizer.ggml.add_padding_token bool = false llama_model_loader: - kv 22: tokenizer.ggml.add_unknown_token bool = false llama_model_loader: - kv 23: tokenizer.ggml.bos_token_id u32 = 2 llama_model_loader: - kv 24: tokenizer.ggml.eos_token_id u32 = 1 llama_model_loader: - kv 25: tokenizer.ggml.merges arr[str,514906] = ["\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n", ... llama_model_loader: - kv 26: tokenizer.ggml.model str = llama llama_model_loader: - kv 27: tokenizer.ggml.padding_token_id u32 = 0 llama_model_loader: - kv 28: tokenizer.ggml.pre str = default llama_model_loader: - kv 29: tokenizer.ggml.scores arr[f32,262145] = [0.000000, 0.000000, 0.000000, 0.0000... llama_model_loader: - kv 30: tokenizer.ggml.token_type arr[i32,262145] = [3, 3, 3, 2, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 31: tokenizer.ggml.tokens arr[str,262145] = ["<pad>", "<eos>", "<bos>", "<unk>", ... llama_model_loader: - kv 32: tokenizer.ggml.unknown_token_id u32 = 3 llama_model_loader: - kv 33: general.quantization_version u32 = 2 llama_model_loader: - kv 34: general.file_type u32 = 15 llama_model_loader: - type f32: 647 tensors llama_model_loader: - type f16: 165 tensors llama_model_loader: - type q4_K: 376 tensors llama_model_loader: - type q6_K: 59 tensors load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect load: special tokens cache size = 7 load: token to piece cache size = 1.9446 MB ggml_backend_cuda_buffer_type_alloc_buffer: allocating 44183.76 MiB on device 0: cudaMalloc failed: out of memory ggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 46330025984 Exception 0xc0000005 0x0 0x58 0x7ff60e0da844 PC=0x7ff60e0da844 signal arrived during external code execution ------------------------------------------------------------------- goroutine 1464 gp=0xc000586700 m=nil [IO wait]: runtime.gopark(0x0?, 0xc00017ef20?, 0xc8?, 0xef?, 0xc00017efcc?) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc003183d58 sp=0xc003183d38 pc=0x7ff60d2a57ce runtime.netpollblock(0x278?, 0xd2403c6?, 0xf6?) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/netpoll.go:575 +0xf7 fp=0xc003183d90 sp=0xc003183d58 pc=0x7ff60d26b697 internal/poll.runtime_pollWait(0x27ee9c40d28, 0x72) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/netpoll.go:351 +0x85 fp=0xc003183db0 sp=0xc003183d90 pc=0x7ff60d2a4965 internal/poll.(*pollDesc).wait(0xb15680?, 0xb1568000000002?, 0x0) C:/hostedtoolcache/windows/go/1.24.0/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc003183dd8 sp=0xc003183db0 pc=0x7ff60d33ad87 internal/poll.execIO(0xc00017ef20, 0x7ff60e470f98) C:/hostedtoolcache/windows/go/1.24.0/x64/src/internal/poll/fd_windows.go:177 +0x105 fp=0xc003183e50 sp=0xc003183dd8 pc=0x7ff60d33c1e5 internal/poll.(*FD).Read(0xc00017ef08, {0xc00021c1c1, 0x1, 0x1}) C:/hostedtoolcache/windows/go/1.24.0/x64/src/internal/poll/fd_windows.go:438 +0x29b fp=0xc003183ef0 sp=0xc003183e50 pc=0x7ff60d33cebb net.(*netFD).Read(0xc00017ef08, {0xc00021c1c1?, 0xc0004aa158?, 0xc003183f70?}) C:/hostedtoolcache/windows/go/1.24.0/x64/src/net/fd_posix.go:55 +0x25 fp=0xc003183f38 sp=0xc003183ef0 pc=0x7ff60d3affc5 net.(*conn).Read(0xc000148300, {0xc00021c1c1?, 0xc0030cd200?, 0x7ff60d6a3320?}) C:/hostedtoolcache/windows/go/1.24.0/x64/src/net/net.go:194 +0x45 fp=0xc003183f80 sp=0xc003183f38 pc=0x7ff60d3bf4a5 net/http.(*connReader).backgroundRead(0xc00021c1b0) C:/hostedtoolcache/windows/go/1.24.0/x64/src/net/http/server.go:690 +0x37 fp=0xc003183fc8 sp=0xc003183f80 pc=0x7ff60d5abd97 net/http.(*connReader).startBackgroundRead.gowrap2() C:/hostedtoolcache/windows/go/1.24.0/x64/src/net/http/server.go:686 +0x25 fp=0xc003183fe0 sp=0xc003183fc8 pc=0x7ff60d5abcc5 runtime.goexit({}) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc003183fe8 sp=0xc003183fe0 pc=0x7ff60d2acfc1 created by net/http.(*connReader).startBackgroundRead in goroutine 54 C:/hostedtoolcache/windows/go/1.24.0/x64/src/net/http/server.go:686 +0xb6 rax 0x27ef39901a0 rbx 0x28331a6d060 rcx 0x0 rdx 0x0 rdi 0x27e804aec48 rsi 0x27e809fb280 rbp 0xe3188ffe30 rsp 0xe3188ffc00 r8 0x0 r9 0x5400 r10 0x1 r11 0x44 r12 0x0 r13 0x27e8028fec0 r14 0x1 r15 0x0 rip 0x7ff60e0da844 rflags 0x10206 cs 0x33 fs 0x53 gs 0x2b [GIN] 2025/03/14 - 11:31:03 | 500 | 36.7943608s | 127.0.0.1 | POST "/api/chat" time=2025-03-14T11:31:05.149+03:30 level=ERROR source=server.go:449 msg="llama runner terminated" error="exit status 2" time=2025-03-14T11:31:09.001+03:30 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.0154224 model=C:\Users\Foren2\.ollama\models\blobs\sha256-afa0ea2ef463c87a1eebb9af070e76a353107493b5d9a62e5e66f65a65409541 time=2025-03-14T11:31:09.251+03:30 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.2650615 model=C:\Users\Foren2\.ollama\models\blobs\sha256-afa0ea2ef463c87a1eebb9af070e76a353107493b5d9a62e5e66f65a65409541 time=2025-03-14T11:31:09.287+03:30 level=INFO source=server.go:105 msg="system memory" total="63.6 GiB" free="35.6 GiB" free_swap="42.6 GiB" time=2025-03-14T11:31:09.300+03:30 level=INFO source=server.go:138 msg=offload library=cuda layers.requested=-1 layers.model=63 layers.offload=16 layers.split="" memory.available="[6.9 GiB]" memory.gpu_overhead="512.0 MiB" memory.required.full="19.5 GiB" memory.required.partial="6.3 GiB" memory.required.kv="992.0 MiB" memory.required.allocations="[6.3 GiB]" memory.weights.total="15.3 GiB" memory.weights.repeating="14.2 GiB" memory.weights.nonrepeating="1.1 GiB" memory.graph.full="522.5 MiB" memory.graph.partial="1.6 GiB" time=2025-03-14T11:31:09.301+03:30 level=INFO source=server.go:185 msg="enabling flash attention" time=2025-03-14T11:31:09.301+03:30 level=WARN source=server.go:193 msg="kv cache type not supported by model" type="" time=2025-03-14T11:31:09.501+03:30 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.514924 model=C:\Users\Foren2\.ollama\models\blobs\sha256-afa0ea2ef463c87a1eebb9af070e76a353107493b5d9a62e5e66f65a65409541 time=2025-03-14T11:31:09.511+03:30 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+" time=2025-03-14T11:31:09.515+03:30 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.add_eot_token default=false time=2025-03-14T11:31:09.519+03:30 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+" time=2025-03-14T11:31:09.525+03:30 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.attention.layer_norm_rms_epsilon default=9.999999974752427e-07 time=2025-03-14T11:31:09.525+03:30 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.local.freq_base default=10000 time=2025-03-14T11:31:09.525+03:30 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.global.freq_base default=1e+06 time=2025-03-14T11:31:09.525+03:30 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.freq_scale default=1 time=2025-03-14T11:31:09.525+03:30 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.final_logit_softcapping default=30 time=2025-03-14T11:31:09.525+03:30 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.mm_tokens_per_image default=256 time=2025-03-14T11:31:09.537+03:30 level=INFO source=server.go:405 msg="starting llama server" cmd="C:\\Users\\Foren2\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model C:\\Users\\Foren2\\.ollama\\models\\blobs\\sha256-afa0ea2ef463c87a1eebb9af070e76a353107493b5d9a62e5e66f65a65409541 --ctx-size 2048 --batch-size 512 --n-gpu-layers 16 --threads 8 --flash-attn --no-mmap --parallel 1 --port 65447" time=2025-03-14T11:31:09.541+03:30 level=INFO source=sched.go:450 msg="loaded runners" count=1 time=2025-03-14T11:31:09.541+03:30 level=INFO source=server.go:585 msg="waiting for llama runner to start responding" time=2025-03-14T11:31:09.544+03:30 level=INFO source=server.go:619 msg="waiting for server to become available" status="llm server error" time=2025-03-14T11:31:09.570+03:30 level=INFO source=runner.go:882 msg="starting ollama engine" time=2025-03-14T11:31:09.572+03:30 level=INFO source=runner.go:938 msg="Server listening on 127.0.0.1:65447" time=2025-03-14T11:31:09.629+03:30 level=WARN source=ggml.go:149 msg="key not found" key=general.name default="" time=2025-03-14T11:31:09.629+03:30 level=WARN source=ggml.go:149 msg="key not found" key=general.description default="" time=2025-03-14T11:31:09.629+03:30 level=INFO source=ggml.go:67 msg="" architecture=gemma3 file_type=Q4_K_M name="" description="" num_tensors=1247 num_key_values=36 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 4070 Laptop GPU, compute capability 8.9, VMM: yes load_backend: loaded CUDA backend from C:\Users\Foren2\AppData\Local\Programs\Ollama\lib\ollama\cuda_v12\ggml-cuda.dll load_backend: loaded CPU backend from C:\Users\Foren2\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-alderlake.dll time=2025-03-14T11:31:09.732+03:30 level=INFO source=ggml.go:109 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX_VNNI=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2025-03-14T11:31:09.795+03:30 level=INFO source=server.go:619 msg="waiting for server to become available" status="llm server loading model" time=2025-03-14T11:31:10.244+03:30 level=INFO source=ggml.go:289 msg="model weights" buffer=CPU size="13.5 GiB" time=2025-03-14T11:31:10.244+03:30 level=INFO source=ggml.go:289 msg="model weights" buffer=CUDA0 size="3.8 GiB" ``` ### OS Windows ### GPU Nvidia ### CPU Intel ### Ollama version 0.6.0
GiteaMirror added the bug label 2026-05-04 13:56:42 -05:00
GiteaMirror commented 2026-05-04 13:56:46 -05:00
Author
Owner
Copy Link

@JulienDeveaux commented on GitHub (Mar 14, 2025):

From your logs allocating 44183.76 MiB on device 0: cudaMalloc failed: out of memory
I got the same, but it should be fixed in 0.6.1

<!-- gh-comment-id:2724043008 --> @JulienDeveaux commented on GitHub (Mar 14, 2025): From your logs `allocating 44183.76 MiB on device 0: cudaMalloc failed: out of memory` I got the same, but it should be fixed in 0.6.1
GiteaMirror commented 2026-05-04 13:56:47 -05:00
Author
Owner
Copy Link

@YonTracks commented on GitHub (Mar 14, 2025):

I want to help? hoping they know what I mean whoever at this point lol.

the memory is being over allocated? most likely because of orphans, the reason for the actual crash is crash first but ollama now does not fully track the orphan/ the memory is stuck in the system (if orphan) else if wrong config but ollama is falling back (the mission is keep running lol), if the libs and packages don't sync, but ollama recovers (that is the mission, keep running.), and auto switching the libs. the official ollama comes with all packages/lib's for gpu acceleration for most all in the one install. the dev build is less, only what we configure.

I see if configured correct, the build? in theory it will be better etc...

edit^: I know I edit more than < stated srry. else lots of messages not sure, learning what is best lol.

but also, I think based on this logic? when not configured correct, and say it is falling back even if gpu is running great, the libs may not be synced, and a gpu driver tool thing will not fully release the gpu and or mutex blocking will crash/ lose track of things.

test the theory lol.

good luck

<!-- gh-comment-id:2724145598 --> @YonTracks commented on GitHub (Mar 14, 2025): I want to help? hoping they know what I mean whoever at this point lol. the memory is being over allocated? most likely because of orphans, the reason for the actual crash is crash first but ollama now does not fully track the orphan/ the memory is stuck in the system (if orphan) else if wrong config but ollama is falling back (the mission is keep running lol), if the libs and packages don't sync, but ollama recovers (that is the mission, keep running.), and auto switching the libs. the official ollama comes with all packages/lib's for gpu acceleration for most all in the one install. the dev build is less, only what we configure. I see if configured correct, the build? in theory it will be better etc... edit^: I know I edit more than < stated srry. else lots of messages not sure, learning what is best lol. but also, I think based on this logic? when not configured correct, and say it is falling back even if gpu is running great, the libs may not be synced, and a gpu driver tool thing will not fully release the gpu and or mutex blocking will crash/ lose track of things. test the theory lol. good luck
GiteaMirror commented 2026-05-04 13:56:48 -05:00
Author
Owner
Copy Link

@wisepmlin commented on GitHub (Mar 14, 2025):

Error: llama runner process has terminated: this model is not supported by your version of Ollama. You may need to upgrade

<!-- gh-comment-id:2724903104 --> @wisepmlin commented on GitHub (Mar 14, 2025): Error: llama runner process has terminated: this model is not supported by your version of Ollama. You may need to upgrade
GiteaMirror commented 2026-05-04 13:56:49 -05:00
Author
Owner
Copy Link

@mkarobi commented on GitHub (Mar 14, 2025):

Error: llama runner process has terminated: this model is not supported by your version of Ollama. You may need to upgrade

i used latest version

<!-- gh-comment-id:2725617901 --> @mkarobi commented on GitHub (Mar 14, 2025): > Error: llama runner process has terminated: this model is not supported by your version of Ollama. You may need to upgrade i used latest version
GiteaMirror commented 2026-05-04 13:56:50 -05:00
Author
Owner
Copy Link

@jmorganca commented on GitHub (Mar 16, 2025):

Hey all this should be fixed in 0.6.1: https://github.com/ollama/ollama/releases/tag/v0.6.1. Let me know if that isn't the case (we're still seeing some OOM errors in rare cases). Merging with https://github.com/ollama/ollama/issues/9791

<!-- gh-comment-id:2727118505 --> @jmorganca commented on GitHub (Mar 16, 2025): Hey all this should be fixed in 0.6.1: https://github.com/ollama/ollama/releases/tag/v0.6.1. Let me know if that isn't the case (we're still seeing some OOM errors in rare cases). Merging with https://github.com/ollama/ollama/issues/9791
GiteaMirror commented 2026-05-04 13:56:51 -05:00
Author
Owner
Copy Link

@karl19 commented on GitHub (Mar 16, 2025):

Still happening but for 4b

<!-- gh-comment-id:2727679995 --> @karl19 commented on GitHub (Mar 16, 2025): Still happening but for 4b <!-- Failed to upload "image.png" -->
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#68436
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?