github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-06 16:11:34 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #8907] qwen2-vl gives error whit image chat #31535

New Issue
Closed
opened 2026-04-22 12:04:33 -05:00 by GiteaMirror · 1 comment
GiteaMirror commented 2026-04-22 12:04:33 -05:00
Owner
Copy Link

Originally created by @uoakinci on GitHub (Feb 7, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/8907

What is the issue?

API or CLI backend fails when trying to chat with qwen2-vl.
Using GGUF from repository "https://huggingface.co/bartowski/Qwen2-VL-7B-Instruct-GGUF" -> "Qwen2-VL-7B-Instruct-Q4_K_M.gguf"

Created on my system with Modelfile:

`FROM Qwen2-VL-7B-Instruct-Q4_K_M.gguf

PARAMETER stop <|im_start|>
PARAMETER stop <|im_end|>

TEMPLATE """
{{ if .Messages }}
{{- if or .System .Tools }}<|im_start|>system
{{ .System }}
{{- if .Tools }}

Tools

You are provided with function signatures within XML tags:
{{- range .Tools }}
{"type": "function", "function": {{ .Function }}}{{- end }}

For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": , "arguments": }
</tool_call>
{{- end }}<|im_end|>
{{ end }}
{{- range $i, $_ := .Messages }}
{{- last := eq (len (slice .Messages $i)) 1 -}}
{{- if eq .Role "user" }}<|im_start|>user
{{ .Content }}<|im_end|>
{{ else if eq .Role "assistant" }}<|im_start|>assistant
{{ if .Content }}{{ .Content }}
{{- else if .ToolCalls }}<tool_call>
{{ range .ToolCalls }}{"name": "{{ .Function.Name }}", "arguments": {{ .Function.Arguments }}}
{{ end }}</tool_call>
{{- end }}{{ if not $last }}<|im_end|>
{{ end }}
{{- else if eq .Role "tool" }}<|im_start|>tool
<tool_response>
{{ .Content }}
</tool_response><|im_end|>
{{ end }}
{{- if and (ne .Role "assistant") $last }}<|im_start|>assistant
{{ end }}
{{- end }}
{{- else }}
{{- if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
{{ end }}{{ .Response }}{{ if .Response }}<|im_end|>{{ end }}
"""
`
Error I'am receiving from the frontend is :

Ollama response returned error: error reading llm response: read tcp 127.0.0.1:55962->127.0.0.1:55960: wsarecv

I think problem comes from llama.cpp backend but I'm not sure.

Relevant log output

[GIN] 2025/02/07 - 05:08:38 | 200 |            0s |       127.0.0.1 | GET      "/"
[GIN] 2025/02/07 - 05:08:40 | 200 |      1.5843ms |       127.0.0.1 | GET      "/api/tags"
[GIN] 2025/02/07 - 05:08:42 | 200 |            0s |       127.0.0.1 | GET      "/api/ps"
time=2025-02-07T05:08:44.432+03:00 level=WARN source=types.go:512 msg="invalid option provided" option=raw
[GIN] 2025/02/07 - 05:08:45 | 200 |            0s |       127.0.0.1 | HEAD     "/"
[GIN] 2025/02/07 - 05:08:45 | 200 |            0s |       127.0.0.1 | GET      "/api/ps"
time=2025-02-07T05:08:46.048+03:00 level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=C:\Users\ME\.ollama\models\blobs\sha256-30f199c2192fce1db0fbbbd484c7b2aa69ccce883890853f9807e1c837405a80 gpu=GPU-4da6527d-2842-38ad-3150-a9b4c62edb2f parallel=4 available=11521118208 required="5.6 GiB"
time=2025-02-07T05:08:46.078+03:00 level=INFO source=server.go:100 msg="system memory" total="125.6 GiB" free="91.9 GiB" free_swap="93.8 GiB"
time=2025-02-07T05:08:46.078+03:00 level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=-1 layers.model=29 layers.offload=29 layers.split="" memory.available="[10.7 GiB]" memory.gpu_overhead="0 B" memory.required.full="5.6 GiB" memory.required.partial="5.6 GiB" memory.required.kv="448.0 MiB" memory.required.allocations="[5.6 GiB]" memory.weights.total="4.1 GiB" memory.weights.repeating="3.7 GiB" memory.weights.nonrepeating="426.4 MiB" memory.graph.full="522.7 MiB" memory.graph.partial="522.7 MiB"
time=2025-02-07T05:08:46.083+03:00 level=INFO source=server.go:381 msg="starting llama server" cmd="C:\\Users\\ME\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --model C:\\Users\\ME\\.ollama\\models\\blobs\\sha256-30f199c2192fce1db0fbbbd484c7b2aa69ccce883890853f9807e1c837405a80 --ctx-size 8192 --batch-size 512 --n-gpu-layers 29 --threads 16 --no-mmap --parallel 4 --port 55960"
time=2025-02-07T05:08:46.086+03:00 level=INFO source=sched.go:449 msg="loaded runners" count=1
time=2025-02-07T05:08:46.086+03:00 level=INFO source=server.go:558 msg="waiting for llama runner to start responding"
time=2025-02-07T05:08:46.086+03:00 level=INFO source=server.go:592 msg="waiting for server to become available" status="llm server error"
time=2025-02-07T05:08:46.110+03:00 level=INFO source=runner.go:936 msg="starting go runner"
time=2025-02-07T05:08:46.121+03:00 level=INFO source=runner.go:937 msg=system info="CPU : LLAMAFILE = 1 | CPU : LLAMAFILE = 1 | cgo(clang)" threads=16
time=2025-02-07T05:08:46.122+03:00 level=INFO source=runner.go:995 msg="Server listening on 127.0.0.1:55960"
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 5080, compute capability 12.0, VMM: yes
load_backend: loaded CUDA backend from C:\Users\ME\AppData\Local\Programs\Ollama\lib\ollama\cuda_v12\ggml-cuda.dll
load_backend: loaded CPU backend from C:\Users\ME\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-icelake.dll
llama_load_model_from_file: using device CUDA0 (NVIDIA GeForce RTX 5080) - 14867 MiB free
llama_model_loader: loaded meta data with 37 key-value pairs and 339 tensors from C:\Users\ME\.ollama\models\blobs\sha256-30f199c2192fce1db0fbbbd484c7b2aa69ccce883890853f9807e1c837405a80 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen2vl
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Qwen2 VL 7B Instruct
llama_model_loader: - kv   3:                           general.finetune str              = Instruct
llama_model_loader: - kv   4:                           general.basename str              = Qwen2-VL
llama_model_loader: - kv   5:                         general.size_label str              = 7B
llama_model_loader: - kv   6:                            general.license str              = apache-2.0
llama_model_loader: - kv   7:                   general.base_model.count u32              = 1
llama_model_loader: - kv   8:                  general.base_model.0.name str              = Qwen2 VL 7B
llama_model_loader: - kv   9:          general.base_model.0.organization str              = Qwen
llama_model_loader: - kv  10:              general.base_model.0.repo_url str              = https://huggingface.co/Qwen/Qwen2-VL-7B
llama_model_loader: - kv  11:                               general.tags arr[str,2]       = ["multimodal", "image-text-to-text"]
llama_model_loader: - kv  12:                          general.languages arr[str,1]       = ["en"]
llama_model_loader: - kv  13:                        qwen2vl.block_count u32              = 28
llama_model_loader: - kv  14:                     qwen2vl.context_length u32              = 32768
llama_model_loader: - kv  15:                   qwen2vl.embedding_length u32              = 3584
llama_model_loader: - kv  16:                qwen2vl.feed_forward_length u32              = 18944
llama_model_loader: - kv  17:               qwen2vl.attention.head_count u32              = 28
llama_model_loader: - kv  18:            qwen2vl.attention.head_count_kv u32              = 4
llama_model_loader: - kv  19:                     qwen2vl.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  20:   qwen2vl.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  21:                          general.file_type u32              = 15
llama_model_loader: - kv  22:            qwen2vl.rope.dimension_sections arr[i32,4]       = [16, 24, 24, 0]
llama_model_loader: - kv  23:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  24:                         tokenizer.ggml.pre str              = qwen2
llama_model_loader: - kv  25:                      tokenizer.ggml.tokens arr[str,152064]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  26:                  tokenizer.ggml.token_type arr[i32,152064]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
time=2025-02-07T05:08:46.337+03:00 level=INFO source=server.go:592 msg="waiting for server to become available" status="llm server loading model"
llama_model_loader: - kv  27:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  28:                tokenizer.ggml.eos_token_id u32              = 151645
llama_model_loader: - kv  29:            tokenizer.ggml.padding_token_id u32              = 151643
llama_model_loader: - kv  30:                tokenizer.ggml.bos_token_id u32              = 151643
llama_model_loader: - kv  31:                    tokenizer.chat_template str              = {% set image_count = namespace(value=...
llama_model_loader: - kv  32:               general.quantization_version u32              = 2
llama_model_loader: - kv  33:                      quantize.imatrix.file str              = /models_out/Qwen2-VL-7B-Instruct-GGUF...
llama_model_loader: - kv  34:                   quantize.imatrix.dataset str              = /training_dir/calibration_datav3.txt
llama_model_loader: - kv  35:             quantize.imatrix.entries_count i32              = 196
llama_model_loader: - kv  36:              quantize.imatrix.chunks_count i32              = 128
llama_model_loader: - type  f32:  141 tensors
llama_model_loader: - type q4_K:  169 tensors
llama_model_loader: - type q6_K:   29 tensors
llm_load_vocab: special tokens cache size = 14
llm_load_vocab: token to piece cache size = 0.9309 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = qwen2vl
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 152064
llm_load_print_meta: n_merges         = 151387
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 32768
llm_load_print_meta: n_embd           = 3584
llm_load_print_meta: n_layer          = 28
llm_load_print_meta: n_head           = 28
llm_load_print_meta: n_head_kv        = 4
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 7
llm_load_print_meta: n_embd_k_gqa     = 512
llm_load_print_meta: n_embd_v_gqa     = 512
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 18944
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 8
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 32768
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: ssm_dt_b_c_rms   = 0
llm_load_print_meta: model type       = 7B
llm_load_print_meta: model ftype      = Q4_K - Medium
llm_load_print_meta: model params     = 7.62 B
llm_load_print_meta: model size       = 4.36 GiB (4.91 BPW) 
llm_load_print_meta: general.name     = Qwen2 VL 7B Instruct
llm_load_print_meta: BOS token        = 151643 '<|endoftext|>'
llm_load_print_meta: EOS token        = 151645 '<|im_end|>'
llm_load_print_meta: EOT token        = 151645 '<|im_end|>'
llm_load_print_meta: PAD token        = 151643 '<|endoftext|>'
llm_load_print_meta: LF token         = 148848 'ÄĬ'
llm_load_print_meta: EOG token        = 151643 '<|endoftext|>'
llm_load_print_meta: EOG token        = 151645 '<|im_end|>'
llm_load_print_meta: max token length = 256
llm_load_tensors: offloading 28 repeating layers to GPU
llm_load_tensors: offloading output layer to GPU
llm_load_tensors: offloaded 29/29 layers to GPU
llm_load_tensors:          CPU model buffer size =   292.36 MiB
llm_load_tensors:        CUDA0 model buffer size =  4168.09 MiB
llama_new_context_with_model: n_seq_max     = 4
llama_new_context_with_model: n_ctx         = 8192
llama_new_context_with_model: n_ctx_per_seq = 2048
llama_new_context_with_model: n_batch       = 2048
llama_new_context_with_model: n_ubatch      = 512
llama_new_context_with_model: flash_attn    = 0
llama_new_context_with_model: freq_base     = 1000000.0
llama_new_context_with_model: freq_scale    = 1
llama_new_context_with_model: n_ctx_per_seq (2048) < n_ctx_train (32768) -- the full capacity of the model will not be utilized
llama_kv_cache_init: kv_size = 8192, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 28, can_shift = 1
llama_kv_cache_init:      CUDA0 KV buffer size =   448.00 MiB
llama_new_context_with_model: KV self size  =  448.00 MiB, K (f16):  224.00 MiB, V (f16):  224.00 MiB
llama_new_context_with_model:  CUDA_Host  output buffer size =     2.38 MiB
llama_new_context_with_model:      CUDA0 compute buffer size =   492.01 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =    23.01 MiB
llama_new_context_with_model: graph nodes  = 986
llama_new_context_with_model: graph splits = 2
time=2025-02-07T05:08:47.088+03:00 level=INFO source=server.go:597 msg="llama runner started in 1.00 seconds"
time=2025-02-07T05:08:47.092+03:00 level=WARN source=runner.go:129 msg="truncating input prompt" limit=2048 prompt=3187 keep=4 new=2048
C:\a\ollama\ollama\ml\backend\ggml\ggml\src\ggml-cuda\rope.cu:423: GGML_ASSERT(sections.v[0] > 0 || sections.v[1] > 0 || sections.v[2] > 0) failed
[GIN] 2025/02/07 - 05:08:47 | 500 |    3.1238383s |       127.0.0.1 | POST     "/api/generate"
time=2025-02-07T05:08:47.586+03:00 level=ERROR source=server.go:422 msg="llama runner terminated" error="exit status 3"

OS

Windows

GPU

Nvidia

CPU

AMD

Ollama version

0.5.8-rc9

Originally created by @uoakinci on GitHub (Feb 7, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/8907 ### What is the issue? API or CLI backend fails when trying to chat with qwen2-vl. Using GGUF from repository "https://huggingface.co/bartowski/Qwen2-VL-7B-Instruct-GGUF" -> "Qwen2-VL-7B-Instruct-Q4_K_M.gguf" Created on my system with Modelfile: `FROM Qwen2-VL-7B-Instruct-Q4_K_M.gguf PARAMETER stop <|im_start|> PARAMETER stop <|im_end|> TEMPLATE """ {{ if .Messages }} {{- if or .System .Tools }}<|im_start|>system {{ .System }} {{- if .Tools }} # Tools You are provided with function signatures within <tools></tools> XML tags: <tools>{{- range .Tools }} {"type": "function", "function": {{ .Function }}}{{- end }} </tools> For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags: <tool_call> {"name": <function-name>, "arguments": <args-json-object>} </tool_call> {{- end }}<|im_end|> {{ end }} {{- range $i, $_ := .Messages }} {{- $last := eq (len (slice $.Messages $i)) 1 -}} {{- if eq .Role "user" }}<|im_start|>user {{ .Content }}<|im_end|> {{ else if eq .Role "assistant" }}<|im_start|>assistant {{ if .Content }}{{ .Content }} {{- else if .ToolCalls }}<tool_call> {{ range .ToolCalls }}{"name": "{{ .Function.Name }}", "arguments": {{ .Function.Arguments }}} {{ end }}</tool_call> {{- end }}{{ if not $last }}<|im_end|> {{ end }} {{- else if eq .Role "tool" }}<|im_start|>tool <tool_response> {{ .Content }} </tool_response><|im_end|> {{ end }} {{- if and (ne .Role "assistant") $last }}<|im_start|>assistant {{ end }} {{- end }} {{- else }} {{- if .System }}<|im_start|>system {{ .System }}<|im_end|> {{ end }}{{ if .Prompt }}<|im_start|>user {{ .Prompt }}<|im_end|> {{ end }}<|im_start|>assistant {{ end }}{{ .Response }}{{ if .Response }}<|im_end|>{{ end }} """ ` Error I'am receiving from the frontend is : > Ollama response returned error: error reading llm response: read tcp 127.0.0.1:55962->127.0.0.1:55960: wsarecv I think problem comes from llama.cpp backend but I'm not sure. ### Relevant log output ```shell [GIN] 2025/02/07 - 05:08:38 | 200 | 0s | 127.0.0.1 | GET "/" [GIN] 2025/02/07 - 05:08:40 | 200 | 1.5843ms | 127.0.0.1 | GET "/api/tags" [GIN] 2025/02/07 - 05:08:42 | 200 | 0s | 127.0.0.1 | GET "/api/ps" time=2025-02-07T05:08:44.432+03:00 level=WARN source=types.go:512 msg="invalid option provided" option=raw [GIN] 2025/02/07 - 05:08:45 | 200 | 0s | 127.0.0.1 | HEAD "/" [GIN] 2025/02/07 - 05:08:45 | 200 | 0s | 127.0.0.1 | GET "/api/ps" time=2025-02-07T05:08:46.048+03:00 level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=C:\Users\ME\.ollama\models\blobs\sha256-30f199c2192fce1db0fbbbd484c7b2aa69ccce883890853f9807e1c837405a80 gpu=GPU-4da6527d-2842-38ad-3150-a9b4c62edb2f parallel=4 available=11521118208 required="5.6 GiB" time=2025-02-07T05:08:46.078+03:00 level=INFO source=server.go:100 msg="system memory" total="125.6 GiB" free="91.9 GiB" free_swap="93.8 GiB" time=2025-02-07T05:08:46.078+03:00 level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=-1 layers.model=29 layers.offload=29 layers.split="" memory.available="[10.7 GiB]" memory.gpu_overhead="0 B" memory.required.full="5.6 GiB" memory.required.partial="5.6 GiB" memory.required.kv="448.0 MiB" memory.required.allocations="[5.6 GiB]" memory.weights.total="4.1 GiB" memory.weights.repeating="3.7 GiB" memory.weights.nonrepeating="426.4 MiB" memory.graph.full="522.7 MiB" memory.graph.partial="522.7 MiB" time=2025-02-07T05:08:46.083+03:00 level=INFO source=server.go:381 msg="starting llama server" cmd="C:\\Users\\ME\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --model C:\\Users\\ME\\.ollama\\models\\blobs\\sha256-30f199c2192fce1db0fbbbd484c7b2aa69ccce883890853f9807e1c837405a80 --ctx-size 8192 --batch-size 512 --n-gpu-layers 29 --threads 16 --no-mmap --parallel 4 --port 55960" time=2025-02-07T05:08:46.086+03:00 level=INFO source=sched.go:449 msg="loaded runners" count=1 time=2025-02-07T05:08:46.086+03:00 level=INFO source=server.go:558 msg="waiting for llama runner to start responding" time=2025-02-07T05:08:46.086+03:00 level=INFO source=server.go:592 msg="waiting for server to become available" status="llm server error" time=2025-02-07T05:08:46.110+03:00 level=INFO source=runner.go:936 msg="starting go runner" time=2025-02-07T05:08:46.121+03:00 level=INFO source=runner.go:937 msg=system info="CPU : LLAMAFILE = 1 | CPU : LLAMAFILE = 1 | cgo(clang)" threads=16 time=2025-02-07T05:08:46.122+03:00 level=INFO source=runner.go:995 msg="Server listening on 127.0.0.1:55960" ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 5080, compute capability 12.0, VMM: yes load_backend: loaded CUDA backend from C:\Users\ME\AppData\Local\Programs\Ollama\lib\ollama\cuda_v12\ggml-cuda.dll load_backend: loaded CPU backend from C:\Users\ME\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-icelake.dll llama_load_model_from_file: using device CUDA0 (NVIDIA GeForce RTX 5080) - 14867 MiB free llama_model_loader: loaded meta data with 37 key-value pairs and 339 tensors from C:\Users\ME\.ollama\models\blobs\sha256-30f199c2192fce1db0fbbbd484c7b2aa69ccce883890853f9807e1c837405a80 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = qwen2vl llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = Qwen2 VL 7B Instruct llama_model_loader: - kv 3: general.finetune str = Instruct llama_model_loader: - kv 4: general.basename str = Qwen2-VL llama_model_loader: - kv 5: general.size_label str = 7B llama_model_loader: - kv 6: general.license str = apache-2.0 llama_model_loader: - kv 7: general.base_model.count u32 = 1 llama_model_loader: - kv 8: general.base_model.0.name str = Qwen2 VL 7B llama_model_loader: - kv 9: general.base_model.0.organization str = Qwen llama_model_loader: - kv 10: general.base_model.0.repo_url str = https://huggingface.co/Qwen/Qwen2-VL-7B llama_model_loader: - kv 11: general.tags arr[str,2] = ["multimodal", "image-text-to-text"] llama_model_loader: - kv 12: general.languages arr[str,1] = ["en"] llama_model_loader: - kv 13: qwen2vl.block_count u32 = 28 llama_model_loader: - kv 14: qwen2vl.context_length u32 = 32768 llama_model_loader: - kv 15: qwen2vl.embedding_length u32 = 3584 llama_model_loader: - kv 16: qwen2vl.feed_forward_length u32 = 18944 llama_model_loader: - kv 17: qwen2vl.attention.head_count u32 = 28 llama_model_loader: - kv 18: qwen2vl.attention.head_count_kv u32 = 4 llama_model_loader: - kv 19: qwen2vl.rope.freq_base f32 = 1000000.000000 llama_model_loader: - kv 20: qwen2vl.attention.layer_norm_rms_epsilon f32 = 0.000001 llama_model_loader: - kv 21: general.file_type u32 = 15 llama_model_loader: - kv 22: qwen2vl.rope.dimension_sections arr[i32,4] = [16, 24, 24, 0] llama_model_loader: - kv 23: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 24: tokenizer.ggml.pre str = qwen2 llama_model_loader: - kv 25: tokenizer.ggml.tokens arr[str,152064] = ["!", "\"", "#", "$", "%", "&", "'", ... llama_model_loader: - kv 26: tokenizer.ggml.token_type arr[i32,152064] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... time=2025-02-07T05:08:46.337+03:00 level=INFO source=server.go:592 msg="waiting for server to become available" status="llm server loading model" llama_model_loader: - kv 27: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",... llama_model_loader: - kv 28: tokenizer.ggml.eos_token_id u32 = 151645 llama_model_loader: - kv 29: tokenizer.ggml.padding_token_id u32 = 151643 llama_model_loader: - kv 30: tokenizer.ggml.bos_token_id u32 = 151643 llama_model_loader: - kv 31: tokenizer.chat_template str = {% set image_count = namespace(value=... llama_model_loader: - kv 32: general.quantization_version u32 = 2 llama_model_loader: - kv 33: quantize.imatrix.file str = /models_out/Qwen2-VL-7B-Instruct-GGUF... llama_model_loader: - kv 34: quantize.imatrix.dataset str = /training_dir/calibration_datav3.txt llama_model_loader: - kv 35: quantize.imatrix.entries_count i32 = 196 llama_model_loader: - kv 36: quantize.imatrix.chunks_count i32 = 128 llama_model_loader: - type f32: 141 tensors llama_model_loader: - type q4_K: 169 tensors llama_model_loader: - type q6_K: 29 tensors llm_load_vocab: special tokens cache size = 14 llm_load_vocab: token to piece cache size = 0.9309 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = qwen2vl llm_load_print_meta: vocab type = BPE llm_load_print_meta: n_vocab = 152064 llm_load_print_meta: n_merges = 151387 llm_load_print_meta: vocab_only = 0 llm_load_print_meta: n_ctx_train = 32768 llm_load_print_meta: n_embd = 3584 llm_load_print_meta: n_layer = 28 llm_load_print_meta: n_head = 28 llm_load_print_meta: n_head_kv = 4 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_swa = 0 llm_load_print_meta: n_embd_head_k = 128 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 7 llm_load_print_meta: n_embd_k_gqa = 512 llm_load_print_meta: n_embd_v_gqa = 512 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-06 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 18944 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 8 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 1000000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_ctx_orig_yarn = 32768 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: ssm_dt_b_c_rms = 0 llm_load_print_meta: model type = 7B llm_load_print_meta: model ftype = Q4_K - Medium llm_load_print_meta: model params = 7.62 B llm_load_print_meta: model size = 4.36 GiB (4.91 BPW) llm_load_print_meta: general.name = Qwen2 VL 7B Instruct llm_load_print_meta: BOS token = 151643 '<|endoftext|>' llm_load_print_meta: EOS token = 151645 '<|im_end|>' llm_load_print_meta: EOT token = 151645 '<|im_end|>' llm_load_print_meta: PAD token = 151643 '<|endoftext|>' llm_load_print_meta: LF token = 148848 'ÄĬ' llm_load_print_meta: EOG token = 151643 '<|endoftext|>' llm_load_print_meta: EOG token = 151645 '<|im_end|>' llm_load_print_meta: max token length = 256 llm_load_tensors: offloading 28 repeating layers to GPU llm_load_tensors: offloading output layer to GPU llm_load_tensors: offloaded 29/29 layers to GPU llm_load_tensors: CPU model buffer size = 292.36 MiB llm_load_tensors: CUDA0 model buffer size = 4168.09 MiB llama_new_context_with_model: n_seq_max = 4 llama_new_context_with_model: n_ctx = 8192 llama_new_context_with_model: n_ctx_per_seq = 2048 llama_new_context_with_model: n_batch = 2048 llama_new_context_with_model: n_ubatch = 512 llama_new_context_with_model: flash_attn = 0 llama_new_context_with_model: freq_base = 1000000.0 llama_new_context_with_model: freq_scale = 1 llama_new_context_with_model: n_ctx_per_seq (2048) < n_ctx_train (32768) -- the full capacity of the model will not be utilized llama_kv_cache_init: kv_size = 8192, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 28, can_shift = 1 llama_kv_cache_init: CUDA0 KV buffer size = 448.00 MiB llama_new_context_with_model: KV self size = 448.00 MiB, K (f16): 224.00 MiB, V (f16): 224.00 MiB llama_new_context_with_model: CUDA_Host output buffer size = 2.38 MiB llama_new_context_with_model: CUDA0 compute buffer size = 492.01 MiB llama_new_context_with_model: CUDA_Host compute buffer size = 23.01 MiB llama_new_context_with_model: graph nodes = 986 llama_new_context_with_model: graph splits = 2 time=2025-02-07T05:08:47.088+03:00 level=INFO source=server.go:597 msg="llama runner started in 1.00 seconds" time=2025-02-07T05:08:47.092+03:00 level=WARN source=runner.go:129 msg="truncating input prompt" limit=2048 prompt=3187 keep=4 new=2048 C:\a\ollama\ollama\ml\backend\ggml\ggml\src\ggml-cuda\rope.cu:423: GGML_ASSERT(sections.v[0] > 0 || sections.v[1] > 0 || sections.v[2] > 0) failed [GIN] 2025/02/07 - 05:08:47 | 500 | 3.1238383s | 127.0.0.1 | POST "/api/generate" time=2025-02-07T05:08:47.586+03:00 level=ERROR source=server.go:422 msg="llama runner terminated" error="exit status 3" ``` ### OS Windows ### GPU Nvidia ### CPU AMD ### Ollama version 0.5.8-rc9
GiteaMirror added the bug label 2026-04-22 12:04:33 -05:00
GiteaMirror commented 2026-04-22 12:04:34 -05:00
Author
Owner
Copy Link

@jmorganca commented on GitHub (Feb 7, 2025):

Hi @uoakinci qwen2 VL is not yet available in Ollama - how token positions are encoded in a batch didn't work with Ollama's prompt caching. Some initial work was done in https://github.com/ollama/ollama/pull/8113

<!-- gh-comment-id:2642004620 --> @jmorganca commented on GitHub (Feb 7, 2025): Hi @uoakinci qwen2 VL is not yet available in Ollama - how token positions are encoded in a batch didn't work with Ollama's prompt caching. Some initial work was done in https://github.com/ollama/ollama/pull/8113
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#31535
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?