github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-07 00:22:43 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #14575] qwen 3.5 models from HuggingFace don't work #35210

New Issue
Open
opened 2026-04-22 19:35:20 -05:00 by GiteaMirror · 30 comments
GiteaMirror commented 2026-04-22 19:35:20 -05:00
Owner
Copy Link

Originally created by @twhlai on GitHub (Mar 3, 2026).
Original GitHub issue: https://github.com/ollama/ollama/issues/14575

What is the issue?

I tried using hf.co/bartowski/Qwen_Qwen3.5-9B-GGUF:Q4_K_M and hf.co/unsloth/Qwen3.5-9B-GGUF:Q6_K, but both give "Error: 500 Internal Server Error: unable to load model". server.log is empty. If it helps, I'm running Windows 11 on an AMD Ryzen 5700G CPU system with 64GB RAM and an Nvidia RTX 3060 12GB.

Relevant log output

OS

Windows

GPU

Nvidia

CPU

AMD

Ollama version

0.17.5

Originally created by @twhlai on GitHub (Mar 3, 2026). Original GitHub issue: https://github.com/ollama/ollama/issues/14575 ### What is the issue? I tried using hf.co/bartowski/Qwen_Qwen3.5-9B-GGUF:Q4_K_M and hf.co/unsloth/Qwen3.5-9B-GGUF:Q6_K, but both give "Error: 500 Internal Server Error: unable to load model". server.log is empty. If it helps, I'm running Windows 11 on an AMD Ryzen 5700G CPU system with 64GB RAM and an Nvidia RTX 3060 12GB. ### Relevant log output ```shell ``` ### OS Windows ### GPU Nvidia ### CPU AMD ### Ollama version 0.17.5
GiteaMirror added the bug label 2026-04-22 19:35:20 -05:00
GiteaMirror commented 2026-04-22 19:35:21 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 3, 2026):

Models downloaded from HuggingFace/ModelScope are split, text and vision GGUFs in separate files. Split models must run on the llama.cpp engine. If the llama.cpp engine doesn't the architecture, the model won't run. The llama.cpp engine in ollama does not support qwen35/qwen35moe/gemma4 architecture yet, a vendor sync (#14134) to merge llama.cpp is required. A text-only version of the model, by removing the vision GGUF, will work.

<!-- gh-comment-id:3989918451 --> @rick-github commented on GitHub (Mar 3, 2026): Models downloaded from HuggingFace/ModelScope are split, text and vision GGUFs in separate files. Split models must run on the llama.cpp engine. If the llama.cpp engine doesn't the architecture, the model won't run. The llama.cpp engine in ollama does not support qwen35/qwen35moe/gemma4 architecture yet, a vendor sync (#14134) to merge llama.cpp is required. A text-only version of the model, [by removing the vision GGUF](https://github.com/ollama/ollama/issues/14503#issuecomment-3986898959), will work.
GiteaMirror commented 2026-04-22 19:35:22 -05:00
Author
Owner
Copy Link

@james-lawrence commented on GitHub (Mar 3, 2026):

whats the eta on that PR merging?

<!-- gh-comment-id:3991309103 --> @james-lawrence commented on GitHub (Mar 3, 2026): whats the eta on that PR merging?
GiteaMirror commented 2026-04-22 19:35:23 -05:00
Author
Owner
Copy Link

@NarutoU-linux commented on GitHub (Mar 4, 2026):

updated to 17.6 (LATEST AS OF POSTING THIS) and for me still this issue persists, UPLOADED LOG ON DEBUG MODE, my laptop is a amd ryzen 7320 u with 8gb ram, if need any more logs i will attach
MODELS USING NOW
1 . hf.co/bartowski/Qwen_Qwen3.5-0.8B-GGUF:Q8_0
2. hf.co/unsloth/Qwen3.5-4B-GGUF:UD-Q4_K_XL
3. hf.co/unsloth/Qwen3.5-2B-GGUF:Q5_K_M

OLLAMA DEBUG.txt

<!-- gh-comment-id:3997462619 --> @NarutoU-linux commented on GitHub (Mar 4, 2026): updated to 17.6 (LATEST AS OF POSTING THIS) and for me still this issue persists, UPLOADED LOG ON DEBUG MODE, my laptop is a amd ryzen 7320 u with 8gb ram, if need any more logs i will attach MODELS USING NOW 1 . hf.co/bartowski/Qwen_Qwen3.5-0.8B-GGUF:Q8_0 2. hf.co/unsloth/Qwen3.5-4B-GGUF:UD-Q4_K_XL 3. hf.co/unsloth/Qwen3.5-2B-GGUF:Q5_K_M [OLLAMA DEBUG.txt](https://github.com/user-attachments/files/25741532/OLLAMA.DEBUG.txt)
GiteaMirror commented 2026-04-22 19:35:24 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 4, 2026):

The issue won't be resolved until #14134 (or similar) is merged. https://github.com/ollama/ollama/issues/14575#issuecomment-3989918451

<!-- gh-comment-id:3997549517 --> @rick-github commented on GitHub (Mar 4, 2026): The issue won't be resolved until #14134 (or similar) is merged. https://github.com/ollama/ollama/issues/14575#issuecomment-3989918451
GiteaMirror commented 2026-04-22 19:35:24 -05:00
Author
Owner
Copy Link

@nonetrix commented on GitHub (Mar 13, 2026):

Is there a way to work around this for now? It can run it just not using the right engine, is there a way to convert the model or something to use the right one?

<!-- gh-comment-id:4053007307 --> @nonetrix commented on GitHub (Mar 13, 2026): Is there a way to work around this for now? It can run it just not using the right engine, is there a way to convert the model or something to use the right one?
GiteaMirror commented 2026-04-22 19:35:24 -05:00
Author
Owner
Copy Link

@davidry commented on GitHub (Mar 13, 2026):

Is there a way to work around this for now? It can run it just not using the right engine, is there a way to convert the model or something to use the right one?

"ollama run qwen3.5:9b" works on 16gb cards.. barley fits into vram

I believe the non-v2 equivalent works with gguf if you downgrade ollama versions.. Haven't tried though
Not ready for dynamic gguf yet - think pending llama.cpp stable release

<!-- gh-comment-id:4053069717 --> @davidry commented on GitHub (Mar 13, 2026): > Is there a way to work around this for now? It can run it just not using the right engine, is there a way to convert the model or something to use the right one? "ollama run qwen3.5:9b" works on 16gb cards.. barley fits into vram I believe the non-v2 equivalent works with gguf if you downgrade ollama versions.. Haven't tried though Not ready for dynamic gguf yet - think pending llama.cpp stable release
GiteaMirror commented 2026-04-22 19:35:25 -05:00
Author
Owner
Copy Link

@beatenyou commented on GitHub (Mar 16, 2026):

Is there a way to work around this for now? It can run it just not using the right engine, is there a way to convert the model or something to use the right one?

Workaround for qwen35moe before officially supported. Seriously, it works.

  1. Custom Modelfile (workaround in Ollama)
    -- Create custom Modelfile that points directly to the blob and explicity sets the renderer/parser to qwen3.5.
  2. Get your Ollama blob path and file for the qwen model (ls /root/.ollama/models/blobs/)
  3. Create a Modelfile with:
    cat > /tmp/qwen35modelfile.txt << 'EOF'
    FROM /root/.ollama/models/blobs/sha256-REPLACE_WITH_YOUR_BLOB
    TEMPLATE {{ .Prompt }}
    RENDERER qwen3.5
    PARSER qwen3.5
    PARAMETER temperature 0.4
    PARAMETER top_p 0.95
    PARAMETER top_k 20
    PARAMETER min_p 0.0
    PARAMETER presence_penalty 0.0
    PARAMETER repeat_penalty 1.0
    PARAMETER num_ctx 32768
    PARAMETER num_predict 4096
    EOF
  4. Create the model
    ollama create qwen35-hauhau -f /tmp/qwen35modelfile.txt
  5. Use new model
    -- ollama list
<!-- gh-comment-id:4070024368 --> @beatenyou commented on GitHub (Mar 16, 2026): > Is there a way to work around this for now? It can run it just not using the right engine, is there a way to convert the model or something to use the right one? Workaround for qwen35moe before officially supported. Seriously, it works. 1. Custom Modelfile (workaround in Ollama) -- Create custom Modelfile that points directly to the blob and explicity sets the renderer/parser to qwen3.5. 2. Get your Ollama blob path and file for the qwen model (ls /root/.ollama/models/blobs/) 3. Create a Modelfile with: cat > /tmp/qwen35modelfile.txt << 'EOF' FROM /root/.ollama/models/blobs/sha256-REPLACE_WITH_YOUR_BLOB TEMPLATE {{ .Prompt }} RENDERER qwen3.5 PARSER qwen3.5 PARAMETER temperature 0.4 PARAMETER top_p 0.95 PARAMETER top_k 20 PARAMETER min_p 0.0 PARAMETER presence_penalty 0.0 PARAMETER repeat_penalty 1.0 PARAMETER num_ctx 32768 PARAMETER num_predict 4096 EOF 4. Create the model ollama create qwen35-hauhau -f /tmp/qwen35modelfile.txt 5. Use new model -- ollama list
GiteaMirror commented 2026-04-22 19:35:25 -05:00
Author
Owner
Copy Link

@Heavy-A commented on GitHub (Mar 20, 2026):

Workaround does not work for me:
llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'qwen35'

full log:
Mar 20 15:02:45 ThinkBook ollama[1819664]: [GIN] 2026/03/20 - 15:02:45 | 200 | 71.27µs | 127.0.0.1 | HEAD "/"
Mar 20 15:02:45 ThinkBook ollama[1819664]: [GIN] 2026/03/20 - 15:02:45 | 200 | 267.533116ms | 127.0.0.1 | POST "/api/show"
Mar 20 15:02:46 ThinkBook ollama[1819664]: [GIN] 2026/03/20 - 15:02:46 | 200 | 262.886893ms | 127.0.0.1 | POST "/api/show"
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: loaded meta data with 46 key-value pairs and 426 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-00fe7986ff5f6b463e62455821146049db6f9313603938a70800d1fb69ef11a4 (version GGUF V3 (latest))
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 0: general.architecture str = qwen35
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 1: general.type str = model
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 2: general.name str = Qwen3.5-4B
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 3: general.basename str = Qwen3.5-4B
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 4: general.quantized_by str = Unsloth
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 5: general.size_label str = 4B
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 6: general.license str = apache-2.0
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 7: general.license.link str = https://huggingface.co/Qwen/Qwen3.5-4...
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 8: general.repo_url str = https://huggingface.co/unsloth
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 9: general.base_model.count u32 = 1
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 10: general.base_model.0.name str = Qwen3.5 4B
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 11: general.base_model.0.organization str = Qwen
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 12: general.base_model.0.repo_url str = https://huggingface.co/Qwen/Qwen3.5-4B
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 13: general.tags arr[str,2] = ["unsloth", "image-text-to-text"]
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 14: qwen35.block_count u32 = 32
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 15: qwen35.context_length u32 = 262144
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 16: qwen35.embedding_length u32 = 2560
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 17: qwen35.feed_forward_length u32 = 9216
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 18: qwen35.attention.head_count u32 = 16
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 19: qwen35.attention.head_count_kv u32 = 4
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 20: qwen35.rope.dimension_sections arr[i32,4] = [11, 11, 10, 0]
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 21: qwen35.rope.freq_base f32 = 10000000.000000
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 22: qwen35.attention.layer_norm_rms_epsilon f32 = 0.000001
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 23: qwen35.attention.key_length u32 = 256
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 24: qwen35.attention.value_length u32 = 256
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 25: qwen35.ssm.conv_kernel u32 = 4
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 26: qwen35.ssm.state_size u32 = 128
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 27: qwen35.ssm.group_count u32 = 16
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 28: qwen35.ssm.time_step_rank u32 = 32
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 29: qwen35.ssm.inner_size u32 = 4096
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 30: qwen35.full_attention_interval u32 = 4
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 31: qwen35.rope.dimension_count u32 = 64
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 32: tokenizer.ggml.model str = gpt2
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 33: tokenizer.ggml.pre str = qwen35
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 34: tokenizer.ggml.tokens arr[str,248320] = ["!", """, "#", "$", "%", "&", "'", ...
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 35: tokenizer.ggml.token_type arr[i32,248320] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 36: tokenizer.ggml.merges arr[str,247587] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 37: tokenizer.ggml.eos_token_id u32 = 248046
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 38: tokenizer.ggml.padding_token_id u32 = 248055
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 39: tokenizer.chat_template str = {%- set image_count = namespace(value...
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 40: general.quantization_version u32 = 2
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 41: general.file_type u32 = 15
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 42: quantize.imatrix.file str = Qwen3.5-4B-GGUF/imatrix_unsloth.gguf
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 43: quantize.imatrix.dataset str = unsloth_calibration_Qwen3.5-4B.txt
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 44: quantize.imatrix.entries_count u32 = 248
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 45: quantize.imatrix.chunks_count u32 = 80
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - type f32: 177 tensors
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - type q8_0: 48 tensors
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - type q4_K: 131 tensors
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - type q5_K: 48 tensors
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - type q6_K: 22 tensors
Mar 20 15:02:46 ThinkBook ollama[1819664]: print_info: file format = GGUF V3 (latest)
Mar 20 15:02:46 ThinkBook ollama[1819664]: print_info: file type = Q4_K - Medium
Mar 20 15:02:46 ThinkBook ollama[1819664]: print_info: file size = 2.54 GiB (5.19 BPW)
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'qwen35'
Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_load_from_file_impl: failed to load model
Mar 20 15:02:46 ThinkBook ollama[1819664]: time=2026-03-20T15:02:46.681Z level=INFO source=sched.go:462 msg="failed to create server" model=qwen35-4b-gguf-unsloth:latest error="unable to load model: /usr/share/ollama/.ollama/models/blobs/sha256-00fe7986ff5f6b463e62455821146049db6f9313603938a70800d1fb69ef11a4"
Mar 20 15:02:46 ThinkBook ollama[1819664]: [GIN] 2026/03/20 - 15:02:46 | 500 | 557.657148ms | 127.0.0.1 | POST "/api/generate"

<!-- gh-comment-id:4098767454 --> @Heavy-A commented on GitHub (Mar 20, 2026): Workaround does not work for me: llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'qwen35' full log: Mar 20 15:02:45 ThinkBook ollama[1819664]: [GIN] 2026/03/20 - 15:02:45 | 200 | 71.27µs | 127.0.0.1 | HEAD "/" Mar 20 15:02:45 ThinkBook ollama[1819664]: [GIN] 2026/03/20 - 15:02:45 | 200 | 267.533116ms | 127.0.0.1 | POST "/api/show" Mar 20 15:02:46 ThinkBook ollama[1819664]: [GIN] 2026/03/20 - 15:02:46 | 200 | 262.886893ms | 127.0.0.1 | POST "/api/show" Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: loaded meta data with 46 key-value pairs and 426 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-00fe7986ff5f6b463e62455821146049db6f9313603938a70800d1fb69ef11a4 (version GGUF V3 (latest)) Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 0: general.architecture str = qwen35 Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 1: general.type str = model Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 2: general.name str = Qwen3.5-4B Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 3: general.basename str = Qwen3.5-4B Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 4: general.quantized_by str = Unsloth Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 5: general.size_label str = 4B Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 6: general.license str = apache-2.0 Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 7: general.license.link str = https://huggingface.co/Qwen/Qwen3.5-4... Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 8: general.repo_url str = https://huggingface.co/unsloth Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 9: general.base_model.count u32 = 1 Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 10: general.base_model.0.name str = Qwen3.5 4B Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 11: general.base_model.0.organization str = Qwen Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 12: general.base_model.0.repo_url str = https://huggingface.co/Qwen/Qwen3.5-4B Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 13: general.tags arr[str,2] = ["unsloth", "image-text-to-text"] Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 14: qwen35.block_count u32 = 32 Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 15: qwen35.context_length u32 = 262144 Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 16: qwen35.embedding_length u32 = 2560 Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 17: qwen35.feed_forward_length u32 = 9216 Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 18: qwen35.attention.head_count u32 = 16 Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 19: qwen35.attention.head_count_kv u32 = 4 Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 20: qwen35.rope.dimension_sections arr[i32,4] = [11, 11, 10, 0] Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 21: qwen35.rope.freq_base f32 = 10000000.000000 Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 22: qwen35.attention.layer_norm_rms_epsilon f32 = 0.000001 Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 23: qwen35.attention.key_length u32 = 256 Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 24: qwen35.attention.value_length u32 = 256 Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 25: qwen35.ssm.conv_kernel u32 = 4 Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 26: qwen35.ssm.state_size u32 = 128 Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 27: qwen35.ssm.group_count u32 = 16 Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 28: qwen35.ssm.time_step_rank u32 = 32 Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 29: qwen35.ssm.inner_size u32 = 4096 Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 30: qwen35.full_attention_interval u32 = 4 Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 31: qwen35.rope.dimension_count u32 = 64 Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 32: tokenizer.ggml.model str = gpt2 Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 33: tokenizer.ggml.pre str = qwen35 Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 34: tokenizer.ggml.tokens arr[str,248320] = ["!", "\"", "#", "$", "%", "&", "'", ... Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 35: tokenizer.ggml.token_type arr[i32,248320] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 36: tokenizer.ggml.merges arr[str,247587] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",... Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 37: tokenizer.ggml.eos_token_id u32 = 248046 Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 38: tokenizer.ggml.padding_token_id u32 = 248055 Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 39: tokenizer.chat_template str = {%- set image_count = namespace(value... Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 40: general.quantization_version u32 = 2 Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 41: general.file_type u32 = 15 Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 42: quantize.imatrix.file str = Qwen3.5-4B-GGUF/imatrix_unsloth.gguf Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 43: quantize.imatrix.dataset str = unsloth_calibration_Qwen3.5-4B.txt Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 44: quantize.imatrix.entries_count u32 = 248 Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - kv 45: quantize.imatrix.chunks_count u32 = 80 Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - type f32: 177 tensors Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - type q8_0: 48 tensors Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - type q4_K: 131 tensors Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - type q5_K: 48 tensors Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_loader: - type q6_K: 22 tensors Mar 20 15:02:46 ThinkBook ollama[1819664]: print_info: file format = GGUF V3 (latest) Mar 20 15:02:46 ThinkBook ollama[1819664]: print_info: file type = Q4_K - Medium Mar 20 15:02:46 ThinkBook ollama[1819664]: print_info: file size = 2.54 GiB (5.19 BPW) Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'qwen35' Mar 20 15:02:46 ThinkBook ollama[1819664]: llama_model_load_from_file_impl: failed to load model Mar 20 15:02:46 ThinkBook ollama[1819664]: time=2026-03-20T15:02:46.681Z level=INFO source=sched.go:462 msg="failed to create server" model=qwen35-4b-gguf-unsloth:latest error="unable to load model: /usr/share/ollama/.ollama/models/blobs/sha256-00fe7986ff5f6b463e62455821146049db6f9313603938a70800d1fb69ef11a4" Mar 20 15:02:46 ThinkBook ollama[1819664]: [GIN] 2026/03/20 - 15:02:46 | 500 | 557.657148ms | 127.0.0.1 | POST "/api/generate"
GiteaMirror commented 2026-04-22 19:35:26 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 20, 2026):

Remove the vision GGUF.

<!-- gh-comment-id:4098863904 --> @rick-github commented on GitHub (Mar 20, 2026): Remove the vision GGUF.
GiteaMirror commented 2026-04-22 19:35:26 -05:00
Author
Owner
Copy Link

@nehSgnaiL commented on GitHub (Mar 29, 2026):

ref to temp fix https://github.com/ollama/ollama/issues/14503#issuecomment-4133511574

<!-- gh-comment-id:4149600013 --> @nehSgnaiL commented on GitHub (Mar 29, 2026): ref to temp fix https://github.com/ollama/ollama/issues/14503#issuecomment-4133511574
GiteaMirror commented 2026-04-22 19:35:26 -05:00
Author
Owner
Copy Link

@whooer commented on GitHub (Mar 30, 2026):

I have the same question. Is there a follow-up plan for resolving this issue?

<!-- gh-comment-id:4153005755 --> @whooer commented on GitHub (Mar 30, 2026): I have the same question. Is there a follow-up plan for resolving this issue?
GiteaMirror commented 2026-04-22 19:35:27 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 30, 2026):

Remove the vision GGUF or wait for the next vendor sync (perhaps #14864).

<!-- gh-comment-id:4153572974 --> @rick-github commented on GitHub (Mar 30, 2026): Remove the vision GGUF or wait for the next vendor sync (perhaps #14864).
GiteaMirror commented 2026-04-22 19:35:27 -05:00
Author
Owner
Copy Link

@joshuachris2001 commented on GitHub (Apr 4, 2026):

How about patching the GGUF model itself to how the Ollama engine expects? it seems the only issue is the tensors that relate to vision, correct this and the model works fully.
Examples: https://ollama.com/fredrezones55/Qwopus3.5, https://ollama.com/fredrezones55/Jan-v2-VL, and https://ollama.com/fredrezones55/qwen3.5-opus
I guess that means we need to work on Chandra OCR 2 as well: https://ollama.com/fredrezones55/chandra-ocr-2 👌
Seems fine now. 👍 perhaps needs a proper harness for best performance.

Image

the model that has been uploaded is the 4B Q8 model with the F16 mmproj vision package; might patch and upload other combinations later...
Any other models where the only the base model functions and their helpful fine-finetunes in Ollama?

<!-- gh-comment-id:4187886126 --> @joshuachris2001 commented on GitHub (Apr 4, 2026): How about patching the GGUF model itself to how the Ollama engine expects? it seems the only issue is the tensors that relate to vision, correct this and the model works fully. Examples: https://ollama.com/fredrezones55/Qwopus3.5, https://ollama.com/fredrezones55/Jan-v2-VL, and https://ollama.com/fredrezones55/qwen3.5-opus I guess that means we need to work on Chandra OCR 2 as well: https://ollama.com/fredrezones55/chandra-ocr-2 👌 Seems fine now. 👍 perhaps needs a proper harness for best performance. <img width="560" height="480" alt="Image" src="https://github.com/user-attachments/assets/80917ab7-f9fc-40aa-96eb-1edfef70a377" /> the model that has been uploaded is the 4B Q8 model with the F16 mmproj vision package; might patch and upload other combinations later... Any other models where the only the base model functions and their helpful fine-finetunes in Ollama?
GiteaMirror commented 2026-04-22 19:35:27 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Apr 4, 2026):

If you just want the standard quants, there's no need to patch models for which safetensors exist, as is the case for these models. Just import them. If you require different quants, then patching is required.

<!-- gh-comment-id:4187897644 --> @rick-github commented on GitHub (Apr 4, 2026): If you just want the standard quants, there's no need to patch models for which safetensors exist, as is the case for these models. Just [import](https://github.com/ollama/ollama/blob/main/docs/import.mdx#Importing-a-model-from-Safetensors-weights) them. If you require different quants, then patching is required.
GiteaMirror commented 2026-04-22 19:35:28 -05:00
Author
Owner
Copy Link

@joshuachris2001 commented on GitHub (Apr 5, 2026):

using safetensors with my limited RAM, where my testing has not shown that ollama create utilizes some form of mmaping; I always get OOM kills. I had to get creative with correcting the disagreement between llama.cpp pure gguf models and Ollama model structures, that also includes the standard quants such as Q4, as even using ollama create to quantify a larger quant model causes my system to OOM kill.
But if you got the resources to handle safetensors directly in Ollama and it works for you; be my guest.

<!-- gh-comment-id:4188061449 --> @joshuachris2001 commented on GitHub (Apr 5, 2026): using safetensors with my limited RAM, where my testing has **not** shown that `ollama create` utilizes some form of mmaping; I always get OOM kills. *I had to get creative with correcting the disagreement between llama.cpp pure gguf models and Ollama model structures*, that also includes the standard quants such as Q4, as even using `ollama create` to quantify a larger quant model causes my system to OOM kill. But if you got the resources to handle safetensors directly in Ollama and it works for you; be my guest.
GiteaMirror commented 2026-04-22 19:35:28 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Apr 5, 2026):

https://github.com/ollama/ollama/issues/11783#issuecomment-3324771183

<!-- gh-comment-id:4188546738 --> @rick-github commented on GitHub (Apr 5, 2026): https://github.com/ollama/ollama/issues/11783#issuecomment-3324771183
GiteaMirror commented 2026-04-22 19:35:28 -05:00
Author
Owner
Copy Link

@tunglt1810 commented on GitHub (Apr 6, 2026):

The issue won't be resolved until #14134 (or similar) is merged. #14575 (comment)

Unfortunately, this PR is now closed

<!-- gh-comment-id:4190101207 --> @tunglt1810 commented on GitHub (Apr 6, 2026): > The issue won't be resolved until [#14134](https://github.com/ollama/ollama/pull/14134) (or similar) is merged. [#14575 (comment)](https://github.com/ollama/ollama/issues/14575#issuecomment-3989918451) Unfortunately, this PR is now closed
GiteaMirror commented 2026-04-22 19:35:29 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Apr 6, 2026):

#14864

<!-- gh-comment-id:4190952549 --> @rick-github commented on GitHub (Apr 6, 2026): #14864
GiteaMirror commented 2026-04-22 19:35:30 -05:00
Author
Owner
Copy Link

@Utku92 commented on GitHub (Apr 10, 2026):

When will the vision part be fixed in GGUF models? We can’t use vision. We’re waiting for an update.

<!-- gh-comment-id:4223244080 --> @Utku92 commented on GitHub (Apr 10, 2026): When will the vision part be fixed in GGUF models? We can’t use vision. We’re waiting for an update.
GiteaMirror commented 2026-04-22 19:35:30 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Apr 10, 2026):

Vision part works fine for models from the ollama library. For models quantized by llama.cpp (HuggingFace, ModelScope, local), a vendor sync needs to happen, which is being prepared in #14864.

<!-- gh-comment-id:4223310983 --> @rick-github commented on GitHub (Apr 10, 2026): Vision part works fine for models from the ollama library. For models quantized by llama.cpp (HuggingFace, ModelScope, local), a vendor sync needs to happen, which is being prepared in #14864.
GiteaMirror commented 2026-04-22 19:35:30 -05:00
Author
Owner
Copy Link

@Utku92 commented on GitHub (Apr 10, 2026):

When I removed the second “from” section in the modelfile, it started working. I couldn’t run the full model as it is normally. I’m not a developer, so I don’t know where I’m making a mistake.

Image
<!-- gh-comment-id:4223460583 --> @Utku92 commented on GitHub (Apr 10, 2026): When I removed the second “from” section in the modelfile, it started working. I couldn’t run the full model as it is normally. I’m not a developer, so I don’t know where I’m making a mistake. <img width="930" height="495" alt="Image" src="https://github.com/user-attachments/assets/4c4af47f-1e2d-408e-8048-85cbf4a88c64" />
GiteaMirror commented 2026-04-22 19:35:31 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Apr 10, 2026):

Yes, but this model will now not process images. Using the full model either requires a vendor sync (#14864) or importing the model from safetensors.

<!-- gh-comment-id:4223487454 --> @rick-github commented on GitHub (Apr 10, 2026): Yes, but this model will now not process images. Using the full model either requires a vendor sync (#14864) or [importing](https://github.com/ollama/ollama/blob/main/docs/import.mdx#Importing-a-model-from-Safetensors-weights) the model from [safetensors](https://huggingface.co/Kooten/gemma-4-31b-abliterated-v1.1).
GiteaMirror commented 2026-04-22 19:35:32 -05:00
Author
Owner
Copy Link

@Utku92 commented on GitHub (Apr 10, 2026):

It’s a bit confusing for me. I was downloading GGUF models from Hugging Face, and Ollama was adding the vision mmproj next to it, which was causing an error. When I removed the second line using the hash, vision was also removed of course. I also tried downloading it separately and using it that way, but it didn’t work. You shared a safetensors model link, but I can’t find many models in that format—there’s a much wider variety available in GGUF. I’ll do some research and see if I can figure it out.

<!-- gh-comment-id:4223557415 --> @Utku92 commented on GitHub (Apr 10, 2026): It’s a bit confusing for me. I was downloading GGUF models from Hugging Face, and Ollama was adding the vision mmproj next to it, which was causing an error. When I removed the second line using the hash, vision was also removed of course. I also tried downloading it separately and using it that way, but it didn’t work. You shared a safetensors model link, but I can’t find many models in that format—there’s a much wider variety available in GGUF. I’ll do some research and see if I can figure it out.
GiteaMirror commented 2026-04-22 19:35:32 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Apr 10, 2026):

All models start as safetensors. GGUF is more widely used because they are a quantized version of the safetensor format, ie they are smaller and can be run with multiple inference engines (llama.cpp, LM Studio, Jan, ollama, etc). Most GGUF models are the result of being quantized using tools from the llama.cpp project, which splits the modes of the model into separate files, one for text weights and one for a/v weights. The ollama engine wants the modes of a multi-modal model in a single file, so the multi-file models need to run on the llama.cpp engine. If the llama.cpp engine has not been updated with support for the model, the multi-file models will not run in ollama.

<!-- gh-comment-id:4223774975 --> @rick-github commented on GitHub (Apr 10, 2026): All models start as safetensors. GGUF is more widely used because they are a quantized version of the safetensor format, ie they are smaller and can be run with multiple inference engines (llama.cpp, LM Studio, Jan, ollama, etc). Most GGUF models are the result of being quantized using tools from the llama.cpp project, which splits the modes of the model into separate files, one for text weights and one for a/v weights. The ollama engine wants the modes of a multi-modal model in a single file, so the multi-file models need to run on the llama.cpp engine. If the llama.cpp engine has not been updated with support for the model, the multi-file models will not run in ollama.
GiteaMirror commented 2026-04-22 19:35:32 -05:00
Author
Owner
Copy Link

@Utku92 commented on GitHub (Apr 10, 2026):

I understand, thank you very much. My goal was actually to run the Gemma 4 31B model in its full form in the interpreter. I have an RTX 3090 with 24 GB VRAM. On Ollama’s website, there aren’t lower-quantized versions of that model. However, I found a lower-quantized 31B version in GGUF format. That’s why I tried a few things, but it didn’t work.

So now I’m using the Gemma 4 26B model with the interpreter, since it fits into VRAM. I preferred not to use LM Studio.

<!-- gh-comment-id:4223828416 --> @Utku92 commented on GitHub (Apr 10, 2026): I understand, thank you very much. My goal was actually to run the Gemma 4 31B model in its full form in the interpreter. I have an RTX 3090 with 24 GB VRAM. On Ollama’s website, there aren’t lower-quantized versions of that model. However, I found a lower-quantized 31B version in GGUF format. That’s why I tried a few things, but it didn’t work. So now I’m using the Gemma 4 26B model with the interpreter, since it fits into VRAM. I preferred not to use LM Studio.
GiteaMirror commented 2026-04-22 19:35:33 -05:00
Author
Owner
Copy Link

@iosub commented on GitHub (Apr 19, 2026):

This was already solvable five months ago, as it appears to be the same underlying issue discussed here: .https://github.com/ollama/ollama/issues/13391#issuecomment-3678317364 However, the wrong decision was made, and it still does not appear to be resolved, based on the results so far, as new split models continue to expose the same problem.

Two possible approaches were outlined in https://github.com/ollama/ollama/issues/13480#issuecomment-3657660086. If useful, I’d be happy to help test either of them.

<!-- gh-comment-id:4275283356 --> @iosub commented on GitHub (Apr 19, 2026): This was already solvable five months ago, as it appears to be the same underlying issue discussed here: .https://github.com/ollama/ollama/issues/13391#issuecomment-3678317364 However, the wrong decision was made, and it still does not appear to be resolved, based on the results so far, as new split models continue to expose the same problem. Two possible approaches were outlined in https://github.com/ollama/ollama/issues/13480#issuecomment-3657660086. If useful, I’d be happy to help test either of them.
GiteaMirror commented 2026-04-22 19:35:33 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Apr 19, 2026):

The official way to run split models is on the llama.cpp engine. When a vendor sync is committed, these models will run.

<!-- gh-comment-id:4275508063 --> @rick-github commented on GitHub (Apr 19, 2026): The official way to run split models is on the llama.cpp engine. When a vendor sync is committed, these models will run.
GiteaMirror commented 2026-04-22 19:35:34 -05:00
Author
Owner
Copy Link

@iosub commented on GitHub (Apr 22, 2026):

The official way to run split models is on the llama.cpp engine. When a vendor sync is committed, these models will run.

Hi, so you’ve changed the official approach since https://github.com/ollama/ollama/pull/13306#issuecomment-3604748281 great. It’s a shame that PR wasn’t merged five months ago; there would have been far fewer complaints about the lack of support for split models.

That said, I really appreciate the work you @rick-github do in support. One of the reasons I opened that PR was exactly to help reduce the volume of repeated issues and lighten that burden a bit.

Thanks

<!-- gh-comment-id:4300364289 --> @iosub commented on GitHub (Apr 22, 2026): > The official way to run split models is on the llama.cpp engine. When a vendor sync is committed, these models will run. Hi, so you’ve changed the official approach since https://github.com/ollama/ollama/pull/13306#issuecomment-3604748281 great. It’s a shame that PR wasn’t merged five months ago; there would have been far fewer complaints about the lack of support for split models. That said, I really appreciate the work you @rick-github do in support. One of the reasons I opened that PR was exactly to help reduce the volume of repeated issues and lighten that burden a bit. Thanks
GiteaMirror commented 2026-04-22 19:35:34 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Apr 22, 2026):

The official way to run split models is on the llama.cpp engine. When a vendor sync is committed, these models will run.

Hi, so you’ve changed the official approach since #13306 (comment) great.

The official approach is unchanged. Split models run on the llama.cpp engine, when that engine is updated, ollama will run split models. How well they run depends on the llama.cpp implementation.

<!-- gh-comment-id:4300493752 --> @rick-github commented on GitHub (Apr 22, 2026): > > The official way to run split models is on the llama.cpp engine. When a vendor sync is committed, these models will run. > > Hi, so you’ve changed the official approach since [#13306 (comment)](https://github.com/ollama/ollama/pull/13306#issuecomment-3604748281) great. The official approach is unchanged. Split models run on the llama.cpp engine, when that engine is updated, ollama will run split models. How well they run depends on the llama.cpp implementation.
GiteaMirror commented 2026-04-22 19:35:35 -05:00
Author
Owner
Copy Link

@iosub commented on GitHub (Apr 22, 2026):

The official way to run split models is on the llama.cpp engine. When a vendor sync is committed, these models will run.

Hi, so you’ve changed the official approach since #13306 (comment) great.

The official approach is unchanged. Split models run on the llama.cpp engine, when that engine is updated, ollama will run split models. How well they run depends on the llama.cpp implementation.

The https://github.com/ollama/ollama/pull/13306#issuecomment-3604748281 comment was "Split vision model support is only for backwards compatibility for legacy models that do not run in the Ollama engine. We won’t continue to extend it to support new models. As a result, qwen3-vl will only be supported in a merged version.

Maybe I misunderstood, but the fact is that since that comment you’ve made vendor updates and still haven’t enabled split models from llama.cpp.
In any case, I assume we’ll have it available in the next up

Thanks

<!-- gh-comment-id:4300636335 --> @iosub commented on GitHub (Apr 22, 2026): > > > The official way to run split models is on the llama.cpp engine. When a vendor sync is committed, these models will run. > > > > > > Hi, so you’ve changed the official approach since [#13306 (comment)](https://github.com/ollama/ollama/pull/13306#issuecomment-3604748281) great. > > The official approach is unchanged. Split models run on the llama.cpp engine, when that engine is updated, ollama will run split models. How well they run depends on the llama.cpp implementation. The https://github.com/ollama/ollama/pull/13306#issuecomment-3604748281 comment was "**Split vision model support is only for backwards compatibility for legacy models that do not run in the Ollama engine. We won’t continue to extend it to support new model**s. As a result, qwen3-vl will only be supported in a merged version. Maybe I misunderstood, but the fact is that since that comment you’ve made vendor updates and still haven’t enabled split models from llama.cpp. In any case, I assume we’ll have it available in the next up Thanks
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#35210
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?