[GH-ISSUE #15235] can not run unsloth gemma-4 #71803

New Issue

GiteaMirror · 2026-05-05T02:33:02-05:00

GiteaMirror commented

2026-05-05 02:33:02 -05:00

Originally created by @iwater on GitHub (Apr 2, 2026).
Original GitHub issue: https://github.com/ollama/ollama/issues/15235

What is the issue?

v0.20 got 500 Internal Server Error

ollama run hf.co/unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M
Error: 500 Internal Server Error: unable to load model: /Users/iwater/.ollama/models/blobs/sha256-8520b6bc9bbd9b6432bc1c37ee460eb861b4d821ddd6f56ef3a7ebcfca7e6005

Relevant log output

OS

macOS

GPU

Apple

CPU

Apple

Ollama version

0.20.0-rc1

Originally created by @iwater on GitHub (Apr 2, 2026). Original GitHub issue: https://github.com/ollama/ollama/issues/15235 ### What is the issue? v0.20 got 500 Internal Server Error ``` ollama run hf.co/unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M Error: 500 Internal Server Error: unable to load model: /Users/iwater/.ollama/models/blobs/sha256-8520b6bc9bbd9b6432bc1c37ee460eb861b4d821ddd6f56ef3a7ebcfca7e6005 ``` ### Relevant log output ```shell ``` ### OS macOS ### GPU Apple ### CPU Apple ### Ollama version 0.20.0-rc1

GiteaMirror added the bug label 2026-05-05 02:33:03 -05:00

GiteaMirror commented

2026-05-05 02:33:07 -05:00

@rick-github commented on GitHub (Apr 3, 2026):

Unlsloth versions of gemma4 will not work until the next vendor sync. The unsloth models (and any other source that uses llama.cpp to quantize the model) are split models, separate text and vision GGUFs. Split models must run on the llama.cpp backend, which does not yet support the gemma4 architecture. The next vendor sync will merge llama.cpp support for gemma4.

@rick-github commented on GitHub (Apr 3, 2026): Unlsloth versions of gemma4 will not work until the next vendor sync. The unsloth models (and any other source that uses llama.cpp to quantize the model) are split models, separate text and vision GGUFs. Split models must run on the llama.cpp backend, which does not yet support the gemma4 architecture. The next vendor sync will merge llama.cpp support for gemma4.

GiteaMirror commented

2026-05-05 02:33:08 -05:00

@yie2984 commented on GitHub (Apr 3, 2026):

me too

@yie2984 commented on GitHub (Apr 3, 2026): me too

GiteaMirror commented

2026-05-05 02:33:10 -05:00

@francisoliverlee commented on GitHub (Apr 3, 2026):

me too
Error: 500 Internal Server Error: unable to load model: /root/.ollama/models/blobs/sha256-27ee8bbb31f274a8e6e37217198bce873611c8c605566d2c2ec95d064c9a213b

@francisoliverlee commented on GitHub (Apr 3, 2026): me too Error: 500 Internal Server Error: unable to load model: /root/.ollama/models/blobs/sha256-27ee8bbb31f274a8e6e37217198bce873611c8c605566d2c2ec95d064c9a213b

GiteaMirror commented

2026-05-05 02:33:13 -05:00

@Cyberschorsch commented on GitHub (Apr 4, 2026):

Meanwhile you can try and "remove" the vision model part:

Run this command to create a Modelfile:
ollama show --modelfile hf.co/unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M > gemma_4_unsloth_modelfile

Then open the gemma_4_unsloth_modelfile and add a # character in front of the second FROM line:

# Modelfile generated by "ollama show"
# To build a new Modelfile based on this, replace FROM with:
# FROM hf.co/unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M

FROM /root/.ollama/models/blobs/sha256-2f8672b0c2cca8dedfb8782815c2769ccdaa6512788f3ee87b32cf117f0dffc1
#FROM /root/.ollama/models/blobs/sha256-fc2ebf4c44528daa2cea7b39891712847ca5e4f87dcf578054a06c46bfe6da27
TEMPLATE "{{ if .System }}<bos><|turn>system
{{ .System }}<turn|>
{{ end }}{{ if .Prompt }}<|turn>user
{{ .Prompt }}<turn|>
{{ end }}<|turn>model
{{ .Response }}<turn|>
"
PARAMETER stop <bos>
PARAMETER stop <|turn>
PARAMETER stop <turn|>
PARAMETER stop <|turn>user

Then run:

ollama create gemma-4-unsloth -f gemma_4_unsloth_modelfile

Now you can run the unsloth version without vision:

ollama run gemma-4-unsloth

@Cyberschorsch commented on GitHub (Apr 4, 2026): Meanwhile you can try and "remove" the vision model part: Run this command to create a Modelfile: `ollama show --modelfile hf.co/unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M > gemma_4_unsloth_modelfile` Then open the gemma_4_unsloth_modelfile and add a # character in front of the second FROM line: ``` # Modelfile generated by "ollama show" # To build a new Modelfile based on this, replace FROM with: # FROM hf.co/unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M FROM /root/.ollama/models/blobs/sha256-2f8672b0c2cca8dedfb8782815c2769ccdaa6512788f3ee87b32cf117f0dffc1 #FROM /root/.ollama/models/blobs/sha256-fc2ebf4c44528daa2cea7b39891712847ca5e4f87dcf578054a06c46bfe6da27 TEMPLATE "{{ if .System }}<bos><|turn>system {{ .System }}<turn|> {{ end }}{{ if .Prompt }}<|turn>user {{ .Prompt }}<turn|> {{ end }}<|turn>model {{ .Response }}<turn|> " PARAMETER stop <bos> PARAMETER stop <|turn> PARAMETER stop <turn|> PARAMETER stop <|turn>user ``` Then run: `ollama create gemma-4-unsloth -f gemma_4_unsloth_modelfile` Now you can run the unsloth version without vision: ollama run gemma-4-unsloth

GiteaMirror commented

2026-05-05 02:33:14 -05:00

@uprootnetworks commented on GitHub (Apr 4, 2026):

Meanwhile you can try and "remove" the vision model part:

Run this command to create a Modelfile: ollama show --modelfile hf.co/unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M > gemma_4_unsloth_modelfile

Then open the gemma_4_unsloth_modelfile and add a # character in front of the second FROM line:
# Modelfile generated by "ollama show"
# To build a new Modelfile based on this, replace FROM with:
# FROM hf.co/unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M

FROM /root/.ollama/models/blobs/sha256-2f8672b0c2cca8dedfb8782815c2769ccdaa6512788f3ee87b32cf117f0dffc1
#FROM /root/.ollama/models/blobs/sha256-fc2ebf4c44528daa2cea7b39891712847ca5e4f87dcf578054a06c46bfe6da27
TEMPLATE "{{ if .System }}<bos><|turn>system
{{ .System }}<turn|>
{{ end }}{{ if .Prompt }}<|turn>user
{{ .Prompt }}<turn|>
{{ end }}<|turn>model
{{ .Response }}<turn|>
"
PARAMETER stop <bos>
PARAMETER stop <|turn>
PARAMETER stop <turn|>
PARAMETER stop <|turn>user
Then run:

ollama create gemma-4-unsloth -f gemma_4_unsloth_modelfile

Now you can run the unsloth version without vision:

ollama run gemma-4-unsloth

This worked for me, thank you!

@uprootnetworks commented on GitHub (Apr 4, 2026): > Meanwhile you can try and "remove" the vision model part: > > Run this command to create a Modelfile: `ollama show --modelfile hf.co/unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M > gemma_4_unsloth_modelfile` > > Then open the gemma_4_unsloth_modelfile and add a # character in front of the second FROM line: > > ``` > # Modelfile generated by "ollama show" > # To build a new Modelfile based on this, replace FROM with: > # FROM hf.co/unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M > > FROM /root/.ollama/models/blobs/sha256-2f8672b0c2cca8dedfb8782815c2769ccdaa6512788f3ee87b32cf117f0dffc1 > #FROM /root/.ollama/models/blobs/sha256-fc2ebf4c44528daa2cea7b39891712847ca5e4f87dcf578054a06c46bfe6da27 > TEMPLATE "{{ if .System }}<bos><|turn>system > {{ .System }}<turn|> > {{ end }}{{ if .Prompt }}<|turn>user > {{ .Prompt }}<turn|> > {{ end }}<|turn>model > {{ .Response }}<turn|> > " > PARAMETER stop <bos> > PARAMETER stop <|turn> > PARAMETER stop <turn|> > PARAMETER stop <|turn>user > ``` > > Then run: > > `ollama create gemma-4-unsloth -f gemma_4_unsloth_modelfile` > > Now you can run the unsloth version without vision: > > ollama run gemma-4-unsloth This worked for me, thank you!

GiteaMirror commented

2026-05-05 02:33:17 -05:00

@aznohonza commented on GitHub (Apr 7, 2026):

This commenting of the 2nd FROM line works but what if I want to use the vision capabilities? I tried updating to the current latest 0.20.3 and keep both FROM lines uncommented but it still fails to load the model.

honza@desktop:~/Documents$ ollama run gemma-4-E4B-it-UD-Q4_K_XL what is this image? ./Pictures/Screenshots/s.png Error: 500 Internal Server Error: unable to load model: /usr/share/ollama/.ollama/models/blobs/sha256-e8cd1a4c5162dfa355c567b4099206793793b5548951f077f376a0076dc0a058

@aznohonza commented on GitHub (Apr 7, 2026): This commenting of the 2nd FROM line works but what if I want to use the vision capabilities? I tried updating to the current latest 0.20.3 and keep both FROM lines uncommented but it still fails to load the model. `honza@desktop:~/Documents$ ollama run gemma-4-E4B-it-UD-Q4_K_XL what is this image? ./Pictures/Screenshots/s.png Error: 500 Internal Server Error: unable to load model: /usr/share/ollama/.ollama/models/blobs/sha256-e8cd1a4c5162dfa355c567b4099206793793b5548951f077f376a0076dc0a058 `

GiteaMirror commented

2026-05-05 02:33:20 -05:00

@rick-github commented on GitHub (Apr 7, 2026):

https://github.com/ollama/ollama/issues/15235#issuecomment-4181221561

@rick-github commented on GitHub (Apr 7, 2026): https://github.com/ollama/ollama/issues/15235#issuecomment-4181221561

GiteaMirror commented

2026-05-05 02:33:23 -05:00

@aznohonza commented on GitHub (Apr 7, 2026):

Thx, so I need to wait for llama.cpp to support it, then for ollama to update to the new llama.cpp version?
If so are there any other quantized versions of gemma4 that currently work with ollama supporting vision?

@aznohonza commented on GitHub (Apr 7, 2026): Thx, so I need to wait for llama.cpp to support it, then for ollama to update to the new llama.cpp version? If so are there any other quantized versions of gemma4 that currently work with ollama supporting vision?

GiteaMirror commented

2026-05-05 02:33:25 -05:00

@Cyberschorsch commented on GitHub (Apr 7, 2026):

llama.cpp should already support it. You could use LocalAI in the meantime with an updated llama.cpp backend which already supports gemma4 models with vision.

@Cyberschorsch commented on GitHub (Apr 7, 2026): llama.cpp should already support it. You could use LocalAI in the meantime with an updated llama.cpp backend which already supports gemma4 models with vision.

GiteaMirror commented

2026-05-05 02:33:27 -05:00

@aznohonza commented on GitHub (Apr 7, 2026):

Ah alright, thx. Can I install localAI without breaking ollama on my system?

@aznohonza commented on GitHub (Apr 7, 2026): Ah alright, thx. Can I install localAI without breaking ollama on my system?

GiteaMirror commented

2026-05-05 02:33:29 -05:00

@rick-github commented on GitHub (Apr 7, 2026):

Or just use the gemma4 model from the ollama library.

@rick-github commented on GitHub (Apr 7, 2026): Or just use the gemma4 model from the ollama library.

GiteaMirror commented

2026-05-05 02:33:30 -05:00

@aznohonza commented on GitHub (Apr 7, 2026):

but that isnt quantized or is it?

@aznohonza commented on GitHub (Apr 7, 2026): but that isnt quantized or is it?

GiteaMirror commented

2026-05-05 02:33:31 -05:00

@rick-github commented on GitHub (Apr 7, 2026):

The ollama library has q4, q8 and bf16 quants of the model. Since you are trying to run the q4 quant from unsloth, you might as well use the q4 from ollama. There will be slight differences in some tensors but it's unlikely to make a significant difference to the output. If you want to use a different quant, for example q6 because you want to trade space for precision without going as large as q8, then you need to wait for the llama.cpp sync.

@rick-github commented on GitHub (Apr 7, 2026): The ollama library has [q4](https://ollama.com/library/gemma4:e4b-it-q4_K_M), [q8](https://ollama.com/library/gemma4:e4b-it-q8_0) and [bf16](https://ollama.com/library/gemma4:e4b-it-bf16) quants of the model. Since you are trying to run the q4 quant from unsloth, you might as well use the q4 from ollama. There will be slight differences in some tensors but it's unlikely to make a significant difference to the output. If you want to use a different quant, for example q6 because you want to trade space for precision without going as large as q8, then you need to wait for the llama.cpp sync.

GiteaMirror commented

2026-05-05 02:33:33 -05:00

@aznohonza commented on GitHub (Apr 7, 2026):

I downloaded the ollama q4 but unfortunately it doesnt have the mmproj included so it does not support vision, so guess Ill just wait.

ollama show --modelfile gemma4:e4b-it-q4_K_M

# Modelfile generated by "ollama show"
# To build a new Modelfile based on this, replace FROM with:
# FROM gemma4:e4b-it-q4_K_M

FROM /usr/share/ollama/.ollama/models/blobs/sha256-4c27e0f5b5adf02ac956c7322bd2ee7636fe3f45a8512c9aba5385242cb6e09a
TEMPLATE {{ .Prompt }}
RENDERER gemma4
PARSER gemma4
PARAMETER top_p 0.95
PARAMETER temperature 1
PARAMETER top_k 64```

@aznohonza commented on GitHub (Apr 7, 2026): I downloaded the ollama q4 but unfortunately it doesnt have the mmproj included so it does not support vision, so guess Ill just wait. ``` ollama show --modelfile gemma4:e4b-it-q4_K_M # Modelfile generated by "ollama show" # To build a new Modelfile based on this, replace FROM with: # FROM gemma4:e4b-it-q4_K_M FROM /usr/share/ollama/.ollama/models/blobs/sha256-4c27e0f5b5adf02ac956c7322bd2ee7636fe3f45a8512c9aba5385242cb6e09a TEMPLATE {{ .Prompt }} RENDERER gemma4 PARSER gemma4 PARAMETER top_p 0.95 PARAMETER temperature 1 PARAMETER top_k 64```

GiteaMirror commented

2026-05-05 02:33:34 -05:00

@rick-github commented on GitHub (Apr 7, 2026):

Ollama models combine the vision (mmproj) and text weights into a single GGUF file, that's why only one FROM line is found in the Modelfile. gemma4:e4b-it-q4_K_M will process images.

$ ollama show gemma4:e4b-it-q4_K_M
...
  Capabilities
    completion    
    vision        
    audio         
    tools         
    thinking

@rick-github commented on GitHub (Apr 7, 2026): Ollama models combine the vision (mmproj) and text weights into a single GGUF file, that's why only one `FROM` line is found in the Modelfile. `gemma4:e4b-it-q4_K_M` will process images. ```console $ ollama show gemma4:e4b-it-q4_K_M ... Capabilities completion vision audio tools thinking ```

GiteaMirror commented

2026-05-05 02:33:36 -05:00

@aznohonza commented on GitHub (Apr 7, 2026):

Oh alright, thanks. And could you please share the way to supply the model with an image? My old method that worked for gemma3 does not seem to work for this gemma4 model.

 honza@desktop:~/Documents$ cat a.py 
import ollama

response = ollama.chat(
    #model="gemma-4-E4B-it-UD-Q4_K_XL",
    #model="gemma-3-12b-it-qat-UD-Q4_K_XL",
    model="gemma4:e4b-it-q4_K_M",
    messages=[
        {"role": "user", "content": "Describe the image", "images": ["s.png"]}
    ],
)

print(response["message"]['role'])
print(response["message"]['content'])
honza@desktop:~/Documents$ python3 a.py 
assistant
I cannot describe an image because no image was provided.

You have pasted a large block of academic or literary notes, but there is no visual content attached.

If you would like me to help you with the text you provided (e.g., summarizing the literary movements, explaining the life of a specific author, or formatting the notes), please let me know!

@aznohonza commented on GitHub (Apr 7, 2026): Oh alright, thanks. And could you please share the way to supply the model with an image? My old method that worked for gemma3 does not seem to work for this gemma4 model. ``` honza@desktop:~/Documents$ cat a.py import ollama response = ollama.chat( #model="gemma-4-E4B-it-UD-Q4_K_XL", #model="gemma-3-12b-it-qat-UD-Q4_K_XL", model="gemma4:e4b-it-q4_K_M", messages=[ {"role": "user", "content": "Describe the image", "images": ["s.png"]} ], ) print(response["message"]['role']) print(response["message"]['content']) honza@desktop:~/Documents$ python3 a.py assistant I cannot describe an image because no image was provided. You have pasted a large block of academic or literary notes, but there is no visual content attached. If you would like me to help you with the text you provided (e.g., summarizing the literary movements, explaining the life of a specific author, or formatting the notes), please let me know! ```

GiteaMirror commented

2026-05-05 02:33:37 -05:00

@rick-github commented on GitHub (Apr 7, 2026):

Works for me.

$ cp image1.png s.png
$ python3 a.py 
assistant
This is a charming, brightly lit portrait of an extremely fluffy, white puppy.

Is s.png an image?

@rick-github commented on GitHub (Apr 7, 2026): Works for me. ```console $ cp image1.png s.png $ python3 a.py assistant This is a charming, brightly lit portrait of an extremely fluffy, white puppy. ``` Is `s.png` an image?

GiteaMirror commented

2026-05-05 02:33:39 -05:00

@aznohonza commented on GitHub (Apr 7, 2026):

Yes s.png is an image. I dont know why it doesnt work for me. But if I pass an image as bytes instead it works fine. (Tried a different image than s.png, though s.png also worked with this new script). I have the latest ollama 0.20.3.

$ cat b.py 
import ollama
from pathlib import Path

# Path to your image
#image_path = "/home/honza/Pictures/Screenshots/s.png"
image_path = "/home/honza/Pictures/wallpapers/3.jpg"

# Read the image as raw bytes (safest approach for multimodal models)
img_bytes = Path(image_path).read_bytes()

response = ollama.chat(
    model="gemma4:e4b-it-q4_K_M",
    messages=[
        {
            "role": "user",
            "content": "Describe the image, who is the main subject?",
            "images": [img_bytes] # Pass the raw bytes instead of the path string
        }
    ],
)

# Use dot notation to access the properties!
print(response.message.role)
print(response.message.content)
$ python3 b.py 
assistant
**Description of the Image:**

This is a highly dramatic, stylized, and cinematic image dominated by a futuristic vehicle. The overall aesthetic screams **Cyberpunk**—a blend of high-tech extravagance and gritty, dystopian urban decay.

**The Main Subject:**
The main subject is a vividly customized, low-slung **sports car**.

**Detailed Description:**

1.  **The Car:** The car is painted a striking, vibrant **mustard yellow or gold**. It is heavily modified with futuristic, aggressive styling. Its most notable features are the mechanical details and the lighting:
    *   **Wheels:** The wheels are massive and highly detailed, featuring glowing **electric blue** accents that give them a mechanical, cybernetic look.
    *   **Body:** The body is angular and sleek, featuring visible glowing trim and graphic decals, including the word "CYBERPUNK" on the side.
    *   **Front:** The front fascia is aggressive, featuring glowing marker slots marked "XXX."
    *   **Impact:** The car appears extremely powerful, almost mechanical in its excess of glowing parts and visible technology.

2.  **The Setting:** The car is parked or driving slowly on a dark, slick, reflective asphalt street. The background is a massive, overwhelming **futuristic cityscape**. The buildings are towering and indistinct, covered in layers of glowing neon advertisements, digital displays, and multi-colored lights (pinks, blues, and purples). The setting feels dense, rainy, or perpetually damp, contributing to the glossy reflection on the ground.

3.  **Mood and Lighting:** The lighting is dark and moody, relying heavily on artificial, saturated light sources. The glowing blue elements of the car are particularly prominent, contrasting sharply with the yellow paint and the deep, mysterious tones of the background. The overall atmosphere is intensely futuristic, dramatic, and restless.

**In summary, the image captures a highly stylized, neon-drenched cyber-race car in a sprawling, dystopian metropolis.**

@aznohonza commented on GitHub (Apr 7, 2026): Yes s.png is an image. I dont know why it doesnt work for me. But if I pass an image as bytes instead it works fine. (Tried a different image than s.png, though s.png also worked with this new script). I have the latest ollama 0.20.3. ``` $ cat b.py import ollama from pathlib import Path # Path to your image #image_path = "/home/honza/Pictures/Screenshots/s.png" image_path = "/home/honza/Pictures/wallpapers/3.jpg" # Read the image as raw bytes (safest approach for multimodal models) img_bytes = Path(image_path).read_bytes() response = ollama.chat( model="gemma4:e4b-it-q4_K_M", messages=[ { "role": "user", "content": "Describe the image, who is the main subject?", "images": [img_bytes] # Pass the raw bytes instead of the path string } ], ) # Use dot notation to access the properties! print(response.message.role) print(response.message.content) $ python3 b.py assistant **Description of the Image:** This is a highly dramatic, stylized, and cinematic image dominated by a futuristic vehicle. The overall aesthetic screams **Cyberpunk**—a blend of high-tech extravagance and gritty, dystopian urban decay. **The Main Subject:** The main subject is a vividly customized, low-slung **sports car**. **Detailed Description:** 1. **The Car:** The car is painted a striking, vibrant **mustard yellow or gold**. It is heavily modified with futuristic, aggressive styling. Its most notable features are the mechanical details and the lighting: * **Wheels:** The wheels are massive and highly detailed, featuring glowing **electric blue** accents that give them a mechanical, cybernetic look. * **Body:** The body is angular and sleek, featuring visible glowing trim and graphic decals, including the word "CYBERPUNK" on the side. * **Front:** The front fascia is aggressive, featuring glowing marker slots marked "XXX." * **Impact:** The car appears extremely powerful, almost mechanical in its excess of glowing parts and visible technology. 2. **The Setting:** The car is parked or driving slowly on a dark, slick, reflective asphalt street. The background is a massive, overwhelming **futuristic cityscape**. The buildings are towering and indistinct, covered in layers of glowing neon advertisements, digital displays, and multi-colored lights (pinks, blues, and purples). The setting feels dense, rainy, or perpetually damp, contributing to the glossy reflection on the ground. 3. **Mood and Lighting:** The lighting is dark and moody, relying heavily on artificial, saturated light sources. The glowing blue elements of the car are particularly prominent, contrasting sharply with the yellow paint and the deep, mysterious tones of the background. The overall atmosphere is intensely futuristic, dramatic, and restless. **In summary, the image captures a highly stylized, neon-drenched cyber-race car in a sprawling, dystopian metropolis.** ```

GiteaMirror commented

2026-05-05 02:33:40 -05:00

@rparo20 commented on GitHub (Apr 8, 2026):

We ran into the same issue with unsloth Gemma 4 GGUFs on Ollama 0.20.

After some debugging, we found the root cause is the split model format + architecture mismatch in Ollama's llama.cpp backend. So we quantized directly from Google's official weights using latest llama.cpp (build 400ac8e, includes the Gemma 4 BOS fix).

Everything loads and runs correctly on Ollama 0.20.2. Tested on real hardware:

Model	Size	VRAM	Mac mini M4 (16GB)	M4 Max (128GB)	Tool Calling
`batiai/gemma4-e2b:q4`	3.2GB	7.1GB	107.8 t/s	132.5 t/s	inconsistent
`batiai/gemma4-e4b:q4`	5.0GB	10GB	57.1 t/s	84.0 t/s	works
`batiai/gemma4-26b:q3`	13GB	20GB	needs 24GB+	70.7 t/s	works
`batiai/gemma4-26b:q4`	16GB	23GB	needs 32GB+	74.9 t/s	works

ollama pull batiai/gemma4-e4b:q4
ollama pull batiai/gemma4-26b:q3

Source weights (BF16) -> GGUF conversion -> Q4_K_M / Q6_K quantization. No re-quantization from existing GGUFs.

HuggingFace repos with full benchmark data:

Hope this helps anyone stuck on this.

@rparo20 commented on GitHub (Apr 8, 2026): We ran into the same issue with unsloth Gemma 4 GGUFs on Ollama 0.20. After some debugging, we found the root cause is the split model format + architecture mismatch in Ollama's llama.cpp backend. So we quantized directly from Google's official weights using latest llama.cpp (build `400ac8e`, includes the Gemma 4 BOS fix). Everything loads and runs correctly on Ollama 0.20.2. Tested on real hardware: | Model | Size | VRAM | Mac mini M4 (16GB) | M4 Max (128GB) | Tool Calling | |-------|------|------|---------------------|----------------|-------------| | `batiai/gemma4-e2b:q4` | 3.2GB | 7.1GB | 107.8 t/s | 132.5 t/s | inconsistent | | `batiai/gemma4-e4b:q4` | 5.0GB | 10GB | 57.1 t/s | 84.0 t/s | works | | `batiai/gemma4-26b:q3` | 13GB | 20GB | needs 24GB+ | 70.7 t/s | works | | `batiai/gemma4-26b:q4` | 16GB | 23GB | needs 32GB+ | 74.9 t/s | works | ``` ollama pull batiai/gemma4-e4b:q4 ollama pull batiai/gemma4-26b:q3 ``` Source weights (BF16) -> GGUF conversion -> Q4_K_M / Q6_K quantization. No re-quantization from existing GGUFs. HuggingFace repos with full benchmark data: - https://huggingface.co/batiai/gemma-4-26B-A4B-it-GGUF - https://huggingface.co/batiai/gemma-4-E4B-it-GGUF - https://huggingface.co/batiai/gemma-4-E2B-it-GGUF Hope this helps anyone stuck on this.

GiteaMirror commented

2026-05-05 02:33:41 -05:00

@rick-github commented on GitHub (Apr 8, 2026):

You realize that these models are not multi-modal, right?

@rick-github commented on GitHub (Apr 8, 2026): You realize that these models are not multi-modal, right?

GiteaMirror commented

2026-05-05 02:33:42 -05:00

@rparo20 commented on GitHub (Apr 11, 2026):

Yes, that's intentional — our GGUFs are text-only by design.

We originally included mmproj (vision) in the Modelfile, but ran into the same issues reported in ollama#15352 and llama.cpp#21402: adding the vision projector caused the model to fail to load entirely on Ollama 0.20.x.

So we ship text-only builds that reliably load and run on Ollama 0.20+. They're also ~1GB smaller per tag since mmproj is excluded. For users who need vision later, we've uploaded the mmproj-BF16.gguf files separately to HuggingFace — they can be added back via Modelfile once the ecosystem catches up:

The goal was "text chat + tool calling that actually works on latest Ollama today," not full multimodal. Apologies if the framing was unclear in the original comment.

@rparo20 commented on GitHub (Apr 11, 2026): Yes, that's intentional — our GGUFs are text-only by design. We originally included mmproj (vision) in the Modelfile, but ran into the same issues reported in [ollama#15352](https://github.com/ollama/ollama/issues/15352) and [llama.cpp#21402](https://github.com/ggml-org/llama.cpp/issues/21402): adding the vision projector caused the model to fail to load entirely on Ollama 0.20.x. So we ship text-only builds that reliably load and run on Ollama 0.20+. They're also ~1GB smaller per tag since mmproj is excluded. For users who need vision later, we've uploaded the `mmproj-BF16.gguf` files separately to HuggingFace — they can be added back via Modelfile once the ecosystem catches up: - https://huggingface.co/batiai/gemma-4-E2B-it-GGUF - https://huggingface.co/batiai/gemma-4-E4B-it-GGUF - https://huggingface.co/batiai/gemma-4-26B-A4B-it-GGUF - https://huggingface.co/batiai/gemma-4-31B-it-GGUF The goal was "text chat + tool calling that actually works on latest Ollama today," not full multimodal. Apologies if the framing was unclear in the original comment.

GiteaMirror commented

2026-05-05 02:33:43 -05:00

@TByte007 commented on GitHub (Apr 12, 2026):

Yes, that's intentional — our GGUFs are text-only by design.

We originally included mmproj (vision) in the Modelfile, but ran into the same issues reported in ollama#15352 and llama.cpp#21402: adding the vision projector caused the model to fail to load entirely on Ollama 0.20.x.

So we ship text-only builds that reliably load and run on Ollama 0.20+. They're also ~1GB smaller per tag since mmproj is excluded. For users who need vision later, we've uploaded the mmproj-BF16.gguf files separately to HuggingFace — they can be added back via Modelfile once the ecosystem catches up:
* https://huggingface.co/batiai/gemma-4-E2B-it-GGUF

* https://huggingface.co/batiai/gemma-4-E4B-it-GGUF

* https://huggingface.co/batiai/gemma-4-26B-A4B-it-GGUF

* https://huggingface.co/batiai/gemma-4-31B-it-GGUF
The goal was "text chat + tool calling that actually works on latest Ollama today," not full multimodal. Apologies if the framing was unclear in the original comment.

Thank you for those models, they work perfectly and are fast. I can even fit 40k context on 3090 :)

@TByte007 commented on GitHub (Apr 12, 2026): > Yes, that's intentional — our GGUFs are text-only by design. > > We originally included mmproj (vision) in the Modelfile, but ran into the same issues reported in [ollama#15352](https://github.com/ollama/ollama/issues/15352) and [llama.cpp#21402](https://github.com/ggml-org/llama.cpp/issues/21402): adding the vision projector caused the model to fail to load entirely on Ollama 0.20.x. > > So we ship text-only builds that reliably load and run on Ollama 0.20+. They're also ~1GB smaller per tag since mmproj is excluded. For users who need vision later, we've uploaded the `mmproj-BF16.gguf` files separately to HuggingFace — they can be added back via Modelfile once the ecosystem catches up: > > * https://huggingface.co/batiai/gemma-4-E2B-it-GGUF > > * https://huggingface.co/batiai/gemma-4-E4B-it-GGUF > > * https://huggingface.co/batiai/gemma-4-26B-A4B-it-GGUF > > * https://huggingface.co/batiai/gemma-4-31B-it-GGUF > > > The goal was "text chat + tool calling that actually works on latest Ollama today," not full multimodal. Apologies if the framing was unclear in the original comment. Thank you for those models, they work perfectly and are fast. I can even fit 40k context on 3090 :)

GiteaMirror commented

2026-05-05 02:33:44 -05:00

@PureBlissAK commented on GitHub (Apr 18, 2026):

🤖 Automated Triage & Analysis Report

Issue: #15235
Analyzed: 2026-04-18T18:22:52.244357

Analysis

Type: unknown
Severity: medium
Components: unknown

Implementation Plan

Effort: medium
Steps:

This issue has been triaged and marked for implementation.

@PureBlissAK commented on GitHub (Apr 18, 2026):  ## 🤖 Automated Triage & Analysis Report **Issue**: #15235 **Analyzed**: 2026-04-18T18:22:52.244357 ### Analysis - **Type**: unknown - **Severity**: medium - **Components**: unknown ### Implementation Plan - **Effort**: medium - **Steps**: *This issue has been triaged and marked for implementation.*

Sign in to join this conversation.

Branches Tags

main

dhiltgen/ci

parth-launch-plan-gating

hoyyeva/anthropic-reference-images-path

parth-anthropic-reference-images-path

brucemacd/download-before-remove

hoyyeva/editor-config-repair

parth-mlx-decode-checkpoints

parth-launch-codex-app

hoyyeva/fix-codex-model-metadata-warning

hoyyeva/qwen

parth/hide-claude-desktop-till-release

hoyyeva/opencode-image-modality

parth-add-claude-code-autoinstall

release_v0.22.0

pdevine/manifest-list

codex/fix-codex-model-metadata-warning

pdevine/addressable-manifest

brucemacd/launch-fetch-reccomended

jmorganca/llama-compat

launch-copilot-cli

hoyyeva/opencode-thinking

release_v0.20.7

parth-auto-save-backup

parth-test

jmorganca/gemma4-audio-replacements

fix-manifest-digest-on-pull

hoyyeva/vscode-improve

brucemacd/install-server-wait

parth/update-claude-docs

brucemac/start-ap-install

pdevine/mlx-update

pdevine/qwen35_vision

drifkin/api-show-fallback

mintlify/image-generation-1773352582

hoyyeva/server-context-length-local-config

jmorganca/faster-reptition-penalties

jmorganca/convert-nemotron

parth-pi-thinking

pdevine/sampling-penalties

jmorganca/fix-create-quantization-memory

dongchen/resumable_transfer_fix

pdevine/sampling-cache-error

jessegross/mlx-usage

hoyyeva/openclaw-config

hoyyeva/app-html

pdevine/qwen3next

brucemacd/sign-sh-install

brucemacd/tui-update

brucemacd/usage-api

jmorganca/launch-empty

fix-app-dist-embed

mxyng/mlx-compile

mxyng/mlx-quant

mxyng/mlx-glm4.7

mxyng/mlx

brucemacd/simplify-model-picker

jmorganca/qwen3-concurrent

fix-glm-4.7-flash-mla-config

drifkin/qwen3-coder-opening-tag

brucemacd/usage-cli

fix-cuda12-fattn-shmem

ollama-imagegen-docs

parth/fix-multiline-inputs

brucemacd/config-docs

mxyng/model-files

mxyng/simple-execute

fix-imagegen-ollama-models

mxyng/async-upload

jmorganca/lazy-no-dtype-changes

imagegen-auto-detect-create

parth/decrease-concurrent-download-hf

fix-mlx-quantize-init

jmorganca/x-cleanup

usage

imagegen-readme

jmorganca/glm-image

mlx-gpu-cd

jmorganca/imagegen-modelfile

parth/agent-skills

parth/agent-allowlist

parth/signed-in-offline

parth/agents

parth/fix-context-chopping

improve-cloud-flow

parth/add-models-websearch

parth/prompt-renderer-mcp

jmorganca/native-settings

jmorganca/download-stream-hash

jmorganca/client2-rebased

brucemacd/oai-chat-req-multipart

jessegross/multi_chunk_reserve

grace/additional-omit-empty

grace/mistral-3-large

mxyng/tokenizer2

mxyng/tokenizer

jessegross/flash

hoyyeva/windows-nacked-app

mxyng/cleanup-attention

grace/deepseek-parser

hoyyeva/remember-unsent-prompt

parth/add-lfs-pointer-error-conversion

parth/olmo2-test2

hoyyeva/ollama-launchagent-plist

nicole/olmo-model

parth/olmo-test

mxyng/remove-embedded

parth/render-template

jmorganca/intellect-3

parth/remove-prealloc-linter

jmorganca/cmd-eval

nicole/nomic-embed-text-fix

mxyng/lint-2

hoyyeva/add-gemini-3-pro-preview

hoyyeva/load-model-list

mxyng/expand-path

mxyng/environ-2

hoyyeva/deeplink-json-encoding

parth/improve-tool-calling-tests

hoyyeva/conversation

hoyyeva/assistant-edit-response

hoyyeva/thinking

origin/brucemacd/invalid-char-i-err

parth/improve-tool-calling

jmorganca/required-omitempty

grace/qwen3-vl-tests

mxyng/iter-client

parth/docs-readme

nicole/embed-test

pdevine/integration-benchstat

parth/remove-generate-cmd

parth/add-toolcall-id

mxyng/server-tests

jmorganca/glm-4.6

jmorganca/gin-h-compat

drifkin/stable-tool-args

pdevine/qwen3-more-thinking

parth/add-websearch-client

nicole/websearch_local

jmorganca/qwen3-coder-updates

grace/deepseek-v3-migration-tests

mxyng/fix-create

jmorganca/cloud-errors

pdevine/parser-tidy

revert-12233-parth/simplify-entrypoints-runner

parth/enable-so-gpt-oss

brucemacd/qwen3vl

jmorganca/readme-simplify

parth/gpt-oss-structured-outputs

revert-12039-jmorganca/tools-braces

mxyng/embeddings

mxyng/gguf

mxyng/benchmark

mxyng/types-null

parth/move-parsing

mxyng/gemma2

jmorganca/docs

mxyng/16-bit

mxyng/create-stdin

pdevine/authorizedkeys

mxyng/quant

parth/opt-in-error-context-window

brucemacd/cache-models

brucemacd/runner-completion

jmorganca/llama-update-6

brucemacd/benchmark-list

brucemacd/partial-read-caps

parth/deepseek-r1-tools

mxyng/omit-array

parth/tool-prefix-temp

brucemacd/runner-test

jmorganca/qwen25vl

brucemacd/model-forward-test-ext

parth/python-function-parsing

jmorganca/cuda-compression-none

drifkin/num-parallel

drifkin/chat-truncation-fix

jmorganca/sync

parth/python-tools-calling

drifkin/array-head-count

brucemacd/create-no-loop

parth/server-enable-content-stream-with-tools

qwen25omni

mxyng/v3

brucemacd/ropeconfig

jmorganca/silence-tokenizer

parth/sample-so-test

parth/sampling-structured-outputs

brucemacd/doc-go-engine

parth/constrained-sampling-json

jmorganca/mistral-wip

brucemacd/mistral-small-convert

parth/sample-unmarshal-json-for-params

brucemacd/jomorganca/mistral

pdevine/bfloat16

jmorganca/mistral

brucemacd/mistral

pdevine/logging

parth/sample-correctness-fix

parth/sample-fix-sorting

jmorgan/sample-fix-sorting-extras

jmorganca/temp-0-images

brucemacd/parallel-embed-models

brucemacd/shim-grammar

jmorganca/fix-gguf-error

bmizerany/nameswork

jmorganca/faster-releases

bmizerany/validatenames

brucemacd/err-no-vocab

brucemacd/rope-config

brucemacd/err-hint

brucemacd/qwen2_5

brucemacd/logprobs

brucemacd/new_runner_graph_bench

progress-flicker

brucemacd/forward-test

brucemacd/go_qwen2

pdevine/gemma2

jmorganca/add-missing-symlink-eval

mxyng/next-debug

parth/set-context-size-openai

brucemacd/next-bpe-bench

brucemacd/next-bpe-test

brucemacd/new_runner_e2e

brucemacd/new_runner_qwen2

pdevine/convert-cohere2

brucemacd/convert-cli

parth/log-probs

mxyng/next-mlx

mxyng/cmd-history

parth/templating

parth/tokenize-detokenize

brucemacd/check-key-register

bmizerany/grammar

jmorganca/vendor-081b29bd

mxyng/func-checks

jmorganca/fix-null-format

parth/fix-default-to-warn-json

jmorganca/qwen2vl

jmorganca/no-concat

parth/cmd-cleanup-SO

brucemacd/check-key-register-structured-err

parth/openai-stream-usage

parth/fix-referencing-so

stream-tools-stop

jmorganca/degin-1

brucemacd/install-path-clean

brucemacd/push-name-validation

brucemacd/browser-key-register

jmorganca/openai-fix-first-message

jmorganca/fix-proxy

jessegross/sample

parth/disallow-streaming-tools

dhiltgen/remove_submodule

jmorganca/ga

jmorganca/mllama

pdevine/newlines

pdevine/geems-2b

jmorganca/llama-bump

mxyng/modelname-7

mxyng/gin-slog

mxyng/modelname-6

jyan/convert-prog

jyan/quant5

paligemma-support

pdevine/import-docs

jmorganca/openai-context

jyan/paligemma

jyan/p2

jyan/palitest

bmizerany/embedspeedup

jmorganca/llama-vit

brucemacd/allow-ollama

royh/ep-methods

royh/whisper

mxyng/api-models

mxyng/fix-memory

jyan/q4_4/8

jyan/ollama-v

royh/stream-tools

roy-embed-parallel

bmizerany/hrm

revert-5963-revert-5924-mxyng/llama3.1-rope

royh/embed-viz

jyan/local2

jyan/auth

jyan/local

jyan/parse-temp

jmorganca/template-mistral

jyan/reord-g

royh-openai-suffixdocs

royh-imgembed

royh-embed-parallel

jyan/quant4

royh-precision

jyan/progress

pdevine/fix-template

jyan/quant3

pdevine/ggla

mxyng/update-registry-domain

jmorganca/ggml-static

mxyng/create-context

jyan/v0.146

mxyng/layers-from-files

build_dist

bmizerany/noseek

royh-ls

royh-name

timeout

mxyng/server-timestamp

bmizerany/nosillyggufslurps

royh-params

jmorganca/llama-cpp-7c26775

royh-openai-delete

royh-show-rigid

jmorganca/enable-fa

jmorganca/no-error-template

jyan/format

royh-testdelete

bmizerany/fastverify

language_support

pdevine/ps-glitches

brucemacd/tokenize

bruce/iq-quants

bmizerany/filepathwithcoloninhost

mxyng/split-bin

bmizerany/client-registry

jmorganca/if-none-match

native

jmorganca/native

jmorganca/batch-embeddings

jmorganca/initcmake

jmorganca/mm

pdevine/showggmlinfo

modenameenforcealphanum

bmizerany/modenameenforcealphanum

jmorganca/done-reason

jmorganca/llama-cpp-8960fe8

ollama.com

bmizerany/filepathnobuild

bmizerany/types/model/defaultfix

rmdisplaylong

nogogen

bmizerany/x

modelfile-readme

bmizerany/replacecolon

jmorganca/limit

jmorganca/execstack

jmorganca/replace-assets

mxyng/tune-concurrency

jmorganca/testing

whitespace-detection

jmorganca/options

upgrade-all

scratch

cuda-search

mattw/airenamer

mattw/allmodelsonhuggingface

mattw/quantcontext

mattw/whatneedstorun

brucemacd/llama-mem-calc

mattw/faq-context

mattw/communitylinks

mattw/noprune

mattw/python-functioncalling

rename

mxyng/install

pulse

remove-first

editor

mattw/selfqueryingretrieval

cgo

mattw/howtoquant

api

matt/streamingapi

format-config

mxyng/extra-args

shell

update-nous-hermes

cp-model

upload-progress

fix-unknown-model

fix-model-names

delete-fix

insecure-registry

ls

deletemodels

progressbar

readme-updates

license-layers

skip-list

list-models

modelpath

matt/examplemodelfiles

distribution

go-opts

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/ollama#71803