github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 05:05:37 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #6632] New Command-r models output nonsense #4174

New Issue
Closed
opened 2026-04-12 15:06:07 -05:00 by GiteaMirror · 7 comments
GiteaMirror commented 2026-04-12 15:06:07 -05:00
Owner
Copy Link

Originally created by @xmaayy on GitHub (Sep 4, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/6632

What is the issue?

The new 4-bit quants of command-r (I dont have the VRAM for higher quants) output nonsense.

≻ ollama run command-r
pulling manifest
pulling 8e0609b8f0fe... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████▏  18 GB
pulling b3741b7b9ce5... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████▏   77 B
pulling 922095537bc1... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████▏ 2.9 KB
pulling 945eaa8b1428... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████▏  13 KB
pulling 36b9655abe6a... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████▏   81 B
pulling 8e63f21e12fb... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████▏  568 B
verifying sha256 digest
writing manifest
success
>>> Hello
 general obligatoireasıası obligatoireasıasıası obligatoire obligatoireası obligatoireasıası obligatoireasıasıası obligatoireasıasıasıasıası obligatoire
obligatoire obligatoireasıasıası obligatoireasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıası

>>> What actions can you perform
ştıhind ministrowieası
SummaryasıştıştıhindhindhindasıhindhindasıasıştıştıhindasıasıasıhindştıştıştıştıasıhindasıhindasıasıasıasıhindasıasıştıştıasıhindasıasıasıasıasıasıasıasıasıasıaSummaryasıştıştıhindhindhindasıhindhindasıasıştıştıhindasıasıasıhindştıştıştıştıasıhindasıhindasıasıasıasıhindasıasıştıştıasıhindasıasıasasıasıasıasıasıasıasıasıasıasıasıasıştıhindştıhindştıhindasıhindasıasıası ministrowie Summaryasıasıhindhindştıştıhindasıasıasıasıasıasıasıasıasıası

Specifying version

≻ ollama run command-r:35b-08-2024-q4_K_S
pulling manifest
pulling 3ed323d43be5... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████▏  18 GB
pulling b3741b7b9ce5... 100% ▕██████████████████████���██████████████████████████████████████████████████████████████████████████████▏   77 B
pulling 922095537bc1... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████▏ 2.9 KB
pulling 945eaa8b1428... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████▏  13 KB
pulling 36b9655abe6a... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████▏   81 B
pulling 93af4240a02c... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████▏  570 B
verifying sha256 digest
writing manifest
success
>>> Hello
 generalası obligatoireasıası GENERası expulsionası obligatoireasıasıası
primarioasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıası

OS

macOS

GPU

Apple

CPU

Apple

Ollama version

0.3.9

Originally created by @xmaayy on GitHub (Sep 4, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/6632 ### What is the issue? The new 4-bit quants of command-r (I dont have the VRAM for higher quants) output nonsense. ``` bash ≻ ollama run command-r pulling manifest pulling 8e0609b8f0fe... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████▏ 18 GB pulling b3741b7b9ce5... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████▏ 77 B pulling 922095537bc1... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████▏ 2.9 KB pulling 945eaa8b1428... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████▏ 13 KB pulling 36b9655abe6a... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████▏ 81 B pulling 8e63f21e12fb... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████▏ 568 B verifying sha256 digest writing manifest success >>> Hello general obligatoireasıası obligatoireasıasıası obligatoire obligatoireası obligatoireasıası obligatoireasıasıası obligatoireasıasıasıasıası obligatoire obligatoire obligatoireasıasıası obligatoireasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıası >>> What actions can you perform ştıhind ministrowieası SummaryasıştıştıhindhindhindasıhindhindasıasıştıştıhindasıasıasıhindştıştıştıştıasıhindasıhindasıasıasıasıhindasıasıştıştıasıhindasıasıasıasıasıasıasıasıasıasıaSummaryasıştıştıhindhindhindasıhindhindasıasıştıştıhindasıasıasıhindştıştıştıştıasıhindasıhindasıasıasıasıhindasıasıştıştıasıhindasıasıasasıasıasıasıasıasıasıasıasıasıasıasıştıhindştıhindştıhindasıhindasıasıası ministrowie Summaryasıasıhindhindştıştıhindasıasıasıasıasıasıasıasıasıası ``` Specifying version ``` bash ≻ ollama run command-r:35b-08-2024-q4_K_S pulling manifest pulling 3ed323d43be5... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████▏ 18 GB pulling b3741b7b9ce5... 100% ▕██████████████████████���██████████████████████████████████████████████████████████████████████████████▏ 77 B pulling 922095537bc1... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████▏ 2.9 KB pulling 945eaa8b1428... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████▏ 13 KB pulling 36b9655abe6a... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████▏ 81 B pulling 93af4240a02c... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████▏ 570 B verifying sha256 digest writing manifest success >>> Hello generalası obligatoireasıası GENERası expulsionası obligatoireasıasıası primarioasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıasıası ``` ### OS macOS ### GPU Apple ### CPU Apple ### Ollama version 0.3.9
GiteaMirror added the memorybugmacos labels 2026-04-12 15:06:07 -05:00
GiteaMirror commented 2026-04-12 15:06:10 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Sep 4, 2024):

Can you add some server logs? It works fine on an Intel/Nvidia system:

$ docker compose exec ollama ollama -v
ollama version is 0.3.9
$ docker compose exec ollama ollama ps
NAME                            ID              SIZE    PROCESSOR       UNTIL   
command-r:35b-08-2024-q4_K_S    e746a62b5017    22 GB   27%/73% CPU/GPU Forever
$ docker compose exec ollama ollama run command-r:35b-08-2024-q4_K_S
>>> Hello
Hello there! How can I assist you today?

>>> What actions can you perform
I am capable of performing various functions to help you with information retrieval, task management, creative writing assistance, and more. Here are some of the actions I can undertake:
- Answering questions on a wide range of topics, from general knowledge to specific domains like science, history, literature, or current affairs.
- Providing concise summaries of complex texts or articles if you need a quick overview.
- Generating creative content such as stories, poems, or even marketing copy based on your prompts and guidelines.
- Offering suggestions for solutions when faced with problems or brainstorming sessions.
- Helping organize schedules by creating to-do lists or outlining steps for projects.
- Conducting research on specific subjects according to your instructions.
<!-- gh-comment-id:2328833531 --> @rick-github commented on GitHub (Sep 4, 2024): Can you add some [server logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md)? It works fine on an Intel/Nvidia system: ``` $ docker compose exec ollama ollama -v ollama version is 0.3.9 $ docker compose exec ollama ollama ps NAME ID SIZE PROCESSOR UNTIL command-r:35b-08-2024-q4_K_S e746a62b5017 22 GB 27%/73% CPU/GPU Forever $ docker compose exec ollama ollama run command-r:35b-08-2024-q4_K_S >>> Hello Hello there! How can I assist you today? >>> What actions can you perform I am capable of performing various functions to help you with information retrieval, task management, creative writing assistance, and more. Here are some of the actions I can undertake: - Answering questions on a wide range of topics, from general knowledge to specific domains like science, history, literature, or current affairs. - Providing concise summaries of complex texts or articles if you need a quick overview. - Generating creative content such as stories, poems, or even marketing copy based on your prompts and guidelines. - Offering suggestions for solutions when faced with problems or brainstorming sessions. - Helping organize schedules by creating to-do lists or outlining steps for projects. - Conducting research on specific subjects according to your instructions. ```
GiteaMirror commented 2026-04-12 15:06:11 -05:00
Author
Owner
Copy Link

@jmorganca commented on GitHub (Sep 4, 2024):

I believe this may be an issue with memory estimation, since the model got slightly smaller it may be offloading one too few layers.

<!-- gh-comment-id:2328906999 --> @jmorganca commented on GitHub (Sep 4, 2024): I believe this may be an issue with memory estimation, since the model got slightly smaller it may be offloading one too few layers.
GiteaMirror commented 2026-04-12 15:06:11 -05:00
Author
Owner
Copy Link

@xmaayy commented on GitHub (Sep 4, 2024):

Agh thanks, it looks like the model is running out of memory. I'm on a 32GB MacOS device so I have to use a smaller context size which works fine with llama-cpp-python on 4_K_S at 2048 context. When I use a model file like this:

  1 FROM command-r:35b-08-2024-q4_K_S
  2 PARAMETER num_ctx 2048

I'd assume that the model would load with a lower context size but looking at the logs its trying to load 8192 context size which exceeds my memory (also something I thought Ollama would catch):

llm_load_tensors: offloading 40 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 41/41 layers to GPU
llm_load_tensors:        CPU buffer size =  1640.62 MiB
llm_load_tensors:      Metal buffer size = 17965.92 MiB
llama_new_context_with_model: n_ctx      = 8192
llama_new_context_with_model: n_batch    = 512
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 4000000.0
llama_new_context_with_model: freq_scale = 1
...
ggml_metal_graph_compute: command buffer 7 failed with status 5
error: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)

Using 3_K_S works for now, so I'll use that to test out this model! Thanks

<!-- gh-comment-id:2329154458 --> @xmaayy commented on GitHub (Sep 4, 2024): Agh thanks, it looks like the model is running out of memory. I'm on a 32GB MacOS device so I have to use a smaller context size which works fine with llama-cpp-python on 4_K_S at 2048 context. When I use a model file like this: ``` dockerfile 1 FROM command-r:35b-08-2024-q4_K_S 2 PARAMETER num_ctx 2048 ``` I'd assume that the model would load with a lower context size but looking at the logs its trying to load 8192 context size which exceeds my memory (also something I thought Ollama would catch): ``` llm_load_tensors: offloading 40 repeating layers to GPU llm_load_tensors: offloading non-repeating layers to GPU llm_load_tensors: offloaded 41/41 layers to GPU llm_load_tensors: CPU buffer size = 1640.62 MiB llm_load_tensors: Metal buffer size = 17965.92 MiB llama_new_context_with_model: n_ctx = 8192 llama_new_context_with_model: n_batch = 512 llama_new_context_with_model: n_ubatch = 512 llama_new_context_with_model: flash_attn = 0 llama_new_context_with_model: freq_base = 4000000.0 llama_new_context_with_model: freq_scale = 1 ... ggml_metal_graph_compute: command buffer 7 failed with status 5 error: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory) ``` Using `3_K_S` works for now, so I'll use that to test out this model! Thanks
GiteaMirror commented 2026-04-12 15:06:12 -05:00
Author
Owner
Copy Link

@xmaayy commented on GitHub (Sep 4, 2024):

Something strange is that it seems to load the model at 8196 context, which fails. But then executes with the correct context of 2048 but the model data has already been ejected from memory so it spews the nonsense

<!-- gh-comment-id:2329166370 --> @xmaayy commented on GitHub (Sep 4, 2024): Something strange is that it seems to load the model at 8196 context, which fails. But then executes with the correct context of 2048 but the model data has already been ejected from memory so it spews the nonsense
GiteaMirror commented 2026-04-12 15:06:12 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Sep 4, 2024):

ollama is using 8192 for total context size because you either have OLLAMA_NUM_PARALLEL=4 or it's unset and ollama is using a default of 4. You can override this behaviour by setting OLLAMA_NUM_PARALLEL=1, although that will only get you back 6K, which may not be materially useful.

<!-- gh-comment-id:2329167029 --> @rick-github commented on GitHub (Sep 4, 2024): ollama is using 8192 for total context size because you either have `OLLAMA_NUM_PARALLEL=4` or it's unset and ollama is using a default of 4. You can override this behaviour by setting `OLLAMA_NUM_PARALLEL=1`, although that will only get you back 6K, which may not be materially useful.
GiteaMirror commented 2026-04-12 15:06:13 -05:00
Author
Owner
Copy Link

@xmaayy commented on GitHub (Sep 4, 2024):

@rick-github That was definitely materially useful! Thank you

$ set -x OLLAMA_NUM_PARALLEL 1
$ ollama run command-r-4KS-limited-ctx
>>> Hello
Hello there! How may I assist you today? Feel free to ask any questions or provide instructions for the task at hand.
<!-- gh-comment-id:2329236474 --> @xmaayy commented on GitHub (Sep 4, 2024): @rick-github That was definitely materially useful! Thank you ``` $ set -x OLLAMA_NUM_PARALLEL 1 $ ollama run command-r-4KS-limited-ctx >>> Hello Hello there! How may I assist you today? Feel free to ask any questions or provide instructions for the task at hand.
GiteaMirror commented 2026-04-12 15:06:14 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Sep 4, 2024):

Unless you skipped some lines in your paste, I don't think this is what helped. OLLAMA_NUM_PARALLEL needs to be set in the server environment, from your paste it looks like you've set it in the client environment.

<!-- gh-comment-id:2329678554 --> @rick-github commented on GitHub (Sep 4, 2024): Unless you skipped some lines in your paste, I don't think this is what helped. OLLAMA_NUM_PARALLEL needs to be set in the server environment, from your paste it looks like you've set it in the client environment.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug macos memory
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#4174
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?