github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-07 08:30:05 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #2709] Ollama hangs on Resampling because token 17158: '<token>' does not meet grammar rules #1621

New Issue
Closed
opened 2026-04-12 11:33:15 -05:00 by GiteaMirror · 6 comments
GiteaMirror commented 2026-04-12 11:33:15 -05:00
Owner
Copy Link

Originally created by @boxabirds on GitHub (Feb 23, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/2709

Situation: I am having ollama get stuck in an infinite loop on ubuntu 22.04 with certain requests. It appears to die, with broken pipes not breaking out and I have to restart the service. When I say "die" I mean no further requests are handled. As the log at INFO level only logs when the request has been sent back, nothing is logged in this scenario.

My approach to solving it:

set OLLAMA_DEBUG=1 and look at the journalctl logs. I've set it in two places:
environment variable:

export OLLAMA_DEBUG=1
set | grep OLLAMA
OLLAMA_DEBUG=1

And in the [Service] of ollama.service

[Unit]
Description=Ollama Service
After=network-online.target

[Service]
ExecStart=/usr/local/bin/ollama serve
User=ollama
Group=ollama
Restart=always
RestartSec=3

Environment="PATH=/home/…<various paths>…:/snap/bin OLLAMA_DEBUG=1"


[Install]
WantedBy=default.target

Then I restarted the server successfully.
sudo systemctl daemon-reload
sudo systemctl restart ollama.service

Expected output: all the slog.Debug and greater requests logged
Observed: only INFO seem to be logged. But the GPU is busy so it's doing SOMETHING.

Anyone know how I can confirm that the debug flag is set correctly?

Or more to the point, anyone know how I can better diagnose the server's infinite loop? It only happens with a particular model, so maybe the GGUF config isn't quite right? It's calebfahlgren/natural-functions:latest

Originally created by @boxabirds on GitHub (Feb 23, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/2709 Situation: I am having ollama get stuck in an infinite loop on ubuntu 22.04 with certain requests. It appears to die, with broken pipes not breaking out and I have to restart the service. When I say "die" I mean no further requests are handled. As the log at INFO level only logs when the request has been sent back, nothing is logged in this scenario. My approach to solving it: set `OLLAMA_DEBUG=1` and look at the journalctl logs. I've set it in two places: environment variable: ``` export OLLAMA_DEBUG=1 set | grep OLLAMA OLLAMA_DEBUG=1 ``` And in the [Service] of ollama.service ``` [Unit] Description=Ollama Service After=network-online.target [Service] ExecStart=/usr/local/bin/ollama serve User=ollama Group=ollama Restart=always RestartSec=3 Environment="PATH=/home/…<various paths>…:/snap/bin OLLAMA_DEBUG=1" [Install] WantedBy=default.target ``` Then I restarted the server successfully. `sudo systemctl daemon-reload` `sudo systemctl restart ollama.service` Expected output: all the slog.Debug and greater requests logged Observed: only INFO seem to be logged. But the GPU is busy so it's doing SOMETHING. Anyone know how I can confirm that the debug flag is set correctly? Or more to the point, anyone know how I can better diagnose the server's infinite loop? It only happens with a particular model, so maybe the GGUF config isn't quite right? It's calebfahlgren/natural-functions:latest
GiteaMirror added the bug label 2026-04-12 11:33:15 -05:00
GiteaMirror commented 2026-04-12 11:33:16 -05:00
Author
Owner
Copy Link

@sid-cypher commented on GitHub (Feb 23, 2024):

Does the loop respect systemd's RestartSec=3 setting?
You could diagnose by changing the ollama.service file and setting ExecStart=ollama serve to run a wrapper script instead, for example to hold the process running and/or dump its envvars.

To see a running processes' environment and check for debug flags, just read it from procfs:

cat /proc/$PID/environ | tr '\0' '\n' | less

Edit: rather than spending time on the inconveniences and overheads of systemd, you could kill the service and just run sudo -u ollama /usr/local/bin/ollama serve directly, then monitor the log output as you run your model in a separate terminal window.

<!-- gh-comment-id:1961374077 --> @sid-cypher commented on GitHub (Feb 23, 2024): Does the loop respect `systemd`'s RestartSec=3 setting? You could diagnose by changing the `ollama.service` file and setting `ExecStart=ollama serve` to run a wrapper script instead, for example to hold the process running and/or dump its envvars. To see a running processes' environment and check for debug flags, just read it from procfs: ``` cat /proc/$PID/environ | tr '\0' '\n' | less ``` Edit: rather than spending time on the inconveniences and overheads of systemd, you could kill the service and just run `sudo -u ollama /usr/local/bin/ollama serve` directly, then monitor the log output as you run your model in a separate terminal window.
GiteaMirror commented 2026-04-12 11:33:17 -05:00
Author
Owner
Copy Link

@boxabirds commented on GitHub (Feb 23, 2024):

Ah that's great running it directly both a) set the environment variable properly and b) I can now see level=DEBUG in the logs. I guess I'm not clear how to alter the ollama.service to set the environment variable properly.

<!-- gh-comment-id:1961568926 --> @boxabirds commented on GitHub (Feb 23, 2024): Ah that's great running it directly both a) set the environment variable properly and b) I can now see `level=DEBUG` in the logs. I guess I'm not clear how to alter the ollama.service to set the environment variable properly.
GiteaMirror commented 2026-04-12 11:33:17 -05:00
Author
Owner
Copy Link

@boxabirds commented on GitHub (Feb 23, 2024):

It's very interesting it seems like instructor is generating prompts that even mistral:7b can't cope with, but more interestingly indeed in a way that causes ollama to barf. I get ollama to get stuck here, not returning at all.

time=2024-02-23T15:58:59.224Z level=DEBUG source=routes.go:1225 msg="chat handler" prompt="[INST] \n                As a genius expert, your task is to understand the content and provide\n                the parsed objects in json that match the following json_schema:\n\n                {'messages': {'items': {'$ref': '#/$defs/MessagePair'}, 'title': 'Messages', 'type': 'array'}}\n                \nHere are some more definitions to adhere too:\n{'MessagePair': {'properties': {'respectful': {'title': 'Respectful', 'type': 'string'}, 'nondisrespectful': {'title': 'Nondisrespectful', 'type': 'string'}}, 'required': ['respectful', 'nondisrespectful'], 'title': 'MessagePair', 'type': 'object'}}\n\n\n                As a genius expert, your task is to understand the content and provide\n                the parsed objects in json that match the following json_schema:\n\n                {'messages': {'items': {'$ref': '#/$defs/MessagePair'}, 'title': 'Messages', 'type': 'array'}}\n                \nHere are some more definitions to adhere too:\n{'MessagePair': {'properties': {'respectful': {'title': 'Respectful', 'type': 'string'}, 'nondisrespectful': {'title': 'Nondisrespectful', 'type': 'string'}}, 'required': ['respectful', 'nondisrespectful'], 'title': 'MessagePair', 'type': 'object'}} Generate 5 pairs of short instant messages, where each pair contains a non-disrespectful (respectful or neutral) message and a corresponding disrespectful message exemplifying 'Dishonesty'. [/INST]" images=0
[1708703939] slot 0 is processing [task id: 0]
[1708703939] slot 0 : in cache: 0 tokens | to process: 370 tokens
[1708703939] slot 0 : kv cache rm - [0, end)
[1708703939] Resampling because token 17158: ' Based' does not meet grammar rules
[1708703941] Resampling because token 12069: 'Please' does not meet grammar rules
[1708703941] Resampling because token 12069: 'Please' does not meet grammar rules
[1708703941] Resampling because token 12069: 'Please' does not meet grammar rules
[1708703941] Resampling because token 12069: 'Please' does not meet grammar rules
[1708703941] Resampling because token 12069: 'Please' does not meet grammar rules
[1708703951] slot 0: context shift - n_keep = 0, n_left = 2046, n_discard = 1023
[1708703959] slot 0: context shift - n_keep = 0, n_left = 2046, n_discard = 1023
[1708703967] slot 0: context shift - n_keep = 0, n_left = 2046, n_discard = 1023
[1708703974] slot 0: context shift - n_keep = 0, n_left = 2046, n_discard = 1023
[1708703982] slot 0: context shift - n_keep = 0, n_left = 2046, n_discard = 1023

It just does this until I kill it, blocking the thread and the socket.

<!-- gh-comment-id:1961680311 --> @boxabirds commented on GitHub (Feb 23, 2024): It's very interesting it seems like `instructor` is generating prompts that even `mistral:7b` can't cope with, but more interestingly indeed in a way that causes ollama to barf. I get `ollama` to get stuck here, not returning at all. ``` time=2024-02-23T15:58:59.224Z level=DEBUG source=routes.go:1225 msg="chat handler" prompt="[INST] \n As a genius expert, your task is to understand the content and provide\n the parsed objects in json that match the following json_schema:\n\n {'messages': {'items': {'$ref': '#/$defs/MessagePair'}, 'title': 'Messages', 'type': 'array'}}\n \nHere are some more definitions to adhere too:\n{'MessagePair': {'properties': {'respectful': {'title': 'Respectful', 'type': 'string'}, 'nondisrespectful': {'title': 'Nondisrespectful', 'type': 'string'}}, 'required': ['respectful', 'nondisrespectful'], 'title': 'MessagePair', 'type': 'object'}}\n\n\n As a genius expert, your task is to understand the content and provide\n the parsed objects in json that match the following json_schema:\n\n {'messages': {'items': {'$ref': '#/$defs/MessagePair'}, 'title': 'Messages', 'type': 'array'}}\n \nHere are some more definitions to adhere too:\n{'MessagePair': {'properties': {'respectful': {'title': 'Respectful', 'type': 'string'}, 'nondisrespectful': {'title': 'Nondisrespectful', 'type': 'string'}}, 'required': ['respectful', 'nondisrespectful'], 'title': 'MessagePair', 'type': 'object'}} Generate 5 pairs of short instant messages, where each pair contains a non-disrespectful (respectful or neutral) message and a corresponding disrespectful message exemplifying 'Dishonesty'. [/INST]" images=0 [1708703939] slot 0 is processing [task id: 0] [1708703939] slot 0 : in cache: 0 tokens | to process: 370 tokens [1708703939] slot 0 : kv cache rm - [0, end) [1708703939] Resampling because token 17158: ' Based' does not meet grammar rules [1708703941] Resampling because token 12069: 'Please' does not meet grammar rules [1708703941] Resampling because token 12069: 'Please' does not meet grammar rules [1708703941] Resampling because token 12069: 'Please' does not meet grammar rules [1708703941] Resampling because token 12069: 'Please' does not meet grammar rules [1708703941] Resampling because token 12069: 'Please' does not meet grammar rules [1708703951] slot 0: context shift - n_keep = 0, n_left = 2046, n_discard = 1023 [1708703959] slot 0: context shift - n_keep = 0, n_left = 2046, n_discard = 1023 [1708703967] slot 0: context shift - n_keep = 0, n_left = 2046, n_discard = 1023 [1708703974] slot 0: context shift - n_keep = 0, n_left = 2046, n_discard = 1023 [1708703982] slot 0: context shift - n_keep = 0, n_left = 2046, n_discard = 1023 ``` It just does this until I kill it, blocking the thread and the socket.
GiteaMirror commented 2026-04-12 11:33:19 -05:00
Author
Owner
Copy Link

@jmorganca commented on GitHub (Feb 23, 2024):

Hi @boxabirds, are you using JSON mode by chance? Sorry you hit this

<!-- gh-comment-id:1962135047 --> @jmorganca commented on GitHub (Feb 23, 2024): Hi @boxabirds, are you using JSON mode by chance? Sorry you hit this
GiteaMirror commented 2026-04-12 11:33:20 -05:00
Author
Owner
Copy Link

@mxyng commented on GitHub (Feb 24, 2024):

Environment="PATH=/home/……:/snap/bin OLLAMA_DEBUG=1"

OLLAMA_DEBUG needs to be on its own Environment line

Environment="PATH=/home/…<various paths>…:/snap/bin"
Environment="OLLAMA_DEBUG=1"
<!-- gh-comment-id:1962201708 --> @mxyng commented on GitHub (Feb 24, 2024): > Environment="PATH=/home/…<various paths>…:/snap/bin OLLAMA_DEBUG=1" `OLLAMA_DEBUG` needs to be on its own `Environment` line ``` Environment="PATH=/home/…<various paths>…:/snap/bin" Environment="OLLAMA_DEBUG=1" ```
GiteaMirror commented 2026-04-12 11:33:20 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Jul 24, 2024):

@boxabirds if you're still having problems, please share a bit more information about how your calling Ollama so we can try to reproduce and understand the problem.

<!-- gh-comment-id:2249014397 --> @dhiltgen commented on GitHub (Jul 24, 2024): @boxabirds if you're still having problems, please share a bit more information about how your calling Ollama so we can try to reproduce and understand the problem.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#1621
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?