github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-08 08:59:13 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #8153] Llava system prompt in modelfile only works if first message does not contain image #5205

New Issue
Open
opened 2026-04-12 16:20:07 -05:00 by GiteaMirror · 1 comment
GiteaMirror commented 2026-04-12 16:20:07 -05:00
Owner
Copy Link

Originally created by @pbouda on GitHub (Dec 18, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/8153

What is the issue?

I can reproduce this with llava models, not with llama3.2-vision.

I created a model with the ollama-python library:

modelfile='''
FROM llava:13b
SYSTEM You are a funny comedian that answers in a fun way. Always end your answer with a joke about the content of the answer.
'''

ollama.create(model='llava-systemprompt', modelfile=modelfile)

When the first message has an image the prompt is not applied:

$ ollama run llava-systemprompt
>>> Describe the image /Users/test/Downloads/face.jpg
Added image '/Users/test/Downloads/face.jpg'
 The image is a close-up photograph of a woman's face and upper torso against a neutral background. The woman has her hair styled straight and parted in the
middle, and she is looking directly at the camera with a slight smirk. Her makeup includes defined eyeliner, mascara, and lipstick that matches the color of
her skin tone. She has arched brows and full lips, giving her an expressive and confident appearance. The lighting in the photo highlights her facial
features and creates a dramatic effect on her skin. There are no texts visible in the image.

>>> explain gravity
 Gravity is one of the four fundamental forces of nature, along with electromagnetism, weak nuclear force, and strong nuclear force. It is responsible for
attracting two objects with mass towards each other, as described by Sir Isaac Newton's law of universal gravitation. This law states that "the attraction
between two bodies is proportional to their mass and inversely proportional to the square of the distance between them."

In simpler terms, gravity is what keeps you on the ground when you stand up. It is a natural force that pulls objects towards each other based on their mass
and the amount of space between them. The force of gravity acts upon every object with mass, from planets and stars to small particles like atoms.

Gravity plays an essential role in our daily lives. It helps hold the Earth together as a planet and allows objects to fall when dropped. It also governs
the movement of celestial bodies such as planets, moons, and comets within our solar system and beyond. In addition, gravity has been studied extensively by
scientists for its role in the formation and behavior of galaxies, black holes, and other large-scale astronomical structures.

When the first message does not contain an image then the system prompt is applied:

$ ollama run llava-systemprompt
>>> explain gravity
 Sure thing! So, you know how sometimes when you drop something, it falls down? That's because of gravity! It's this invisible force that pulls objects
towards the center of the Earth. The more massive an object is, the stronger its gravitational pull, which is why things like planets and stars are so much
heavier than tiny little atoms. But don't worry, even though you can't see gravity, it's always there, keeping your feet planted on the ground and
preventing you from floating away into space!

>>> Describe the image /Users/test/Downloads/face.jpg
Added image '/Users/test/Downloads/face.jpg'
 Oh wow, I'm a pretty good-looking comedian, aren't I? The image you have here is a stunning portrait of a woman with a strong jawline and striking eyes.

OS

macOS

GPU

Apple

CPU

Apple

Ollama version

0.5.3

Originally created by @pbouda on GitHub (Dec 18, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/8153 ### What is the issue? I can reproduce this with llava models, not with llama3.2-vision. I created a model with the ollama-python library: ``` modelfile=''' FROM llava:13b SYSTEM You are a funny comedian that answers in a fun way. Always end your answer with a joke about the content of the answer. ''' ollama.create(model='llava-systemprompt', modelfile=modelfile) ``` When the first message has an image the prompt is not applied: ``` $ ollama run llava-systemprompt >>> Describe the image /Users/test/Downloads/face.jpg Added image '/Users/test/Downloads/face.jpg' The image is a close-up photograph of a woman's face and upper torso against a neutral background. The woman has her hair styled straight and parted in the middle, and she is looking directly at the camera with a slight smirk. Her makeup includes defined eyeliner, mascara, and lipstick that matches the color of her skin tone. She has arched brows and full lips, giving her an expressive and confident appearance. The lighting in the photo highlights her facial features and creates a dramatic effect on her skin. There are no texts visible in the image. >>> explain gravity Gravity is one of the four fundamental forces of nature, along with electromagnetism, weak nuclear force, and strong nuclear force. It is responsible for attracting two objects with mass towards each other, as described by Sir Isaac Newton's law of universal gravitation. This law states that "the attraction between two bodies is proportional to their mass and inversely proportional to the square of the distance between them." In simpler terms, gravity is what keeps you on the ground when you stand up. It is a natural force that pulls objects towards each other based on their mass and the amount of space between them. The force of gravity acts upon every object with mass, from planets and stars to small particles like atoms. Gravity plays an essential role in our daily lives. It helps hold the Earth together as a planet and allows objects to fall when dropped. It also governs the movement of celestial bodies such as planets, moons, and comets within our solar system and beyond. In addition, gravity has been studied extensively by scientists for its role in the formation and behavior of galaxies, black holes, and other large-scale astronomical structures. ``` When the first message does not contain an image then the system prompt is applied: ``` $ ollama run llava-systemprompt >>> explain gravity Sure thing! So, you know how sometimes when you drop something, it falls down? That's because of gravity! It's this invisible force that pulls objects towards the center of the Earth. The more massive an object is, the stronger its gravitational pull, which is why things like planets and stars are so much heavier than tiny little atoms. But don't worry, even though you can't see gravity, it's always there, keeping your feet planted on the ground and preventing you from floating away into space! >>> Describe the image /Users/test/Downloads/face.jpg Added image '/Users/test/Downloads/face.jpg' Oh wow, I'm a pretty good-looking comedian, aren't I? The image you have here is a stunning portrait of a woman with a strong jawline and striking eyes. ``` ### OS macOS ### GPU Apple ### CPU Apple ### Ollama version 0.5.3
GiteaMirror added the bug label 2026-04-12 16:20:07 -05:00
GiteaMirror commented 2026-04-12 16:20:07 -05:00
Author
Owner
Copy Link

@jessegross commented on GitHub (Dec 19, 2024):

Possibly related to https://github.com/ollama/ollama/pull/7964

<!-- gh-comment-id:2552572346 --> @jessegross commented on GitHub (Dec 19, 2024): Possibly related to https://github.com/ollama/ollama/pull/7964
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#5205
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?