Bug Summary:
Multimodal chat only responds to first query after a second image is input.
Steps to Reproduce:
Give any prompt, observe a normal response.
Add an image, ask a question about it, observe a normal response
Add a second image, ask a question about it, receive a response to the prompt from step 1. All future responses will be to prompt 1.
Expected Behavior:
I am able to ask questions about any number of single images within one chat. The conversation should not be stuck responding to only one of my prompts.
Actual Behavior:
After uploading a second image, the model only responds to the first-ever prompt I gave it.
Environment
Operating System: Ubuntu 20.04
Browser (if applicable): Firefox 121
Reproduction Details
Confirmation:
I have read and followed all the instructions provided in the README.md.
I have reviewed the troubleshooting.md document.
I have included the browser console logs.
I have included the Docker container logs.
Logs and Screenshots
Browser Console Logs:
n/a no useful info
Docker Container Logs:
n/a just useless request logs
Screenshots (if applicable):
BROKEN - interacting with 2 images
WORKING - interacting with one image
Installation Method
Docker
Additional Information
No custom config. I am able to interact with multiple subsequent images when using the ollama CLI, which is why I'm filing a bug on the UI
Originally created by @hammeiam on GitHub (Jan 24, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/561
# Bug Report
## Description
**Bug Summary:**
Multimodal chat only responds to first query after a second image is input.
**Steps to Reproduce:**
1. Give any prompt, observe a normal response.
2. Add an image, ask a question about it, observe a normal response
3. Add a second image, ask a question about it, receive a response to the prompt from step 1. All future responses will be to prompt 1.
**Expected Behavior:**
I am able to ask questions about any number of single images within one chat. The conversation should not be stuck responding to only one of my prompts.
**Actual Behavior:**
After uploading a second image, the model only responds to the first-ever prompt I gave it.
## Environment
- **Operating System:** Ubuntu 20.04
- **Browser (if applicable):** Firefox 121
## Reproduction Details
**Confirmation:**
- [x] I have read and followed all the instructions provided in the README.md.
- [x] I have reviewed the troubleshooting.md document.
- [x] I have included the browser console logs.
- [x] I have included the Docker container logs.
## Logs and Screenshots
**Browser Console Logs:**
n/a no useful info
**Docker Container Logs:**
n/a just useless request logs
**Screenshots (if applicable):**
BROKEN - interacting with 2 images
WORKING - interacting with one image
## Installation Method
Docker
## Additional Information
No custom config. I am able to interact with multiple subsequent images when using the ollama CLI, which is why I'm filing a bug on the UI
Hi,Thanks for creating this issue! I'll close this issue as the behaviour you see is caused by limitations of llava model (local models in general). Feel free to make a suggestion if I missed anything, Thanks!
<!-- gh-comment-id:1909043235 -->
@tjbck commented on GitHub (Jan 24, 2024):
Hi,Thanks for creating this issue! I'll close this issue as the behaviour you see is caused by limitations of llava model (local models in general). Feel free to make a suggestion if I missed anything, Thanks!
Hey @tjbck thanks for taking a look but I don't think that's quite right - the model itself can handle multiple images, I believe the issue is elsewhere.
<!-- gh-comment-id:1909053338 -->
@hammeiam commented on GitHub (Jan 24, 2024):
Hey @tjbck thanks for taking a look but I don't think that's quite right - the model itself can handle multiple images, I believe the issue is elsewhere.
One thing you can do to verify it's a model issue is testing the model using Ollama CLI, let us know if CLI doesn't seem to experience the same issue!
<!-- gh-comment-id:1909056150 -->
@tjbck commented on GitHub (Jan 24, 2024):
One thing you can do to verify it's a model issue is testing the model using Ollama CLI, let us know if CLI doesn't seem to experience the same issue!
As I wrote in the "notes" section at the end of my ticket:
I am able to interact with multiple subsequent images when using the ollama CLI, which is why I'm filing a bug on the UI
Here's another example using two images in one prompt which works fine. It's just something about images in subsequent prompts that breaks.
<!-- gh-comment-id:1909058201 -->
@hammeiam commented on GitHub (Jan 24, 2024):
As I wrote in the "notes" section at the end of my ticket:
> I am able to interact with multiple subsequent images when using the ollama CLI, which is why I'm filing a bug on the UI
Here's another example using two images in one prompt which works fine. It's just something about images in subsequent prompts that breaks.
Hmm, If thats the case, one quick fix could be that we just remove all the previous images when sending a request to Ollama, but I'm not entirely sure if that's the right approach as feels a bit hacky to me, although I believe that's how Ollama is handling multi-images based on what you suggested. Any thoughts?
<!-- gh-comment-id:1909070655 -->
@tjbck commented on GitHub (Jan 24, 2024):
Hmm, If thats the case, one quick fix could be that we just remove all the previous images when sending a request to Ollama, but I'm not entirely sure if that's the right approach as feels a bit hacky to me, although I believe that's how Ollama is handling multi-images based on what you suggested. Any thoughts?
Yeah it seems like that might resolve this issue. Since multimodal is still nascent, a more permanent solution may come up in the future. I also wonder if some array flattening for images isn't happening when it should be?
<!-- gh-comment-id:1909181277 -->
@hammeiam commented on GitHub (Jan 25, 2024):
Yeah it seems like that might resolve this issue. Since multimodal is still nascent, a more permanent solution may come up in the future. I also wonder if some array flattening for images isn't happening when it should be?
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @hammeiam on GitHub (Jan 24, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/561
Bug Report
Description
Bug Summary:
Multimodal chat only responds to first query after a second image is input.
Steps to Reproduce:
Expected Behavior:
I am able to ask questions about any number of single images within one chat. The conversation should not be stuck responding to only one of my prompts.
Actual Behavior:
After uploading a second image, the model only responds to the first-ever prompt I gave it.
Environment
Reproduction Details
Confirmation:
Logs and Screenshots
Browser Console Logs:
n/a no useful info
Docker Container Logs:
n/a just useless request logs
Screenshots (if applicable):
BROKEN - interacting with 2 images
WORKING - interacting with one image
Installation Method
Docker
Additional Information
No custom config. I am able to interact with multiple subsequent images when using the ollama CLI, which is why I'm filing a bug on the UI
@tjbck commented on GitHub (Jan 24, 2024):
Hi,Thanks for creating this issue! I'll close this issue as the behaviour you see is caused by limitations of llava model (local models in general). Feel free to make a suggestion if I missed anything, Thanks!
@hammeiam commented on GitHub (Jan 24, 2024):
Hey @tjbck thanks for taking a look but I don't think that's quite right - the model itself can handle multiple images, I believe the issue is elsewhere.
@tjbck commented on GitHub (Jan 24, 2024):
One thing you can do to verify it's a model issue is testing the model using Ollama CLI, let us know if CLI doesn't seem to experience the same issue!
@hammeiam commented on GitHub (Jan 24, 2024):
As I wrote in the "notes" section at the end of my ticket:
Here's another example using two images in one prompt which works fine. It's just something about images in subsequent prompts that breaks.
@tjbck commented on GitHub (Jan 24, 2024):
Hmm, If thats the case, one quick fix could be that we just remove all the previous images when sending a request to Ollama, but I'm not entirely sure if that's the right approach as feels a bit hacky to me, although I believe that's how Ollama is handling multi-images based on what you suggested. Any thoughts?
@hammeiam commented on GitHub (Jan 25, 2024):
Yeah it seems like that might resolve this issue. Since multimodal is still nascent, a more permanent solution may come up in the future. I also wonder if some array flattening for images isn't happening when it should be?
@hammeiam commented on GitHub (Feb 3, 2024):
@tjbck great minds think alike https://github.com/ollama/ollama/commit/38296ab352ea7dd029f24cea5353b581f53f678b
@tjbck commented on GitHub (Feb 4, 2024):
#642 should work as intended, let me know if you encounter any issues!