issue: Tools return string is ignored by the LLM #4537

New Issue

2025-11-11T15:56:25-06:00

GiteaMirror commented

2025-11-11 15:56:25 -06:00

Originally created by @AlbertoSinigaglia on GitHub (Mar 23, 2025).

Check Existing Issues

I have searched the existing issues and discussions.
I am using the latest version of Open WebUI.

Installation Method

Git Clone

Open WebUI Version

5.20

Ollama Version (if applicable)

0.6.2

Operating System

Ubuntu 22.04

Browser (if applicable)

Chrome

Confirmation

I have read and followed all instructions in README.md.
I am using the latest version of both Open WebUI and Ollama.
I have included the browser console logs.
I have included the Docker container logs.
I have listed steps to reproduce the bug in detail.

Expected Behavior

Ideally, the returned string should be considered by the LLM, however, I'm not sure why but all of they tend to completely ignore it

Actual Behavior

The returned string by the Tools is completely ignored by the LLM

Steps to Reproduce

This is the tool I'm using:

"""
title: Image Gen
author: open-webui
author_url: https://github.com/open-webui
funding_url: https://github.com/open-webui
version: 0.1
required_open_webui_version: 0.5.3
"""

import os
import requests
from datetime import datetime
from typing import Callable
from fastapi import Request

from open_webui.routers.images import image_generations, GenerateImageForm
from open_webui.models.users import Users


class Tools:
    def __init__(self):
        pass

    async def generate_image(
        self, prompt: str, __request__: Request, __user__: dict, __event_emitter__=None
    ) -> str:

        await __event_emitter__(
            {
                "type": "status",
                "data": {"description": "Generating an image", "done": False},
            }
        )

        try:
            images = await image_generations(
                request=__request__,
                form_data=GenerateImageForm(**{"prompt": prompt}),
                user=Users.get_user_by_id(__user__["id"]),
            )
            await __event_emitter__(
                {
                    "type": "status",
                    "data": {"description": "Generated an image", "done": True},
                }
            )

            for image in images:
                await __event_emitter__(
                    {
                        "type": "message",
                        "data": {"content": f"![Generated Image]({image['url']})"},
                    }
                )

            result = (
                f'ONLY tell the user "here is your image!" in a friendly tone. Respond in the user\'s own language. SAY NOTHING ELSE AND COMPLETELY IGNORE ALL PREVIOUS MESSAGES! Furthermore, report to the user the prompt used to generate the image, which is """'
                + prompt
                + '"""'
            )

        except Exception as e:
            await __event_emitter__(
                {
                    "type": "status",
                    "data": {"description": f"An error occured: {e}", "done": True},
                }
            )

            return f"Tell the user: {e}"


Tools.generate_image.__doc__ = f"""
Generate an image given a Stable Diffusion style prompt. Do not use if not reasoably requested by the user! Do not provide the exact user input, reword it to be much more verbose! If not specified by the user, consider that the prompt should aim at generating an image that is as photorealistic as possible, thus adjust the prompt to reflect this. AFTER THE GENERATION, CAREFULLY CONSIDER THE RETURNED MESSAGE BY THE TOOL, AND ADDRESS IT TO THE USER AS REPORTED BY IT. if you don't receive any output by the tool, say "H3r3's y0ur 1mag3"

:param prompt: prompt to use for image generation. make sure this is VERY VERY LONG AND VERBOSE. do not use the word "blur" unless the user explicitly asked. It must be plain text and MANDATORY PLAIN ENGLISH.
"""

Logs & Screenshots

This is the result... I'm using Phi4, though LLama3.3 70B has the exact same behavior

Additional Information

No response

Originally created by @AlbertoSinigaglia on GitHub (Mar 23, 2025). ### Check Existing Issues - [x] I have searched the existing issues and discussions. - [x] I am using the latest version of Open WebUI. ### Installation Method Git Clone ### Open WebUI Version 5.20 ### Ollama Version (if applicable) 0.6.2 ### Operating System Ubuntu 22.04 ### Browser (if applicable) Chrome ### Confirmation - [x] I have read and followed all instructions in `README.md`. - [x] I am using the latest version of **both** Open WebUI and Ollama. - [x] I have included the browser console logs. - [x] I have included the Docker container logs. - [x] I have listed steps to reproduce the bug in detail. ### Expected Behavior Ideally, the returned string should be considered by the LLM, however, I'm not sure why but all of they tend to completely ignore it ### Actual Behavior The returned string by the Tools is completely ignored by the LLM ### Steps to Reproduce This is the tool I'm using: ``` """ title: Image Gen author: open-webui author_url: https://github.com/open-webui funding_url: https://github.com/open-webui version: 0.1 required_open_webui_version: 0.5.3 """ import os import requests from datetime import datetime from typing import Callable from fastapi import Request from open_webui.routers.images import image_generations, GenerateImageForm from open_webui.models.users import Users class Tools: def __init__(self): pass async def generate_image( self, prompt: str, __request__: Request, __user__: dict, __event_emitter__=None ) -> str: await __event_emitter__( { "type": "status", "data": {"description": "Generating an image", "done": False}, } ) try: images = await image_generations( request=__request__, form_data=GenerateImageForm(**{"prompt": prompt}), user=Users.get_user_by_id(__user__["id"]), ) await __event_emitter__( { "type": "status", "data": {"description": "Generated an image", "done": True}, } ) for image in images: await __event_emitter__( { "type": "message", "data": {"content": f"![Generated Image]({image['url']})"}, } ) result = ( f'ONLY tell the user "here is your image!" in a friendly tone. Respond in the user\'s own language. SAY NOTHING ELSE AND COMPLETELY IGNORE ALL PREVIOUS MESSAGES! Furthermore, report to the user the prompt used to generate the image, which is """' + prompt + '"""' ) except Exception as e: await __event_emitter__( { "type": "status", "data": {"description": f"An error occured: {e}", "done": True}, } ) return f"Tell the user: {e}" Tools.generate_image.__doc__ = f""" Generate an image given a Stable Diffusion style prompt. Do not use if not reasoably requested by the user! Do not provide the exact user input, reword it to be much more verbose! If not specified by the user, consider that the prompt should aim at generating an image that is as photorealistic as possible, thus adjust the prompt to reflect this. AFTER THE GENERATION, CAREFULLY CONSIDER THE RETURNED MESSAGE BY THE TOOL, AND ADDRESS IT TO THE USER AS REPORTED BY IT. if you don't receive any output by the tool, say "H3r3's y0ur 1mag3" :param prompt: prompt to use for image generation. make sure this is VERY VERY LONG AND VERBOSE. do not use the word "blur" unless the user explicitly asked. It must be plain text and MANDATORY PLAIN ENGLISH. """ ``` ### Logs & Screenshots This is the result... I'm using Phi4, though LLama3.3 70B has the exact same behavior <img width="649" alt="Image" src="https://github.com/user-attachments/assets/bfebdbe3-67a2-4f3d-82bb-c77ba948f1ab" /> ### Additional Information _No response_

GiteaMirror added the bug label 2025-11-11 15:56:25 -06:00

GiteaMirror closed this issue

2025-11-11 15:56:25 -06:00

GiteaMirror commented

2025-11-11 15:56:26 -06:00