mirror of
https://github.com/open-webui/open-webui.git
synced 2026-07-16 23:21:44 -05:00
[GH-ISSUE #6298] Models not working with RAG UNLESS imported via Openwebui's experimental feature (but broken response) #14308
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: github-starred/open-webui#14308
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @antonio-lamberti on GitHub (Oct 20, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/6298
Discussed in https://github.com/open-webui/open-webui/discussions/6297
Imported models, either with the new ollama pull hf.co/bartowski/Hermes-3-Llama-3.1-8B-GGUF:Q6_K_Lor via pulling a model on Ollama webside, result in models NOT being able to properly retrieve RAG context in openwebui.
If, instead, I DOWNLOAD locally the gguf file and THEN I import it in openwebui via the experimental feature, it works with RAG, but the response has bad formatting (lots of <|end_of_text|><|begin_of_text|>://->}<|end_of_text|><|begin_of_text|>://->}
Why on earth is that?!
The 1st one I imported via experimental feature, downloading first the gguf from hf...the 2nd one is downloaded either via pull from Ollama website or hf directly
Model imported via experimental feature has good response, but the formatting is totally bad...
Models imported via pull (either from ollama or with the new hf method) have no idea of the context, although it's correctly passed to the llm.
Ps. Modelfile content when importing via experimental feature is like this:
TEMPLATE """{{ .System }}
USER: {{ .Prompt }}
ASSISTANT: """
PARAMETER num_ctx 4096
PARAMETER stop ""
PARAMETER stop "USER:"
PARAMETER stop "ASSISTANT:"
I have then tried to use, in my example, params + template as the modelfile from the website https://ollama.com/taozhiyuai/hermes-2-theta-llama-3:8b-q8_0 but the results is the same as importing directly from HF / Ollama: the llm seems unaware of the context, that is instead being properly retrieved.