github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-17 21:55:43 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #11960] Latest Ollama does not give satisfactory results for RAG using LangChain #7940

New Issue
Closed
opened 2026-04-12 20:06:16 -05:00 by GiteaMirror · 6 comments
GiteaMirror commented 2026-04-12 20:06:16 -05:00
Owner
Copy Link

Originally created by @robinpaul85 on GitHub (Aug 19, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/11960

What is the issue?

The attached python code provides satisfactory results with Ollama v0.7.0. When I upgraded to the latest v0.11.4 it does not provide satisfactory results, even though I am still using the same model. The final output should provide a JSON with result as snv_indel/cnv/sv_fusion/hierarchial/DGE (in v0.7.0). But with v0.11.4 it gives "data_type_classification" which does not make sense.

I am attaching the python code (pp_rag2.txt, file renamed to txt because I was not able to upload .py file extension), the RAG document (ai_docs2.txt), log file for ollama v0.11.4 (ollama_0.11.4_log.txt), log file for ollama v0.7.0 (ollama_0.7.0_log.txt) when running the attached python code. v0.7.0 was run on my mac natively whereas v0.11.4 was run via docker.

pp_rag2.txt

ollama_0.7.0_log.txt

ollama_0.11.4_log.txt

ai_docs2.txt

Relevant log output

OS

No response

GPU

No response

CPU

No response

Ollama version

No response

Originally created by @robinpaul85 on GitHub (Aug 19, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/11960 ### What is the issue? The attached python code provides satisfactory results with Ollama v0.7.0. When I upgraded to the latest v0.11.4 it does not provide satisfactory results, even though I am still using the same model. The final output should provide a JSON with result as snv_indel/cnv/sv_fusion/hierarchial/DGE (in v0.7.0). But with v0.11.4 it gives "data_type_classification" which does not make sense. I am attaching the python code (pp_rag2.txt, file renamed to txt because I was not able to upload .py file extension), the RAG document (ai_docs2.txt), log file for ollama v0.11.4 (ollama_0.11.4_log.txt), log file for ollama v0.7.0 (ollama_0.7.0_log.txt) when running the attached python code. v0.7.0 was run on my mac natively whereas v0.11.4 was run via docker. [pp_rag2.txt](https://github.com/user-attachments/files/21848277/pp_rag2.txt) [ollama_0.7.0_log.txt](https://github.com/user-attachments/files/21848242/ollama_0.7.0_log.txt) [ollama_0.11.4_log.txt](https://github.com/user-attachments/files/21848244/ollama_0.11.4_log.txt) [ai_docs2.txt](https://github.com/user-attachments/files/21848239/ai_docs2.txt) ### Relevant log output ```shell ``` ### OS _No response_ ### GPU _No response_ ### CPU _No response_ ### Ollama version _No response_
GiteaMirror added the bug label 2026-04-12 20:06:16 -05:00
GiteaMirror commented 2026-04-12 20:06:18 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 19, 2025):

There are a number of factors that have caused this.

Up until 0.9.0, ollama tool_call parsing was a bit lax, in that if a tool call looked vaguely JSON-like, ollama would try to extract a tool_call.. Starting in 0.9.1, ollama started adhering more closely to the tool_call definition in the template. For example, in llama3-groq-tool-use:latest the template defines a tool_call as

<tool_call>
{{ range .ToolCalls }}{"name": "{{ .Function.Name }}", "arguments": {{ .Function.Arguments }}}{{ end }}
</tool_call>

That is, the <tool_call></tool_call> tags are required to identify a tool_call.

Applying a format to a completion request can actually prevent a model from generating a correct tool call. In this case, the template definition requires the tool call to be surrounded by <tool_call></tool_call> tags, but this is not valid JSON, so the output is just the JSON part of the tool call. With the more constrained parsing of 0.9.2+, the model output is no longer interpreted as a tool call.

The role of the LLM in the langchain call graph is a bit vague. If the graph is invoked without format="json", the model will respond with a followup question rather than a tool call, because the model doesn't know its role. The only reason it seems to respond with a tool call is that the JSON grammar forces the model to output a JSON structure. It's not actually a tool call, but because of the lax parsing in 0.9.0, it was interpreted as one.

llama3-groq-tool-use:latest is pretty ancient as models go. More recent models are better and faster instruction followers, with about the same footprint.

--- pp_rag2.py.orig	2025-08-19 09:21:03.309643199 +0200
+++ pp_rag2.py	2025-08-20 00:51:10.161573483 +0200
@@ -12,8 +12,8 @@
 import os
 
 #os.environ["OLLAMA_HOST"] = "http://0.0.0.0:8000" #Using remote ollama server
-local_llm = "llama3-groq-tool-use:latest"
-llm_json_mode = ChatOllama(model=local_llm, temperature=0, format="json")
+local_llm = "qwen3:4b-instruct-2507-q8_0"
+llm_json_mode = ChatOllama(model=local_llm, temperature=0)
 
 loader = TextLoader("ai_docs2.txt")
 content = loader.load()[0].page_content 
@@ -53,7 +53,7 @@
 tools = [retriever_tool]
 
 chat_template = ChatPromptTemplate.from_messages([
-    ("system", "I am an assistant that figures out the type of data (differential gene expression (DGE), hierarchial clustering, SNV, CNV and SV) a particular dataset in ProteinPaint (PP) contains"),
+    ("system", "You are an assistant that figures out the type of data (differential gene expression (DGE), hierarchial clustering, SNV, CNV and SV) a particular dataset in ProteinPaint (PP) contains"),
     ("human", "{input}"),
     MessagesPlaceholder(variable_name="agent_scratchpad")
 ])
<!-- gh-comment-id:3202641456 --> @rick-github commented on GitHub (Aug 19, 2025): There are a number of factors that have caused this. Up until 0.9.0, ollama tool_call parsing was a bit lax, in that if a tool call looked vaguely JSON-like, ollama would try to extract a tool_call.. Starting in 0.9.1, ollama started adhering more closely to the tool_call definition in the template. For example, in llama3-groq-tool-use:latest the template defines a tool_call as ``` <tool_call> {{ range .ToolCalls }}{"name": "{{ .Function.Name }}", "arguments": {{ .Function.Arguments }}}{{ end }} </tool_call> ``` That is, the `<tool_call></tool_call>` tags are required to identify a tool_call. Applying a `format` to a completion request can actually [prevent](https://github.com/ollama/ollama/issues/11037) a model from generating a correct tool call. In this case, the template definition requires the tool call to be surrounded by `<tool_call></tool_call>` tags, but this is not valid JSON, so the output is just the JSON part of the tool call. With the more constrained parsing of 0.9.2+, the model output is no longer interpreted as a tool call. The role of the LLM in the langchain call graph is a bit vague. If the graph is invoked without `format="json"`, the model will respond with a followup question rather than a tool call, because the model doesn't know its role. The only reason it seems to respond with a tool call is that the JSON grammar forces the model to output a JSON structure. It's not actually a tool call, but because of the lax parsing in 0.9.0, it was interpreted as one. llama3-groq-tool-use:latest is pretty ancient as models go. More recent models are better and faster instruction followers, with about the same footprint. ```diff --- pp_rag2.py.orig 2025-08-19 09:21:03.309643199 +0200 +++ pp_rag2.py 2025-08-20 00:51:10.161573483 +0200 @@ -12,8 +12,8 @@ import os #os.environ["OLLAMA_HOST"] = "http://0.0.0.0:8000" #Using remote ollama server -local_llm = "llama3-groq-tool-use:latest" -llm_json_mode = ChatOllama(model=local_llm, temperature=0, format="json") +local_llm = "qwen3:4b-instruct-2507-q8_0" +llm_json_mode = ChatOllama(model=local_llm, temperature=0) loader = TextLoader("ai_docs2.txt") content = loader.load()[0].page_content @@ -53,7 +53,7 @@ tools = [retriever_tool] chat_template = ChatPromptTemplate.from_messages([ - ("system", "I am an assistant that figures out the type of data (differential gene expression (DGE), hierarchial clustering, SNV, CNV and SV) a particular dataset in ProteinPaint (PP) contains"), + ("system", "You are an assistant that figures out the type of data (differential gene expression (DGE), hierarchial clustering, SNV, CNV and SV) a particular dataset in ProteinPaint (PP) contains"), ("human", "{input}"), MessagesPlaceholder(variable_name="agent_scratchpad") ]) ```
GiteaMirror commented 2026-04-12 20:06:19 -05:00
Author
Owner
Copy Link

@robinpaul85 commented on GitHub (Aug 20, 2025):

What models in the (8-12billion) range do you recommend after I upgrade to the latest ollama version?

<!-- gh-comment-id:3208226818 --> @robinpaul85 commented on GitHub (Aug 20, 2025): What models in the (8-12billion) range do you recommend after I upgrade to the latest ollama version?
GiteaMirror commented 2026-04-12 20:06:20 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 20, 2025):

It depends on your requirements and how the models perform. Since you are limited to tool using models, the qwen3 family are quite good. Other tools using models can be found here. Note that it may be worthwhile to consider smaller models that aren't as heavily quantized, eg q8_0. But you will likely have to tune the prompt for the different models to get the best results.

<!-- gh-comment-id:3208265781 --> @rick-github commented on GitHub (Aug 20, 2025): It depends on your requirements and how the models perform. Since you are limited to tool using models, the qwen3 family are quite good. Other tools using models can be found [here](https://ollama.com/search?c=tools&o=newest). Note that it may be worthwhile to consider smaller models that aren't as heavily quantized, eg q8_0. But you will likely have to tune the prompt for the different models to get the best results.
GiteaMirror commented 2026-04-12 20:06:21 -05:00
Author
Owner
Copy Link

@robinpaul85 commented on GitHub (Aug 22, 2025):

I used the llm PetrosStav/gemma3-tools:12b I still get no output with the latest ollama {'input': [SystemMessage(content="Generate classification for the user query into summary, dge, hierarchial, snv_indel, cnv, variant_calling, sv_fusion and none categories. Return output in JSON with ALWAYS a single word answer, that is 'summary' for summary plot,'dge' for differential gene expression, 'hierarchial' for hierarchial clustering, 'snv_indel' for SNV/Indel, 'cnv' for CNV and 'sv_fusion' for SV/fusion, 'variant_calling' for variant calling, 'surivial' for survival data, 'none' for none of the previously described categories. The answer should always be in lower case", additional_kwargs={}, response_metadata={}), HumanMessage(content='Does aspirin leads to decrease in death rates among Africans?', additional_kwargs={}, response_metadata={})], 'output': '{"name": "data_type_classification", "parameters": {"query": "Does aspirin leads to decrease in death rates among Africans?"}}'}

I tried removing format = 'json' but that led to a non-json string. The problem with a lot of the models is a lot of them do not support tool calls.

<!-- gh-comment-id:3213015862 --> @robinpaul85 commented on GitHub (Aug 22, 2025): I used the llm `PetrosStav/gemma3-tools:12b` I still get no output with the latest ollama `{'input': [SystemMessage(content="Generate classification for the user query into summary, dge, hierarchial, snv_indel, cnv, variant_calling, sv_fusion and none categories. Return output in JSON with ALWAYS a single word answer, that is 'summary' for summary plot,'dge' for differential gene expression, 'hierarchial' for hierarchial clustering, 'snv_indel' for SNV/Indel, 'cnv' for CNV and 'sv_fusion' for SV/fusion, 'variant_calling' for variant calling, 'surivial' for survival data, 'none' for none of the previously described categories. The answer should always be in lower case", additional_kwargs={}, response_metadata={}), HumanMessage(content='Does aspirin leads to decrease in death rates among Africans?', additional_kwargs={}, response_metadata={})], 'output': '{"name": "data_type_classification", "parameters": {"query": "Does aspirin leads to decrease in death rates among Africans?"}}'}` I tried removing `format = 'json'` but that led to a non-json string. The problem with a lot of the models is a lot of them do not support tool calls.
GiteaMirror commented 2026-04-12 20:06:22 -05:00
Author
Owner
Copy Link

@robinpaul85 commented on GitHub (Aug 22, 2025):

I have a somewhat unrelated question, for RAG I need to provide the embeddings to the model. In this api documentation I see how embeddings are generated but I don't see a REST API for how embeddings are passed by an agent for retrieval during RAG. Can you point to some documentation which shows an example how to carry our RAG using ollama REST API?

If the REST API supports it, then I can create my own API in my programming language and not have to depend on off the shelf software.

<!-- gh-comment-id:3213039455 --> @robinpaul85 commented on GitHub (Aug 22, 2025): I have a somewhat unrelated question, for RAG I need to provide the embeddings to the model. In [this](https://github.com/ollama/ollama/blob/main/docs/api.md) api documentation I see how embeddings are generated but I don't see a REST API for how embeddings are passed by an agent for retrieval during RAG. Can you point to some documentation which shows an example how to carry our RAG using ollama REST API? If the REST API supports it, then I can create my own API in my programming language and not have to depend on off the shelf software.
GiteaMirror commented 2026-04-12 20:06:23 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 22, 2025):

As I explained, adding format = 'json' constrains the output of the model and prevents it from generating a valid tool call.

Langchain can be configured to use ad-hoc tooling, ie non-native tools.

The embeddings aren't sent to the model. A quick overview of how RAG works:

  1. The corpus of data is converted into embeddings which are associated with the original text, and stored in a database.
  2. The user sends a query to the client.
  3. The query is converted to embeddings.
  4. The client searches for embeddings in the database that closely match the embeddings of the query.
  5. The text associated with the matched embeddings is retrieved.
  6. The retrieved text is sent as context along with the original query to the LLM.
  7. The LLM uses the context to formulate an answer to the query and returns it to the client.
  8. The client sends the response to the user.
<!-- gh-comment-id:3215778371 --> @rick-github commented on GitHub (Aug 22, 2025): As I explained, adding `format = 'json'` constrains the output of the model and prevents it from generating a valid tool call. Langchain can be configured to use [ad-hoc tooling](https://python.langchain.com/docs/how_to/tools_prompting/), ie non-native tools. The embeddings aren't sent to the model. A quick overview of how RAG works: 1. The corpus of data is converted into embeddings which are associated with the original text, and stored in a database. 2. The user sends a query to the client. 3. The query is converted to embeddings. 4. The client searches for embeddings in the database that closely match the embeddings of the query. 5. The text associated with the matched embeddings is retrieved. 6. The retrieved text is sent as context along with the original query to the LLM. 7. The LLM uses the context to formulate an answer to the query and returns it to the client. 8. The client sends the response to the user.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#7940
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?