I'm researching the usage of small LLM for common tasks and after trying some models (particularly phi4 and deepseek r1:7b) I suspect that not all the context retrieved in the web searches is passed to the model. In the example below, the UI reports having searched 50 sites but in the response only a few of them are taken into account. In fact, model's response does not include many of the information requested, complaining about not having been able to find it in the context ("not specified in context"), while, if you check the provided links manually, the information is indeed there.
Why is this happening? Is it a problem of the web search feature? Does it only add a briefing of the page to the context (and not the entire page)? Or is it a problem with the context of the models being too sort? Any insight will be appreciated.
Note: to add more context, I am using DuckDuckGo search with web search feature.
Originally created by @joanlofe on GitHub (Jan 27, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/8983
I'm researching the usage of small LLM for common tasks and after trying some models (particularly phi4 and deepseek r1:7b) I suspect that not all the context retrieved in the web searches is passed to the model. In the example below, the UI reports having searched 50 sites but in the response only a few of them are taken into account. In fact, model's response does not include many of the information requested, complaining about not having been able to find it in the context ("not specified in context"), while, if you check the provided links manually, the information is indeed there.
Why is this happening? Is it a problem of the web search feature? Does it only add a briefing of the page to the context (and not the entire page)? Or is it a problem with the context of the models being too sort? Any insight will be appreciated.
**Note**: to add more context, I am using DuckDuckGo search with web search feature.
I am new like a few days new but pretty sure the 3 is the default setting where you enable web search.
<!-- gh-comment-id:2615987827 -->
@BeZazz commented on GitHub (Jan 27, 2025):
I am new like a few days new but pretty sure the 3 is the default setting where you enable web search.
LLMs have a limited context window size. When this is exceeded, the model starts to "forget" or drop the oldest information in its context to make room for new data. This process is often referred to as "recency bias". Any text that falls outside of the LLM's context window will be discarded and not considered by the model when generating responses. If the information provided is too much for the model to handle, it may struggle to maintain a consistent line of thought or generate relevant responses. This is because it might not have access to all the necessary context while making predictions. And while you can expand the default context window size of 2048 that Ollama sets on all models downloaded by it, this comes with trade-offs like increased memory usage and processing time. Even with expanded windows, there's still a limit to how much information can be effectively used in a single response generation process.
<!-- gh-comment-id:2616138193 -->
@silentoplayz commented on GitHub (Jan 27, 2025):
LLMs have a limited context window size. When this is exceeded, the model starts to "forget" or drop the oldest information in its context to make room for new data. This process is often referred to as "recency bias". Any text that falls outside of the LLM's context window will be discarded and not considered by the model when generating responses. If the information provided is too much for the model to handle, it may struggle to maintain a consistent line of thought or generate relevant responses. This is because it might not have access to all the necessary context while making predictions. And while you can expand the default context window size of `2048` that Ollama sets on all models downloaded by it, this comes with trade-offs like increased memory usage and processing time. Even with expanded windows, there's still a limit to how much information can be effectively used in a single response generation process.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @joanlofe on GitHub (Jan 27, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/8983
I'm researching the usage of small LLM for common tasks and after trying some models (particularly phi4 and deepseek r1:7b) I suspect that not all the context retrieved in the web searches is passed to the model. In the example below, the UI reports having searched 50 sites but in the response only a few of them are taken into account. In fact, model's response does not include many of the information requested, complaining about not having been able to find it in the context ("not specified in context"), while, if you check the provided links manually, the information is indeed there.
Why is this happening? Is it a problem of the web search feature? Does it only add a briefing of the page to the context (and not the entire page)? Or is it a problem with the context of the models being too sort? Any insight will be appreciated.
Note: to add more context, I am using DuckDuckGo search with web search feature.
@BeZazz commented on GitHub (Jan 27, 2025):
I am new like a few days new but pretty sure the 3 is the default setting where you enable web search.
@silentoplayz commented on GitHub (Jan 27, 2025):
LLMs have a limited context window size. When this is exceeded, the model starts to "forget" or drop the oldest information in its context to make room for new data. This process is often referred to as "recency bias". Any text that falls outside of the LLM's context window will be discarded and not considered by the model when generating responses. If the information provided is too much for the model to handle, it may struggle to maintain a consistent line of thought or generate relevant responses. This is because it might not have access to all the necessary context while making predictions. And while you can expand the default context window size of
2048that Ollama sets on all models downloaded by it, this comes with trade-offs like increased memory usage and processing time. Even with expanded windows, there's still a limit to how much information can be effectively used in a single response generation process.