mirror of
https://github.com/open-webui/open-webui.git
synced 2026-07-15 21:19:39 -05:00
[GH-ISSUE #9912] Domain Filter List in WebSearch #135422
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: github-starred/open-webui#135422
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @liucoj on GitHub (Feb 13, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/9912
Hi, On the latest version 0.5.11, I noticed that the new option 'Domain Filter List' in WebSearch doesn't work as expected. When I tried to insert a couple of domains separated by commas (e.g., example.com, example2.com), the filtered domains were not considered when retrieving results. I'm using Tavily as an API.
I couldn't find any instructions about this new option on official OWUI channels, such as Twitter, Discord, documentation, or here. Although I noticed that when saving preferences after inserting domains into the filter list, only the Tavily API update, concurrent requests, and connections are updated in the logs - not the domain filters themselves.
Any ideas?
@oatmealm commented on GitHub (Feb 13, 2025):
Funny for me with exa the filter returns results only from the sites I want to filter out. Either .wikipedia.org or !.wikipedia.org cause the search to only return results from this site.
@liucoj commented on GitHub (Feb 13, 2025):
I think Tavily module is not ready for this option, I didn't find anything related to "filter_list: Optional[list[str]]" within the code (tavily.py) as with the other search modules.
@tjbck commented on GitHub (Feb 13, 2025):
It's only supported by a select few providers, PR welcome here!
@liucoj commented on GitHub (Feb 14, 2025):
@tjbck I made a simple implementation in tavily.py (in bold ) and it works! I took as an example the searching module (searxng.py) working module. Considering that, in UI, as an admin, the search module permits only 3 options (max results , max concurrent requests , domain filter list ), and there aren't any other options to consider, I'll make a PR with these implementations. Do I miss something?
#i rewrite this function
def search_tavily(
api_key: str,
query: str,
count: int,
filter_list: Optional[list[str]] = None,
) -> list[SearchResult]:
#Add include_domain as requested by Tavily API
url = "https://api.tavily.com/search"
data = {"query": query, "api_key": api_key}
include_domain = filter_list
response = requests.post(url, include_domain, json=data)
response.raise_for_status()
@nachogmd commented on GitHub (Feb 14, 2025):
I will create a pull request which handles both "included" and "excluded" if this makes sense.
I struggled too with this feature until I noticed what it does is to exclude those domains
It should involve passing all search engines a white list and a black list. I would need advise on how to name configuration parameters as if this approach is taken ideally we should have RAG_WEB_SEARCH_DOMAIN_FILTER_LIST as
Best regards,
n
@tjbck commented on GitHub (Feb 15, 2025):
@nachogmd env var naming should follow the existing convention
e.g.
RAG_WEB_SEARCH_ALLOWED_DOMAIN_LIST,RAG_WEB_SEARCH_BLOCKED_DOMAIN_LIST@glottisfaun0000 commented on GitHub (Feb 15, 2025):
So if I wanted to filter so my web searches only use websites with .edu or .gov tld's, what would I put in this field?
@abdessalaam commented on GitHub (Feb 16, 2025):
Perhaps even 3 categories:
PREFERREDoption would always try to include that domain, while still searching everywhere else (e.g. ifALLOWEDnot set)@tjbck commented on GitHub (Feb 16, 2025):
Or perhaps it maybe more prudent/intutive to introduce a widely used syntax (e.g.
!prefix to indicate blocked domain)@nachogmd commented on GitHub (Feb 16, 2025):
Seacxng uses "!images Donald Duck" (for example) to specify the tab which you want to search (in this case images)
The only way I found to "filter" results in searxng is using https://docs.searxng.org/dev/plugins/hostnames.html
Other search engines might allow using google like sintax site:mydomain.com
@nachogmd commented on GitHub (Feb 16, 2025):
The issue I am facing is that is quite easy to filter out all search results leading to 0 search results. But it is the feature, isn't it?
I like RAG_WEB_SEARCH_PREFERRED_DOMAIN_LIST so that it reorders the results and moves to the top of the list those domains since OpenWebUI will likely only use a subset of the found results
@nachogmd commented on GitHub (Feb 16, 2025):
Actually stating in the front end which engines allow real domain filtering (not based on the results) would make the trick since I beleive users won't always expect OpenWebUI to filter out results and bring 0 if, say, in the first page every domain has been blocker
@lowstz commented on GitHub (Feb 22, 2025):
Examples of domain filter lists
https://en.wikipedia.org/wiki/Wikipedia:Reliable_sources/Perennial_sources
e80d9bbea7/knowledge_storm/storm_wiki/modules/retriever.py@ER-EPR commented on GitHub (Apr 17, 2025):
Same for me, it's still the same in the latest release. I listed example.com, and never get any result for two days before I realize the search domain filter maybe wrong.
@svenseeberg commented on GitHub (Aug 14, 2025):
Yes, Google and DuckDuckGo for example support queries like
search term site:github.com OR site:gitlab.com. Having this for search engines, which support it, would be very helpful.It should be relatively easy to achieve this result by, for example, modifying the following lines:
https://github.com/open-webui/open-webui/blob/main/backend/open_webui/retrieval/web/duckduckgo.py#L27-L36