Hi, On the latest version 0.5.11, I noticed that the new option 'Domain Filter List' in WebSearch doesn't work as expected. When I tried to insert a couple of domains separated by commas (e.g., example.com, example2.com), the filtered domains were not considered when retrieving results. I'm using Tavily as an API.
I couldn't find any instructions about this new option on official OWUI channels, such as Twitter, Discord, documentation, or here. Although I noticed that when saving preferences after inserting domains into the filter list, only the Tavily API update, concurrent requests, and connections are updated in the logs - not the domain filters themselves.
Any ideas?
Originally created by @liucoj on GitHub (Feb 13, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/9912
Hi, On the latest version 0.5.11, I noticed that the new option 'Domain Filter List' in WebSearch doesn't work as expected. When I tried to insert a couple of domains separated by commas (e.g., example.com, example2.com), the filtered domains were not considered when retrieving results. I'm using Tavily as an API.
I couldn't find any instructions about this new option on official OWUI channels, such as Twitter, Discord, documentation, or here. Although I noticed that when saving preferences after inserting domains into the filter list, only the Tavily API update, concurrent requests, and connections are updated in the logs - not the domain filters themselves.
Any ideas?
Funny for me with exa the filter returns results only from the sites I want to filter out. Either .wikipedia.org or !.wikipedia.org cause the search to only return results from this site.
<!-- gh-comment-id:2656876466 -->
@oatmealm commented on GitHub (Feb 13, 2025):
Funny for me with exa the filter returns results only from the sites I want to filter out. Either *.wikipedia.org or !*.wikipedia.org cause the search to only return results from this site.
I think Tavily module is not ready for this option, I didn't find anything related to "filter_list: Optional[list[str]]" within the code (tavily.py) as with the other search modules.
<!-- gh-comment-id:2656961831 -->
@liucoj commented on GitHub (Feb 13, 2025):
I think Tavily module is not ready for this option, I didn't find anything related to "filter_list: Optional[list[str]]" within the code (tavily.py) as with the other search modules.
@tjbck I made a simple implementation in tavily.py (in bold ) and it works! I took as an example the searching module (searxng.py) working module. Considering that, in UI, as an admin, the search module permits only 3 options (max results , max concurrent requests , domain filter list ), and there aren't any other options to consider, I'll make a PR with these implementations. Do I miss something?
#Add include_domain as requested by Tavily API
url = "https://api.tavily.com/search"
data = {"query": query, "api_key": api_key} include_domain = filter_list
response = requests.post(url, include_domain, json=data)
response.raise_for_status()
<!-- gh-comment-id:2659352066 -->
@liucoj commented on GitHub (Feb 14, 2025):
@tjbck I made a simple implementation in tavily.py (in **bold** ) and it works! I took as an example the searching module (searxng.py) working module. Considering that, in UI, as an admin, the search module permits only 3 options (max results , max concurrent requests , domain filter list ), and there aren't any other options to consider, I'll make a PR with these implementations. Do I miss something?
#i rewrite this function
def search_tavily(
api_key: str,
query: str,
count: int,
**filter_list: Optional[list[str]] = None**,
) -> list[SearchResult]:
#Add include_domain as requested by Tavily API
url = "https://api.tavily.com/search"
data = {"query": query, "api_key": api_key}
**include_domain = filter_list**
response = requests.post(url, include_domain, json=data)
response.raise_for_status()
I will create a pull request which handles both "included" and "excluded" if this makes sense.
I struggled too with this feature until I noticed what it does is to exclude those domains
It should involve passing all search engines a white list and a black list. I would need advise on how to name configuration parameters as if this approach is taken ideally we should have RAG_WEB_SEARCH_DOMAIN_FILTER_LIST as
RAG_WEB_SEARCH_WHITE_FILTER_LIST
RAG_WEB_SEARCH_BLACK_FILTER_LIST
Best regards,
n
<!-- gh-comment-id:2660348720 -->
@nachogmd commented on GitHub (Feb 14, 2025):
I will create a pull request which handles both "included" and "excluded" if this makes sense.
I struggled too with this feature until I noticed what it does is to exclude those domains
It should involve passing all search engines a white list and a black list. I would need advise on how to name configuration parameters as if this approach is taken ideally we should have RAG_WEB_SEARCH_DOMAIN_FILTER_LIST as
- RAG_WEB_SEARCH_WHITE_FILTER_LIST
- RAG_WEB_SEARCH_BLACK_FILTER_LIST
Best regards,
n
@nachogmd env var naming should follow the existing convention
e.g. RAG_WEB_SEARCH_ALLOWED_DOMAIN_LIST, RAG_WEB_SEARCH_BLOCKED_DOMAIN_LIST
<!-- gh-comment-id:2660634872 -->
@tjbck commented on GitHub (Feb 15, 2025):
@nachogmd env var naming should follow the existing convention
e.g. `RAG_WEB_SEARCH_ALLOWED_DOMAIN_LIST`, `RAG_WEB_SEARCH_BLOCKED_DOMAIN_LIST`
@glottisfaun0000 commented on GitHub (Feb 15, 2025):
So if I wanted to filter so my web searches only use websites with .edu or .gov tld's, what would I put in this field?
<!-- gh-comment-id:2661144003 -->
@glottisfaun0000 commented on GitHub (Feb 15, 2025):
So if I wanted to filter so my web searches only use websites with .edu or .gov tld's, what would I put in this field?
the PREFERRED option would always try to include that domain, while still searching everywhere else (e.g. if ALLOWED not set)
<!-- gh-comment-id:2661153980 -->
@abdessalaam commented on GitHub (Feb 16, 2025):
Perhaps even 3 categories:
```
RAG_WEB_SEARCH_ALLOWED_DOMAIN_LIST
RAG_WEB_SEARCH_BLOCKED_DOMAIN_LIST
RAG_WEB_SEARCH_PREFERRED_DOMAIN_LIST
```
- the ```PREFERRED``` option would always try to include that domain, while still searching everywhere else (e.g. if ```ALLOWED``` not set)
Or perhaps it maybe more prudent/intutive to introduce a widely used syntax (e.g. ! prefix to indicate blocked domain)
<!-- gh-comment-id:2661183947 -->
@tjbck commented on GitHub (Feb 16, 2025):
Or perhaps it maybe more prudent/intutive to introduce a widely used syntax (e.g. `!` prefix to indicate blocked domain)
Other search engines might allow using google like sintax site:mydomain.com
<!-- gh-comment-id:2661415785 -->
@nachogmd commented on GitHub (Feb 16, 2025):
> Or perhaps it maybe more prudent/intutive to introduce a widely used syntax (e.g. `!` prefix to indicate blocked domain)
Seacxng uses "!images Donald Duck" (for example) to specify the tab which you want to search (in this case images)
The only way I found to "filter" results in searxng is using https://docs.searxng.org/dev/plugins/hostnames.html
Other search engines might allow using google like sintax site:mydomain.com
* the `PREFERRED` option would always try to include that domain, while still searching everywhere else (e.g. if `ALLOWED` not set)
The issue I am facing is that is quite easy to filter out all search results leading to 0 search results. But it is the feature, isn't it?
I like RAG_WEB_SEARCH_PREFERRED_DOMAIN_LIST so that it reorders the results and moves to the top of the list those domains since OpenWebUI will likely only use a subset of the found results
<!-- gh-comment-id:2661417260 -->
@nachogmd commented on GitHub (Feb 16, 2025):
> Perhaps even 3 categories:
>
> ```
> RAG_WEB_SEARCH_ALLOWED_DOMAIN_LIST
> RAG_WEB_SEARCH_BLOCKED_DOMAIN_LIST
> RAG_WEB_SEARCH_PREFERRED_DOMAIN_LIST
> ```
>
> * the `PREFERRED` option would always try to include that domain, while still searching everywhere else (e.g. if `ALLOWED` not set)
The issue I am facing is that is quite easy to filter out all search results leading to 0 search results. But it is the feature, isn't it?
I like RAG_WEB_SEARCH_PREFERRED_DOMAIN_LIST so that it reorders the results and moves to the top of the list those domains since OpenWebUI will likely only use a subset of the found results
Other search engines might allow using google like sintax site:mydomain.com
Actually stating in the front end which engines allow real domain filtering (not based on the results) would make the trick since I beleive users won't always expect OpenWebUI to filter out results and bring 0 if, say, in the first page every domain has been blocker
<!-- gh-comment-id:2661417817 -->
@nachogmd commented on GitHub (Feb 16, 2025):
> > Or perhaps it maybe more prudent/intutive to introduce a widely used syntax (e.g. `!` prefix to indicate blocked domain)
>
> Seacxng uses "!images Donald Duck" (for example) to specify the tab which you want to search (in this case images)
>
> The only way I found to "filter" results in searxng is using https://docs.searxng.org/dev/plugins/hostnames.html
>
> Other search engines might allow using google like sintax site:mydomain.com
Actually stating in the front end which engines allow real domain filtering (not based on the results) would make the trick since I beleive users won't always expect OpenWebUI to filter out results and bring 0 if, say, in the first page every domain has been blocker
Funny for me with exa the filter returns results only from the sites I want to filter out. Either .wikipedia.org or !.wikipedia.org cause the search to only return results from this site.
Same for me, it's still the same in the latest release. I listed example.com, and never get any result for two days before I realize the search domain filter maybe wrong.
<!-- gh-comment-id:2812446553 -->
@ER-EPR commented on GitHub (Apr 17, 2025):
> Funny for me with exa the filter returns results only from the sites I want to filter out. Either _.wikipedia.org or !_.wikipedia.org cause the search to only return results from this site.
Same for me, it's still the same in the latest release. I listed example.com, and never get any result for two days before I realize the search domain filter maybe wrong.
Other search engines might allow using google like sintax site:mydomain.com
Yes, Google and DuckDuckGo for example support queries like search term site:github.com OR site:gitlab.com. Having this for search engines, which support it, would be very helpful.
It should be relatively easy to achieve this result by, for example, modifying the following lines:
withDDGS()asddgs:# build the filter into the search query:query=query+"site:"+" site:".join(filter_list)# Use the ddgs.text() method to perform the searchtry:search_results=ddgs.text(query,safesearch="moderate",max_results=count,backend="lite")exceptRatelimitExceptionase:log.error(f"RatelimitException: {e}")# The following lines are not necessary anymore:# if filter_list:# search_results = get_filtered_results(search_results, filter_list)
<!-- gh-comment-id:3188840584 -->
@svenseeberg commented on GitHub (Aug 14, 2025):
> Other search engines might allow using google like sintax site:mydomain.com
Yes, Google and DuckDuckGo for example support queries like `search term site:github.com OR site:gitlab.com`. Having this for search engines, which support it, would be very helpful.
It should be relatively easy to achieve this result by, for example, modifying the following lines:
https://github.com/open-webui/open-webui/blob/main/backend/open_webui/retrieval/web/duckduckgo.py#L27-L36
```python
with DDGS() as ddgs:
# build the filter into the search query:
query = query + "site:"+" site:".join(filter_list)
# Use the ddgs.text() method to perform the search
try:
search_results = ddgs.text(
query, safesearch="moderate", max_results=count, backend="lite"
)
except RatelimitException as e:
log.error(f"RatelimitException: {e}")
# The following lines are not necessary anymore:
# if filter_list:
# search_results = get_filtered_results(search_results, filter_list)
```
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @liucoj on GitHub (Feb 13, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/9912
Hi, On the latest version 0.5.11, I noticed that the new option 'Domain Filter List' in WebSearch doesn't work as expected. When I tried to insert a couple of domains separated by commas (e.g., example.com, example2.com), the filtered domains were not considered when retrieving results. I'm using Tavily as an API.
I couldn't find any instructions about this new option on official OWUI channels, such as Twitter, Discord, documentation, or here. Although I noticed that when saving preferences after inserting domains into the filter list, only the Tavily API update, concurrent requests, and connections are updated in the logs - not the domain filters themselves.
Any ideas?
@oatmealm commented on GitHub (Feb 13, 2025):
Funny for me with exa the filter returns results only from the sites I want to filter out. Either .wikipedia.org or !.wikipedia.org cause the search to only return results from this site.
@liucoj commented on GitHub (Feb 13, 2025):
I think Tavily module is not ready for this option, I didn't find anything related to "filter_list: Optional[list[str]]" within the code (tavily.py) as with the other search modules.
@tjbck commented on GitHub (Feb 13, 2025):
It's only supported by a select few providers, PR welcome here!
@liucoj commented on GitHub (Feb 14, 2025):
@tjbck I made a simple implementation in tavily.py (in bold ) and it works! I took as an example the searching module (searxng.py) working module. Considering that, in UI, as an admin, the search module permits only 3 options (max results , max concurrent requests , domain filter list ), and there aren't any other options to consider, I'll make a PR with these implementations. Do I miss something?
#i rewrite this function
def search_tavily(
api_key: str,
query: str,
count: int,
filter_list: Optional[list[str]] = None,
) -> list[SearchResult]:
#Add include_domain as requested by Tavily API
url = "https://api.tavily.com/search"
data = {"query": query, "api_key": api_key}
include_domain = filter_list
response = requests.post(url, include_domain, json=data)
response.raise_for_status()
@nachogmd commented on GitHub (Feb 14, 2025):
I will create a pull request which handles both "included" and "excluded" if this makes sense.
I struggled too with this feature until I noticed what it does is to exclude those domains
It should involve passing all search engines a white list and a black list. I would need advise on how to name configuration parameters as if this approach is taken ideally we should have RAG_WEB_SEARCH_DOMAIN_FILTER_LIST as
Best regards,
n
@tjbck commented on GitHub (Feb 15, 2025):
@nachogmd env var naming should follow the existing convention
e.g.
RAG_WEB_SEARCH_ALLOWED_DOMAIN_LIST,RAG_WEB_SEARCH_BLOCKED_DOMAIN_LIST@glottisfaun0000 commented on GitHub (Feb 15, 2025):
So if I wanted to filter so my web searches only use websites with .edu or .gov tld's, what would I put in this field?
@abdessalaam commented on GitHub (Feb 16, 2025):
Perhaps even 3 categories:
PREFERREDoption would always try to include that domain, while still searching everywhere else (e.g. ifALLOWEDnot set)@tjbck commented on GitHub (Feb 16, 2025):
Or perhaps it maybe more prudent/intutive to introduce a widely used syntax (e.g.
!prefix to indicate blocked domain)@nachogmd commented on GitHub (Feb 16, 2025):
Seacxng uses "!images Donald Duck" (for example) to specify the tab which you want to search (in this case images)
The only way I found to "filter" results in searxng is using https://docs.searxng.org/dev/plugins/hostnames.html
Other search engines might allow using google like sintax site:mydomain.com
@nachogmd commented on GitHub (Feb 16, 2025):
The issue I am facing is that is quite easy to filter out all search results leading to 0 search results. But it is the feature, isn't it?
I like RAG_WEB_SEARCH_PREFERRED_DOMAIN_LIST so that it reorders the results and moves to the top of the list those domains since OpenWebUI will likely only use a subset of the found results
@nachogmd commented on GitHub (Feb 16, 2025):
Actually stating in the front end which engines allow real domain filtering (not based on the results) would make the trick since I beleive users won't always expect OpenWebUI to filter out results and bring 0 if, say, in the first page every domain has been blocker
@lowstz commented on GitHub (Feb 22, 2025):
Examples of domain filter lists
https://en.wikipedia.org/wiki/Wikipedia:Reliable_sources/Perennial_sources
https://github.com/stanford-oval/storm/blob/e80d9bbea7362141a479940dabb751c1f244e4b6/knowledge_storm/storm_wiki/modules/retriever.py
@ER-EPR commented on GitHub (Apr 17, 2025):
Same for me, it's still the same in the latest release. I listed example.com, and never get any result for two days before I realize the search domain filter maybe wrong.
@svenseeberg commented on GitHub (Aug 14, 2025):
Yes, Google and DuckDuckGo for example support queries like
search term site:github.com OR site:gitlab.com. Having this for search engines, which support it, would be very helpful.It should be relatively easy to achieve this result by, for example, modifying the following lines:
https://github.com/open-webui/open-webui/blob/main/backend/open_webui/retrieval/web/duckduckgo.py#L27-L36