403 Error with websearch using searxng engine #1131

@justinh-rahb commented on GitHub (Jun 4, 2024):

Tutorial page is on docs now:
https://docs.openwebui.com/tutorial/web_search/

@justinh-rahb commented on GitHub (Jun 4, 2024): Tutorial page is on docs now: https://docs.openwebui.com/tutorial/web_search/

GiteaMirror commented

@andrewginns commented on GitHub (Jun 4, 2024):

Most public instances do not have JSON format enabled, the recommendation is to self host an instance if you want to use it.

Ah I see, thanks

Tutorial page is on docs now:
https://docs.openwebui.com/tutorial/web_search/

Will follow this, much appreciated!

@andrewginns commented on GitHub (Jun 4, 2024): > Most public instances do not have JSON format enabled, the recommendation is to self host an instance if you want to use it. Ah I see, thanks > Tutorial page is on docs now: https://docs.openwebui.com/tutorial/web_search/ Will follow this, much appreciated!

GiteaMirror commented

@sail1369 commented on GitHub (Jun 10, 2024):

I tried the fixes and followed the guide, at first with no luck, as i used the "3. Alternative: Docker Run", from the guide,
to make it work,
Step 1:
i went into the container and edited the settings file (/etc/searxng/settings.yml), for good measure i also did it in (/usr/local/searxng/searx/settings.yml) in both these files i added :
formats:
- html
- json #this line

Step 2:
set the websearch url as only the base url: http://localhost:8080/

hoping maybe this will help some people,
seems to work in my case, but no idea why

@sail1369 commented on GitHub (Jun 10, 2024): I tried the fixes and followed the guide, at first with no luck, as i used the "3. Alternative: Docker Run", from the guide, to make it work, Step 1: i went into the container and edited the settings file (/etc/searxng/settings.yml), for good measure i also did it in (/usr/local/searxng/searx/settings.yml) in both these files i added : formats: - html - json #this line Step 2: set the websearch url as only the base url: http://localhost:8080/ hoping maybe this will help some people, seems to work in my case, but no idea why

GiteaMirror commented

@rootcli commented on GitHub (Jul 12, 2024):

I've added all the settings that worked for me in discussions
Hope it helps

@rootcli commented on GitHub (Jul 12, 2024): I've added all the settings that worked for me in [discussions](https://github.com/open-webui/open-webui/discussions/3827#discussion-6926731) Hope it helps

GiteaMirror commented

@iplayfast commented on GitHub (Jul 29, 2024):

Not working for me. I even tried using public addresses eg https://searxng.site/ and got "too many requests" error. But I can access them through the browser. Looks to me like open-webui has problems.

@iplayfast commented on GitHub (Jul 29, 2024): Not working for me. I even tried using public addresses eg https://searxng.site/ and got "too many requests" error. But I can access them through the browser. Looks to me like open-webui has problems.

GiteaMirror commented

@justinh-rahb commented on GitHub (Jul 30, 2024):

Not working for me. I even tried using public addresses eg https://searxng.site/ and got "too many requests" error. But I can access them through the browser. Looks to me like open-webui has problems.

It doesn't if you follow the docs:
https://docs.openwebui.com/tutorial/web_search#searxng-docker

Specifically the settings.yml file must have json output format enabled:

search:
  safe_search: 0
  autocomplete: ""
  default_lang: ""
  formats:
    - html
    - json

Most public instances of SearXNG only have html format enabled, or restrict access to API usage.

@justinh-rahb commented on GitHub (Jul 30, 2024): > Not working for me. I even tried using public addresses eg https://searxng.site/ and got "too many requests" error. But I can access them through the browser. Looks to me like open-webui has problems. It doesn't if you follow the docs: https://docs.openwebui.com/tutorial/web_search#searxng-docker Specifically the `settings.yml` file must have `json` output format enabled: ```yaml search: safe_search: 0 autocomplete: "" default_lang: "" formats: - html - json ``` Most public instances of SearXNG only have `html` format enabled, or restrict access to API usage.

GiteaMirror commented

@fahadshery commented on GitHub (Aug 16, 2024):

formats:
- html
- json


Most public instances of SearXNG only have `html` format enabled, or restrict access to API usage.

This fixed it for me. Thank you!
just had to add - json under formats in the settings.yml file

@fahadshery commented on GitHub (Aug 16, 2024): > formats: > - html > - json > ``` > > Most public instances of SearXNG only have `html` format enabled, or restrict access to API usage. This fixed it for me. Thank you! just had to add `- json` under formats in the `settings.yml` file

GiteaMirror commented

@grimaldiss commented on GitHub (Aug 26, 2024):

where do i find settings.yml?

@grimaldiss commented on GitHub (Aug 26, 2024): where do i find settings.yml?

GiteaMirror commented

@rootcli commented on GitHub (Aug 27, 2024):

where do i find settings.yml?

https://github.com/open-webui/open-webui/issues/2824#issuecomment-2225356957
👆🏻

@rootcli commented on GitHub (Aug 27, 2024): > where do i find settings.yml? https://github.com/open-webui/open-webui/issues/2824#issuecomment-2225356957 👆🏻

GiteaMirror commented

@interzone2 commented on GitHub (Oct 7, 2024):

I've followed all instructions above but still get forbidden.

Relevant settings.yml :

formats:
   - html
   - json

server:
 # Is overwritten by ${SEARXNG_PORT} and ${SEARXNG_BIND_ADDRESS}
 port: 8080
 bind_address: "0.0.0.0"
 # public URL of the instance, to ensure correct inbound links. Is overwritten
 # by ${SEARXNG_URL}.
 base_url: http://138.68.159.187/search  # "http://example.com/location"
 # rate limit the number of request on the instance, block some bots.
 # Is overwritten by ${SEARXNG_LIMITER}
 limiter: true
 # enable features designed only for public instances.
 # Is overwritten by ${SEARXNG_PUBLIC_INSTANCE}
 public_instance: true

 # If your instance owns a /etc/searxng/settings.yml file, then set the following
 # values there.

 secret_key: "ccb1fe5de699f7a1589a1b11376da0ee9053006565bbb17fc2fbd2c01f5a0d72"  # Is overwritten by ${SEARXNG_SECRET}
 # Proxy image results through SearXNG. Is overwritten by ${SEARXNG_IMAGE_PROXY}
 image_proxy: false
 # 1.0 and 1.1 are supported
 http_protocol_version: "1.0"
 # POST queries are more secure as they don't show up in history but may cause
 # problems when using Firefox containers
 method: "GET"
 default_http_headers:
   X-Content-Type-Options: nosniff
   X-Download-Options: noopen
   X-Robots-Tag: noindex, nofollow
   Referrer-Policy: no-referrer

redis:
 # URL to connect redis database. Is overwritten by ${SEARXNG_REDIS_URL}.
 # https://docs.searxng.org/admin/settings/settings_redis.html#settings-redis
 url: redis://redis:6379/0

http://xxx.xx.xx.xx/search - is live

deno app attempting to communicate with searxng:

import { Readability } from "jsr:@paoramen/cheer-reader";

import ollama from "npm:ollama";
import * as cheerio from "npm:cheerio@1.0.0";

const searchUrl = Deno.env.get("SEARCH_URL");
const query = Deno.args.join(" ");



console.log(`Query: ${query}`);
const urls = await getNewsUrls(query);
const alltexts = await getCleanedText(urls);
await answerQuery(query, alltexts);

async function getNewsUrls(query: string) {
  try {
    const searchResults = await fetch(`${searchUrl}?q=${query}&format=json`);
    if (!searchResults.ok) {
      console.error('Error fetching search results:', searchResults.statusText);
      return [];
    }
    const contentType = searchResults.headers.get("content-type");
    if (contentType && contentType.includes("application/json")) {
      const searchResultsJson: { results: Array<{ url: string }> } = await searchResults.json();
      return searchResultsJson.results.map((result) => result.url).slice(0, 1);
    } else {
      console.error("Unexpected response type, expected JSON");
      console.error(await searchResults.text()); // Log the HTML response for debugging
      return [];
    }
  } catch (error) {
    console.error('Error during fetch:', error);
    return [];
  }
}


async function getCleanedText(urls: string[]) {
	const texts = [];
	for await (const url of urls) {
		const getUrl = await fetch(url);
		console.log(`Fetching ${url}`);
		const html = await getUrl.text();
		const text = htmlToText(html);
		texts.push(`Source: ${url}\n${text}\n\n`);
	}
	return texts;
}

function htmlToText(html: string) {
	const $ = cheerio.load(html);

  // Thanks to the comment on the YouTube video from @eliaspereirah for suggesting 
  // using Mozilla Readability. I used a variant that made it easier to use with 
  // cheerio. Definitely simplifies things
		const text = new Readability($).parse();

  // What I had before

	// $("script, source, style, head, img, svg, a, form, link, iframe").remove();
	// $("*").removeClass();
	// $("*").each((_, el) => {
	// 	if (el.type === "tag" || el.type === "script" || el.type === "style") {
	// 		for (const attr of Object.keys(el.attribs || {})) {
	// 			if (attr.startsWith("data-")) {
	// 				$(el).removeAttr(attr);
	// 			}
	// 		}
	// 	}
	// });
	// const text = $("body").text().replace(/\s+/g, " ");

	return text.textContent;
}

async function answerQuery(query: string, texts: string[]) {
	const result = await ollama.generate({
		model: "llama3.2:1b",
		prompt: `${query}. Summarize the information and provide an answer. Use only the information in the following articles to answer the question: ${texts.join("\n\n")}`,
		stream: true,
		options: {
			num_ctx: 16000,
		},
	});
	for await (const chunk of result) {
		if (chunk.done !== true) {
			await Deno.stdout.write(new TextEncoder().encode(chunk.response));
		}
	}
}

Search command: SEARCH_URL="http://xxx.xx.xx.xx/search" deno run --allow-net --allow-env --watch main.ts "breaking news"

Reply:

Watcher Process started.
SEARCH_URL: http://xxx.xx.xx.xx/search
Query: breaking news
Error fetching search results: Forbidden
However, I need the articles you're referring to. Please share the summaries or links to the articles, and I'll be happy to help summarize the information for your question.

No luck as yet!

@interzone2 commented on GitHub (Oct 7, 2024): I've followed all instructions above but still get forbidden. Relevant settings.yml : ``` formats: - html - json server: # Is overwritten by ${SEARXNG_PORT} and ${SEARXNG_BIND_ADDRESS} port: 8080 bind_address: "0.0.0.0" # public URL of the instance, to ensure correct inbound links. Is overwritten # by ${SEARXNG_URL}. base_url: http://138.68.159.187/search # "http://example.com/location" # rate limit the number of request on the instance, block some bots. # Is overwritten by ${SEARXNG_LIMITER} limiter: true # enable features designed only for public instances. # Is overwritten by ${SEARXNG_PUBLIC_INSTANCE} public_instance: true # If your instance owns a /etc/searxng/settings.yml file, then set the following # values there. secret_key: "ccb1fe5de699f7a1589a1b11376da0ee9053006565bbb17fc2fbd2c01f5a0d72" # Is overwritten by ${SEARXNG_SECRET} # Proxy image results through SearXNG. Is overwritten by ${SEARXNG_IMAGE_PROXY} image_proxy: false # 1.0 and 1.1 are supported http_protocol_version: "1.0" # POST queries are more secure as they don't show up in history but may cause # problems when using Firefox containers method: "GET" default_http_headers: X-Content-Type-Options: nosniff X-Download-Options: noopen X-Robots-Tag: noindex, nofollow Referrer-Policy: no-referrer redis: # URL to connect redis database. Is overwritten by ${SEARXNG_REDIS_URL}. # https://docs.searxng.org/admin/settings/settings_redis.html#settings-redis url: redis://redis:6379/0 ``` http://xxx.xx.xx.xx/search - is live deno app attempting to communicate with searxng: ``` import { Readability } from "jsr:@paoramen/cheer-reader"; import ollama from "npm:ollama"; import * as cheerio from "npm:cheerio@1.0.0"; const searchUrl = Deno.env.get("SEARCH_URL"); const query = Deno.args.join(" "); console.log(`Query: ${query}`); const urls = await getNewsUrls(query); const alltexts = await getCleanedText(urls); await answerQuery(query, alltexts); async function getNewsUrls(query: string) { try { const searchResults = await fetch(`${searchUrl}?q=${query}&format=json`); if (!searchResults.ok) { console.error('Error fetching search results:', searchResults.statusText); return []; } const contentType = searchResults.headers.get("content-type"); if (contentType && contentType.includes("application/json")) { const searchResultsJson: { results: Array<{ url: string }> } = await searchResults.json(); return searchResultsJson.results.map((result) => result.url).slice(0, 1); } else { console.error("Unexpected response type, expected JSON"); console.error(await searchResults.text()); // Log the HTML response for debugging return []; } } catch (error) { console.error('Error during fetch:', error); return []; } } async function getCleanedText(urls: string[]) { const texts = []; for await (const url of urls) { const getUrl = await fetch(url); console.log(`Fetching ${url}`); const html = await getUrl.text(); const text = htmlToText(html); texts.push(`Source: ${url}\n${text}\n\n`); } return texts; } function htmlToText(html: string) { const $ = cheerio.load(html); // Thanks to the comment on the YouTube video from @eliaspereirah for suggesting // using Mozilla Readability. I used a variant that made it easier to use with // cheerio. Definitely simplifies things const text = new Readability($).parse(); // What I had before // $("script, source, style, head, img, svg, a, form, link, iframe").remove(); // $("*").removeClass(); // $("*").each((_, el) => { // if (el.type === "tag" || el.type === "script" || el.type === "style") { // for (const attr of Object.keys(el.attribs || {})) { // if (attr.startsWith("data-")) { // $(el).removeAttr(attr); // } // } // } // }); // const text = $("body").text().replace(/\s+/g, " "); return text.textContent; } async function answerQuery(query: string, texts: string[]) { const result = await ollama.generate({ model: "llama3.2:1b", prompt: `${query}. Summarize the information and provide an answer. Use only the information in the following articles to answer the question: ${texts.join("\n\n")}`, stream: true, options: { num_ctx: 16000, }, }); for await (const chunk of result) { if (chunk.done !== true) { await Deno.stdout.write(new TextEncoder().encode(chunk.response)); } } } ``` Search command: `SEARCH_URL="http://xxx.xx.xx.xx/search" deno run --allow-net --allow-env --watch main.ts "breaking news"` Reply: ``` Watcher Process started. SEARCH_URL: http://xxx.xx.xx.xx/search Query: breaking news Error fetching search results: Forbidden However, I need the articles you're referring to. Please share the summaries or links to the articles, and I'll be happy to help summarize the information for your question. No luck as yet! ```

GiteaMirror commented

@aayusharyan commented on GitHub (Nov 20, 2024):

Tutorial page is on docs now: https://docs.openwebui.com/tutorial/web_search/

This page is no longer there on the website. Instead following this URL - https://docs.openwebui.com/features/web_search
However, same issue.

@aayusharyan commented on GitHub (Nov 20, 2024): > Tutorial page is on docs now: https://docs.openwebui.com/tutorial/web_search/ This page is no longer there on the website. Instead following this URL - https://docs.openwebui.com/features/web_search However, same issue.

GiteaMirror commented

@magentaFire commented on GitHub (Dec 6, 2024):

I tried the fixes and followed the guide, at first with no luck, as i used the "3. Alternative: Docker Run", from the guide, to make it work, Step 1: i went into the container and edited the settings file (/etc/searxng/settings.yml), for good measure i also did it in (/usr/local/searxng/searx/settings.yml) in both these files i added : formats: - html - json #this line

Step 2: set the websearch url as only the base url: http://localhost:8080/

hoping maybe this will help some people, seems to work in my case, but no idea why

This worked for me with some variants:

I have my instances of searxng, ollama and open webui on WSL ubuntu 24.04
My WSL instance is configured for mirrored networking (https://learn.microsoft.com/en-us/windows/wsl/networking)
I'd highlight that I needed to edit settings.yml files as above by entering in the docker container:

docker exec -it searxng sh
vi /etc/searxng/settings.yml 
// add -json line under "formats:"
// change bind_address: "0.0.0.0"  because I'm exposing my IP to my LAN through mirrored networking

Finally I added websearch url as only the base url: http://wsl-ip:3001/
and that works!!

BUT..... Also I don't know why 🫤

@magentaFire commented on GitHub (Dec 6, 2024): > I tried the fixes and followed the guide, at first with no luck, as i used the "3. Alternative: Docker Run", from the guide, to make it work, Step 1: i went into the container and edited the settings file (/etc/searxng/settings.yml), for good measure i also did it in (/usr/local/searxng/searx/settings.yml) in both these files i added : formats: - html - json #this line > > Step 2: set the websearch url as only the base url: http://localhost:8080/ > > hoping maybe this will help some people, seems to work in my case, but no idea why This worked for me with some variants: * I have my instances of searxng, ollama and open webui on WSL ubuntu 24.04 * My WSL instance is configured for mirrored networking (https://learn.microsoft.com/en-us/windows/wsl/networking) I'd highlight that I needed to edit settings.yml files as above by entering in the docker container: ``` docker exec -it searxng sh vi /etc/searxng/settings.yml // add -json line under "formats:" // change bind_address: "0.0.0.0" because I'm exposing my IP to my LAN through mirrored networking ``` Finally I added websearch url as only the base url: http://wsl-ip:3001/ and that works!! BUT..... Also I don't know why 🫤

GiteaMirror commented

@Atomique commented on GitHub (Dec 6, 2024):

I had the same problem, but already set anything suggested here. Problem was that I set the Base URL "correct". I set it to SEARXNG_BASE_URL=https://sub.domain.tld but this wasnt working with my open webui connecting to it. Its a bit weird. I changed to SEARXNG_BASE_URL=http://0.0.0.0:3001 in the container environment and also have set the following in my settings.yml:

# see https://docs.searxng.org/admin/settings/settings.html#settings-use-default-settings
use_default_settings: true
server:
  # base_url is defined in the SEARXNG_BASE_URL environment variable, see .env and docker-compose.yml
  secret_key: "<REDACTED>"  # change this!
  limiter: false  # can be disabled for a private instance
  image_proxy: true
  port: 8080
  bind_address: "0.0.0.0"

ui:
  static_use_hash: true

redis:
  url: redis://redis:6379/0

search:
  safe_search: 0
  autocomplete: ""
  default_lang: ""
  formats:
    - html
    - json

@Atomique commented on GitHub (Dec 6, 2024): I had the same problem, but already set anything suggested here. Problem was that I set the Base URL "correct". I set it to SEARXNG_BASE_URL=https://sub.domain.tld but this wasnt working with my open webui connecting to it. Its a bit weird. I changed to SEARXNG_BASE_URL=http://0.0.0.0:3001 in the container environment and also have set the following in my settings.yml: ```yml # see https://docs.searxng.org/admin/settings/settings.html#settings-use-default-settings use_default_settings: true server: # base_url is defined in the SEARXNG_BASE_URL environment variable, see .env and docker-compose.yml secret_key: "<REDACTED>" # change this! limiter: false # can be disabled for a private instance image_proxy: true port: 8080 bind_address: "0.0.0.0" ui: static_use_hash: true redis: url: redis://redis:6379/0 search: safe_search: 0 autocomplete: "" default_lang: "" formats: - html - json ```

GiteaMirror commented