* fix: apply the SSRF checks to redirect targets on every web fetch path
Two guards protect server-side fetches: a private-IP check and the operator's `WEB_FETCH_FILTER_LIST`. Neither reached a redirect hop on the aiohttp paths, and the filter list never reached one on the requests paths either.
aiohttp answers IP-literal hosts itself without consulting a resolver, so `_SSRFSafeResolver` was never invoked for a hop such as `http://169.254.169.254/` and the private-IP check simply did not run. With redirect following enabled, a submitted public URL that redirects to an IP literal reached loopback, RFC1918 and cloud-metadata addresses, and the response body was returned to the caller. The filter list was consulted only in `validate_url`, on the originally submitted URL, so a redirect to a filter-listed host was fetched without it ever being applied.
`_SSRFSafeResolver` is replaced by `_SSRFSafeConnector`, which hooks `_resolve_host` so the IP check also covers the IP-literal shortcut and both DNS cache paths. The filter list moves to a per-request hook on each transport, `connect()` for aiohttp and `send()` for the requests adapter, because those see the request destination: at the connection layer a proxied request presents the proxy's host, and a pooled connection skips resolution entirely. This covers every hop, including redirects, on all five aiohttp call sites and both requests sessions. The Playwright loader already validated each hop and is unchanged.
Both gaps required `AIOHTTP_CLIENT_ALLOW_REDIRECTS=true`, which is not the default.
Two behaviour changes for operators. The filter list now applies to redirect targets rather than only to submitted URLs. Under a forward proxy it is evaluated against the request destination instead of the proxy, which also fixes allowlist entries rejecting every fetch in proxied deployments.
* refac: match the web fetch filter list against resolved addresses
The filter list is now evaluated against the hostname together with the addresses it resolves to, at URL validation and on each connection, on both transports. An IPv6 address is also matched by the IPv4 address it carries.
* refac: screen outbound fetch addresses against reserved ranges ipaddress misses
`ipaddress.is_global` was the only test behind the web-fetch address check, and it answers a narrower question than "may we fetch this". Several special-purpose ranges are globally routable by registry while nothing on them is a legitimate destination, so they passed. Classification now screens those ranges on top of `is_global`, and applies the same screen to the IPv4 address embedded in an IPv6 transition encoding rather than only to the literal. All three checkpoints share the predicate, so they all inherit it.
The range list is the exact complement of what CPython's `ipaddress` already models, checked entry by entry against both IANA special-purpose registries. Prefixes IANA marks globally reachable are deliberately left out, so no real destination changes behaviour. Verified against 31 addresses covering every entry, their transition-encoded forms, and public controls in both families: 31/31 expected after, 18/31 before.
* refac: match web fetch filter entries that name an address or a range
A filter entry that parses as an address or a CIDR range is matched by containment rather than by DNS label suffix, so a range covers the addresses inside it and an address matches however it is spelled. A range entry previously matched nothing at all, silently.
The built-in list gains the special-purpose networks that ipaddress.is_global reports as reachable while nothing on them is a legitimate destination, so taking an address out of reach is a WEB_FETCH_FILTER_LIST change rather than a release. Those entries hold whether or not local web fetch is enabled; the private-address rule still follows the toggle.
The binary branch of the web fetch read the entire response body into memory
before writing it out. It now streams in blocks, applies the configured file
size limit the same way the sibling URL endpoint already does, and removes the
temporary file when a download fails partway instead of leaving it behind.
With `DATABASE_ENABLE_IAM_TOKEN_AUTH=true` and `VECTOR_DB=pgvector`, startup failed at vector store initialisation with `fe_sendauth: no password supplied`, so the two features could not be used together.
`PgvectorClient` builds its own engine and never got the `do_connect` listener that refreshes the RDS IAM token, and the `ScopedSession` branch that would have reused the instrumented main engine is unreachable because `PGVECTOR_DB_URL` defaults to `DATABASE_URL` and is therefore never falsy.
The pgvector engine now goes through `enable_iam_token_auth()` like the main and Alembic engines. Since a token authenticates exactly one host/port/user, that function now attaches the listener only to engines pointing at the same target, so a `PGVECTOR_DB_URL` aimed at a separate database keeps the password from its own URL instead of having it overwritten; the skip is logged with both identities.
Fixes#27752
`x in d` and `x in d.keys()` are identical for a plain dict, so the `.keys()` call builds a throwaway view and reads as if it were doing something. Both sites operate on a plain dict: `combined` in `merge_and_sort_query_results` is a local `dict()`, and `ui_settings` comes from `UserSettings.model_dump()` where `ui` is annotated `dict | None` and is already guarded against None on the preceding line.
No behaviour change, and no measurable speedup either, so this is a readability cleanup rather than a performance one.
Sites where `.keys()` is load-bearing are left alone: the `list(d.keys())` snapshots taken before mutating during iteration, and the places where `.keys()` is the iteration or comprehension source rather than a membership test.
The DuckDuckGo search path catches RatelimitException from the ddgs library. That exception is defined by the library but never raised anywhere in it, checked against the pinned 9.14.4 and against 9.11.3, so the handler could never run. The two fallbacks around it were dead for the same reason: ddgs.text() returns a non-empty list or raises, so None and an empty list are not outcomes it can produce.
Removing all three leaves one call and changes nothing observable. A refused or rate limited search already came out as a failed search, with the error shown to the user and the traceback in the log, and it still does.
The backend argument is now passed as backend or 'auto' rather than conditionally omitted, because 'auto' is the library's own default for that parameter, so every configured value including unset and empty resolves exactly as before. Verified by running the old and the new function side by side against a stubbed library covering normal results, the domain filter, all four backend settings and a failing search, with identical results in every case.
The Mistral OCR loader has a full async pipeline beside its synchronous one: an async load, its own upload, signed URL, OCR, delete and retry helpers, a pooled session and a batch loader on top. The only way in was the batch loader, which nothing calls, so the entire async half was unreachable. Everything that loads documents goes through the synchronous path, and the shared loader entry point runs it in a worker thread. The Datalab loader carries a public request status poller with no caller either, since its own load inlines the polling it needs.
With the async half gone, the retry classifier's two aiohttp branches can no longer be reached, since the only retried calls are synchronous, so those go with it along with the aiohttp import that existed solely to feed them, and a timeout attribute that nothing reads any more. The class docstring loses the three bullets that only described the removed pipeline, and four docstrings stop calling themselves the sync version of something that no longer has an async counterpart.
This removes around 350 lines and leaves one code path per loader instead of one live path and one that cannot be entered.
The Playwright loader's route interceptor now performs each intercepted request with the same requests/aiohttp clients the other web loader paths already use and fulfills the page with that response, rather than having the browser issue it. Redirect handling, header forwarding and cookie delivery to the browser are unchanged.
Two consequences worth knowing. Page requests now leave from the backend instead of the browser, so with PLAYWRIGHT_WS_URL set they originate from a different host, and TLS is verified against certifi plus AIOHTTP_CLIENT_SSL_CERT_FILE rather than the browser's own trust store. And because the synchronous interceptor blocks, sub-resources on that path fetch one at a time: 30 assets at 40ms went from 2.01s to 3.01s, and 8 assets at 500ms from 1.05s to 4.50s. The asynchronous path is unaffected, at 0.65s and 1.05s respectively.
SRC_LOG_LEVELS became an empty dict when per-module log levels were dropped, and env.py keeps it only as a legacy name. opengauss.py is the last thing in the tree that still indexes it, at module scope, so importing the module raises KeyError: 'RAG' and any deployment on VECTOR_DB=opengauss dies the first time it touches the vector store. The factory imports it lazily, which is why nothing else trips over it. Deleting the line is the whole fix: every other vector backend takes getLogger(__name__) and inherits the root level.
colbert.py passes an argument to a message with no placeholder to consume it:
log.info('ColBERT: Loading model', name)
At INFO, which is the default, logging evaluates 'ColBERT: Loading model' % ('colbert-ir/colbertv2.0',) and raises TypeError: not all arguments converted during string formatting. The record is swallowed by handleError, so loading a ColBERT reranker prints '--- Logging error ---' plus a traceback to stderr instead of the model name. Adding %s prints the name and drops the traceback.
Since v0.11.0 shipped aiodns, aiohttp silently switched every outbound request from the OS resolver to c-ares. On some Windows hosts the bundled c-ares 1.34.6 (pycares 5) discovers only 127.0.0.1:53 as nameserver, so every external provider lookup fails (#28013). In Docker the long-lived c-ares channel intermittently stops resolving container names while Docker's embedded DNS keeps answering, which wipes the Ollama model list and fails all in-flight chats with a misleading "Model not found" (#28215).
This restores the pre-0.11 ThreadedResolver (OS resolver) by default and gates the c-ares path behind a new env var, AIOHTTP_CLIENT_ASYNC_DNS_RESOLVER, off by default. The event-loop DNS perf improvement is now opt-in for deployments whose resolver setup is known to work with c-ares, instead of a process-wide side effect of the package being installed.
aiodns is also downgraded and pinned to 3.6.1 (pycares<5), the last release before the broken c-ares 1.34.6 build, so opting in does not hit the Windows regression. The hardcoded AsyncResolver in the Mistral OCR loader now follows the same switch. Simply removing aiodns instead was not an option because opting in would then be impossible, and #28215 showed the Docker failure is c-ares itself, not aiodns 4.x.
Fetching a URL and saving it were reported as one thing. Everything from
reading the URL to writing the vector database sat inside a single try,
whose handler blamed the knowledge base, so a page that could not be
fetched, parsed or resolved was reported as a knowledge base error even
though nothing had reached the knowledge base yet. Reading the URL now has
its own handler that names the URL, and the knowledge base message is left
to the step that actually touches it.
When YouTube refused a transcript the reason was discarded earlier still:
the loader caught the error, logged it, and returned an empty document
list, so the empty result failed downstream and even the salvageable
explanation was gone before a message was produced. The loader now raises
YoutubeTranscriptError carrying a readable reason, mapped from the
transcript library's own exception types. Blocked requests mention that a
proxy can be configured, and disabled, age restricted, unavailable and
missing language cases each say what actually happened.
URLs that attach successfully are unaffected.
Raising GLOBAL_LOG_LEVEL to WARNING buys quieter output but not less work: 241 INFO call sites interpolate their payload into an f-string before the logging call gets to drop it. The heaviest is get_doc, which logs every chunk id and metadata dict in a collection, so on the full-context retrieval path that is the entire knowledge base, once per chat request.
That one line at WARNING, CPython 3.12:
| knowledge base | payload | before | after |
| -------------- | ------- | -------- | ------- |
| top-k of 3 | 1.2 kB | 3.8 us | 0.07 us |
| 500 chunks | 201 kB | 583.6 us | 0.08 us |
| 5000 chunks | 2.0 MB | 5.8 ms | 0.15 us |
The lazy form log.info('query_doc:result %s %s', result.ids, result.metadatas) hands the payload to record.getMessage(), which the InterceptHandler only reaches once a record has passed the level check. Output at INFO is byte-identical. Two sites that already built their message eagerly, one str concat and one % operator, move to the same lazy form.
GLOBAL_LOG_LEVEL defaults to INFO, so every log.debug(...) in the backend is discarded, but the message is built first: 187 call sites interpolate their payload into an f-string before the logging call runs, so the work happens on every request and the result is thrown away. The worst one sits in process_chat_payload and stringifies the whole request body, full conversation history included, once per chat completion.
That one line with DEBUG disabled, CPython 3.12:
| conversation | payload | before | after |
| ------------ | ------- | -------- | ------- |
| 4 messages | 1.2 kB | 3.4 us | 0.07 us |
| 20 messages | 17 kB | 24.8 us | 0.07 us |
| 60 messages | 123 kB | 216.6 us | 0.07 us |
The lazy form log.debug('form_data: %s', form_data) hands the payload to record.getMessage(), which the InterceptHandler only reaches once a record has passed the level check. With DEBUG enabled the emitted lines are byte-identical, f'{x=}' sites included: those map to %r. MistralLoader._debug_log callers get the same treatment, since that wrapper already forwards *args.
`_json_to_metadata` parses the metadata of every result row returned by search and get. It now goes through `JSONCodec`, which selects orjson when `ENABLE_ORJSON` is set.
The text it parses is produced by Oracle's own `JSON_SERIALIZE`, and the column is a native `JSON` type, so the database normalises whatever was written and the reader never depends on the writer's escaping.
The matching `_metadata_to_json` write deliberately keeps stdlib `json`: it passes `default=self._decimal_handler`, orjson accepts none of stdlib's keyword arguments, and dropping the handler would turn a currently successful insert of a `Decimal` into a hard failure. The read side has no such constraint.
With `ENABLE_ORJSON` unset, which is the default, `JSONCodec` is stdlib `json` and this call site behaves exactly as before.
Fourteen modules import `json` without using it. Ruff flags every one with F401, and a word-boundary search for `json` in each file matches only the import line itself, including inside strings, comments and annotations.
Two exclusions, both deliberate. Migration files are left alone: the import is equally dead there, but those files are frozen history and not worth the churn. `models/chats.py` has the same dead import and is handled in its own change, so it is skipped here to avoid two changes touching the same line.
No behaviour change.
The Valkey backend serializes chunk metadata on every insert and parses it back on every result row in `get` and `query`. Both directions now go through `JSONCodec`, which selects orjson when `ENABLE_ORJSON` is set.
The stored `metadata_json` field is never matched against as text. `_build_filter_expression` only emits TAG predicates, and the TAG fields are `id`, `hash`, `file_id`, `source` and `knowledge_base_id`; `metadata_json` appears only as a return field that is immediately re-parsed. So rows written with escaped non-ASCII and rows written raw are indistinguishable to every reader, and no migration is needed.
`process_metadata` already stringifies datetimes and strips null bytes and lone surrogates before the write, so the two backends cannot disagree about what is serializable here.
Both read `except` clauses widen from `(json.JSONDecodeError, TypeError)` to `(ValueError, TypeError)`. The codec falls back to engineio's codec, which installs `parse_int=_safe_int` and raises a bare `ValueError` for integer literals longer than 100 characters; the narrower clause would have let that escape and abort a search instead of yielding empty metadata. `json.JSONDecodeError` is a `ValueError` subclass, so this is a strict superset. That removes the module's last use of stdlib `json`, so the import goes with it.
With `ENABLE_ORJSON` unset, which is the default, `JSONCodec` is stdlib `json` and this call site behaves exactly as before.
RAG vector search runs in a thread pool, but then calls `future.result()` on the event loop thread, so the whole worker freezes until every collection answers. Every other user's token stream stops for that long. It's the default retrieval path.
Now `asyncio.gather` over `asyncio.to_thread`, matching what `routers/retrieval.py:2779` already does for the same call.
Measured with 3 queries across 4 collections, 60 ms search, and a second request wanting a turn every 5 ms:
| | before | after |
|---|---|---|
| RAG call | 62.0 ms | 61.2 ms |
| other request's turns | 0 | 7 |
| its worst stall | 62.5 ms | 16.0 ms |
Same results, same order, same `(result, error)` contract. Cancellation now lands mid-search instead of after every thread finishes. Threads move from an unbounded per-call pool to the loop's bounded shared one.
The web-search-* namespace was the one collection namespace filter_accessible_collections admitted unconditionally for any non-admin user, on both read and write, unlike file-*, user-memory-* and knowledge bases which are owner-scoped. process_web_search now mints these ephemeral per-query collections as web-search-{user.id}-<hash>, and the access helper only admits web-search-{requester.id}-* names, so a web-search collection is readable and writable only by the user who created it (admins keep their bypass). The collections hold transient public web-search results and their names are non-enumerable query hashes, so there was no demonstrated cross-user access path; this removes the namespace exception so the per-user scoping the other namespaces enforce also covers web-search.
Co-authored-by: rexpository <rexpository@users.noreply.github.com>
The .msg branch routed to langchain's OutlookMessageLoader, which requires the extract_msg package. extract_msg pins beautifulsoup4<4.14, but we pin unstructured==0.22.31 (needs beautifulsoup4>=4.14.3) and beautifulsoup4==4.14.3, so extract_msg can never be installed alongside the current dependency set. As a result the .msg path could not function on any supported install: uploads failed at runtime with an ImportError, and adding the missing package broke the build with an unsatisfiable resolver error.
Switch to UnstructuredEmailLoader, which parses .msg through unstructured's partition_msg (backed by python-oxmsg). Both are already shipped, so .msg uploads work with no new dependency and no version conflict. Attachment partitioning is disabled to preserve the previous body-only extraction behaviour.
Fixes#26690
Custom per-connection headers can now forward the user's groups to
upstream backends via two new template placeholders:
- {{USER_GROUPS}}: comma-separated group names
- {{USER_GROUP_IDS}}: comma-separated group ids
The group lookup is async, so get_custom_headers becomes an async
wrapper around the sync template substitution (parse_custom_headers)
and fetches groups lazily — only when a header value actually
references a groups placeholder. The external document loader path
runs in a worker thread without an event loop, so Loader.aload
prefetches the groups before offloading and passes them through to
ExternalDocumentLoader.
Claude-Session: https://claude.ai/code/session_01EbBEfTyu8fFJmC13rnQthT
Co-authored-by: Claude <noreply@anthropic.com>
Add self-hosted OpenSERP as a web search engine option. OpenSERP
provides browser-rendered search across Google, Bing, Yandex, Baidu,
DuckDuckGo, and Ecosia with no API keys required.
- New module: retrieval/web/openserp.py (async, uses aiohttp session pool)
- Config: OPENSERP_BASE_URL env var (defaults to http://localhost:7070)
- Routing: search_web() dispatch for 'openserp' engine
- Follows existing patterns (searxng, brave)
Co-authored-by: crustopher-lgtm <crustopher-lgtm@users.noreply.github.com>
`_detect_text_encoding()` hands the complete file to `chardet.detect()`. chardet is pure Python and costs roughly 1.3 seconds per megabyte, so uploading a large non-UTF-8 text file stalls for seconds inside encoding detection alone. A 4 MiB Shift-JIS file spends 6.4 seconds there. The UTF-8 fast path above it means only non-UTF-8 files reach this, which in practice are exactly the CJK documents the surrounding code was written to handle, so the slow case and the case that matters are the same case.
Detection does not need the whole file. It needs the bytes that are actually not UTF-8, and `UnicodeDecodeError.start` from the fast-path decode already says where those begin, so this samples a 256 KiB window around that offset.
Two things make that safe rather than merely fast.
Centring the window on the first non-UTF-8 byte instead of the file head is what keeps the common case correct. A plain head sample makes chardet report ascii for a file that is ASCII for its first few hundred KiB and only turns CJK later, and the method then falls through to latin-1 instead of the right codec.
The window still cannot help when a stray byte, a pasted Windows-1252 artifact for example, sits hundreds of KiB ahead of the real payload: the sample is then almost pure ASCII and carries no signal. So when the sample holds almost no non-ASCII bytes and is a strict subset of the file, detection falls back to the whole buffer. That case pays the old cost, which is the right trade, because it is precisely the case where sampling would otherwise be wrong. Without this guard a Cyrillic document with a stray leading byte was detected as ISO-8859-1 rather than windows-1251, which is silent mojibake.
Measured, with the encoding returned identical in every case:
| file | before | after |
|---|---|---|
| shift_jis 4 MiB | 6402ms | 755ms |
| gb18030 4 MiB | 3199ms | 449ms |
| big5 4 MiB | 2926ms | 413ms |
| euc-jp 4 MiB | 2456ms | 413ms |
| euc-kr 4 MiB | 2382ms | 468ms |
| latin-1 4 MiB | 1902ms | 394ms |
| gb18030 1 MiB | 807ms | 376ms |
| ascii head then gb18030 tail | 533ms | 294ms |
| stray byte then cp1251 payload | 496ms | 1051ms |
| any UTF-8 file | 8ms | 0ms |
29 cases, all returning an identical encoding before and after: six encodings at 100 KiB, 1 MiB and 4 MiB, three layouts where the non-UTF-8 bytes only begin beyond the window, four where a stray byte is separated from the payload, plus plain UTF-8, UTF-8 CJK and an empty file. The stray-byte rows are slower than before because they scan twice, once over the window and once over the whole buffer. They are the pathological shape, and correctness wins there.
The residual time is now the decode-and-validate loop below, which walks the file once per candidate codec, and `_has_cjk_characters`, which is a per-character Python loop over the decoded text. Both are the same "full scan for a detection decision" pattern and could take a bounded prefix too. That is left alone here.
`alazy_load()` builds every BeautifulSoup tree inline in an async function, so a web search that pulls in ten pages stops the entire worker for the whole time it spends parsing. Nothing else on that worker runs during it: not other users' token streams, not health checks, not socket.io traffic. Parsing is CPU work and it belongs in a thread.
Measured over 37 real pages, 13.5 MiB total, with a 5ms ticker sampling event-loop lag:
| | wall | worst loop stall | ticker fired |
|---|---|---|---|
| inline, html.parser (today) | 1793.8ms | 1788.8ms | 1 time |
| offloaded, html.parser | 1872.9ms | 82.9ms | 88 times |
| inline, lxml | 1346.7ms | 1341.8ms | 1 time |
| offloaded, lxml | 1445.4ms | 37.0ms | 118 times |
Today the loop is not merely slow during a batch, it is gone: a 5ms timer fired exactly once across 1.8 seconds. After the change it fires normally and the worst single stall drops by a factor of 20 to 36. The cost is 4 to 7 percent more wall time for the batch itself, from the thread handoffs, which is the right trade for a server handling more than one user.
Three details behind the shape of the change:
`get_text()` is only 2 percent of the cost (34ms against 1706ms of parsing over the corpus), so the whole per-page unit moves into the thread rather than the parse alone. Splitting them measured worse on both axes.
The offload is per page, not per batch. Handing the whole batch to one thread measured worse than either (2081ms wall, 235ms worst stall), so the loop is yielded to between pages.
The metadata block in `alazy_load()` was a duplicate of the module-level `extract_metadata()`, field for field, and `lazy_load()` was already using the shared helper. The new helper calls it too, which is why the diff removes more lines than it adds. The `ascrape_all()` override goes with it: it was a verbatim copy of the inherited implementation and `alazy_load()` was its only caller, so anything still calling it now gets the identical parent method, which resolves `self._unpack_fetch_results` to the override this class keeps.
Verified by feeding the real loader a 37 page corpus and comparing every resulting Document against the implementation this replaces:
```
PASS one Document per url (37)
PASS every Document identical to the pre-change implementation (0 differ)
PASS parsing ran off the main thread
PASS event loop kept running during parsing (90 ticks)
```
Both `page_content` and `metadata` are byte-identical on all 37 pages. This is independent of the parser in use and composes with switching the default parser to lxml: that change makes the stalls shorter, this one takes them off the loop.
Timothy Jaeryang Baek
G30andGitHub
