I have read and followed all the instructions provided in the README.md.
I am on the latest version of both Open WebUI and Ollama.
I have included the browser console logs.
I have included the Docker container logs.
I have provided the exact steps to reproduce the bug in the "Steps to Reproduce" section below.
Expected Behavior:
Logging in with OIDC token and using custom root ca certificate.
Actual Behavior:
Different error messages and exceptions thrown, see description.
Description
Bug Summary:
I'm trying to use keycloak as a SSO solution for open webui, but I keep running into SSL errors. My setup (all docker) currently is:
nginx -> Reverse Proxy with SSL Termination
keycloak -> can be reached as http://keycloak-keycloak-1:8080 (direct) or https://keycloak.local (With the reverse proxy)
Open WebUI -> can be reached as http://llm-open-web-ui-1:8080 (direct) or https://llm.local (With the reverse proxy)
I'm also running a custom DNS server that resolves all local domain names to the correct IP address. I'm using a custom root ca certificate for my ssl certificates because all of this is setup in my internal network.
This is the open webui part of my docker-compose file
open-web-ui:image:ghcr.io/open-webui/open-webui:mainrestart:unless-stoppednetworks:- nginx_nginxvolumes:- open-webui:/app/backend/data- /data/docker_data/open-webui/ca-certificates:/usr/local/share/ca-certificates/nas-ca:ro# contains my custom ca certificateenvironment:OLLAMA_BASE_URL:http://llm-ollama-1:11434ENABLE_LOGIN_FORM:falseENABLE_SIGNUP:falseOAUTH_CLIENT_ID:open-webuiOAUTH_CLIENT_SECRET:[...]# super secret key replaced for questionOPENID_PROVIDER_URL:https://keycloak.local/realms/nas-local/.well-known/openid-configurationOAUTH_PROVIDER_NAME:KeycloakOAUTH_SCOPES:openid email profile ENABLE_OAUTH_SIGNUP:trueREQUESTS_CA_BUNDLE:/usr/local/share/ca-certificates/nas-ca/nas-rootCa.crt MAIN_LOG_LEVEL:debug
Now for my issue, when using https://keycloak.local/realms/nas-local/.well-known/openid-configuration as OPENID_PROVIDER_URL after clicking the sign in with keycloak button I just get a internal server error in my browser and the server logs (Case 1 below) show
httpcore.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)
and a bunch of other exception messages.
Changing the OPENID_PROVIDER_URL to http://keycloak-keycloak-1:8080/realms/nas-local/.well-known/openid-configuration I get a bit further in the webui. After logging in trough keycloak I get a json error The email or password provided is incorrect. Please check for typos and try logging in again. in my browser. The server logs (Case 2 below) show
OAuth callback error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)
But no exception messages from python.
I think the issue is my custom CA that is not recognized by python. I've tried following the documentation for setting up a custom ca store. And I've also tried running the command update-ca-certificates after starting the docker container. This shows 1 added certificate and I can curl my keycloak container with https and get a good response.
I've also tried to get a response with python from the command line. This also shows a good response and python isn't complaining about a missing certificate.
Reproduction Details
Steps to Reproduce:
Create a custom root ca
Create a ssl certificate, for all the necessary domains, signed by the custom root ca
Deploy a DNS server or add the necessary DNS entries in your local host files
Deploy an nginx reverse proxy with custom root ca and ssl certificate
Deploy keycloak and create the necessary nginx config for the reverse proxy and ssl termination
Deploy open webui and ollama and create the necessary nginx config for the reverse proxy and ssl termination
Add the custom root ca to open webui containers
Configure OAuth as shown above
Try to log in
Logs and Screenshots
Browser Console Logs:
Not relevant (I think)
Docker Container Logs:
Case 1
INFO: 192.168.192.28:0 - "GET /oauth/oidc/login HTTP/1.0" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 72, in map_httpcore_exceptions
d
File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 377, in handle_async_request
resp = await self._pool.handle_async_request(req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 216, in handle_async_request
raise exc from None
File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 196, in handle_async_request
response = await connection.handle_async_request(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection.py", line 99, in handle_async_request
raise exc
File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection.py", line 76, in handle_async_request
stream = await self._connect(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection.py", line 154, in _connect
stream = await stream.start_tls(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 69, in start_tls
with map_exceptions(exc_map):
File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
raise to_exc(exc) from exc
httpcore.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
await super().__call__(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
await self.app(scope, receive, _send)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/sessions.py", line 85, in __call__
await self.app(scope, receive, send_wrapper)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 189, in __call__
with collapse_excgroups():
File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.11/site-packages/starlette/_utils.py", line 93, in collapse_excgroups
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__
response = await self.dispatch_func(request, call_next)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/backend/open_webui/main.py", line 901, in inspect_websocket
return await call_next(request)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 165, in call_next
raise app_exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 151, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 189, in __call__
with collapse_excgroups():
File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.11/site-packages/starlette/_utils.py", line 93, in collapse_excgroups
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__
response = await self.dispatch_func(request, call_next)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/backend/open_webui/main.py", line 880, in update_embedding_function
response = await call_next(request)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 165, in call_next
raise app_exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 151, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 189, in __call__
with collapse_excgroups():
File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.11/site-packages/starlette/_utils.py", line 93, in collapse_excgroups
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__
response = await self.dispatch_func(request, call_next)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/backend/open_webui/main.py", line 871, in check_url
response = await call_next(request)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 165, in call_next
raise app_exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 151, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 189, in __call__
with collapse_excgroups():
File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.11/site-packages/starlette/_utils.py", line 93, in collapse_excgroups
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__
response = await self.dispatch_func(request, call_next)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/backend/open_webui/main.py", line 857, in commit_session_after_request
response = await call_next(request)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 165, in call_next
raise app_exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 151, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 189, in __call__
with collapse_excgroups():
File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.11/site-packages/starlette/_utils.py", line 93, in collapse_excgroups
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__
response = await self.dispatch_func(request, call_next)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/backend/open_webui/utils/security_headers.py", line 11, in dispatch
response = await call_next(request)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 165, in call_next
raise app_exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 151, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
await self.app(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 189, in __call__
with collapse_excgroups():
File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.11/site-packages/starlette/_utils.py", line 93, in collapse_excgroups
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__
response = await self.dispatch_func(request, call_next)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/backend/open_webui/main.py", line 836, in dispatch
response = await call_next(request)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 165, in call_next
raise app_exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 151, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 189, in __call__
with collapse_excgroups():
File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.11/site-packages/starlette/_utils.py", line 93, in collapse_excgroups
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__
response = await self.dispatch_func(request, call_next)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/backend/open_webui/main.py", line 757, in dispatch
return await call_next(request)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 165, in call_next
raise app_exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 151, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 189, in __call__
with collapse_excgroups():
File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.11/site-packages/starlette/_utils.py", line 93, in collapse_excgroups
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__
response = await self.dispatch_func(request, call_next)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/backend/open_webui/main.py", line 538, in dispatch
return await call_next(request)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 165, in call_next
raise app_exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 151, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
await route.handle(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
await self.app(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
response = await func(request)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
return await dependant.call(**values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/backend/open_webui/main.py", line 2374, in oauth_login
return await oauth_manager.handle_login(provider, request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/backend/open_webui/utils/oauth.py", line 130, in handle_login
return await client.authorize_redirect(request, redirect_uri)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/authlib/integrations/starlette_client/apps.py", line 34, in authorize_redirect
rv = await self.create_authorization_url(redirect_uri, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/authlib/integrations/base_client/async_app.py", line 95, in create_authorization_url
metadata = await self.load_server_metadata()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/authlib/integrations/base_client/async_app.py", line 76, in load_server_metadata
resp = await client.request('GET', self._server_metadata_url, withhold_token=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/authlib/integrations/httpx_client/oauth2_client.py", line 90, in request
return await super().request(
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1585, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1674, in send
response = await self._send_handling_auth(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1702, in _send_handling_auth
response = await self._send_handling_redirects(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1739, in _send_handling_redirects
response = await self._send_single_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1776, in _send_single_request
response = await transport.handle_async_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 376, in handle_async_request
with map_httpcore_exceptions():
File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 89, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)
Case 2:
INFO: 192.168.192.28:0 - "GET /oauth/oidc/login HTTP/1.0" 302 Found
WARNI [open_webui.utils.oauth] OAuth callback error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)
DEBUG [open_webui.main] Commit session after request
INFO: 192.168.192.28:0 - "GET /oauth/oidc/callback?state=OUMVmepAxJd0tJ7vJaQkgWJ3lzOCuF&session_state=fc1cbf66-a7dc-4828-af49-c11a898c6240&iss=https%3A%2F%2Fkeycloak.nas.local%2Frealms%2Fnas-local&code=db6982d2-ffff-41b8-b0be-383d75d438e1.fc1cbf66-a7dc-4828-af49-c11a898c6240.05e0d2d4-f73b-4fdd-95b9-054190e2d9b5 HTTP/1.0" 400 Bad Request
DEBUG [open_webui.main] Commit session after request
Screenshots/Screen Recordings (if applicable):
Additional Information
I've also used my keycloak deployment as SSO for other self hosted applications, I think I can exclude that as an general issue. I also assume that docker is not the issue here, as all my other stuff is hosted with docker and works with this setup.
Originally created by @joejoe87577 on GitHub (Nov 8, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/6793
# Bug Report
## Installation Method
docker
## Environment
- **Open WebUI Version:** v0.3.35 (7228b39064ac28e1240bf8998f2a35535c6f7ef5)
- **Ollama (if applicable):** 0.3.14
- **Operating System:** Debian GNU/Linux 12 (bookworm)
- **Browser (if applicable):** Firefox 131.0
**Confirmation:**
- [x] I have read and followed all the instructions provided in the README.md.
- [x] I am on the latest version of both Open WebUI and Ollama.
- [ ] I have included the browser console logs.
- [x] I have included the Docker container logs.
- [x] I have provided the exact steps to reproduce the bug in the "Steps to Reproduce" section below.
## Expected Behavior:
Logging in with OIDC token and using custom root ca certificate.
## Actual Behavior:
Different error messages and exceptions thrown, see description.
## Description
**Bug Summary:**
I'm trying to use keycloak as a SSO solution for open webui, but I keep running into SSL errors. My setup (all docker) currently is:
- nginx -> Reverse Proxy with SSL Termination
- keycloak -> can be reached as `http://keycloak-keycloak-1:8080` (direct) or `https://keycloak.local` (With the reverse proxy)
- Open WebUI -> can be reached as `http://llm-open-web-ui-1:8080` (direct) or `https://llm.local` (With the reverse proxy)
I'm also running a custom DNS server that resolves all local domain names to the correct IP address. I'm using a custom root ca certificate for my ssl certificates because all of this is setup in my internal network.
This is the open webui part of my docker-compose file
```yml
open-web-ui:
image: ghcr.io/open-webui/open-webui:main
restart: unless-stopped
networks:
- nginx_nginx
volumes:
- open-webui:/app/backend/data
- /data/docker_data/open-webui/ca-certificates:/usr/local/share/ca-certificates/nas-ca:ro # contains my custom ca certificate
environment:
OLLAMA_BASE_URL: http://llm-ollama-1:11434
ENABLE_LOGIN_FORM: false
ENABLE_SIGNUP: false
OAUTH_CLIENT_ID: open-webui
OAUTH_CLIENT_SECRET: [...] # super secret key replaced for question
OPENID_PROVIDER_URL: https://keycloak.local/realms/nas-local/.well-known/openid-configuration
OAUTH_PROVIDER_NAME: Keycloak
OAUTH_SCOPES: openid email profile
ENABLE_OAUTH_SIGNUP: true
REQUESTS_CA_BUNDLE: /usr/local/share/ca-certificates/nas-ca/nas-rootCa.crt
MAIN_LOG_LEVEL: debug
```
Now for my issue, when using `https://keycloak.local/realms/nas-local/.well-known/openid-configuration` as `OPENID_PROVIDER_URL` after clicking the sign in with keycloak button I just get a internal server error in my browser and the server logs (Case 1 below) show
```
httpcore.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)
```
and a bunch of other exception messages.
Changing the `OPENID_PROVIDER_URL` to `http://keycloak-keycloak-1:8080/realms/nas-local/.well-known/openid-configuration` I get a bit further in the webui. After logging in trough keycloak I get a json error `The email or password provided is incorrect. Please check for typos and try logging in again.` in my browser. The server logs (Case 2 below) show
```
OAuth callback error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)
```
But no exception messages from python.
I think the issue is my custom CA that is not recognized by python. I've tried following the [documentation](https://docs.openwebui.com/tutorials/integrations/custom-ca/) for setting up a custom ca store. And I've also tried running the command `update-ca-certificates` after starting the docker container. This shows 1 added certificate and I can curl my keycloak container with `https` and get a good response.
I've also tried to get a response with python from the command line. This also shows a good response and python isn't complaining about a missing certificate.
## Reproduction Details
**Steps to Reproduce:**
1. Create a custom root ca
2. Create a ssl certificate, for all the necessary domains, signed by the custom root ca
3. Deploy a DNS server or add the necessary DNS entries in your local host files
4. Deploy an nginx reverse proxy with custom root ca and ssl certificate
5. Deploy keycloak and create the necessary nginx config for the reverse proxy and ssl termination
6. Deploy open webui and ollama and create the necessary nginx config for the reverse proxy and ssl termination
7. Add the custom root ca to open webui containers
8. Configure OAuth as shown above
9. Try to log in
## Logs and Screenshots
**Browser Console Logs:**
Not relevant (I think)
**Docker Container Logs:**
### Case 1
```
INFO: 192.168.192.28:0 - "GET /oauth/oidc/login HTTP/1.0" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 72, in map_httpcore_exceptions
d
File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 377, in handle_async_request
resp = await self._pool.handle_async_request(req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 216, in handle_async_request
raise exc from None
File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 196, in handle_async_request
response = await connection.handle_async_request(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection.py", line 99, in handle_async_request
raise exc
File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection.py", line 76, in handle_async_request
stream = await self._connect(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection.py", line 154, in _connect
stream = await stream.start_tls(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 69, in start_tls
with map_exceptions(exc_map):
File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
raise to_exc(exc) from exc
httpcore.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
await super().__call__(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
await self.app(scope, receive, _send)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/sessions.py", line 85, in __call__
await self.app(scope, receive, send_wrapper)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 189, in __call__
with collapse_excgroups():
File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.11/site-packages/starlette/_utils.py", line 93, in collapse_excgroups
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__
response = await self.dispatch_func(request, call_next)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/backend/open_webui/main.py", line 901, in inspect_websocket
return await call_next(request)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 165, in call_next
raise app_exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 151, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 189, in __call__
with collapse_excgroups():
File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.11/site-packages/starlette/_utils.py", line 93, in collapse_excgroups
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__
response = await self.dispatch_func(request, call_next)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/backend/open_webui/main.py", line 880, in update_embedding_function
response = await call_next(request)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 165, in call_next
raise app_exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 151, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 189, in __call__
with collapse_excgroups():
File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.11/site-packages/starlette/_utils.py", line 93, in collapse_excgroups
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__
response = await self.dispatch_func(request, call_next)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/backend/open_webui/main.py", line 871, in check_url
response = await call_next(request)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 165, in call_next
raise app_exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 151, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 189, in __call__
with collapse_excgroups():
File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.11/site-packages/starlette/_utils.py", line 93, in collapse_excgroups
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__
response = await self.dispatch_func(request, call_next)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/backend/open_webui/main.py", line 857, in commit_session_after_request
response = await call_next(request)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 165, in call_next
raise app_exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 151, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 189, in __call__
with collapse_excgroups():
File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.11/site-packages/starlette/_utils.py", line 93, in collapse_excgroups
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__
response = await self.dispatch_func(request, call_next)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/backend/open_webui/utils/security_headers.py", line 11, in dispatch
response = await call_next(request)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 165, in call_next
raise app_exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 151, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
await self.app(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 189, in __call__
with collapse_excgroups():
File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.11/site-packages/starlette/_utils.py", line 93, in collapse_excgroups
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__
response = await self.dispatch_func(request, call_next)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/backend/open_webui/main.py", line 836, in dispatch
response = await call_next(request)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 165, in call_next
raise app_exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 151, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 189, in __call__
with collapse_excgroups():
File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.11/site-packages/starlette/_utils.py", line 93, in collapse_excgroups
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__
response = await self.dispatch_func(request, call_next)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/backend/open_webui/main.py", line 757, in dispatch
return await call_next(request)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 165, in call_next
raise app_exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 151, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 189, in __call__
with collapse_excgroups():
File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.11/site-packages/starlette/_utils.py", line 93, in collapse_excgroups
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__
response = await self.dispatch_func(request, call_next)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/backend/open_webui/main.py", line 538, in dispatch
return await call_next(request)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 165, in call_next
raise app_exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 151, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
await route.handle(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
await self.app(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
response = await func(request)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
return await dependant.call(**values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/backend/open_webui/main.py", line 2374, in oauth_login
return await oauth_manager.handle_login(provider, request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/backend/open_webui/utils/oauth.py", line 130, in handle_login
return await client.authorize_redirect(request, redirect_uri)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/authlib/integrations/starlette_client/apps.py", line 34, in authorize_redirect
rv = await self.create_authorization_url(redirect_uri, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/authlib/integrations/base_client/async_app.py", line 95, in create_authorization_url
metadata = await self.load_server_metadata()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/authlib/integrations/base_client/async_app.py", line 76, in load_server_metadata
resp = await client.request('GET', self._server_metadata_url, withhold_token=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/authlib/integrations/httpx_client/oauth2_client.py", line 90, in request
return await super().request(
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1585, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1674, in send
response = await self._send_handling_auth(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1702, in _send_handling_auth
response = await self._send_handling_redirects(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1739, in _send_handling_redirects
response = await self._send_single_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1776, in _send_single_request
response = await transport.handle_async_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 376, in handle_async_request
with map_httpcore_exceptions():
File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 89, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)
```
### Case 2:
```
INFO: 192.168.192.28:0 - "GET /oauth/oidc/login HTTP/1.0" 302 Found
WARNI [open_webui.utils.oauth] OAuth callback error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)
DEBUG [open_webui.main] Commit session after request
INFO: 192.168.192.28:0 - "GET /oauth/oidc/callback?state=OUMVmepAxJd0tJ7vJaQkgWJ3lzOCuF&session_state=fc1cbf66-a7dc-4828-af49-c11a898c6240&iss=https%3A%2F%2Fkeycloak.nas.local%2Frealms%2Fnas-local&code=db6982d2-ffff-41b8-b0be-383d75d438e1.fc1cbf66-a7dc-4828-af49-c11a898c6240.05e0d2d4-f73b-4fdd-95b9-054190e2d9b5 HTTP/1.0" 400 Bad Request
DEBUG [open_webui.main] Commit session after request
```
**Screenshots/Screen Recordings (if applicable):**
-
## Additional Information
I've also used my keycloak deployment as SSO for other self hosted applications, I think I can exclude that as an general issue. I also assume that docker is not the issue here, as all my other stuff is hosted with docker and works with this setup.
I had to add SSL_CERT_FILE as environment variable instead of REQUESTS_CA_BUNDLE to get this to work. Do you want to change the documentation or should I write a quick pull request?
<!-- gh-comment-id:2465538046 -->
@joejoe87577 commented on GitHub (Nov 8, 2024):
Ok, I've found the fix.
I had to add `SSL_CERT_FILE` as environment variable instead of `REQUESTS_CA_BUNDLE` to get this to work. Do you want to change the [documentation](https://docs.openwebui.com/tutorials/integrations/custom-ca/) or should I write a quick pull request?
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @joejoe87577 on GitHub (Nov 8, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/6793
Bug Report
Installation Method
docker
Environment
Open WebUI Version: v0.3.35 (
7228b39064)Ollama (if applicable): 0.3.14
Operating System: Debian GNU/Linux 12 (bookworm)
Browser (if applicable): Firefox 131.0
Confirmation:
Expected Behavior:
Logging in with OIDC token and using custom root ca certificate.
Actual Behavior:
Different error messages and exceptions thrown, see description.
Description
Bug Summary:
I'm trying to use keycloak as a SSO solution for open webui, but I keep running into SSL errors. My setup (all docker) currently is:
http://keycloak-keycloak-1:8080(direct) orhttps://keycloak.local(With the reverse proxy)http://llm-open-web-ui-1:8080(direct) orhttps://llm.local(With the reverse proxy)I'm also running a custom DNS server that resolves all local domain names to the correct IP address. I'm using a custom root ca certificate for my ssl certificates because all of this is setup in my internal network.
This is the open webui part of my docker-compose file
Now for my issue, when using
https://keycloak.local/realms/nas-local/.well-known/openid-configurationasOPENID_PROVIDER_URLafter clicking the sign in with keycloak button I just get a internal server error in my browser and the server logs (Case 1 below) showand a bunch of other exception messages.
Changing the
OPENID_PROVIDER_URLtohttp://keycloak-keycloak-1:8080/realms/nas-local/.well-known/openid-configurationI get a bit further in the webui. After logging in trough keycloak I get a json errorThe email or password provided is incorrect. Please check for typos and try logging in again.in my browser. The server logs (Case 2 below) showBut no exception messages from python.
I think the issue is my custom CA that is not recognized by python. I've tried following the documentation for setting up a custom ca store. And I've also tried running the command
update-ca-certificatesafter starting the docker container. This shows 1 added certificate and I can curl my keycloak container withhttpsand get a good response.I've also tried to get a response with python from the command line. This also shows a good response and python isn't complaining about a missing certificate.
Reproduction Details
Steps to Reproduce:
Logs and Screenshots
Browser Console Logs:
Not relevant (I think)
Docker Container Logs:
Case 1
Case 2:
Screenshots/Screen Recordings (if applicable):
Additional Information
I've also used my keycloak deployment as SSO for other self hosted applications, I think I can exclude that as an general issue. I also assume that docker is not the issue here, as all my other stuff is hosted with docker and works with this setup.
@joejoe87577 commented on GitHub (Nov 8, 2024):
Ok, I've found the fix.
I had to add
SSL_CERT_FILEas environment variable instead ofREQUESTS_CA_BUNDLEto get this to work. Do you want to change the documentation or should I write a quick pull request?@tjbck commented on GitHub (Nov 9, 2024):
Feel free to update the docs! Thanks!