feat: CA truststore support #583

@strikeoncmputrz commented on GitHub (Apr 4, 2024):

This would be an excellent feature. I'd rather not have to run my openai compatible inference server in http mode but I'm using a private public key infrastructure

@strikeoncmputrz commented on GitHub (Apr 4, 2024): This would be an excellent feature. I'd rather not have to run my openai compatible inference server in http mode but I'm using a private public key infrastructure

GiteaMirror commented

@tjbck commented on GitHub (Apr 14, 2024):

Feel free to make a PR!

@tjbck commented on GitHub (Apr 14, 2024): Feel free to make a PR!

GiteaMirror commented

@theobjectivedad commented on GitHub (Jun 19, 2024):

For folks stuck on this, here is a quick hack you can use ... basically you can just map /etc/ssl/certs/ca-certificates.crt in the container to a another file that contains your own trust chain.

In this example, I have my internal root CA trusted on the host machine and can map it directly into the container (don't forget to make it read only just in case):

docker run -it --rm \
  --publish=3000:8080 \
  --volume=open-webui:/app/backend/data \
  --volume=/etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt:ro \
  ghcr.io/open-webui/open-webui:main

For a larger-scale deployment you can basically do the same thing, ex in Kubernetes maybe you are using something like trust-manager to manage your truststore within the cluster. You can mount the trust-manager managed bundle to /etc/ssl/certs/ca-certificates.crt inside the open webui pod.

This isn't a best practice and and environment variable configuration to add an additional truststore would be cleaner but this will work in the interim. As an aside, I tried to set REQUESTS_CA_BUNDLE in the environment but it didn't work for me.

@theobjectivedad commented on GitHub (Jun 19, 2024): For folks stuck on this, here is a quick hack you can use ... basically you can just map ``/etc/ssl/certs/ca-certificates.crt`` in the container to a another file that contains your own trust chain. In this example, I have my internal root CA trusted on the host machine and can map it directly into the container (don't forget to make it read only just in case): ```shell docker run -it --rm \ --publish=3000:8080 \ --volume=open-webui:/app/backend/data \ --volume=/etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt:ro \ ghcr.io/open-webui/open-webui:main ``` For a larger-scale deployment you can basically do the same thing, ex in Kubernetes maybe you are using something like [trust-manager](https://cert-manager.io/docs/trust/trust-manager/) to manage your truststore within the cluster. You can mount the trust-manager managed bundle to ``/etc/ssl/certs/ca-certificates.crt`` inside the open webui pod. This isn't a best practice and and environment variable configuration to add an additional truststore would be cleaner but this will work in the interim. As an aside, I tried to set ``REQUESTS_CA_BUNDLE`` in the environment but it didn't work for me.

GiteaMirror commented

@hanahige commented on GitHub (Jul 16, 2024):

I'm not able to get the fix above to work, I've pulled my CA bundle from my AD Domain and added it into the container as mentioned above, however I still get a SSL: CERTIFICATE_VERIFY_FAILED error.
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)
however I did notice that the above method allows me to curl the .well-known address without having to ignore validation. It just seems like Open WebUI doesn't also respect that the CA root for my domain is trusted once added to the container CA trust.
I'm running Open WebUI 0.3.7, and Authentik 2024.4.7 behind a NGINX reverse proxy.

@hanahige commented on GitHub (Jul 16, 2024): I'm not able to get the fix above to work, I've pulled my CA bundle from my AD Domain and added it into the container as mentioned above, however I still get a SSL: CERTIFICATE_VERIFY_FAILED error. `ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)` however I did notice that the above method allows me to curl the .well-known address without having to ignore validation. It just seems like Open WebUI doesn't also respect that the CA root for my domain is trusted once added to the container CA trust. I'm running Open WebUI 0.3.7, and Authentik 2024.4.7 behind a NGINX reverse proxy.

GiteaMirror commented

@KizzyCode commented on GitHub (Jul 30, 2024):

FWIW, you can force python to use the system certificate store via REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt in the environment.

My (trimmed down) docker-compose.yml looks like this:

services:
  openwebui:
    image: ghcr.io/open-webui/open-webui:main
    volumes:
      - /var/containers/openwebui:/app/backend/data:rw
      - /etc/containers/openwebui/compusrv.crt:/etc/ssl/certs/ca-certificates.crt:ro
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    environment:
      - WEBUI_NAME=compusrv
      - ENABLE_SIGNUP=False
      - ENABLE_COMMUNITY_SHARING=False
      - WEBUI_SESSION_COOKIE_SAME_SITE=strict
      - WEBUI_SESSION_COOKIE_SECURE=True
      - ENABLE_OLLAMA_API=False
      - REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

where /etc/containers/openwebui/compusrv.crt is simply my self-signed certificate (i.e. you don't even need to build a full truststore, if you don't want to).

@KizzyCode commented on GitHub (Jul 30, 2024): FWIW, you can force python to use the system certificate store via `REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt` in the environment. My (trimmed down) docker-compose.yml looks like this: ```yml services: openwebui: image: ghcr.io/open-webui/open-webui:main volumes: - /var/containers/openwebui:/app/backend/data:rw - /etc/containers/openwebui/compusrv.crt:/etc/ssl/certs/ca-certificates.crt:ro - /etc/timezone:/etc/timezone:ro - /etc/localtime:/etc/localtime:ro environment: - WEBUI_NAME=compusrv - ENABLE_SIGNUP=False - ENABLE_COMMUNITY_SHARING=False - WEBUI_SESSION_COOKIE_SAME_SITE=strict - WEBUI_SESSION_COOKIE_SECURE=True - ENABLE_OLLAMA_API=False - REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt ``` where `/etc/containers/openwebui/compusrv.crt` is simply my self-signed certificate (i.e. you don't even need to build a full truststore, if you don't want to).

GiteaMirror commented

@nomppy commented on GitHub (Jul 30, 2024):

As an alternative to what @KizzyCode said and @theobjectivedad said about setting the REQUESTS_CA_BUNDLE env + mounting the host store, you can also add the following commands in the Dockerfile

COPY <CorporateSSL.crt> /usr/local/share/ca-certificates/
RUN update-ca-certificates

btw, I also have set

ENV PIP_CERT=/etc/ssl/certs/ca-certificates.crt \
    REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

Note that if you're building with the Dockerfile, you will also need to add the cert for the frontend build stage

@nomppy commented on GitHub (Jul 30, 2024): As an alternative to what @KizzyCode said and @theobjectivedad said about setting the `REQUESTS_CA_BUNDLE` env + mounting the host store, you can also add the following commands in the Dockerfile ``` COPY <CorporateSSL.crt> /usr/local/share/ca-certificates/ RUN update-ca-certificates ``` btw, I also have set ``` ENV PIP_CERT=/etc/ssl/certs/ca-certificates.crt \ REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt ``` Note that if you're building with the Dockerfile, you will also need to add the cert for the frontend `build` stage

GiteaMirror commented

@christiangierschner commented on GitHub (Aug 6, 2024):

FWIW, you can force python to use the system certificate store via REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt in the environment.

My (trimmed down) docker-compose.yml looks like this:

services:
  openwebui:
    image: ghcr.io/open-webui/open-webui:main
    volumes:
      - /var/containers/openwebui:/app/backend/data:rw
      - /etc/containers/openwebui/compusrv.crt:/etc/ssl/certs/ca-certificates.crt:ro
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    environment:
      - WEBUI_NAME=compusrv
      - ENABLE_SIGNUP=False
      - ENABLE_COMMUNITY_SHARING=False
      - WEBUI_SESSION_COOKIE_SAME_SITE=strict
      - WEBUI_SESSION_COOKIE_SECURE=True
      - ENABLE_OLLAMA_API=False
      - REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

where /etc/containers/openwebui/compusrv.crt is simply my self-signed certificate (i.e. you don't even need to build a full truststore, if you don't want to).

Sadly, this does not work for me...

i still get SSL: CERTIFICATE_VERIFY_FAILED if i hit "Continue with OAuth2" on login screen

however if i exec into the container, this works:

>>> import requests
>>> response = requests.get('https://internal-keycloak-url/auth')
>>> print(response)
<Response [200]>

seems like webui is somehow not using the rootca

any ideas?

@christiangierschner commented on GitHub (Aug 6, 2024): > FWIW, you can force python to use the system certificate store via `REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt` in the environment. > > My (trimmed down) docker-compose.yml looks like this: > > ```yaml > services: > openwebui: > image: ghcr.io/open-webui/open-webui:main > volumes: > - /var/containers/openwebui:/app/backend/data:rw > - /etc/containers/openwebui/compusrv.crt:/etc/ssl/certs/ca-certificates.crt:ro > - /etc/timezone:/etc/timezone:ro > - /etc/localtime:/etc/localtime:ro > environment: > - WEBUI_NAME=compusrv > - ENABLE_SIGNUP=False > - ENABLE_COMMUNITY_SHARING=False > - WEBUI_SESSION_COOKIE_SAME_SITE=strict > - WEBUI_SESSION_COOKIE_SECURE=True > - ENABLE_OLLAMA_API=False > - REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt > ``` > > where `/etc/containers/openwebui/compusrv.crt` is simply my self-signed certificate (i.e. you don't even need to build a full truststore, if you don't want to). Sadly, this does not work for me... i still get SSL: CERTIFICATE_VERIFY_FAILED if i hit "Continue with OAuth2" on login screen however if i exec into the container, this works: ``` >>> import requests >>> response = requests.get('https://internal-keycloak-url/auth') >>> print(response) <Response [200]> ``` seems like webui is somehow not using the rootca any ideas?

GiteaMirror commented

@nomppy commented on GitHub (Aug 6, 2024):

What does your compose file look like?

@nomppy commented on GitHub (Aug 6, 2024): What does your compose file look like?

GiteaMirror commented

@christiangierschner commented on GitHub (Aug 7, 2024):

services:
  ollama:
    image: ghcr.io/open-webui/open-webui:v0.3.11
    volumes:
      - ollama-webui:/app/backend/data
      - /usr/local/share/ca-certificates/rootca.crt:/etc/ssl/certs/ca-certificates.crt:ro
    environment:
      OLLAMA_BASE_URL: http://ollama-api:11434
      ENABLE_OAUTH_SIGNUP: true
      ENABLE_LOGIN_FORM: false
      ENABLE_SIGNUP: false
      OAUTH_MERGE_ACCOUNTS_BY_EMAIL: true
      OAUTH_USERNAME_CLAIM: name
      OAUTH_CLIENT_ID: ollama
      OAUTH_CLIENT_SECRET:
      OPENID_PROVIDER_URL: https://$KEYCLOAKURL/auth/realms/$REALMNAME/.well-known/openid-configuration
      OAUTH_PROVIDER_NAME: Keycloak
      REQUESTS_CA_BUNDLE: /etc/ssl/certs/ca-certificates.crt
    extra_hosts:
      - host.docker.internal:host-gateway
    restart: unless-stopped
    networks:
      - backend
      - frontend

    labels:
      - traefik.enable=true
      - traefik.http.routers.ollama.rule=Host(`$HOST_NAME`)
      - traefik.http.routers.ollama.tls.certresolver=internal-acme
      - traefik.http.routers.ollama.service=ollama
      - traefik.http.routers.ollama.entrypoints=https
      - traefik.http.services.ollama.loadbalancer.server.port=8080

@christiangierschner commented on GitHub (Aug 7, 2024): ``` services: ollama: image: ghcr.io/open-webui/open-webui:v0.3.11 volumes: - ollama-webui:/app/backend/data - /usr/local/share/ca-certificates/rootca.crt:/etc/ssl/certs/ca-certificates.crt:ro environment: OLLAMA_BASE_URL: http://ollama-api:11434 ENABLE_OAUTH_SIGNUP: true ENABLE_LOGIN_FORM: false ENABLE_SIGNUP: false OAUTH_MERGE_ACCOUNTS_BY_EMAIL: true OAUTH_USERNAME_CLAIM: name OAUTH_CLIENT_ID: ollama OAUTH_CLIENT_SECRET: OPENID_PROVIDER_URL: https://$KEYCLOAKURL/auth/realms/$REALMNAME/.well-known/openid-configuration OAUTH_PROVIDER_NAME: Keycloak REQUESTS_CA_BUNDLE: /etc/ssl/certs/ca-certificates.crt extra_hosts: - host.docker.internal:host-gateway restart: unless-stopped networks: - backend - frontend labels: - traefik.enable=true - traefik.http.routers.ollama.rule=Host(`$HOST_NAME`) - traefik.http.routers.ollama.tls.certresolver=internal-acme - traefik.http.routers.ollama.service=ollama - traefik.http.routers.ollama.entrypoints=https - traefik.http.services.ollama.loadbalancer.server.port=8080

GiteaMirror commented

@nomppy commented on GitHub (Aug 7, 2024):

Is /usr/local/share/ca-certificates/roocat.crt the complete CA used by your system. If it's just your custom root CA, then you are not giving python any of the usual certs usually present in /etc/ssl/certs/ca-certificates.crt.

Can you try update-ca-certificates on your host and then mapping /etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt:ro?

@nomppy commented on GitHub (Aug 7, 2024): Is `/usr/local/share/ca-certificates/roocat.crt` the complete CA used by your system. If it's just your custom root CA, then you are not giving python any of the usual certs usually present in /etc/ssl/certs/ca-certificates.crt. Can you try `update-ca-certificates` on your host and then mapping `/etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt:ro`?

GiteaMirror commented

@christiangierschner commented on GitHub (Aug 9, 2024):

i tried both...none works
our hosts all have our rootca implmented with update-ca-certificates

i get Internal Server Error and SSL: CERTIFICATE_VERIFY_FAILED

services:
  ollama:
    image: ghcr.io/open-webui/open-webui:v0.3.12
    volumes:
      - ollama-webui:/app/backend/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
      - /etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt:ro
      - /usr/local/share/ca-certificates/rootca.crt:/usr/local/share/ca-certificates/rootca.crt
    environment:
      OLLAMA_BASE_URL: http://ollama-api:11434
      ENABLE_OAUTH_SIGNUP: true
      ENABLE_LOGIN_FORM: false
      ENABLE_SIGNUP: false
      OAUTH_MERGE_ACCOUNTS_BY_EMAIL: true
      OAUTH_USERNAME_CLAIM: name
      OAUTH_CLIENT_ID: ollama
      OAUTH_CLIENT_SECRET:
      OPENID_PROVIDER_URL: https://$KEYCLOAKURL/auth/realms/$REALMNAME/.well-known/openid-configuration
      OAUTH_PROVIDER_NAME: Keycloak
      REQUESTS_CA_BUNDLE: /etc/ssl/certs/ca-certificates.crt
    extra_hosts:
      - host.docker.internal:host-gateway
    restart: unless-stopped
    networks:
      - backend
      - frontend

    labels:
      - traefik.enable=true
      - traefik.http.routers.ollama.rule=Host(`$HOST_NAME`)
      - traefik.http.routers.ollama.tls.certresolver=ukd-internal
      - traefik.http.routers.ollama.service=ollama
      - traefik.http.routers.ollama.entrypoints=https
      - traefik.http.services.ollama.loadbalancer.server.port=8080

@christiangierschner commented on GitHub (Aug 9, 2024): i tried both...none works our hosts all have our rootca implmented with ```update-ca-certificates``` i get Internal Server Error and SSL: CERTIFICATE_VERIFY_FAILED ``` services: ollama: image: ghcr.io/open-webui/open-webui:v0.3.12 volumes: - ollama-webui:/app/backend/data - /etc/timezone:/etc/timezone:ro - /etc/localtime:/etc/localtime:ro - /etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt:ro - /usr/local/share/ca-certificates/rootca.crt:/usr/local/share/ca-certificates/rootca.crt environment: OLLAMA_BASE_URL: http://ollama-api:11434 ENABLE_OAUTH_SIGNUP: true ENABLE_LOGIN_FORM: false ENABLE_SIGNUP: false OAUTH_MERGE_ACCOUNTS_BY_EMAIL: true OAUTH_USERNAME_CLAIM: name OAUTH_CLIENT_ID: ollama OAUTH_CLIENT_SECRET: OPENID_PROVIDER_URL: https://$KEYCLOAKURL/auth/realms/$REALMNAME/.well-known/openid-configuration OAUTH_PROVIDER_NAME: Keycloak REQUESTS_CA_BUNDLE: /etc/ssl/certs/ca-certificates.crt extra_hosts: - host.docker.internal:host-gateway restart: unless-stopped networks: - backend - frontend labels: - traefik.enable=true - traefik.http.routers.ollama.rule=Host(`$HOST_NAME`) - traefik.http.routers.ollama.tls.certresolver=ukd-internal - traefik.http.routers.ollama.service=ollama - traefik.http.routers.ollama.entrypoints=https - traefik.http.services.ollama.loadbalancer.server.port=8080

GiteaMirror commented

@nomppy commented on GitHub (Aug 9, 2024):

Can you manually check that /etcssl/certs/ca-certificates.crt contains the cert in rootca.crt?
How did you obtain rootca.crt?
Do you can SSL error only when you click login with oAuth, or also for other things? I.e., can you try to log in with username/password and see if you get an error for requests to e.g. OpenAI?

@nomppy commented on GitHub (Aug 9, 2024): - Can you manually check that `/etcssl/certs/ca-certificates.crt` contains the cert in rootca.crt? - How did you obtain `rootca.crt`? - Do you can SSL error only when you click login with oAuth, or also for other things? I.e., can you try to log in with username/password and see if you get an error for requests to e.g. OpenAI?

GiteaMirror commented