github-starred/komodo
2
0
Fork 1
mirror of https://github.com/moghtech/komodo.git synced 2026-05-06 08:55:40 -05:00
Code Issues 3.5k Packages Projects Releases 103 Wiki Activity

[GH-ISSUE #130] Self signed certificates for OIDC #5861

New Issue
Closed
opened 2026-04-24 18:50:26 -05:00 by GiteaMirror · 24 comments
GiteaMirror commented 2026-04-24 18:50:26 -05:00
Owner
Copy Link

Originally created by @tedstriker on GitHub (Oct 16, 2024).
Original GitHub issue: https://github.com/moghtech/komodo/issues/130

Is there a way to use accept self signed certificates of the authentication server when using OIDC?

Usually a CA certificate chain can be added with reference to the file with environment variables like SSL_CERT_FILE: /etc/ssl/certs/ca-certificates.crt for PHP based projects or NODE_EXTRA_CA_CERTS=/etc/ssl/certs/ca-certificates.crt for NodeJS based ones.

With Komodo all those known variables don't work and I get the following error:

thread 'tokio-runtime-worker' panicked at bin/core/src/auth/oidc/client.rs:66:4:
called `Result::unwrap()` on an `Err` value: Failed to init default OIDC client
Caused by:
    0: Failed to get OIDC /.well-known/openid-configuration
    1: Request failed
    2: request failed
    3: error sending request for url (https://auth.example.com/application/o/komodo/.well-known/openid-configuration): error trying to connect: invalid peer certificate: UnknownIssuer
    4: error trying to connect: invalid peer certificate: UnknownIssuer
    5: invalid peer certificate: UnknownIssuer
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Error: task 17 panicked

Any ideas how to make the self signed CA known to the app?

Originally created by @tedstriker on GitHub (Oct 16, 2024). Original GitHub issue: https://github.com/moghtech/komodo/issues/130 Is there a way to use accept self signed certificates of the authentication server when using OIDC? Usually a CA certificate chain can be added with reference to the file with environment variables like `SSL_CERT_FILE: /etc/ssl/certs/ca-certificates.crt` for PHP based projects or `NODE_EXTRA_CA_CERTS=/etc/ssl/certs/ca-certificates.crt` for NodeJS based ones. With Komodo all those known variables don't work and I get the following error: ``` thread 'tokio-runtime-worker' panicked at bin/core/src/auth/oidc/client.rs:66:4: called `Result::unwrap()` on an `Err` value: Failed to init default OIDC client Caused by: 0: Failed to get OIDC /.well-known/openid-configuration 1: Request failed 2: request failed 3: error sending request for url (https://auth.example.com/application/o/komodo/.well-known/openid-configuration): error trying to connect: invalid peer certificate: UnknownIssuer 4: error trying to connect: invalid peer certificate: UnknownIssuer 5: invalid peer certificate: UnknownIssuer note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace Error: task 17 panicked ``` Any ideas how to make the self signed CA known to the app?
GiteaMirror added the bugsetup labels 2026-04-24 18:50:26 -05:00
GiteaMirror commented 2026-04-24 18:50:28 -05:00
Author
Owner
Copy Link

@mbecker20 commented on GitHub (Oct 16, 2024):

Hey, you can package your root CA certificate into the container with some simple modifications. Currently from setup you should have two files {database}.compose.yaml and compose.env.

Add your root cert root.crt next to these ones, and create an an additional file called core.Dockerfile with these contents:

FROM ghcr.io/mbecker20/komodo:latest # or latest-aarch64
ADD root.crt:/etc/ssl/certs
RUN update-ca-certificates

Now modify the core: service in {database}.compose.yaml:

services:
  ...(unchanged)

  core:
    ## Comment out 'image' and replace with 'build'
    # image: ghcr.io/mbecker20/komodo:${COMPOSE_KOMODO_IMAGE_TAG:-latest}
    build:
      context: .
      dockerfile: core.Dockerfile
    ...(unchanged)

This will also allow git to work with an internal git provider behind an internal CA. Users running Periphery in container can do the same thing for Periphery to allow it to reach internal git provider. Periphery running in systemd will just use the host certs, so the root CA needs to be added to /etc/ssl/certs and run update-ca-certificates (basically what is happening in the Dockerfile)

From https://stackoverflow.com/questions/26028971/docker-container-ssl-certificates

<!-- gh-comment-id:2417266896 --> @mbecker20 commented on GitHub (Oct 16, 2024): Hey, you can package your root CA certificate into the container with some simple modifications. Currently from setup you should have two files `{database}.compose.yaml` and `compose.env`. Add your root cert `root.crt` next to these ones, and create an an additional file called `core.Dockerfile` with these contents: ``` FROM ghcr.io/mbecker20/komodo:latest # or latest-aarch64 ADD root.crt:/etc/ssl/certs RUN update-ca-certificates ``` Now modify the `core:` service in `{database}.compose.yaml`: ``` services: ...(unchanged) core: ## Comment out 'image' and replace with 'build' # image: ghcr.io/mbecker20/komodo:${COMPOSE_KOMODO_IMAGE_TAG:-latest} build: context: . dockerfile: core.Dockerfile ...(unchanged) ``` This will also allow `git` to work with an internal git provider behind an internal CA. Users running Periphery in container can do the same thing for Periphery to allow it to reach internal git provider. Periphery running in `systemd` will just use the host certs, so the root CA needs to be added to `/etc/ssl/certs` and run `update-ca-certificates` (basically what is happening in the Dockerfile) From https://stackoverflow.com/questions/26028971/docker-container-ssl-certificates
GiteaMirror commented 2026-04-24 18:50:29 -05:00
Author
Owner
Copy Link

@tedstriker commented on GitHub (Nov 4, 2024):

Thank you for your elaborate explanation.

Usually something like this:

volumes:
      - /etc/ssl/certs/myCAfile.pem:/usr/local/share/ca-certificates/myCAfile.crt:ro
      - /etc/ssl/certs/myCAfile.pem:/etc/ssl/certs/c4d1e9a0.0:ro
      - /etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt:ro

is enough to introduce a new certificates to the system.
The 2nd line is just the same file as in the first line. For some reason update-ca-certificates uses the files hash as its filename. Most times it is neccesary to reference those certificates in specific ENV vars like mentioned initially. update-ca-certificates run on the host in that case.

Is there a possibility to just bind the ca certificate file via volumes in the docker-compose.yaml instead of creating an entire image just to add a certificate file in the appropriate place?

<!-- gh-comment-id:2455450565 --> @tedstriker commented on GitHub (Nov 4, 2024): Thank you for your elaborate explanation. Usually something like this: ``` volumes: - /etc/ssl/certs/myCAfile.pem:/usr/local/share/ca-certificates/myCAfile.crt:ro - /etc/ssl/certs/myCAfile.pem:/etc/ssl/certs/c4d1e9a0.0:ro - /etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt:ro ``` is enough to introduce a new certificates to the system. The 2nd line is just the same file as in the first line. For some reason `update-ca-certificates` uses the files hash as its filename. Most times it is neccesary to reference those certificates in specific ENV vars like mentioned initially. `update-ca-certificates` run on the host in that case. Is there a possibility to just bind the ca certificate file via `volumes` in the `docker-compose.yaml` instead of creating an entire image just to add a certificate file in the appropriate place?
GiteaMirror commented 2026-04-24 18:50:31 -05:00
Author
Owner
Copy Link

@tedstriker commented on GitHub (Nov 4, 2024):

I got update-ca-certificates running inside the container. It added the certificate properly, but unfortunately it is not recognized by Komodo.

These changes to the compose file seemed to work.

services:
  ...
  core:
    entrypoint: |
    sh -c "update-ca-certificates ; /app/core"
  ...
volumes:
  - /etc/ssl/certs/myCAfile.pem:/usr/local/share/ca-certificates/myCAfile.crt:ro

Do you have another idea what could be tried next?

here's the log excerpt after a fresh container start:

Updating certificates in /etc/ssl/certs...
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
2024-11-04T19:04:58.298682Z  INFO core: Komodo Core version: v1.16.8
2024-11-04T19:04:58.298712Z  INFO core: CoreConfig { title: "Komodo", host: "https://komodo.example.com", port: 9120, passkey: "##############", ui_write_disabled: false, disable_confirm_dialog: false, first_server: "https://periphery:8120", frontend_path: "/app/frontend", database: DatabaseConfig { uri: "", address: "mongo:27017", username: "##############", password: "##############", app_name: "komodo_core", db_name: "komodo" }, local_auth: true, transparent_mode: false, enable_new_users: true, disable_user_registration: false, disable_non_admin_create: false, jwt_secret: "##############", jwt_ttl: ThreeDay, oidc_enabled: true, oidc_provider: "https://sso.example.com/application/o/komodo/", oidc_redirect_host: "", oidc_client_id: "##############", oidc_client_secret: "##############", oidc_use_full_email: false, oidc_additional_audiences: [], google_oauth: OauthCredentials { enabled: false, id: "", secret: "" }, github_oauth: OauthCredentials { enabled: false, id: "", secret: "" }, webhook_secret: "##############", webhook_base_url: "", github_webhook_app: GithubWebhookAppConfig { app_id: 0, installations: [], pk_path: "/github/private-key.pem" }, logging: LogConfig { level: Info, stdio: Standard, otlp_endpoint: "", opentelemetry_service_name: "Komodo" }, keep_stats_for_days: 14, keep_alerts_for_days: 14, resource_poll_interval: FiveMinutes, monitoring_interval: FifteenSeconds, aws: AwsCredentials { access_key_id: "", secret_access_key: "" }, hetzner: HetznerCredentials { token: "" }, git_providers: [], docker_registries: [], secrets: {}, ssl_enabled: false, ssl_key_file: "/config/ssl/key.pem", ssl_cert_file: "/config/ssl/cert.pem", sync_directory: "/syncs", repo_directory: "/repo-cache", action_directory: "/action-cache" }
thread 'tokio-runtime-worker' panicked at bin/core/src/auth/oidc/client.rs:66:4:
called `Result::unwrap()` on an `Err` value: Failed to init default OIDC client
Caused by:
    0: Failed to get OIDC /.well-known/openid-configuration
    1: Request failed
    2: request failed
    3: error sending request for url (https://sso.example.com/application/o/komodo/.well-known/openid-configuration): error trying to connect: invalid peer certificate: UnknownIssuer
    4: error trying to connect: invalid peer certificate: UnknownIssuer
    5: invalid peer certificate: UnknownIssuer
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Error: task 17 panicked
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
2024-11-04T19:05:03.833526Z  INFO core: Komodo Core version: v1.16.8
<!-- gh-comment-id:2455510705 --> @tedstriker commented on GitHub (Nov 4, 2024): I got `update-ca-certificates` running inside the container. It added the certificate properly, but unfortunately it is not recognized by Komodo. These changes to the compose file seemed to work. ``` services: ... core: entrypoint: | sh -c "update-ca-certificates ; /app/core" ... volumes: - /etc/ssl/certs/myCAfile.pem:/usr/local/share/ca-certificates/myCAfile.crt:ro ``` Do you have another idea what could be tried next? here's the log excerpt after a fresh container start: ``` Updating certificates in /etc/ssl/certs... 1 added, 0 removed; done. Running hooks in /etc/ca-certificates/update.d... done. 2024-11-04T19:04:58.298682Z INFO core: Komodo Core version: v1.16.8 2024-11-04T19:04:58.298712Z INFO core: CoreConfig { title: "Komodo", host: "https://komodo.example.com", port: 9120, passkey: "##############", ui_write_disabled: false, disable_confirm_dialog: false, first_server: "https://periphery:8120", frontend_path: "/app/frontend", database: DatabaseConfig { uri: "", address: "mongo:27017", username: "##############", password: "##############", app_name: "komodo_core", db_name: "komodo" }, local_auth: true, transparent_mode: false, enable_new_users: true, disable_user_registration: false, disable_non_admin_create: false, jwt_secret: "##############", jwt_ttl: ThreeDay, oidc_enabled: true, oidc_provider: "https://sso.example.com/application/o/komodo/", oidc_redirect_host: "", oidc_client_id: "##############", oidc_client_secret: "##############", oidc_use_full_email: false, oidc_additional_audiences: [], google_oauth: OauthCredentials { enabled: false, id: "", secret: "" }, github_oauth: OauthCredentials { enabled: false, id: "", secret: "" }, webhook_secret: "##############", webhook_base_url: "", github_webhook_app: GithubWebhookAppConfig { app_id: 0, installations: [], pk_path: "/github/private-key.pem" }, logging: LogConfig { level: Info, stdio: Standard, otlp_endpoint: "", opentelemetry_service_name: "Komodo" }, keep_stats_for_days: 14, keep_alerts_for_days: 14, resource_poll_interval: FiveMinutes, monitoring_interval: FifteenSeconds, aws: AwsCredentials { access_key_id: "", secret_access_key: "" }, hetzner: HetznerCredentials { token: "" }, git_providers: [], docker_registries: [], secrets: {}, ssl_enabled: false, ssl_key_file: "/config/ssl/key.pem", ssl_cert_file: "/config/ssl/cert.pem", sync_directory: "/syncs", repo_directory: "/repo-cache", action_directory: "/action-cache" } thread 'tokio-runtime-worker' panicked at bin/core/src/auth/oidc/client.rs:66:4: called `Result::unwrap()` on an `Err` value: Failed to init default OIDC client Caused by: 0: Failed to get OIDC /.well-known/openid-configuration 1: Request failed 2: request failed 3: error sending request for url (https://sso.example.com/application/o/komodo/.well-known/openid-configuration): error trying to connect: invalid peer certificate: UnknownIssuer 4: error trying to connect: invalid peer certificate: UnknownIssuer 5: invalid peer certificate: UnknownIssuer note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace Error: task 17 panicked Updating certificates in /etc/ssl/certs... 0 added, 0 removed; done. Running hooks in /etc/ca-certificates/update.d... done. 2024-11-04T19:05:03.833526Z INFO core: Komodo Core version: v1.16.8 ```
GiteaMirror commented 2026-04-24 18:50:31 -05:00
Author
Owner
Copy Link

@mbecker20 commented on GitHub (Jan 3, 2025):

Hey, apologies for the long delay. There is a reason I mention to do it via custom build. I went through same process as you, and it didn't work like you notice. But finally I tried the method I recommended to you, because I saw it mentioned in that stack overflow link before. I think it makes a difference whether the crt is added as a layer via custom build, versus mounted into container.

You can avoid a dedicated dockerfile by using dockerfile_inline:

services:
  ...(unchanged)

  core:
    ## Comment out 'image' and replace with 'build'
    # image: ghcr.io/mbecker20/komodo:${COMPOSE_KOMODO_IMAGE_TAG:-latest}
    build:
      context: /folder/containing/root_ca # use a context with the root ca on the host to deploy core on.
      dockerfile_inline: |
          FROM ghcr.io/mbecker20/komodo:latest
          ADD root_ca.crt:/etc/ssl/certs
          RUN update-ca-certificates
    ...(unchanged)

Please let me know how it works out for you, or whether you already got it to work another way. I definitely didn't try everything possible using mounts instead of custom build, figuring out this method would also be appreciated!

<!-- gh-comment-id:2569821581 --> @mbecker20 commented on GitHub (Jan 3, 2025): Hey, apologies for the long delay. There is a reason I mention to do it via custom build. I went through same process as you, and it didn't work like you notice. But finally I tried the method I recommended to you, because I saw it mentioned in that stack overflow link before. I think it makes a difference whether the crt is added as a layer via custom build, versus mounted into container. You can avoid a dedicated dockerfile by using `dockerfile_inline`: ```yaml services: ...(unchanged) core: ## Comment out 'image' and replace with 'build' # image: ghcr.io/mbecker20/komodo:${COMPOSE_KOMODO_IMAGE_TAG:-latest} build: context: /folder/containing/root_ca # use a context with the root ca on the host to deploy core on. dockerfile_inline: | FROM ghcr.io/mbecker20/komodo:latest ADD root_ca.crt:/etc/ssl/certs RUN update-ca-certificates ...(unchanged) ``` Please let me know how it works out for you, or whether you already got it to work another way. I definitely didn't try everything possible using mounts instead of custom build, figuring out this method would also be appreciated!
GiteaMirror commented 2026-04-24 18:50:32 -05:00
Author
Owner
Copy Link

@tedstriker commented on GitHub (Jan 13, 2025):

No worries, and thank you for taking the time to elaborate on your approach and provide an example.

I gave it a try and confirmed that the certificate has been successfully copied and populated using update-ca-certificates with wget in the console. It worked for wget.

Unfortunately, the outcome remains the same as when mounting the certificate files directly, when it comes to enabling OIDC. They just get ignored.

It seems like the module responsible for establishing the HTTPS connection has its own certificate store. So I had a look around.
This might have something to do with the reqwest module and the issue discussed here.

Key Note from the Linked Issue

The post mentions using the rustls-tls-native-roots feature with reqwest to enable the use of OS-native root certificates:

reqwest = { version = "0.11", default-features = false, features = ["rustls-tls-native-roots"] }

This is further explained in the documentation here.

Its successor, rustls-platform-verifier, is the recommended package to replace rustls-tls-native-roots.

To summarize, while I’m not a Rust developer, it might be worth trying to modify line 50 in Cargo.toml to use the following configuration:

reqwest = { version = "0.12.9", default-features = false, features = ["json", "rustls-platform-verifier"] }

This would use rustls-platform-verifier instead.

What do you think about this approach?

<!-- gh-comment-id:2588368668 --> @tedstriker commented on GitHub (Jan 13, 2025): No worries, and thank you for taking the time to elaborate on your approach and provide an example. I gave it a try and confirmed that the certificate has been successfully copied and populated using `update-ca-certificates` with `wget` in the console. It worked for `wget`. Unfortunately, the outcome remains the same as when mounting the certificate files directly, when it comes to enabling OIDC. They just get ignored. It seems like the module responsible for establishing the HTTPS connection has its own certificate store. So I had a look around. This might have something to do with the `reqwest` module and the issue discussed [here](https://github.com/seanmonstar/reqwest/issues/1554). ### Key Note from the Linked Issue The post mentions using the `rustls-tls-native-roots` feature with `reqwest` to enable the use of OS-native root certificates: ```toml reqwest = { version = "0.11", default-features = false, features = ["rustls-tls-native-roots"] } ``` This is further explained in the documentation [here](https://crates.io/crates/rustls-native-certs). Its successor, [rustls-platform-verifier](https://github.com/rustls/rustls-platform-verifier), is the recommended package to replace `rustls-tls-native-roots`. To summarize, while I’m not a Rust developer, it might be worth trying to modify line 50 in [Cargo.toml](https://github.com/mbecker20/komodo/blob/main/Cargo.toml) to use the following configuration: ``` reqwest = { version = "0.12.9", default-features = false, features = ["json", "rustls-platform-verifier"] } ``` This would use `rustls-platform-verifier` instead. What do you think about this approach?
GiteaMirror commented 2026-04-24 18:50:34 -05:00
Author
Owner
Copy Link

@mbecker20 commented on GitHub (Jan 15, 2025):

Thanks for looking into that, I think you are correct about the issue being reqwest crate using rustls not picking up certs.

Komodo uses the openidconnect-rs rust crate for the OIDC implementation: https://github.com/ramosbugs/openidconnect-rs. It depends on this OIDC crate's features, given here: 5632960ea9/Cargo.toml (L27)

The OIDC crate then depends on a lower level oauth2-rs crate (made by same author), this is the crate which chooses the features which reqwest uses: 9a2b746f76/Cargo.toml (L23)

I think those crates need to be updated to allow crate consumers to choose rustls-native-roots feature of reqwest crate: 3099f30f08/Cargo.toml (L52). Maybe this issue could be brought to @ramosbugs attention?

<!-- gh-comment-id:2591786483 --> @mbecker20 commented on GitHub (Jan 15, 2025): Thanks for looking into that, I think you are correct about the issue being reqwest crate using rustls not picking up certs. Komodo uses the `openidconnect-rs` rust crate for the OIDC implementation: https://github.com/ramosbugs/openidconnect-rs. It depends on this OIDC crate's features, given here: https://github.com/ramosbugs/openidconnect-rs/blob/5632960ea94ceda3a17c818d45c26648cc6bee8b/Cargo.toml#L27 The OIDC crate then depends on a lower level `oauth2-rs` crate (made by same author), this is the crate which chooses the features which reqwest uses: https://github.com/ramosbugs/oauth2-rs/blob/9a2b746f76c5d0f9a7a02a1916bd509668fcaee3/Cargo.toml#L23 I think those crates need to be updated to allow crate consumers to choose `rustls-native-roots` feature of `reqwest` crate: https://github.com/seanmonstar/reqwest/blob/3099f30f089854f8109c49e13898d2f88b9fc0f2/Cargo.toml#L52. Maybe this issue could be brought to @ramosbugs attention?
GiteaMirror commented 2026-04-24 18:50:34 -05:00
Author
Owner
Copy Link

@tedstriker commented on GitHub (Jan 16, 2025):

Tried to get in touch and it got declined. From what I've seen it seems that no change on the oauth2-rs side is necessary.
Regarding the why I'm lost.
Maybe the reasoning and explanation in the related closed merge request tells you more than it does for me?

<!-- gh-comment-id:2596137609 --> @tedstriker commented on GitHub (Jan 16, 2025): Tried to get in touch and it got declined. From what I've seen it _seems_ that no change on the oauth2-rs side is necessary. Regarding the _why_ I'm lost. Maybe the reasoning and explanation in the related [closed merge request](https://github.com/ramosbugs/oauth2-rs/pull/287) tells you more than it does for me?
GiteaMirror commented 2026-04-24 18:50:35 -05:00
Author
Owner
Copy Link

@tylerhunt commented on GitHub (Jan 20, 2025):

@mbecker20 This comment makes it sounds like this can be done by the application without requiring the libraries in the middle to be updated: https://github.com/ramosbugs/oauth2-rs/pull/287#issuecomment-2380046784

<!-- gh-comment-id:2601300009 --> @tylerhunt commented on GitHub (Jan 20, 2025): @mbecker20 This comment makes it sounds like this can be done by the application without requiring the libraries in the middle to be updated: https://github.com/ramosbugs/oauth2-rs/pull/287#issuecomment-2380046784
GiteaMirror commented 2026-04-24 18:50:36 -05:00
Author
Owner
Copy Link

@tylerhunt commented on GitHub (Jan 28, 2025):

I had to make a couple changes to the suggested inline Dockerfile to get the certificate registered properly:

services:
  core:
    build:
      dockerfile_inline: |
        FROM ghcr.io/mbecker20/komodo:${COMPOSE_KOMODO_IMAGE_TAG:-latest}
        ADD certs /usr/local/share/ca-certificates
        RUN update-ca-certificates

I'm now able to access the OIDC server from within the container:

localhost:~/komodo# docker exec -it komodo-core /bin/bash
root@666c00403029:/app# openssl s_client -verify_return_error -connect id.home.arpa:443 < /dev/null
…
SSL handshake has read 2228 bytes and written 368 bytes
Verification: OK

I also updated the Cargo.toml files with the rustls-tls-native-roots feature and rebuilt the project.

diff --git a/Cargo.toml b/Cargo.toml
index 91e88896..f9399760 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -47,7 +47,7 @@ mungos = "1.1.0"
 svi = "1.0.1"
 
 # ASYNC
-reqwest = { version = "0.12.9", default-features = false, features = ["json", "rustls-tls"] }
+reqwest = { version = "0.12.9", default-features = false, features = ["json", "rustls-tls", "rustls-tls-native-roots"] }
 tokio = { version = "1.41.1", features = ["full"] }
 tokio-util = "0.7.12"
 futures = "0.3.31"
@@ -94,6 +94,7 @@ nom_pem = "4.0.0"
 bcrypt = "0.16.0"
 base64 = "0.22.1"
 rustls = "0.23.18"
+rustls-native-certs = "0.8.1"
 hmac = "0.12.1"
 sha2 = "0.10.8"
 rand = "0.8.5"
diff --git a/bin/core/Cargo.toml b/bin/core/Cargo.toml
index cec4c2bf..150d340a 100644
--- a/bin/core/Cargo.toml
+++ b/bin/core/Cargo.toml
@@ -60,6 +60,7 @@ anyhow.workspace = true
 bcrypt.workspace = true
 base64.workspace = true
 rustls.workspace = true
+rustls-native-certs.workspace = true
 tokio.workspace = true
 serde.workspace = true
 regex.workspace = true

But this still doesn't solve the invalid peer certificate: UnknownIssuer error when the OIDC client tries to connect.

Docker log output 
komodo-core        | thread 'tokio-runtime-worker' panicked at bin/core/src/auth/oidc/client.rs:66:4:
komodo-core        | called `Result::unwrap()` on an `Err` value: Failed to init default OIDC client
komodo-core        | 
komodo-core        | Caused by:
komodo-core        |     0: Failed to get OIDC /.well-known/openid-configuration
komodo-core        |     1: Request failed
komodo-core        |     2: request failed
komodo-core        |     3: error sending request for url (https://id.home.arpa/application/o/komodo/.well-known/openid-configuration): error trying to connect: invalid peer certificate: UnknownIssuer
komodo-core        |     4: error trying to connect: invalid peer certificate: UnknownIssuer
komodo-core        |     5: invalid peer certificate: UnknownIssuer
komodo-core        | 
komodo-core        | Stack backtrace:
komodo-core        |    0: <tokio::future::maybe_done::MaybeDone<Fut> as core::future::future::Future>::poll
komodo-core        |    1: <core::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
komodo-core        |    2: core::app::{{closure}}
komodo-core        |    3: tokio::runtime::task::core::Core<T,S>::poll
komodo-core        |    4: tokio::runtime::task::harness::Harness<T,S>::poll
komodo-core        |    5: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
komodo-core        |    6: tokio::runtime::scheduler::multi_thread::worker::Context::run
komodo-core        |    7: tokio::runtime::context::runtime::enter_runtime
komodo-core        |    8: tokio::runtime::scheduler::multi_thread::worker::run
komodo-core        |    9: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
komodo-core        |   10: tokio::runtime::task::core::Core<T,S>::poll
komodo-core        |   11: tokio::runtime::task::harness::Harness<T,S>::poll
komodo-core        |   12: tokio::runtime::blocking::pool::Inner::run
komodo-core        |   13: std::sys::backtrace::__rust_begin_short_backtrace
komodo-core        |   14: core::ops::function::FnOnce::call_once{{vtable.shim}}
komodo-core        |   15: std::sys::pal::unix::thread::Thread::new::thread_start
komodo-core        |   16: start_thread
komodo-core        |   17: clone
<!-- gh-comment-id:2620032305 --> @tylerhunt commented on GitHub (Jan 28, 2025): I had to make a couple changes to the suggested inline `Dockerfile` to get the certificate registered properly: ```yaml services: core: build: dockerfile_inline: | FROM ghcr.io/mbecker20/komodo:${COMPOSE_KOMODO_IMAGE_TAG:-latest} ADD certs /usr/local/share/ca-certificates RUN update-ca-certificates ``` I'm now able to access the OIDC server from within the container: ``` localhost:~/komodo# docker exec -it komodo-core /bin/bash root@666c00403029:/app# openssl s_client -verify_return_error -connect id.home.arpa:443 < /dev/null … SSL handshake has read 2228 bytes and written 368 bytes Verification: OK ``` I also updated the `Cargo.toml` files with the `rustls-tls-native-roots` feature and rebuilt the project. ```diff diff --git a/Cargo.toml b/Cargo.toml index 91e88896..f9399760 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -47,7 +47,7 @@ mungos = "1.1.0" svi = "1.0.1" # ASYNC -reqwest = { version = "0.12.9", default-features = false, features = ["json", "rustls-tls"] } +reqwest = { version = "0.12.9", default-features = false, features = ["json", "rustls-tls", "rustls-tls-native-roots"] } tokio = { version = "1.41.1", features = ["full"] } tokio-util = "0.7.12" futures = "0.3.31" @@ -94,6 +94,7 @@ nom_pem = "4.0.0" bcrypt = "0.16.0" base64 = "0.22.1" rustls = "0.23.18" +rustls-native-certs = "0.8.1" hmac = "0.12.1" sha2 = "0.10.8" rand = "0.8.5" diff --git a/bin/core/Cargo.toml b/bin/core/Cargo.toml index cec4c2bf..150d340a 100644 --- a/bin/core/Cargo.toml +++ b/bin/core/Cargo.toml @@ -60,6 +60,7 @@ anyhow.workspace = true bcrypt.workspace = true base64.workspace = true rustls.workspace = true +rustls-native-certs.workspace = true tokio.workspace = true serde.workspace = true regex.workspace = true ``` But this still doesn't solve the `invalid peer certificate: UnknownIssuer` error when the OIDC client tries to connect. <details> <summary>Docker log output</summary> ``` komodo-core | thread 'tokio-runtime-worker' panicked at bin/core/src/auth/oidc/client.rs:66:4: komodo-core | called `Result::unwrap()` on an `Err` value: Failed to init default OIDC client komodo-core | komodo-core | Caused by: komodo-core | 0: Failed to get OIDC /.well-known/openid-configuration komodo-core | 1: Request failed komodo-core | 2: request failed komodo-core | 3: error sending request for url (https://id.home.arpa/application/o/komodo/.well-known/openid-configuration): error trying to connect: invalid peer certificate: UnknownIssuer komodo-core | 4: error trying to connect: invalid peer certificate: UnknownIssuer komodo-core | 5: invalid peer certificate: UnknownIssuer komodo-core | komodo-core | Stack backtrace: komodo-core | 0: <tokio::future::maybe_done::MaybeDone<Fut> as core::future::future::Future>::poll komodo-core | 1: <core::future::poll_fn::PollFn<F> as core::future::future::Future>::poll komodo-core | 2: core::app::{{closure}} komodo-core | 3: tokio::runtime::task::core::Core<T,S>::poll komodo-core | 4: tokio::runtime::task::harness::Harness<T,S>::poll komodo-core | 5: tokio::runtime::scheduler::multi_thread::worker::Context::run_task komodo-core | 6: tokio::runtime::scheduler::multi_thread::worker::Context::run komodo-core | 7: tokio::runtime::context::runtime::enter_runtime komodo-core | 8: tokio::runtime::scheduler::multi_thread::worker::run komodo-core | 9: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll komodo-core | 10: tokio::runtime::task::core::Core<T,S>::poll komodo-core | 11: tokio::runtime::task::harness::Harness<T,S>::poll komodo-core | 12: tokio::runtime::blocking::pool::Inner::run komodo-core | 13: std::sys::backtrace::__rust_begin_short_backtrace komodo-core | 14: core::ops::function::FnOnce::call_once{{vtable.shim}} komodo-core | 15: std::sys::pal::unix::thread::Thread::new::thread_start komodo-core | 16: start_thread komodo-core | 17: clone ``` </details>
GiteaMirror commented 2026-04-24 18:50:37 -05:00
Author
Owner
Copy Link

@tedstriker commented on GitHub (Feb 1, 2025):

The problem wasn't that the certificates weren't registered properly on OS level, because this works alright.
Issue is, they don't get recognized by komodo itself, due to the discussed behavior of reqwest.

I wonder what happend, if the usage of rustls-tls feature gets replaced entirely with rustls-tls-native-roots (or its successor: rustls-platform-verifier)
From there on, no reqwest call should be possible without supporting OS level ca-certificates, or did I miss something?

<!-- gh-comment-id:2628876720 --> @tedstriker commented on GitHub (Feb 1, 2025): The problem wasn't that the certificates weren't registered properly on OS level, because this works alright. Issue is, they don't get recognized by komodo itself, due to the discussed behavior of `reqwest`. I wonder what happend, if the usage of `rustls-tls` feature gets replaced entirely with `rustls-tls-native-roots` (or its successor: [rustls-platform-verifier](https://github.com/rustls/rustls-platform-verifier)) From there on, no `reqwest` call should be possible without supporting OS level ca-certificates, or did I miss something?
GiteaMirror commented 2026-04-24 18:50:37 -05:00
Author
Owner
Copy Link

@tylerhunt commented on GitHub (Feb 1, 2025):

rustls-tls-native-roots isn't a Cargo package, it's a feature of reqwest. It looks like what this does is configure use of the package rustls-native-certs.

I'm not sure rustls-platform-verifier is the answer here, as it looks to be designed to solve this problem on non-Linux platforms. Their README itself seems to suggest that if you're using Linux, then rustle-native-certs + webpki is the right choice.

<!-- gh-comment-id:2628964154 --> @tylerhunt commented on GitHub (Feb 1, 2025): `rustls-tls-native-roots` isn't a Cargo package, it's a [feature](https://github.com/seanmonstar/reqwest/blob/master/Cargo.toml#L52) of `reqwest`. It looks like what this does is configure use of the package `rustls-native-certs`. I'm not sure `rustls-platform-verifier` is the answer here, as it looks to be designed to solve this problem on non-Linux platforms. [Their README itself seems to suggest](https://github.com/rustls/rustls-platform-verifier?tab=readme-ov-file#deployment-considerations) that if you're using Linux, then `rustle-native-certs` + `webpki` is the right choice.
GiteaMirror commented 2026-04-24 18:50:38 -05:00
Author
Owner
Copy Link

@draeron commented on GitHub (Feb 1, 2025):

Definitly not an expert on rust, but my understanding is that pretty much all other languages either support native CA loading (ex: go) or can be set through somekind of ENV (ex: java, nodejs).

<!-- gh-comment-id:2629152052 --> @draeron commented on GitHub (Feb 1, 2025): Definitly not an expert on rust, but my understanding is that pretty much all other languages either support native CA loading (ex: go) or can be set through somekind of ENV (ex: java, nodejs).
GiteaMirror commented 2026-04-24 18:50:38 -05:00
Author
Owner
Copy Link

@mbecker20 commented on GitHub (Mar 6, 2025):

This should be fixed in 1.17.0-dev-4 release: https://github.com/moghtech/komodo/releases/tag/v1.17.0-dev-4. It now uses reqwest with rustls-tls-native-roots feature.

I tested with self hosted Zitadel behind TLS with internal certs and it worked.

<!-- gh-comment-id:2702716139 --> @mbecker20 commented on GitHub (Mar 6, 2025): This should be fixed in 1.17.0-dev-4 release: https://github.com/moghtech/komodo/releases/tag/v1.17.0-dev-4. It now uses reqwest with `rustls-tls-native-roots` feature. I tested with self hosted Zitadel behind TLS with internal certs and it worked.
GiteaMirror commented 2026-04-24 18:50:39 -05:00
Author
Owner
Copy Link

@tylerhunt commented on GitHub (Mar 7, 2025):

@mbecker20 Works for me, although the email address doesn't seem to be coming through when auto-registering a user. I'll create a new issue for that. (Never mind. It looks like my OIDC provider is passing through an empty email address.)

<!-- gh-comment-id:2707002198 --> @tylerhunt commented on GitHub (Mar 7, 2025): @mbecker20 Works for me, ~although the email address doesn't seem to be coming through when auto-registering a user. I'll create a new issue for that~. (Never mind. It looks like my OIDC provider is passing through an empty email address.)
GiteaMirror commented 2026-04-24 18:50:40 -05:00
Author
Owner
Copy Link

@somerandomguy17 commented on GitHub (Apr 6, 2025):

I'm on v1.17.0 and followed all the steps in this chain (adding the certificate via dockerfile_inline) and am still not able to get self-signed certs to work. It looks like the certificate is recognized inside the container

openssl s_client -verify_return_error -connect <reverse proxy> < /dev/null
CONNECTED(00000003)

---
SSL handshake has read 1379 bytes and written 386 bytes
Verification: OK
---

but the container fails to start with the following error:

called `Result::unwrap()` on an `Err` value: Failed to initialize OIDC client.

Caused by:

    0: Failed to get OIDC /.well-known/openid-configuration

    1: Request failed

    2: client error

    3: error sending request for url (https://<OIDC provider URL>/.well-known/openid-configuration)

    4: client error (Connect)

    5: invalid peer certificate: BadSignature

note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Error: task 5 panicked with message "called `Result::unwrap()` on an `Err` value: Failed to initialize OIDC client.\n\nCaused by:\n    0: Failed to get OIDC /.well-known/openid-configuration\n    1: Request failed\n    2: client error\n    3: error sending request for url (https://<OIDC provider URL>/.well-known/openid-configuration)\n    4: client error (Connect)\n    5: invalid peer certificate: BadSignature"`

It looks like komodo doesn't believe the cert is valid (bad signature) but this same certificate is used for other services with no issue, so I don't think its a certificate issue. Any idea on what I need to do to get this working?

Happy to create a new issue if necessary!

<!-- gh-comment-id:2781544224 --> @somerandomguy17 commented on GitHub (Apr 6, 2025): I'm on v1.17.0 and followed all the steps in this chain (adding the certificate via dockerfile_inline) and am still not able to get self-signed certs to work. It looks like the certificate is recognized inside the container ``` openssl s_client -verify_return_error -connect <reverse proxy> < /dev/null CONNECTED(00000003) --- SSL handshake has read 1379 bytes and written 386 bytes Verification: OK --- ``` but the container fails to start with the following error: ``` called `Result::unwrap()` on an `Err` value: Failed to initialize OIDC client. Caused by: 0: Failed to get OIDC /.well-known/openid-configuration 1: Request failed 2: client error 3: error sending request for url (https://<OIDC provider URL>/.well-known/openid-configuration) 4: client error (Connect) 5: invalid peer certificate: BadSignature note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace Error: task 5 panicked with message "called `Result::unwrap()` on an `Err` value: Failed to initialize OIDC client.\n\nCaused by:\n 0: Failed to get OIDC /.well-known/openid-configuration\n 1: Request failed\n 2: client error\n 3: error sending request for url (https://<OIDC provider URL>/.well-known/openid-configuration)\n 4: client error (Connect)\n 5: invalid peer certificate: BadSignature"` ``` It looks like komodo doesn't believe the cert is valid (bad signature) but this same certificate is used for other services with no issue, so I don't think its a certificate issue. Any idea on what I need to do to get this working? Happy to create a new issue if necessary!
GiteaMirror commented 2026-04-24 18:50:41 -05:00
Author
Owner
Copy Link

@tedstriker commented on GitHub (Apr 12, 2025):

I think the issue is a new error.
My impression is, that the ca certificate added to your os cert store and the certifcate presented from your OIDC provider aren't matching up properly.
Did you make 100% sure,

  • URLs to OIDC
  • provided ca certificates are valid and are the real parents of the server certificates, that are provided upon connection?
  • if a reverse proxy is involved: Does it become the endpoint of the TLS connection instead of the OIDC provider?
<!-- gh-comment-id:2798727678 --> @tedstriker commented on GitHub (Apr 12, 2025): I think the issue is a new error. My impression is, that the ca certificate added to your os cert store and the certifcate presented from your OIDC provider aren't matching up properly. Did you make 100% sure, - URLs to OIDC - provided ca certificates are valid and are the real parents of the server certificates, that are provided upon connection? - if a reverse proxy is involved: Does it become the endpoint of the TLS connection instead of the OIDC provider?
GiteaMirror commented 2026-04-24 18:50:41 -05:00
Author
Owner
Copy Link

@somerandomguy17 commented on GitHub (Apr 12, 2025):

There is a reverse proxy involved but it uses acertificate signed by the same root CA. I have confirmed the URL to the OIDC is valid (the url in the "error sending request" is my OIDC provider's config and I can rezzsolve it via curl inside the container).

<!-- gh-comment-id:2798869492 --> @somerandomguy17 commented on GitHub (Apr 12, 2025): There is a reverse proxy involved but it uses acertificate signed by the same root CA. I have confirmed the URL to the OIDC is valid (the url in the "error sending request" is my OIDC provider's config and I can rezzsolve it via curl inside the container).
GiteaMirror commented 2026-04-24 18:50:42 -05:00
Author
Owner
Copy Link

@somerandomguy17 commented on GitHub (Apr 27, 2025):

I spent a bunch of time debugging this weekend and have figured out the issue is somewhere with how rustls-tls-native-roots validates certificates. I rebuilt the image using the native-tls feature of reqwest and it works perfectly. From what I'm reading (and I don't claim to be an expert on rust or tls) rustls does additional verification beyond the native tls verication

<!-- gh-comment-id:2833669552 --> @somerandomguy17 commented on GitHub (Apr 27, 2025): I spent a bunch of time debugging this weekend and have figured out the issue is somewhere with how rustls-tls-native-roots validates certificates. I rebuilt the image using the native-tls feature of reqwest and it works perfectly. From what I'm reading (and I don't claim to be an expert on rust or tls) rustls does additional verification beyond the native tls verication
GiteaMirror commented 2026-04-24 18:50:43 -05:00
Author
Owner
Copy Link

@Mr3ase commented on GitHub (Jul 8, 2025):

Regarding the original issue, I managed to get it to work without an additional file, in the .yml :

  1. Pass your root cert to the core container through volumes
    - /path/to/cert.crt:/usr/local/share/ca-certificates/cert.crt:ro

  2. Override the launch command to update CA certs first then start Komodo Core normally. Add this to core:

      command: >
          bash -c "update-ca-certificates && core"
  1. Confirm there is no error showing through docker logs <core container name>
<!-- gh-comment-id:3048975906 --> @Mr3ase commented on GitHub (Jul 8, 2025): Regarding the original issue, I managed to get it to work without an additional file, in the .yml : 1. Pass your root cert to the core container through volumes ` - /path/to/cert.crt:/usr/local/share/ca-certificates/cert.crt:ro` 2. Override the launch command to update CA certs first then start Komodo Core normally. Add this to core: ``` command: > bash -c "update-ca-certificates && core" ``` 3. Confirm there is no error showing through `docker logs <core container name>`
GiteaMirror commented 2026-04-24 18:50:43 -05:00
Author
Owner
Copy Link

@yujqiao commented on GitHub (Dec 1, 2025):

Hey, apologies for the long delay. There is a reason I mention to do it via custom build. I went through same process as you, and it didn't work like you notice. But finally I tried the method I recommended to you, because I saw it mentioned in that stack overflow link before. I think it makes a difference whether the crt is added as a layer via custom build, versus mounted into container.

You can avoid a dedicated dockerfile by using dockerfile_inline:

services:
...(unchanged)

core:
## Comment out 'image' and replace with 'build'
# image: ghcr.io/mbecker20/komodo:${COMPOSE_KOMODO_IMAGE_TAG:-latest}
build:
context: /folder/containing/root_ca # use a context with the root ca on the host to deploy core on.
dockerfile_inline: |
FROM ghcr.io/mbecker20/komodo:latest
ADD root_ca.crt:/etc/ssl/certs
RUN update-ca-certificates
...(unchanged)

Please let me know how it works out for you, or whether you already got it to work another way. I definitely didn't try everything possible using mounts instead of custom build, figuring out this method would also be appreciated!

For anyone following this patch, the image name should be updated to ghcr.io/moghtech/komodo-core

<!-- gh-comment-id:3597482306 --> @yujqiao commented on GitHub (Dec 1, 2025): > Hey, apologies for the long delay. There is a reason I mention to do it via custom build. I went through same process as you, and it didn't work like you notice. But finally I tried the method I recommended to you, because I saw it mentioned in that stack overflow link before. I think it makes a difference whether the crt is added as a layer via custom build, versus mounted into container. > > You can avoid a dedicated dockerfile by using `dockerfile_inline`: > > services: > ...(unchanged) > > core: > ## Comment out 'image' and replace with 'build' > # image: ghcr.io/mbecker20/komodo:${COMPOSE_KOMODO_IMAGE_TAG:-latest} > build: > context: /folder/containing/root_ca # use a context with the root ca on the host to deploy core on. > dockerfile_inline: | > FROM ghcr.io/mbecker20/komodo:latest > ADD root_ca.crt:/etc/ssl/certs > RUN update-ca-certificates > ...(unchanged) > > Please let me know how it works out for you, or whether you already got it to work another way. I definitely didn't try everything possible using mounts instead of custom build, figuring out this method would also be appreciated! For anyone following this patch, the image name should be updated to ghcr.io/moghtech/komodo-core
GiteaMirror commented 2026-04-24 18:50:44 -05:00
Author
Owner
Copy Link

@SapuSeven commented on GitHub (Dec 2, 2025):

What ended up working for me was the following (either as Dockerfile or inline):

FROM ghcr.io/moghtech/komodo-core:latest
COPY root.crt /usr/local/share/ca-certificates/root.crt
RUN update-ca-certificates
<!-- gh-comment-id:3599599211 --> @SapuSeven commented on GitHub (Dec 2, 2025): What ended up working for me was the following (either as Dockerfile or inline): ```Dockerfile FROM ghcr.io/moghtech/komodo-core:latest COPY root.crt /usr/local/share/ca-certificates/root.crt RUN update-ca-certificates ```
GiteaMirror commented 2026-04-24 18:50:45 -05:00
Author
Owner
Copy Link

@b3nw commented on GitHub (Dec 18, 2025):

was this fixed both in core and periphery? running into this issue Komodo Periphery version: v1.19.5

<!-- gh-comment-id:3667871425 --> @b3nw commented on GitHub (Dec 18, 2025): was this fixed both in core and periphery? running into this issue Komodo Periphery version: v1.19.5
GiteaMirror commented 2026-04-24 18:50:46 -05:00
Author
Owner
Copy Link

@Dnny44 commented on GitHub (Dec 31, 2025):

I also just started running into this issue with v1.19.5 but with Lets Encrypt Certs, not Self Signed.

<!-- gh-comment-id:3702242095 --> @Dnny44 commented on GitHub (Dec 31, 2025): I also just started running into this issue with v1.19.5 but with Lets Encrypt Certs, not Self Signed.
GiteaMirror commented 2026-04-24 18:50:46 -05:00
Author
Owner
Copy Link

@yungtechboy1 commented on GitHub (Mar 19, 2026):

Make sure you have the correct certificates installed on the system then add this to the Env Var for Core.

DENO_TLS_CA_STORE=system,mozilla

V2.0.0-dev-124

<!-- gh-comment-id:4093435271 --> @yungtechboy1 commented on GitHub (Mar 19, 2026): Make sure you have the correct certificates installed on the system then add this to the Env Var for Core. `DENO_TLS_CA_STORE=system,mozilla` V2.0.0-dev-124
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
1.19.0
1.19.1
1.19.2
1.19.3
1.19.4
2.0.0
bug
documentation
done
enhancement
help wanted
info
pull-request
Mirrored from GitHub Pull Request
 question
security
seen 👀
setup
thinking 🤔
ui
v2
No Label bug setup
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/komodo#5861
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?