github-starred/pangolin
2
0
Fork 0
mirror of https://github.com/fosrl/pangolin.git synced 2026-07-17 07:04:36 -05:00
Code Issues 1.6k Packages Projects Releases 79 Wiki Activity

[GH-ISSUE #2978] Certificate: Failed (1.18.2) #19375

New Issue
Closed
opened 2026-05-21 20:20:58 -05:00 by GiteaMirror · 19 comments
GiteaMirror commented 2026-05-21 20:20:58 -05:00
Owner
Copy Link

Originally created by @LaserPew on GitHub (May 3, 2026).
Original GitHub issue: https://github.com/fosrl/pangolin/issues/2978

Describe the Bug

Hi,

I currently have 2 public resources (in 2 different orgs) that show their domain status as failed even tough the certificates are working completely fine:
Image

These are the domain configs:
Image
Image

Both certificates are pulled by Traefik via HTTP-01. I have other domains using DNS-01 so this is what my config/traefik/traefik_config.yml looks like:

...
certificatesResolvers:
  letsencrypt:
    acme:
      dnsChallenge:
        provider: "hetzner"
      email: "certs@example.com"
      storage: "/letsencrypt/acme.json"
      caServer: "https://acme-v02.api.letsencrypt.org/directory"
  letsencrypt-http:
    acme:
      httpChallenge:
        entryPoint: web
      email: "certs@example.com"
      storage: "/letsencrypt/acme-http.json"
      caServer: "https://acme-v02.api.letsencrypt.org/directory"
...

As I have configured two different acme storages I set the following config in config/privateConfig.yml:

acme:
  acme_json_path: "config/letsencrypt/"

In Pangolins logs I see this error repeatedly when visiting the respective Public Ressource Pages in the Dashboard:

pangolin  | 2026-05-03T12:58:36+00:00 [warn]: Certificate not found for domain: t2e83di3n8ajdu5
pangolin  | 2026-05-03T12:58:39+00:00 [warn]: Certificate not found for domain: 87t2a0yus8h0btc

Debugging this with sqlite, I see the certs (only those with HTTP-01) missing in the certificates table:

sqlite3 config/db/db.sqlite "SELECT * FROM domains"
...
87t2a0yus8h0btc|subdomain.domain1.com|0|wildcard|1|0|0|letsencrypt-http|0|
t2e83di3n8ajdu5|domain2.com|0|wildcard|1|0|0|letsencrypt-http|0|
...

sqlite3 config/db/db.sqlite "SELECT certId,domainId,status FROM certificates"
21|domain1|valid
22|domain1|valid
23|6ed3s1wkk1jxxlb|valid
25|domain1|valid
26|domain1|valid

Having a look at config/letsencrypt/acme-http.json, they look correct:

{
  "letsencrypt-http": {
    "Account": {
      "Email": "certs@example.com",
      "Registration": {
        "body": { "status": "valid" },
        "uri": "https://acme-v02.api.letsencrypt.org/acme/acct/1234567890"
      },
      "PrivateKey": "<redacted>",
      "KeyType": "4096"
    },
    "Certificates": [
      {
        "domain": {
          "main": "subdomain.domain2.com"
        },
        "certificate": "<redacted>",
        "key": "<redacted>",
        "Store": "default"
      },
      {
        "domain": { "main": "subdomain.subdomain.domain1.com" },
        "certificate": "<redacted>",
        "key": "<redacted>",
        "Store": "default"
      }
    ]
  }
}

Already tried this "fix": https://github.com/fosrl/pangolin/issues/2935#issuecomment-4348272408
But certificates don't show up in the certificates table.
No other certSync errors in the logs.

Environment

  • OS Type & Version: Debian 13
  • Pangolin Version: ee-1.18.2
  • Gerbil Version: 1.3.1
  • Traefik Version: 3.6.15
  • Newt Version: 1.12.3

To Reproduce

  • Update to 1.18.2
  • Create domains with HTTP-01
  • Create public resources using these domains
  • Certificates are generated but shown as failed (even tough they are working)

Expected Behavior

All certificates shown as valid.

Originally created by @LaserPew on GitHub (May 3, 2026). Original GitHub issue: https://github.com/fosrl/pangolin/issues/2978 ### Describe the Bug Hi, I currently have 2 public resources (in 2 different orgs) that show their domain status as failed even tough the certificates are working completely fine: <img width="1232" height="371" alt="Image" src="https://github.com/user-attachments/assets/b168c984-2619-410c-a3e7-030ad475fd68" /> These are the domain configs: <img width="1228" height="721" alt="Image" src="https://github.com/user-attachments/assets/4476d9d9-e18f-42fe-878b-39468740fb04" /> <img width="1232" height="718" alt="Image" src="https://github.com/user-attachments/assets/8ddd4c7f-dfa8-4821-b7b7-419dbbdf99fb" /> Both certificates are pulled by Traefik via HTTP-01. I have other domains using DNS-01 so this is what my `config/traefik/traefik_config.yml` looks like: ``` ... certificatesResolvers: letsencrypt: acme: dnsChallenge: provider: "hetzner" email: "certs@example.com" storage: "/letsencrypt/acme.json" caServer: "https://acme-v02.api.letsencrypt.org/directory" letsencrypt-http: acme: httpChallenge: entryPoint: web email: "certs@example.com" storage: "/letsencrypt/acme-http.json" caServer: "https://acme-v02.api.letsencrypt.org/directory" ... ``` As I have configured two different acme storages I set the following config in `config/privateConfig.yml`: ``` acme: acme_json_path: "config/letsencrypt/" ``` In Pangolins logs I see this error repeatedly when visiting the respective Public Ressource Pages in the Dashboard: ``` pangolin | 2026-05-03T12:58:36+00:00 [warn]: Certificate not found for domain: t2e83di3n8ajdu5 pangolin | 2026-05-03T12:58:39+00:00 [warn]: Certificate not found for domain: 87t2a0yus8h0btc ``` Debugging this with sqlite, I see the certs (only those with HTTP-01) missing in the certificates table: ``` sqlite3 config/db/db.sqlite "SELECT * FROM domains" ... 87t2a0yus8h0btc|subdomain.domain1.com|0|wildcard|1|0|0|letsencrypt-http|0| t2e83di3n8ajdu5|domain2.com|0|wildcard|1|0|0|letsencrypt-http|0| ... sqlite3 config/db/db.sqlite "SELECT certId,domainId,status FROM certificates" 21|domain1|valid 22|domain1|valid 23|6ed3s1wkk1jxxlb|valid 25|domain1|valid 26|domain1|valid ``` Having a look at `config/letsencrypt/acme-http.json`, they look correct: ```json { "letsencrypt-http": { "Account": { "Email": "certs@example.com", "Registration": { "body": { "status": "valid" }, "uri": "https://acme-v02.api.letsencrypt.org/acme/acct/1234567890" }, "PrivateKey": "<redacted>", "KeyType": "4096" }, "Certificates": [ { "domain": { "main": "subdomain.domain2.com" }, "certificate": "<redacted>", "key": "<redacted>", "Store": "default" }, { "domain": { "main": "subdomain.subdomain.domain1.com" }, "certificate": "<redacted>", "key": "<redacted>", "Store": "default" } ] } } ``` Already tried this "fix": https://github.com/fosrl/pangolin/issues/2935#issuecomment-4348272408 But certificates don't show up in the certificates table. No other certSync errors in the logs. ### Environment - OS Type & Version: Debian 13 - Pangolin Version: ee-1.18.2 - Gerbil Version: 1.3.1 - Traefik Version: 3.6.15 - Newt Version: 1.12.3 ### To Reproduce - Update to 1.18.2 - Create domains with HTTP-01 - Create public resources using these domains - Certificates are generated but shown as failed (even tough they are working) ### Expected Behavior All certificates shown as valid.
GiteaMirror commented 2026-05-21 20:21:02 -05:00
Author
Owner
Copy Link

@AstralDestiny commented on GitHub (May 3, 2026):

Bleh sorry initial page broke on me a bit,

So EE, and the domain is a nested subdomain or one?

<!-- gh-comment-id:4366310112 --> @AstralDestiny commented on GitHub (May 3, 2026): Bleh sorry initial page broke on me a bit, So EE, and the domain is a nested subdomain or one?
GiteaMirror commented 2026-05-21 20:21:03 -05:00
Author
Owner
Copy Link

@LaserPew commented on GitHub (May 3, 2026):

Bleh sorry initial page broke on me a bit,

So EE, and the domain is a nested subdomain or one?

Yes, I'm on EE. Regarding the domains, I have two (see screenshots of domain config from original post):

  • domain2.com (just the base domain is configured in Pangolin and my public resources are all subdomains of it)
  • subdomain.domain1.com (I just have access to a specific subdomain of domain1.com, so I have configured subdomain.domain1.com in Pangolin and my resources are under my-example-resource.subdomain.domain1.com)

(Domains are of course redacted for privacy reasons)

This is my (redacted) resource on the nested domain:
Image

<!-- gh-comment-id:4366349543 --> @LaserPew commented on GitHub (May 3, 2026): > Bleh sorry initial page broke on me a bit, > > So EE, and the domain is a nested subdomain or one? Yes, I'm on EE. Regarding the domains, I have two (see screenshots of domain config from original post): - `domain2.com` (just the base domain is configured in Pangolin and my public resources are all subdomains of it) - `subdomain.domain1.com` (I just have access to a specific subdomain of `domain1.com`, so I have configured `subdomain.domain1.com` in Pangolin and my resources are under `my-example-resource.subdomain.domain1.com`) (Domains are of course redacted for privacy reasons) This is my (redacted) resource on the nested domain: <img width="1234" height="384" alt="Image" src="https://github.com/user-attachments/assets/b1605f1c-5983-46d3-84e3-83dcec6f7ed5" />
GiteaMirror commented 2026-05-21 20:21:04 -05:00
Author
Owner
Copy Link

@AstralDestiny commented on GitHub (May 3, 2026):

Curious on the askinf ro the t2 and 87 domains then?

<!-- gh-comment-id:4366353640 --> @AstralDestiny commented on GitHub (May 3, 2026): Curious on the askinf ro the t2 and 87 domains then?
GiteaMirror commented 2026-05-21 20:21:07 -05:00
Author
Owner
Copy Link

@LaserPew commented on GitHub (May 3, 2026):

Curious on the askinf ro the t2 and 87 domains then?

Sorry, don't quite get the question. Can you maybe elaborate?

<!-- gh-comment-id:4366365646 --> @LaserPew commented on GitHub (May 3, 2026): > Curious on the askinf ro the t2 and 87 domains then? Sorry, don't quite get the question. Can you maybe elaborate?
GiteaMirror commented 2026-05-21 20:21:08 -05:00
Author
Owner
Copy Link

@AstralDestiny commented on GitHub (May 3, 2026):

Image

This part.

<!-- gh-comment-id:4366396276 --> @AstralDestiny commented on GitHub (May 3, 2026): <img width="1080" height="941" alt="Image" src="https://github.com/user-attachments/assets/5c585b89-3658-4e3b-967e-1a91c99f174b" /> This part.
GiteaMirror commented 2026-05-21 20:21:10 -05:00
Author
Owner
Copy Link

@LaserPew commented on GitHub (May 3, 2026):

I see this message in the Pangolin logs whenever I open the Public Resources page on the Dashboard.
When inspecting the network tab, I can see the following request with Status 404:

GET https://pangolin.example.com/api/v1/org/myorg/certificate/t2e83di3n8ajdu5/subdomain.domain2.com

{
    "data": null,
    "success": false,
    "error": true,
    "message": "Certificate not found",
    "status": 404,
    "stack": null
}

This seems to be in line with the certificates missing from the certificates table in sqlite. But they are in fact not missing, they are available in the acme-http.json file and the resources are also served correctly, only Pangolin seems to have a problem with them.

Im kinda stuck and don't know how to debug this further.

<!-- gh-comment-id:4366413329 --> @LaserPew commented on GitHub (May 3, 2026): I see this message in the Pangolin logs whenever I open the Public Resources page on the Dashboard. When inspecting the network tab, I can see the following request with Status 404: GET `https://pangolin.example.com/api/v1/org/myorg/certificate/t2e83di3n8ajdu5/subdomain.domain2.com` ```json { "data": null, "success": false, "error": true, "message": "Certificate not found", "status": 404, "stack": null } ``` This seems to be in line with the certificates missing from the `certificates` table in sqlite. But they are in fact not missing, they are available in the `acme-http.json` file and the resources are also served correctly, only Pangolin seems to have a problem with them. Im kinda stuck and don't know how to debug this further.
GiteaMirror commented 2026-05-21 20:21:12 -05:00
Author
Owner
Copy Link

@LaurenceJJones commented on GitHub (May 3, 2026):

Pangolin looks for the default acme.json if you have modified the path to go to acme-http.json you must inform pangolin to look for this file.

ref: https://docs.pangolin.net/self-host/advanced/private-config-file#acme-configuration

<!-- gh-comment-id:4366415952 --> @LaurenceJJones commented on GitHub (May 3, 2026): Pangolin looks for the default `acme.json` if you have modified the path to go to `acme-http.json` you must inform pangolin to look for this file. ref: https://docs.pangolin.net/self-host/advanced/private-config-file#acme-configuration
GiteaMirror commented 2026-05-21 20:21:14 -05:00
Author
Owner
Copy Link

@AstralDestiny commented on GitHub (May 3, 2026):

They did in the above to point to a folder.

Though looking that should likely point to an absolute path and not a relative path..

<!-- gh-comment-id:4366417849 --> @AstralDestiny commented on GitHub (May 3, 2026): They did in the above to point to a folder. Though looking that should likely point to an absolute path and not a relative path..
GiteaMirror commented 2026-05-21 20:21:19 -05:00
Author
Owner
Copy Link

@LaurenceJJones commented on GitHub (May 3, 2026):

Yes but the folder option simply is looking for files named "acme.json"

cf596d980f/server/private/lib/acmeCertSync.ts (L501-L505)

so to the OP are you splitting the json files?

<!-- gh-comment-id:4366422782 --> @LaurenceJJones commented on GitHub (May 3, 2026): Yes but the folder option simply is looking for files named "acme.json" https://github.com/fosrl/pangolin/blob/cf596d980fe2c5ba3ebabcfa7a8f628f34912c2c/server/private/lib/acmeCertSync.ts#L501-L505 so to the OP are you splitting the json files?
GiteaMirror commented 2026-05-21 20:21:20 -05:00
Author
Owner
Copy Link

@AstralDestiny commented on GitHub (May 3, 2026):

Docs should be updated then.. docs say,
Path to the acme.json file or a directory containing more than one acme json file produced by Traefik (or another ACME client). Pangolin reads this file to extract certificates for synchronization and will look for all files in the specified directory if a directory is provided. The file must be in the format produced by Traefik’s ACME integration. This file is typically mounted as a volume from your ACME client container.

They have a acme-http.json and a acme.json

<!-- gh-comment-id:4366425005 --> @AstralDestiny commented on GitHub (May 3, 2026): Docs should be updated then.. docs say, Path to the acme.json file or a directory containing more than one acme json file produced by Traefik (or another ACME client). Pangolin reads this file to extract certificates for synchronization and will look for all files in the specified directory if a directory is provided. The file must be in the format produced by Traefik’s ACME integration. This file is typically mounted as a volume from your ACME client container. They have a acme-http.json and a acme.json
GiteaMirror commented 2026-05-21 20:21:24 -05:00
Author
Owner
Copy Link

@LaserPew commented on GitHub (May 3, 2026):

Yes but the folder option simply is looking for files named "acme.json"

pangolin/server/private/lib/acmeCertSync.ts

Lines 501 to 505 in cf596d9

if (entry.isDirectory()) {
results.push(...findAcmeJsonFiles(fullPath));
} else if (entry.isFile() && entry.name === "acme.json") {
results.push(fullPath);
}

Ah yes, that of course might be the problem.
I'm splitting the acme.json file because I have two different certResolvers in Traefik and was worried that pointing both of them at acme.json might cause problems.

<!-- gh-comment-id:4366426036 --> @LaserPew commented on GitHub (May 3, 2026): > Yes but the folder option simply is looking for files named "acme.json" > > [pangolin/server/private/lib/acmeCertSync.ts](https://github.com/fosrl/pangolin/blob/cf596d980fe2c5ba3ebabcfa7a8f628f34912c2c/server/private/lib/acmeCertSync.ts#L501-L505) > > Lines 501 to 505 in [cf596d9](/fosrl/pangolin/commit/cf596d980fe2c5ba3ebabcfa7a8f628f34912c2c) > > if (entry.isDirectory()) { > results.push(...findAcmeJsonFiles(fullPath)); > } else if (entry.isFile() && entry.name === "acme.json") { > results.push(fullPath); > } Ah yes, that of course might be the problem. I'm splitting the acme.json file because I have two different certResolvers in Traefik and was worried that pointing both of them at `acme.json` might cause problems.
GiteaMirror commented 2026-05-21 20:21:24 -05:00
Author
Owner
Copy Link

@LaurenceJJones commented on GitHub (May 3, 2026):

So to the op you can do this

letsencrypt/dns/acme.json
letsencrypt/http/acme.json

the end file has to be named acme.json but the folders can be nested. As the function searches all folders till it finds acme.json's

<!-- gh-comment-id:4366427240 --> @LaurenceJJones commented on GitHub (May 3, 2026): So to the op you can do this ``` letsencrypt/dns/acme.json letsencrypt/http/acme.json ``` the end file has to be named `acme.json` but the folders can be nested. As the function searches all folders till it finds `acme.json`'s
GiteaMirror commented 2026-05-21 20:21:25 -05:00
Author
Owner
Copy Link

@AstralDestiny commented on GitHub (May 3, 2026):

You could have multiple providers all using the same acme.json, Traefik will properly handle that.

<!-- gh-comment-id:4366427612 --> @AstralDestiny commented on GitHub (May 3, 2026): You could have multiple providers all using the same acme.json, Traefik will properly handle that.
GiteaMirror commented 2026-05-21 20:21:27 -05:00
Author
Owner
Copy Link

@LaserPew commented on GitHub (May 3, 2026):

You could have multiple providers all using the same acme.json, Traefik will properly handle that.

Ah okay, Traefik docs weren't really helpful here. Will try that, should make the setup much more simple.

<!-- gh-comment-id:4366431247 --> @LaserPew commented on GitHub (May 3, 2026): > You could have multiple providers all using the same acme.json, Traefik will properly handle that. Ah okay, Traefik docs weren't really helpful here. Will try that, should make the setup much more simple.
GiteaMirror commented 2026-05-21 20:21:34 -05:00
Author
Owner
Copy Link

@LaserPew commented on GitHub (May 3, 2026):

So to the op you can do this

letsencrypt/dns/acme.json
letsencrypt/http/acme.json

the end file has to be named acme.json but the folders can be nested. As the function searches all folders till it finds acme.json's

Okay get it now. I think that this should be reflected in the docs. Or maybe the implementation can be updated to allow files that are not named exactly acme.json?

<!-- gh-comment-id:4366433204 --> @LaserPew commented on GitHub (May 3, 2026): > So to the op you can do this > > ``` > letsencrypt/dns/acme.json > letsencrypt/http/acme.json > ``` > > the end file has to be named `acme.json` but the folders can be nested. As the function searches all folders till it finds `acme.json`'s Okay get it now. I think that this should be reflected in the docs. Or maybe the implementation can be updated to allow files that are not named exactly `acme.json`?
GiteaMirror commented 2026-05-21 20:21:42 -05:00
Author
Owner
Copy Link

@LaurenceJJones commented on GitHub (May 3, 2026):

Path to the acme.json file or a directory containing more than one acme json file produced by Traefik

I agree it should say "Directory containing one or more acme.json files nested within folders" and probably a warning to say "end file must be named acme.json" 😆 then an example like I shown above.

<!-- gh-comment-id:4366434916 --> @LaurenceJJones commented on GitHub (May 3, 2026): > Path to the acme.json file or a directory containing more than one acme json file produced by Traefik I agree it should say "Directory containing one or more acme.json files nested within folders" and probably a warning to say "end file must be named acme.json" 😆 then an example like I shown above.
GiteaMirror commented 2026-05-21 20:21:47 -05:00
Author
Owner
Copy Link

@AstralDestiny commented on GitHub (May 3, 2026):

I assumed it just looked at anything called in there that was a .json and matches how the acme.json should look like honestly.

<!-- gh-comment-id:4366438583 --> @AstralDestiny commented on GitHub (May 3, 2026): I assumed it just looked at anything called in there that was a .json and matches how the acme.json should look like honestly.
GiteaMirror commented 2026-05-21 20:21:48 -05:00
Author
Owner
Copy Link

@LaserPew commented on GitHub (May 3, 2026):

I have reconfigured the certResolvers in Traefik to both point to acme.json now, removed the privateConfig.yml, cleared the existing acme*.json files and restarted pangolin: everything works as expected now. 💯

Thank you so much for the help!

<!-- gh-comment-id:4366445948 --> @LaserPew commented on GitHub (May 3, 2026): I have reconfigured the certResolvers in Traefik to both point to acme.json now, removed the privateConfig.yml, cleared the existing acme*.json files and restarted pangolin: everything works as expected now. 💯 Thank you so much for the help!
GiteaMirror commented 2026-05-21 20:21:49 -05:00
Author
Owner
Copy Link

@AstralDestiny commented on GitHub (May 3, 2026):

Cheers.

<!-- gh-comment-id:4366450125 --> @AstralDestiny commented on GitHub (May 3, 2026): Cheers.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
api
authentication
bug
config
dependencies
docker
documentation
enhancement
good first issue
help wanted
Improvement
Look Into
needs investigating
networking
new feature
non-critical bug
potential bug
pull-request
Mirrored from GitHub Pull Request
 question
reverse proxy
Security
stale
ui
wontfix
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/pangolin#19375
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?