github-starred/pangolin
2
0
Fork 0
mirror of https://github.com/fosrl/pangolin.git synced 2026-03-09 07:15:04 -05:00
Code Issues 94 Packages Projects Releases 48 Wiki Activity

Unable to perform an HTTP Let's Encrypt certificate request behind Pangolin reverse proxy #155

New Issue
Closed
opened 2025-11-13 11:51:44 -06:00 by GiteaMirror · 6 comments
GiteaMirror commented 2025-11-13 11:51:44 -06:00
Owner
Copy Link

Originally created by @q20 on GitHub (Mar 17, 2025).

Good day!

I have a Home Assistant installation which requires a local certificate for secure MQTT. Rather than self-sign, I have previously used the Let's Encrypt addon with DNS/API request to cloudflare, allowing me to deploy any certificate of my choosing.

Now, however, I have opted to deploy Pangolin as reverse proxy, negating the need to store those API credentials in plain text on the HA installation, as Pangolin correctly registers a valid cert for its reverse proxy, ha.example.com. This reverse proxy works via Pangolin exactly as expected.

Local MQTT is still an issue, though. When attempting an HTTP certificate request for the same FQDN, ha.example.com, the request errors with the following:

[10:36:18] INFO: Selected http verification
[10:36:18] INFO: Detecting existing certificate type for ha.example.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for ha.example.com
Certbot failed to authenticate some domains (authenticator: standalone). The Certificate Authority reported these problems:
  Domain: ha.example.com
  Type:   unauthorized
  Detail: x.x.x.x: Invalid response from http://ha.example.com/.well-known/acme-challenge/xhkbMb_wLRna08hjE0cdcm0jC0fdHK8KIVGHfTp1akI: 404
Hint: The Certificate Authority failed to download the challenge files from the temporary standalone webserver started by Certbot on port 80. Ensure that the listed domains point to this machine and that it can accept inbound connections from the internet.
Some challenges have failed.

Let's just assume that it's absolutely necessary Home Assistant is able to deploy its own, managed, Let's Encrypt cert before workarounds are suggested. 😜

I have a hunch this is not working because Pangolin upgrades all requests to HTTPS, so the HTTP challenge, on port 80, fails. But that's just my gut feeling. When I disable SSL for this resource in Pangolin, Let's Encrypt fails with the same error.

Any ideas? How can Pangolin resources successfully request their own certificates?

Originally created by @q20 on GitHub (Mar 17, 2025). Good day! I have a Home Assistant installation which requires a local certificate for secure MQTT. Rather than self-sign, I have previously used the Let's Encrypt addon with DNS/API request to cloudflare, allowing me to deploy any certificate of my choosing. Now, however, I have opted to deploy Pangolin as reverse proxy, negating the need to store those API credentials in plain text on the HA installation, as Pangolin correctly registers a valid cert for its reverse proxy, `ha.example.com`. This reverse proxy works via Pangolin exactly as expected. Local MQTT is still an issue, though. When attempting an HTTP certificate request for the same FQDN, `ha.example.com`, the request errors with the following: ``` [10:36:18] INFO: Selected http verification [10:36:18] INFO: Detecting existing certificate type for ha.example.com Saving debug log to /var/log/letsencrypt/letsencrypt.log Saving debug log to /var/log/letsencrypt/letsencrypt.log Requesting a certificate for ha.example.com Certbot failed to authenticate some domains (authenticator: standalone). The Certificate Authority reported these problems: Domain: ha.example.com Type: unauthorized Detail: x.x.x.x: Invalid response from http://ha.example.com/.well-known/acme-challenge/xhkbMb_wLRna08hjE0cdcm0jC0fdHK8KIVGHfTp1akI: 404 Hint: The Certificate Authority failed to download the challenge files from the temporary standalone webserver started by Certbot on port 80. Ensure that the listed domains point to this machine and that it can accept inbound connections from the internet. Some challenges have failed. ``` Let's just assume that it's absolutely necessary Home Assistant is able to deploy its own, managed, Let's Encrypt cert before workarounds are suggested. 😜 I have a hunch this is not working because Pangolin upgrades all requests to HTTPS, so the HTTP challenge, on port 80, fails. But that's just my gut feeling. When I disable SSL for this resource in Pangolin, Let's Encrypt fails with the same error. Any ideas? How can Pangolin resources successfully request their own certificates?
GiteaMirror added the stale label 2025-11-13 11:51:44 -06:00
GiteaMirror commented 2025-11-13 11:51:44 -06:00
Author
Owner
Copy Link

@miloschwartz commented on GitHub (Mar 17, 2025):

Invalid response from http://ha.example.com/.well-known/acme-challenge/xhkbMb_wLRna08hjE0cdcm0jC0fdHK8KIVGHfTp1akI: 404
Hint: The Certificate Authority failed to download the challenge files from the temporary standalone webserver started by Certbot on port 80. Ensure that the listed domains point to this machine and that it can accept inbound connections from the internet.

This usually indicates the HTTP-01 challenge has failed when attempting to validate the certificate. This is probably happening because port 80 is not open on the HA instance since it's now being proxied. I think you would have to use DNS validation here (DNS-01 challenge) instead, or find a work around.

@miloschwartz commented on GitHub (Mar 17, 2025): > Invalid response from http://ha.example.com/.well-known/acme-challenge/xhkbMb_wLRna08hjE0cdcm0jC0fdHK8KIVGHfTp1akI: 404 > Hint: The Certificate Authority failed to download the challenge files from the temporary standalone webserver started by Certbot on port 80. Ensure that the listed domains point to this machine and that it can accept inbound connections from the internet. This usually indicates the [HTTP-01 challenge](https://docs.certifytheweb.com/docs/http-validation/) has failed when attempting to validate the certificate. This is probably happening because port 80 is not open on the HA instance since it's now being proxied. I think you would have to use DNS validation here (DNS-01 challenge) instead, or find a work around.
GiteaMirror commented 2025-11-13 11:51:44 -06:00
Author
Owner
Copy Link

@q20 commented on GitHub (Mar 17, 2025):

Let's just assume that it's absolutely necessary Home Assistant is able to deploy its own, managed, Let's Encrypt cert before workarounds are suggested. 😜

This is why I'm suggesting that this perhaps be looked at as a feature request:
Is it possible to allow Pangolin-proxied hosts to successfully complete Let’s Encrypt HTTP certificate challenges independently?

@q20 commented on GitHub (Mar 17, 2025): > Let's just assume that it's absolutely necessary Home Assistant is able to deploy its own, managed, Let's Encrypt cert before workarounds are suggested. 😜 This is why I'm suggesting that this perhaps be looked at as a feature request: Is it possible to allow Pangolin-proxied hosts to successfully complete Let’s Encrypt HTTP certificate challenges independently?
GiteaMirror commented 2025-11-13 11:51:45 -06:00
Author
Owner
Copy Link

@rayjaymor85 commented on GitHub (Mar 23, 2025):

Is it possible to allow Pangolin-proxied hosts to successfully complete Let’s Encrypt HTTP certificate challenges independently?

As far as I know, this would take a fair degree of engineering and the use-case for this is very niche.

I had to do something very similar to get Nextcloud working as I am very insistent on not exposing it to the internet and only running it behind a VPN, making SSL certbot quite tricky.

I ended up doing the DNS-01 challenge method, this docker command worked well for me in that it generated the required keys.

The downside is this needs to be done every 90 days, I'm sure there is an ACME tool out there than can automate this if you use Cloudflare as that's what PfSense and Proxmox uses but I've not looked much into it just yet especially as I'm learning Ansible atm anyway and confident this can be handled there.

docker run --rm -it -v "/etc/letsencrypt:/etc/letsencrypt" -v "/var/lib/letsencrypt:/var/lib/letsencrypt" certbot/certbot certonly --manual --preferred-challenges dns

Just take the generated certificates and line them up to where Home Assistant needs them.

Note that in my Nextcloud case, I didn't need to use my reverse proxy anymore so not sure if this causes any clashes with Traefik.

@rayjaymor85 commented on GitHub (Mar 23, 2025): > Is it possible to allow Pangolin-proxied hosts to successfully complete Let’s Encrypt HTTP certificate challenges independently? As far as I know, this would take a fair degree of engineering and the use-case for this is _very_ niche. I had to do something very similar to get Nextcloud working as I am very insistent on not exposing it to the internet and only running it behind a VPN, making SSL certbot quite tricky. I ended up doing the DNS-01 challenge method, this docker command worked well for me in that it generated the required keys. The downside is this needs to be done every 90 days, I'm sure there is an ACME tool out there than can automate this if you use Cloudflare as that's what PfSense and Proxmox uses but I've not looked much into it just yet especially as I'm learning Ansible atm anyway and confident this can be handled there. `docker run --rm -it -v "/etc/letsencrypt:/etc/letsencrypt" -v "/var/lib/letsencrypt:/var/lib/letsencrypt" certbot/certbot certonly --manual --preferred-challenges dns` Just take the generated certificates and line them up to where Home Assistant needs them. Note that in my Nextcloud case, I didn't need to use my reverse proxy anymore so not sure if this causes any clashes with Traefik.
GiteaMirror commented 2025-11-13 11:51:45 -06:00
Author
Owner
Copy Link

@q20 commented on GitHub (Mar 24, 2025):

As far as I know, this would take a fair degree of engineering and the use-case for this is very niche.

Ah, I guess I had assumed this would be far easier to implement than it clearly is.
Yes, I guess my use-case is niche, but I could imagine more users making use of the feature, were it available.

Case in point:
Pangolin is used to secure access to multiple micro-services at multiple sites. These services are secured by their own 2FA, so Pangolin simply brokers the connection and is responsible for managing the certificate for each. Let's say we go with wildcards, leveraging Cloudflare's API. Great, now we have those micro-services secured for service1.siteA.example.com, service1.siteB.example.com, etc.
On-site, we might now have a device, service2 wanting to securely connect to service1.siteA.example.com. Split-DNS ensures service1 is resolved locally, meaning we can connect to it directly, without looping back via internet.
Unfortunately, locally, service1.siteA.example.com is using a self-signed certificate, so service2 refuses connection.
The way I do things now, is I (as suggested) use a DNA-01 challenge to provide any number of valid certificates to be used internally, on-site.
This comes with a glaring caveat, security-wise - Cloudflare cannot issue DNS edit API keys for a sub-zone (sub-domain), which means every site has an API key able to rewrite/edit/modify the entire root example.com DNS zone. Absolutely a potential security issue.
I thought Pangolin might allow an HTTP challenge for all its unprotected resources, permitting service1.siteA.example.com to successfully manage its own, non-wildcard certificate locally.

Does that make sense? It's purely a security concern: distributing root edit rights to multiple sites has me feeling uneasy.

@q20 commented on GitHub (Mar 24, 2025): > As far as I know, this would take a fair degree of engineering and the use-case for this is _very_ niche. Ah, I guess I had assumed this would be far easier to implement than it clearly is. Yes, I guess my use-case is niche, but I could imagine more users making use of the feature, were it available. Case in point: Pangolin is used to secure access to multiple micro-services at multiple sites. These services are secured by their own 2FA, so Pangolin simply brokers the connection and is responsible for managing the certificate for each. Let's say we go with wildcards, leveraging Cloudflare's API. Great, now we have those micro-services secured for `service1.siteA.example.com`, `service1.siteB.example.com`, etc. On-site, we might now have a device, `service2` wanting to securely connect to `service1.siteA.example.com`. Split-DNS ensures `service1` is resolved locally, meaning we can connect to it directly, without looping back via internet. Unfortunately, locally, `service1.siteA.example.com` is using a self-signed certificate, so `service2` refuses connection. The way I do things now, is I (as suggested) use a DNA-01 challenge to provide any number of valid certificates to be used internally, on-site. This comes with a glaring caveat, security-wise - Cloudflare cannot issue DNS edit API keys for a sub-zone (sub-domain), which means every site has an API key able to rewrite/edit/modify the entire root `example.com` DNS zone. Absolutely a potential security issue. I thought Pangolin might allow an HTTP challenge for all its unprotected resources, permitting `service1.siteA.example.com` to successfully manage its own, non-wildcard certificate locally. Does that make sense? It's purely a security concern: distributing root edit rights to multiple sites has me feeling uneasy.
GiteaMirror commented 2025-11-13 11:51:45 -06:00
Author
Owner
Copy Link

@github-actions[bot] commented on GitHub (Apr 11, 2025):

This issue has been automatically marked as stale due to 14 days of inactivity. It will be closed in 14 days if no further activity occurs.

@github-actions[bot] commented on GitHub (Apr 11, 2025): This issue has been automatically marked as stale due to 14 days of inactivity. It will be closed in 14 days if no further activity occurs.
GiteaMirror commented 2025-11-13 11:51:45 -06:00
Author
Owner
Copy Link

@github-actions[bot] commented on GitHub (Apr 26, 2025):

This issue has been automatically closed due to inactivity. If you believe this is still relevant, please open a new issue with up-to-date information.

@github-actions[bot] commented on GitHub (Apr 26, 2025): This issue has been automatically closed due to inactivity. If you believe this is still relevant, please open a new issue with up-to-date information.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
api
authentication
bug
config
dependencies
docker
documentation
enhancement
good first issue
help wanted
Improvement
Look Into
needs investigating
networking
new feature
non-critical bug
potential bug
pull-request
Mirrored from GitHub Pull Request
 question
reverse proxy
Security
stale
ui
No Label stale
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/pangolin#155
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?