github-starred/pangolin
2
0
Fork 0
mirror of https://github.com/fosrl/pangolin.git synced 2026-07-16 06:08:33 -05:00
Code Issues 1.6k Packages Projects Releases 79 Wiki Activity

[GH-ISSUE #2801] Traefik can't connect all of a sudden #23536

New Issue
Closed
opened 2026-06-05 06:24:32 -05:00 by GiteaMirror · 5 comments
GiteaMirror commented 2026-06-05 06:24:32 -05:00
Owner
Copy Link

Originally created by @roadkingvrod on GitHub (Apr 7, 2026).
Original GitHub issue: https://github.com/fosrl/pangolin/issues/2801

Describe the Bug

Running EE. Not sure if it's tied to the upgrade or something got corrupted.

-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme
2026-04-07T03:58:12Z ERR Provider error, retrying in 494.617452ms error="cannot fetch configuration data: do fetch request: Get "http://pangolin:3001/api/v1/traefik-config": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" providerName=http
2026-04-07T03:58:18Z ERR Provider error, retrying in 262.285775ms error="cannot fetch configuration data: do fetch request: Get "http://pangolin:3001/api/v1/traefik-config": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" providerName=http
2026-04-07T03:58:23Z ERR Provider error, retrying in 558.980307ms error="cannot fetch configuration data: do fetch request: Get "http://pangolin:3001/api/v1/traefik-config": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" providerName=http

Environment

  • OS Type & Version: (e.g., Ubuntu 22.04) - Ubuntu docker
  • Pangolin Version: Latest
  • Gerbil Version: Latest
  • Traefik Version:Latest
  • Newt Version: Latest
  • Olm Version: (if applicable)

To Reproduce

Launch the compose file.

Expected Behavior

Work like it did before.

Originally created by @roadkingvrod on GitHub (Apr 7, 2026). Original GitHub issue: https://github.com/fosrl/pangolin/issues/2801 ### Describe the Bug Running EE. Not sure if it's tied to the upgrade or something got corrupted. -v02.api.letsencrypt.org/directory providerName=letsencrypt.acme 2026-04-07T03:58:12Z ERR Provider error, retrying in 494.617452ms error="cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" providerName=http 2026-04-07T03:58:18Z ERR Provider error, retrying in 262.285775ms error="cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" providerName=http 2026-04-07T03:58:23Z ERR Provider error, retrying in 558.980307ms error="cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" providerName=http ### Environment - OS Type & Version: (e.g., Ubuntu 22.04) - Ubuntu docker - Pangolin Version: Latest - Gerbil Version: Latest - Traefik Version:Latest - Newt Version: Latest - Olm Version: (if applicable) ### To Reproduce Launch the compose file. ### Expected Behavior Work like it did before.
GiteaMirror added the stale label 2026-06-05 06:24:32 -05:00
GiteaMirror commented 2026-06-05 06:24:34 -05:00
Author
Owner
Copy Link

@LaurenceJJones commented on GitHub (Apr 7, 2026):

Can you check that pangolin the container is running? typically traefik wont start if pangolin is unhealthy.

Or check for any errors within the pangolin container logs.

<!-- gh-comment-id:4198564003 --> @LaurenceJJones commented on GitHub (Apr 7, 2026): Can you check that pangolin the container is running? typically traefik wont start if pangolin is unhealthy. Or check for any errors within the pangolin container logs.
GiteaMirror commented 2026-06-05 06:24:35 -05:00
Author
Owner
Copy Link

@roadkingvrod commented on GitHub (Apr 8, 2026):

Thanks for responding. After some digging around, it appears that the logging issue is back. I haven't made any recent changes (still had them at 3 days), but after this current version, everything was very slow. Once I turned off logging again, it sped right up.

Not sure if it was a regression or maybe my disk just can't keep up (it's a home lab)

<!-- gh-comment-id:4203127425 --> @roadkingvrod commented on GitHub (Apr 8, 2026): Thanks for responding. After some digging around, it appears that the logging issue is back. I haven't made any recent changes (still had them at 3 days), but after this current version, everything was very slow. Once I turned off logging again, it sped right up. Not sure if it was a regression or maybe my disk just can't keep up (it's a home lab)
GiteaMirror commented 2026-06-05 06:24:36 -05:00
Author
Owner
Copy Link

@m-elsharkawi commented on GitHub (Apr 13, 2026):

I found what was causing a similar issue for me. Docker bridge networking on the host was broken by stale nftables raw PREROUTING rules left behind after the outage and network recreation.

Symptoms were:

  • host -> container worked
  • container name resolution worked
  • but container -> container traffic on the same Docker bridge timed out

In my case, pangolin and gerbil were on the same bridge, but nft list ruleset showed old rules like these still dropping traffic:

ip daddr 172.18.0.2 iifname != "docker_gwbridge" drop
ip daddr 172.18.0.3 iifname != "docker_gwbridge" drop

Those rules were stale and no longer matched the current bridge.

What fixed it:

  1. Inspect the nftables raw rules:
sudo nft -a list chain ip raw PREROUTING

  1. Identify the stale drop rules for the affected container IPs.

  2. Delete the bad rules by handle:

sudo nft delete rule ip raw PREROUTING handle <handle>

In my case, removing the stale rules immediately restored container-to-container connectivity and Pangolin started working again.

A Docker restart alone did not remove those stale rules, so checking nft directly was the key.

<!-- gh-comment-id:4238557700 --> @m-elsharkawi commented on GitHub (Apr 13, 2026): I found what was causing a similar issue for me. Docker bridge networking on the host was broken by stale nftables raw PREROUTING rules left behind after the outage and network recreation. Symptoms were: * host -> container worked * container name resolution worked * but container -> container traffic on the same Docker bridge timed out In my case, `pangolin` and `gerbil` were on the same bridge, but `nft list ruleset` showed old rules like these still dropping traffic: ```bash ip daddr 172.18.0.2 iifname != "docker_gwbridge" drop ip daddr 172.18.0.3 iifname != "docker_gwbridge" drop ``` Those rules were stale and no longer matched the current bridge. What fixed it: 1. Inspect the nftables raw rules: ```bash sudo nft -a list chain ip raw PREROUTING ``` 2. Identify the stale drop rules for the affected container IPs. 3. Delete the bad rules by handle: ```bash sudo nft delete rule ip raw PREROUTING handle <handle> ``` In my case, removing the stale rules immediately restored container-to-container connectivity and Pangolin started working again. A Docker restart alone did not remove those stale rules, so checking `nft` directly was the key.
GiteaMirror commented 2026-06-05 06:24:36 -05:00
Author
Owner
Copy Link

@github-actions[bot] commented on GitHub (Apr 28, 2026):

This issue has been automatically marked as stale due to 14 days of inactivity. It will be closed in 14 days if no further activity occurs.

<!-- gh-comment-id:4331529570 --> @github-actions[bot] commented on GitHub (Apr 28, 2026): This issue has been automatically marked as stale due to 14 days of inactivity. It will be closed in 14 days if no further activity occurs.
GiteaMirror commented 2026-06-05 06:24:36 -05:00
Author
Owner
Copy Link

@github-actions[bot] commented on GitHub (May 13, 2026):

This issue has been automatically closed due to inactivity. If you believe this is still relevant, please open a new issue with up-to-date information.

<!-- gh-comment-id:4436075993 --> @github-actions[bot] commented on GitHub (May 13, 2026): This issue has been automatically closed due to inactivity. If you believe this is still relevant, please open a new issue with up-to-date information.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
api
authentication
bug
config
dependencies
docker
documentation
enhancement
good first issue
help wanted
Improvement
Look Into
needs investigating
networking
new feature
non-critical bug
potential bug
pull-request
Mirrored from GitHub Pull Request
 question
reverse proxy
Security
stale
ui
wontfix
No Label stale
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/pangolin#23536
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?