github-starred/pangolin
2
0
Fork 0
mirror of https://github.com/fosrl/pangolin.git synced 2026-05-08 13:49:15 -05:00
Code Issues 564 Packages Projects Releases 72 Wiki Activity

[GH-ISSUE #2506] Overnight, all requests to pangolin port 443 return 404 #2176

New Issue
Closed
opened 2026-04-16 09:12:52 -05:00 by GiteaMirror · 8 comments
GiteaMirror commented 2026-04-16 09:12:52 -05:00
Owner
Copy Link

Originally created by @bernhardkaindl on GitHub (Feb 19, 2026).
Original GitHub issue: https://github.com/fosrl/pangolin/issues/2506

Describe the Bug

Logs look similar to #2346 [Traefik unable to reach "http://pangolin:3001/api/v1/traefik-config\"], cc @SteelyxYT
(closed by the bot automatically due to inactivity)

Stock pangolin on a VPS of Oracle Frankfurt(Germany), installed using the install script on January 27 with crowdsec

It looks like it starts when trying to do ACME renew with

Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused"

It looks like the VPS had lost connectivity to traefik servers, restarting the docker compose restart recovered the pangolin stack:

docker compose logs |grep -B1 -e unable -e error= -e traefik |grep -v -e country -e --
traefik   | {"level":"info","time":"2026-02-17T10:28:37Z","message":"Starting provider *acme.Provider"}
traefik   | {"level":"info","providerName":"letsencrypt.acme","acmeCA":"https://acme-v02.api.letsencrypt.org/directory","time":"2026-02-17T10:28:37Z","message":"Testing certificate renew..."}
traefik   | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:28:37Z","message":"Provider error, retrying in 511.376622ms"}
traefik   | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:28:38Z","message":"Provider error, retrying in 528.639607ms"}
traefik   | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:28:38Z","message":"Provider error, retrying in 939.621738ms"}
traefik   | {"level":"error","entryPointName":"web","routerName":"main-app-router-redirect@file","error":"invalid middleware \"badger@file\" configuration: invalid middleware type or middleware does not exist","time":"2026-02-17T10:28:39Z"}
traefik   | {"level":"error","entryPointName":"websecure","routerName":"ws-router@file","error":"invalid middleware \"badger@file\" configuration: invalid middleware type or middleware does not exist","time":"2026-02-17T10:28:39Z"}
traefik   | {"level":"error","entryPointName":"websecure","routerName":"api-router@file","error":"invalid middleware \"badger@file\" configuration: invalid middleware type or middleware does not exist","time":"2026-02-17T10:28:39Z"}
traefik   | {"level":"error","entryPointName":"websecure","routerName":"next-router@file","error":"invalid middleware \"badger@file\" configuration: invalid middleware type or middleware does not exist","time":"2026-02-17T10:28:39Z"}
traefik   | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:28:39Z","message":"Provider error, retrying in 1.516928941s"}
traefik   | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:28:41Z","message":"Provider error, retrying in 1.782065636s"}
traefik   | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:28:42Z","message":"Provider error, retrying in 5.667753233s"}
traefik   | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:28:48Z","message":"Provider error, retrying in 4.333497678s"}
traefik   | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:28:52Z","message":"Provider error, retrying in 9.323227148s"}
traefik   | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:02Z","message":"Provider error, retrying in 9.180967847s"}
traefik   | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:11Z","message":"Provider error, retrying in 503.662399ms"}
traefik   | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:11Z","message":"Provider error, retrying in 455.8412ms"}
traefik   | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:12Z","message":"Provider error, retrying in 1.28502377s"}
traefik   | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:13Z","message":"Provider error, retrying in 2.335269879s"}
traefik   | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:16Z","message":"Provider error, retrying in 3.362738654s"}
traefik   | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:19Z","message":"Provider error, retrying in 5.011281582s"}
traefik   | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:24Z","message":"Provider error, retrying in 3.265452158s"}
traefik   | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:27Z","message":"Provider error, retrying in 8.027231327s"}
traefik   | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:35Z","message":"Provider error, retrying in 12.611991543s"}
traefik   | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:48Z","message":"Provider error, retrying in 373.699452ms"}
traefik   | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:48Z","message":"Provider error, retrying in 474.142047ms"}
traefik   | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:49Z","message":"Provider error, retrying in 1.139402557s"}
traefik   | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:50Z","message":"Provider error, retrying in 1.691150148s"}
traefik   | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:52Z","message":"Provider error, retrying in 2.197721067s"}
traefik   | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:54Z","message":"Provider error, retrying in 3.4589871s"}
traefik   | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:57Z","message":"Provider error, retrying in 8.002826262s"}
traefik   | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:30:05Z","message":"Provider error, retrying in 9.597922893s"}
traefik   | {"level":"error","entryPointName":"web","routerName":"main-app-router-redirect@file","error":"invalid middleware \"badger@file\" configuration: invalid middleware type or middleware does not exist","time":"2026-02-17T10:30:17Z"}
traefik   | {"level":"error","entryPointName":"websecure","routerName":"api-router@file","error":"invalid middleware \"badger@file\" configuration: invalid middleware type or middleware does not exist","time":"2026-02-17T10:30:17Z"}
traefik   | {"level":"error","entryPointName":"websecure","routerName":"next-router@file","error":"invalid middleware \"badger@file\" configuration: invalid middleware type or middleware does not exist","time":"2026-02-17T10:30:17Z"}
traefik   | {"level":"error","entryPointName":"websecure","routerName":"ws-router@file","error":"invalid middleware \"badger@file\" configuration: invalid middleware type or middleware does not exist","time":"2026-02-17T10:30:17Z"}
traefik   | {"level":"error","entryPointName":"websecure","routerName":"8-ocis-router@http","error":"invalid middleware \"badger@http\" configuration: invalid middleware type or middleware does not exist","time":"2026-02-17T10:30:17Z"}
traefik   | {"level":"warn","error":"Get \"https://update.traefik.io/repos/traefik/traefik/releases\": dial tcp: lookup update.traefik.io on 127.0.0.11:53: server misbehaving","time":"2026-02-17T10:38:37Z","message":"Error checking new version"}
traefik   | {"level":"error","error":"read tcp 172.18.0.3:443->81.29.142.6:28134: read: connection timed out","time":"2026-02-17T18:10:35Z","message":"Error while peeking client hello bytes"}
traefik   | {"level":"warn","error":"Get \"https://update.traefik.io/repos/traefik/traefik/releases\": dial tcp: lookup update.traefik.io on 127.0.0.11:53: server misbehaving","time":"2026-02-18T10:28:37Z","message":"Error checking new version"}
traefik   | {"level":"info","providerName":"letsencrypt.acme","acmeCA":"https://acme-v02.api.letsencrypt.org/directory","time":"2026-02-18T10:28:37Z","message":"Testing certificate renew..."}
traefik   | {"level":"error","error":"read tcp 172.18.0.3:443->81.29.142.100:33682: read: connection reset by peer","time":"2026-02-18T21:33:47Z","message":"Error while peeking client hello bytes"}

Environment

  • Pangolin Version: ee-1.15.1
  • Gerbil Version: 1.3.0
  • Traefik Version: 3.6.7
  • Newt Version: not used
  • Olm Version: not used

To Reproduce

Appeared overnight without any action.

Expected Behavior

Continue serving requests as is.

Originally created by @bernhardkaindl on GitHub (Feb 19, 2026). Original GitHub issue: https://github.com/fosrl/pangolin/issues/2506 ### Describe the Bug Logs look similar to #2346 `[Traefik unable to reach "http://pangolin:3001/api/v1/traefik-config\"]`, cc @SteelyxYT (closed by the bot automatically due to inactivity) Stock pangolin on a VPS of Oracle Frankfurt(Germany), installed using the install script on January 27 with crowdsec It looks like it starts when trying to do ACME renew with ``` Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused" ``` It looks like the VPS had lost connectivity to traefik servers, restarting the `docker compose restart` recovered the pangolin stack: ```ml docker compose logs |grep -B1 -e unable -e error= -e traefik |grep -v -e country -e -- traefik | {"level":"info","time":"2026-02-17T10:28:37Z","message":"Starting provider *acme.Provider"} traefik | {"level":"info","providerName":"letsencrypt.acme","acmeCA":"https://acme-v02.api.letsencrypt.org/directory","time":"2026-02-17T10:28:37Z","message":"Testing certificate renew..."} traefik | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:28:37Z","message":"Provider error, retrying in 511.376622ms"} traefik | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:28:38Z","message":"Provider error, retrying in 528.639607ms"} traefik | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:28:38Z","message":"Provider error, retrying in 939.621738ms"} traefik | {"level":"error","entryPointName":"web","routerName":"main-app-router-redirect@file","error":"invalid middleware \"badger@file\" configuration: invalid middleware type or middleware does not exist","time":"2026-02-17T10:28:39Z"} traefik | {"level":"error","entryPointName":"websecure","routerName":"ws-router@file","error":"invalid middleware \"badger@file\" configuration: invalid middleware type or middleware does not exist","time":"2026-02-17T10:28:39Z"} traefik | {"level":"error","entryPointName":"websecure","routerName":"api-router@file","error":"invalid middleware \"badger@file\" configuration: invalid middleware type or middleware does not exist","time":"2026-02-17T10:28:39Z"} traefik | {"level":"error","entryPointName":"websecure","routerName":"next-router@file","error":"invalid middleware \"badger@file\" configuration: invalid middleware type or middleware does not exist","time":"2026-02-17T10:28:39Z"} traefik | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:28:39Z","message":"Provider error, retrying in 1.516928941s"} traefik | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:28:41Z","message":"Provider error, retrying in 1.782065636s"} traefik | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:28:42Z","message":"Provider error, retrying in 5.667753233s"} traefik | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:28:48Z","message":"Provider error, retrying in 4.333497678s"} traefik | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:28:52Z","message":"Provider error, retrying in 9.323227148s"} traefik | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:02Z","message":"Provider error, retrying in 9.180967847s"} traefik | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:11Z","message":"Provider error, retrying in 503.662399ms"} traefik | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:11Z","message":"Provider error, retrying in 455.8412ms"} traefik | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:12Z","message":"Provider error, retrying in 1.28502377s"} traefik | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:13Z","message":"Provider error, retrying in 2.335269879s"} traefik | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:16Z","message":"Provider error, retrying in 3.362738654s"} traefik | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:19Z","message":"Provider error, retrying in 5.011281582s"} traefik | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:24Z","message":"Provider error, retrying in 3.265452158s"} traefik | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:27Z","message":"Provider error, retrying in 8.027231327s"} traefik | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:35Z","message":"Provider error, retrying in 12.611991543s"} traefik | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:48Z","message":"Provider error, retrying in 373.699452ms"} traefik | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:48Z","message":"Provider error, retrying in 474.142047ms"} traefik | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:49Z","message":"Provider error, retrying in 1.139402557s"} traefik | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:50Z","message":"Provider error, retrying in 1.691150148s"} traefik | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:52Z","message":"Provider error, retrying in 2.197721067s"} traefik | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:54Z","message":"Provider error, retrying in 3.4589871s"} traefik | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:29:57Z","message":"Provider error, retrying in 8.002826262s"} traefik | {"level":"error","providerName":"http","error":"cannot fetch configuration data: do fetch request: Get \"http://pangolin:3001/api/v1/traefik-config\": dial tcp 172.18.0.5:3001: connect: connection refused","time":"2026-02-17T10:30:05Z","message":"Provider error, retrying in 9.597922893s"} traefik | {"level":"error","entryPointName":"web","routerName":"main-app-router-redirect@file","error":"invalid middleware \"badger@file\" configuration: invalid middleware type or middleware does not exist","time":"2026-02-17T10:30:17Z"} traefik | {"level":"error","entryPointName":"websecure","routerName":"api-router@file","error":"invalid middleware \"badger@file\" configuration: invalid middleware type or middleware does not exist","time":"2026-02-17T10:30:17Z"} traefik | {"level":"error","entryPointName":"websecure","routerName":"next-router@file","error":"invalid middleware \"badger@file\" configuration: invalid middleware type or middleware does not exist","time":"2026-02-17T10:30:17Z"} traefik | {"level":"error","entryPointName":"websecure","routerName":"ws-router@file","error":"invalid middleware \"badger@file\" configuration: invalid middleware type or middleware does not exist","time":"2026-02-17T10:30:17Z"} traefik | {"level":"error","entryPointName":"websecure","routerName":"8-ocis-router@http","error":"invalid middleware \"badger@http\" configuration: invalid middleware type or middleware does not exist","time":"2026-02-17T10:30:17Z"} traefik | {"level":"warn","error":"Get \"https://update.traefik.io/repos/traefik/traefik/releases\": dial tcp: lookup update.traefik.io on 127.0.0.11:53: server misbehaving","time":"2026-02-17T10:38:37Z","message":"Error checking new version"} traefik | {"level":"error","error":"read tcp 172.18.0.3:443->81.29.142.6:28134: read: connection timed out","time":"2026-02-17T18:10:35Z","message":"Error while peeking client hello bytes"} traefik | {"level":"warn","error":"Get \"https://update.traefik.io/repos/traefik/traefik/releases\": dial tcp: lookup update.traefik.io on 127.0.0.11:53: server misbehaving","time":"2026-02-18T10:28:37Z","message":"Error checking new version"} traefik | {"level":"info","providerName":"letsencrypt.acme","acmeCA":"https://acme-v02.api.letsencrypt.org/directory","time":"2026-02-18T10:28:37Z","message":"Testing certificate renew..."} traefik | {"level":"error","error":"read tcp 172.18.0.3:443->81.29.142.100:33682: read: connection reset by peer","time":"2026-02-18T21:33:47Z","message":"Error while peeking client hello bytes"} ``` ### Environment - Pangolin Version: ee-1.15.1 - Gerbil Version: 1.3.0 - Traefik Version: 3.6.7 - Newt Version: not used - Olm Version: not used ### To Reproduce Appeared overnight without any action. ### Expected Behavior Continue serving requests as is.
GiteaMirror added the stale label 2026-04-16 09:12:52 -05:00
GiteaMirror commented 2026-04-16 09:12:53 -05:00
Author
Owner
Copy Link

@SteelyxYT commented on GitHub (Feb 19, 2026):

Does your site still return the dashboard? Personally I got 404 on all except the pangolin dashboard.

<!-- gh-comment-id:3926389202 --> @SteelyxYT commented on GitHub (Feb 19, 2026): Does your site still return the dashboard? Personally I got 404 on all except the pangolin dashboard.
GiteaMirror commented 2026-04-16 09:12:53 -05:00
Author
Owner
Copy Link

@xupefei commented on GitHub (Feb 19, 2026):

I got the same issue after my cloud provider restarted my VPS. All resources are inaccessible (Chrome message: CONNECTION_REFUSED), including the Pangolin dashboard. I have to run docker compose restart to bring it back online.

Here's the error message from gerbil:

Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp 172.20.0.3:3001: connect: connection refused
<!-- gh-comment-id:3928788567 --> @xupefei commented on GitHub (Feb 19, 2026): I got the same issue after my cloud provider restarted my VPS. All resources are inaccessible (Chrome message: CONNECTION_REFUSED), including the Pangolin dashboard. I have to run `docker compose restart` to bring it back online. Here's the error message from `gerbil`: ``` Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp 172.20.0.3:3001: connect: connection refused ```
GiteaMirror commented 2026-04-16 09:12:53 -05:00
Author
Owner
Copy Link

@miloschwartz commented on GitHub (Feb 25, 2026):

Are containers crashing or locking up? This stuff tends to happen when on extremely resource constrained VPS and things start to lock up. The errors indicate the Pangolin and/or Gerbil containers become inaccessible while Traefik stays online. Manually check if they go unresponsive next time and see if there are any logs from these containers to indicate something is up

<!-- gh-comment-id:3961770967 --> @miloschwartz commented on GitHub (Feb 25, 2026): Are containers crashing or locking up? This stuff tends to happen when on extremely resource constrained VPS and things start to lock up. The errors indicate the Pangolin and/or Gerbil containers become inaccessible while Traefik stays online. Manually check if they go unresponsive next time and see if there are any logs from these containers to indicate something is up
GiteaMirror commented 2026-04-16 09:12:54 -05:00
Author
Owner
Copy Link

@bernhardkaindl commented on GitHub (Mar 2, 2026):

In my case, the containers themselves showed no errors, and the server had plenty of spare memory. The containers consume 3GB of memory, and the VPS had at least 12GB, I think even 24GB of RAM at the time. I checked some of the containers to see if they were responding using curl.

<!-- gh-comment-id:3981453209 --> @bernhardkaindl commented on GitHub (Mar 2, 2026): In my case, the containers themselves showed no errors, and the server had plenty of spare memory. The containers consume 3GB of memory, and the VPS had at least 12GB, I think even 24GB of RAM at the time. I checked some of the containers to see if they were responding using `curl`.
GiteaMirror commented 2026-04-16 09:12:54 -05:00
Author
Owner
Copy Link

@esoadamo commented on GitHub (Mar 2, 2026):

I had similar issue a few days back when everything started returning 404 for no apparent reason. I am running Pangolin's Traefik behind a Caddy reverse proxy and had to enable SNI to be sent to the Traefik as well. This is my Caddy config now:

example.com, *.example, {
  reverse_proxy gerbil:443 {
    # 1. Explicitly pass the original host headers to Traefik's router
    header_up Host {host}
    header_up X-Forwarded-Host {host}
    # 2. Handle the TLS connection
    transport http {
      # Ignore the expired/broken Pangolin certificate
      tls_insecure_skip_verify
      # Send SNI 
      tls_server_name {host}
    }
  }
}
<!-- gh-comment-id:3982602341 --> @esoadamo commented on GitHub (Mar 2, 2026): I had similar issue a few days back when everything started returning 404 for no apparent reason. I am running Pangolin's Traefik behind a Caddy reverse proxy and had to enable SNI to be sent to the Traefik as well. This is my Caddy config now: ``` example.com, *.example, { reverse_proxy gerbil:443 { # 1. Explicitly pass the original host headers to Traefik's router header_up Host {host} header_up X-Forwarded-Host {host} # 2. Handle the TLS connection transport http { # Ignore the expired/broken Pangolin certificate tls_insecure_skip_verify # Send SNI tls_server_name {host} } } } ```
GiteaMirror commented 2026-04-16 09:12:54 -05:00
Author
Owner
Copy Link

@github-actions[bot] commented on GitHub (Mar 17, 2026):

This issue has been automatically marked as stale due to 14 days of inactivity. It will be closed in 14 days if no further activity occurs.

<!-- gh-comment-id:4071531607 --> @github-actions[bot] commented on GitHub (Mar 17, 2026): This issue has been automatically marked as stale due to 14 days of inactivity. It will be closed in 14 days if no further activity occurs.
GiteaMirror commented 2026-04-16 09:12:54 -05:00
Author
Owner
Copy Link

@github-actions[bot] commented on GitHub (Mar 31, 2026):

This issue has been automatically closed due to inactivity. If you believe this is still relevant, please open a new issue with up-to-date information.

<!-- gh-comment-id:4159048597 --> @github-actions[bot] commented on GitHub (Mar 31, 2026): This issue has been automatically closed due to inactivity. If you believe this is still relevant, please open a new issue with up-to-date information.
GiteaMirror commented 2026-04-16 09:12:55 -05:00
Author
Owner
Copy Link

@m-elsharkawi commented on GitHub (Apr 13, 2026):

I found what was causing a similar issue for me. Docker bridge networking on the host was broken by stale nftables raw PREROUTING rules left behind after the outage and network recreation.

Symptoms were:

  • host -> container worked
  • container name resolution worked
  • but container -> container traffic on the same Docker bridge timed out

In my case, pangolin and gerbil were on the same bridge, but nft list ruleset showed old rules like these still dropping traffic:

ip daddr 172.18.0.2 iifname != "docker_gwbridge" drop
ip daddr 172.18.0.3 iifname != "docker_gwbridge" drop

Those rules were stale and no longer matched the current bridge.

What fixed it:

  1. Inspect the nftables raw rules:
sudo nft -a list chain ip raw PREROUTING

  1. Identify the stale drop rules for the affected container IPs.

  2. Delete the bad rules by handle:

sudo nft delete rule ip raw PREROUTING handle <handle>

In my case, removing the stale rules immediately restored container-to-container connectivity and Pangolin started working again.

A Docker restart alone did not remove those stale rules, so checking nft directly was the key.

<!-- gh-comment-id:4238565914 --> @m-elsharkawi commented on GitHub (Apr 13, 2026): I found what was causing a similar issue for me. Docker bridge networking on the host was broken by stale nftables raw PREROUTING rules left behind after the outage and network recreation. Symptoms were: * host -> container worked * container name resolution worked * but container -> container traffic on the same Docker bridge timed out In my case, `pangolin` and `gerbil` were on the same bridge, but `nft list ruleset` showed old rules like these still dropping traffic: ```bash ip daddr 172.18.0.2 iifname != "docker_gwbridge" drop ip daddr 172.18.0.3 iifname != "docker_gwbridge" drop ``` Those rules were stale and no longer matched the current bridge. What fixed it: 1. Inspect the nftables raw rules: ```bash sudo nft -a list chain ip raw PREROUTING ``` 2. Identify the stale drop rules for the affected container IPs. 3. Delete the bad rules by handle: ```bash sudo nft delete rule ip raw PREROUTING handle <handle> ``` In my case, removing the stale rules immediately restored container-to-container connectivity and Pangolin started working again. A Docker restart alone did not remove those stale rules, so checking `nft` directly was the key.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
api
authentication
bug
config
dependencies
docker
documentation
enhancement
good first issue
help wanted
Improvement
Look Into
needs investigating
networking
new feature
non-critical bug
potential bug
pull-request
Mirrored from GitHub Pull Request
 question
reverse proxy
Security
stale
ui
wontfix
No Label stale
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/pangolin#2176
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?