github-starred/gerbil
2
0
Fork 0
mirror of https://github.com/fosrl/gerbil.git synced 2026-05-07 04:09:58 -05:00
Code Issues 43 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #51] Gerbil having several identical panics then proceeds to start #232

New Issue
Open
opened 2026-04-27 18:09:27 -05:00 by GiteaMirror · 2 comments
GiteaMirror commented 2026-04-27 18:09:27 -05:00
Owner
Copy Link

Originally created by @wittypixel on GitHub (Jan 19, 2026).
Original GitHub issue: https://github.com/fosrl/gerbil/issues/51

Originally assigned to: @oschwartz10612 on GitHub.

Describe the Bug

After spinning up pangolin container stack (pangolin, traefik, gerbil, crowdsec) connections are stopped and a panic can be seeing in the gerbil logs.

shortly after gerbil continues to go about it's business and seems to establish connection with the remote host.

redact@redact:~$ sudo docker logs gerbil -f
INFO: 2026/01/14 06:16:14 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config
INFO: 2026/01/14 06:16:14 Created WireGuard interface wg0
INFO: 2026/01/14 06:16:14 Assigned IP address 100.89.128.1/24 to interface wg0
INFO: 2026/01/14 06:16:14 Attempting to delete existing MSS clamping rule for chain INPUT
INFO: 2026/01/14 06:16:14 Attempting to delete existing MSS clamping rule for chain OUTPUT
INFO: 2026/01/14 06:16:14 Attempting to delete existing MSS clamping rule for chain FORWARD
INFO: 2026/01/14 06:16:14 Adding MSS clamping rule for chain INPUT
INFO: 2026/01/14 06:16:14 Successfully added and verified MSS clamping rule for chain INPUT
INFO: 2026/01/14 06:16:14 Adding MSS clamping rule for chain OUTPUT
INFO: 2026/01/14 06:16:14 Successfully added and verified MSS clamping rule for chain OUTPUT
INFO: 2026/01/14 06:16:14 Adding MSS clamping rule for chain FORWARD
INFO: 2026/01/14 06:16:14 Successfully added and verified MSS clamping rule for chain FORWARD
INFO: 2026/01/14 06:16:14 WireGuard interface wg0 created and configured
INFO: 2026/01/14 06:16:14 Received initial mappings: {"mappings":{}}
INFO: 2026/01/14 06:16:14 Loaded 0 initial proxy mappings
INFO: 2026/01/14 06:16:14 UDP server listening on :21820
INFO: 2026/01/14 06:16:14 Starting HTTP server on :3004
2026/01/14 06:23:36 Stopping SNI Proxy...
2026/01/14 06:23:36 All connections closed gracefully
2026/01/14 06:23:36 SNI Proxy stopped
INFO: 2026/01/14 06:23:36 Context cancelled, shutting down
panic: close of closed channel

goroutine 1 [running]:
github.com/fosrl/gerbil/relay.(*UDPProxyServer).Stop(0xc0001f8000)
        /app/relay/relay.go:237 +0x99
main.main()
        /app/main.go:420 +0x217a
INFO: 2026/01/14 06:23:53 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config
ERROR: 2026/01/14 06:23:53 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host
ERROR: 2026/01/14 06:23:53 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host
INFO: 2026/01/14 06:23:58 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config
ERROR: 2026/01/14 06:23:58 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp 172.19.0.3:3001: connect: connection refused
ERROR: 2026/01/14 06:23:58 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp 172.19.0.3:3001: connect: connection refused
INFO: 2026/01/14 06:24:03 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config
INFO: 2026/01/14 06:24:03 Created WireGuard interface wg0
INFO: 2026/01/14 06:24:03 Assigned IP address 100.89.128.1/24 to interface wg0
INFO: 2026/01/14 06:24:03 Attempting to delete existing MSS clamping rule for chain INPUT
INFO: 2026/01/14 06:24:03 Attempting to delete existing MSS clamping rule for chain OUTPUT
INFO: 2026/01/14 06:24:03 Attempting to delete existing MSS clamping rule for chain FORWARD
INFO: 2026/01/14 06:24:03 Adding MSS clamping rule for chain INPUT
INFO: 2026/01/14 06:24:03 Successfully added and verified MSS clamping rule for chain INPUT
INFO: 2026/01/14 06:24:03 Adding MSS clamping rule for chain OUTPUT
INFO: 2026/01/14 06:24:03 Successfully added and verified MSS clamping rule for chain OUTPUT
INFO: 2026/01/14 06:24:03 Adding MSS clamping rule for chain FORWARD
INFO: 2026/01/14 06:24:03 Successfully added and verified MSS clamping rule for chain FORWARD
INFO: 2026/01/14 06:24:03 WireGuard interface wg0 created and configured
INFO: 2026/01/14 06:24:03 Received initial mappings: {"mappings":{}}
INFO: 2026/01/14 06:24:03 Loaded 0 initial proxy mappings
INFO: 2026/01/14 06:24:03 UDP server listening on :21820
INFO: 2026/01/14 06:24:03 Starting HTTP server on :3004
2026/01/14 06:31:54 Stopping SNI Proxy...
2026/01/14 06:31:54 All connections closed gracefully
2026/01/14 06:31:54 SNI Proxy stopped
INFO: 2026/01/14 06:31:55 Context cancelled, shutting down
panic: close of closed channel

goroutine 1 [running]:
github.com/fosrl/gerbil/relay.(*UDPProxyServer).Stop(0xc000152280)
        /app/relay/relay.go:237 +0x99
main.main()
        /app/main.go:420 +0x217a
INFO: 2026/01/14 06:32:42 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config
ERROR: 2026/01/14 06:32:42 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host
ERROR: 2026/01/14 06:32:42 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host
INFO: 2026/01/14 06:32:47 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config
ERROR: 2026/01/14 06:32:47 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host
ERROR: 2026/01/14 06:32:47 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host
INFO: 2026/01/14 06:32:52 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config
ERROR: 2026/01/14 06:32:52 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host
ERROR: 2026/01/14 06:32:52 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host
INFO: 2026/01/14 06:32:57 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config
ERROR: 2026/01/14 06:32:57 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host
ERROR: 2026/01/14 06:32:57 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host
INFO: 2026/01/14 06:33:02 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config
ERROR: 2026/01/14 06:33:02 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host
ERROR: 2026/01/14 06:33:02 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host
INFO: 2026/01/14 06:33:07 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config
ERROR: 2026/01/14 06:33:07 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host
ERROR: 2026/01/14 06:33:07 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host
INFO: 2026/01/14 06:33:12 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config
ERROR: 2026/01/14 06:33:12 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host
ERROR: 2026/01/14 06:33:12 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host
INFO: 2026/01/14 06:33:17 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config
ERROR: 2026/01/14 06:33:17 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host
ERROR: 2026/01/14 06:33:17 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host
INFO: 2026/01/14 06:33:22 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config
ERROR: 2026/01/14 06:33:22 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host
ERROR: 2026/01/14 06:33:22 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host
INFO: 2026/01/14 06:33:27 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config
ERROR: 2026/01/14 06:33:27 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host
ERROR: 2026/01/14 06:33:27 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host
INFO: 2026/01/14 06:33:32 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config
ERROR: 2026/01/14 06:33:32 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host
ERROR: 2026/01/14 06:33:32 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host
INFO: 2026/01/14 06:33:37 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config
ERROR: 2026/01/14 06:33:37 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host
ERROR: 2026/01/14 06:33:37 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host
INFO: 2026/01/14 06:33:42 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config
ERROR: 2026/01/14 06:33:42 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp 172.19.0.4:3001: connect: connection refused
ERROR: 2026/01/14 06:33:42 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp 172.19.0.4:3001: connect: connection refused
INFO: 2026/01/14 06:33:47 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config
INFO: 2026/01/14 06:33:50 Created WireGuard interface wg0
INFO: 2026/01/14 06:33:50 Assigned IP address 100.89.128.1/24 to interface wg0
INFO: 2026/01/14 06:33:50 Attempting to delete existing MSS clamping rule for chain INPUT
INFO: 2026/01/14 06:33:50 Attempting to delete existing MSS clamping rule for chain OUTPUT
INFO: 2026/01/14 06:33:50 Attempting to delete existing MSS clamping rule for chain FORWARD
INFO: 2026/01/14 06:33:50 Adding MSS clamping rule for chain INPUT
INFO: 2026/01/14 06:33:50 Successfully added and verified MSS clamping rule for chain INPUT
INFO: 2026/01/14 06:33:50 Adding MSS clamping rule for chain OUTPUT
INFO: 2026/01/14 06:33:50 Successfully added and verified MSS clamping rule for chain OUTPUT
INFO: 2026/01/14 06:33:50 Adding MSS clamping rule for chain FORWARD
INFO: 2026/01/14 06:33:50 Successfully added and verified MSS clamping rule for chain FORWARD
INFO: 2026/01/14 06:33:50 WireGuard interface wg0 created and configured
INFO: 2026/01/14 06:33:50 Received initial mappings: {"mappings":{}}
INFO: 2026/01/14 06:33:50 Loaded 0 initial proxy mappings
INFO: 2026/01/14 06:33:50 UDP server listening on :21820
INFO: 2026/01/14 06:33:50 Starting HTTP server on :3004
INFO: 2026/01/15 05:27:18 Clearing connections for added peer with WG IP: 100.89.128.4
INFO: 2026/01/15 05:27:18 Cleared 0 connections for WG IP: 100.89.128.4
INFO: 2026/01/15 05:27:18 Peer xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx= added successfully
INFO: 2026/01/15 05:27:18 Cleared 0 sessions for WG IP: **PUB.LIC.IP.ADDRESS**

Environment

  • OS Type & Version: Ubuntu 24.04.3 LTS
  • Pangolin Version: fosrl/pangolin:1.14.1
  • Gerbil Version: fosrl/gerbil:1.3.0
  • Traefik Version: traefik:v3.6
  • Newt Version: 1.8.1 [running on remote server]

To Reproduce

I typically notice this behavior occurring when restarting the stack related to pangolin. I restart the container stack directly using the terminal and sudo docker compose commands.

Attempted to pull the latest container also to no avail, the error was still present.

Expected Behavior

i believe this one is pretty clear, the should be no panic.

Originally created by @wittypixel on GitHub (Jan 19, 2026). Original GitHub issue: https://github.com/fosrl/gerbil/issues/51 Originally assigned to: @oschwartz10612 on GitHub. ### Describe the Bug After spinning up pangolin container stack (pangolin, traefik, gerbil, crowdsec) connections are stopped and a panic can be seeing in the gerbil logs. shortly after gerbil continues to go about it's business and seems to establish connection with the remote host. ``` redact@redact:~$ sudo docker logs gerbil -f INFO: 2026/01/14 06:16:14 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config INFO: 2026/01/14 06:16:14 Created WireGuard interface wg0 INFO: 2026/01/14 06:16:14 Assigned IP address 100.89.128.1/24 to interface wg0 INFO: 2026/01/14 06:16:14 Attempting to delete existing MSS clamping rule for chain INPUT INFO: 2026/01/14 06:16:14 Attempting to delete existing MSS clamping rule for chain OUTPUT INFO: 2026/01/14 06:16:14 Attempting to delete existing MSS clamping rule for chain FORWARD INFO: 2026/01/14 06:16:14 Adding MSS clamping rule for chain INPUT INFO: 2026/01/14 06:16:14 Successfully added and verified MSS clamping rule for chain INPUT INFO: 2026/01/14 06:16:14 Adding MSS clamping rule for chain OUTPUT INFO: 2026/01/14 06:16:14 Successfully added and verified MSS clamping rule for chain OUTPUT INFO: 2026/01/14 06:16:14 Adding MSS clamping rule for chain FORWARD INFO: 2026/01/14 06:16:14 Successfully added and verified MSS clamping rule for chain FORWARD INFO: 2026/01/14 06:16:14 WireGuard interface wg0 created and configured INFO: 2026/01/14 06:16:14 Received initial mappings: {"mappings":{}} INFO: 2026/01/14 06:16:14 Loaded 0 initial proxy mappings INFO: 2026/01/14 06:16:14 UDP server listening on :21820 INFO: 2026/01/14 06:16:14 Starting HTTP server on :3004 2026/01/14 06:23:36 Stopping SNI Proxy... 2026/01/14 06:23:36 All connections closed gracefully 2026/01/14 06:23:36 SNI Proxy stopped INFO: 2026/01/14 06:23:36 Context cancelled, shutting down panic: close of closed channel goroutine 1 [running]: github.com/fosrl/gerbil/relay.(*UDPProxyServer).Stop(0xc0001f8000) /app/relay/relay.go:237 +0x99 main.main() /app/main.go:420 +0x217a INFO: 2026/01/14 06:23:53 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config ERROR: 2026/01/14 06:23:53 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host ERROR: 2026/01/14 06:23:53 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host INFO: 2026/01/14 06:23:58 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config ERROR: 2026/01/14 06:23:58 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp 172.19.0.3:3001: connect: connection refused ERROR: 2026/01/14 06:23:58 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp 172.19.0.3:3001: connect: connection refused INFO: 2026/01/14 06:24:03 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config INFO: 2026/01/14 06:24:03 Created WireGuard interface wg0 INFO: 2026/01/14 06:24:03 Assigned IP address 100.89.128.1/24 to interface wg0 INFO: 2026/01/14 06:24:03 Attempting to delete existing MSS clamping rule for chain INPUT INFO: 2026/01/14 06:24:03 Attempting to delete existing MSS clamping rule for chain OUTPUT INFO: 2026/01/14 06:24:03 Attempting to delete existing MSS clamping rule for chain FORWARD INFO: 2026/01/14 06:24:03 Adding MSS clamping rule for chain INPUT INFO: 2026/01/14 06:24:03 Successfully added and verified MSS clamping rule for chain INPUT INFO: 2026/01/14 06:24:03 Adding MSS clamping rule for chain OUTPUT INFO: 2026/01/14 06:24:03 Successfully added and verified MSS clamping rule for chain OUTPUT INFO: 2026/01/14 06:24:03 Adding MSS clamping rule for chain FORWARD INFO: 2026/01/14 06:24:03 Successfully added and verified MSS clamping rule for chain FORWARD INFO: 2026/01/14 06:24:03 WireGuard interface wg0 created and configured INFO: 2026/01/14 06:24:03 Received initial mappings: {"mappings":{}} INFO: 2026/01/14 06:24:03 Loaded 0 initial proxy mappings INFO: 2026/01/14 06:24:03 UDP server listening on :21820 INFO: 2026/01/14 06:24:03 Starting HTTP server on :3004 2026/01/14 06:31:54 Stopping SNI Proxy... 2026/01/14 06:31:54 All connections closed gracefully 2026/01/14 06:31:54 SNI Proxy stopped INFO: 2026/01/14 06:31:55 Context cancelled, shutting down panic: close of closed channel goroutine 1 [running]: github.com/fosrl/gerbil/relay.(*UDPProxyServer).Stop(0xc000152280) /app/relay/relay.go:237 +0x99 main.main() /app/main.go:420 +0x217a INFO: 2026/01/14 06:32:42 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config ERROR: 2026/01/14 06:32:42 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host ERROR: 2026/01/14 06:32:42 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host INFO: 2026/01/14 06:32:47 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config ERROR: 2026/01/14 06:32:47 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host ERROR: 2026/01/14 06:32:47 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host INFO: 2026/01/14 06:32:52 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config ERROR: 2026/01/14 06:32:52 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host ERROR: 2026/01/14 06:32:52 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host INFO: 2026/01/14 06:32:57 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config ERROR: 2026/01/14 06:32:57 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host ERROR: 2026/01/14 06:32:57 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host INFO: 2026/01/14 06:33:02 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config ERROR: 2026/01/14 06:33:02 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host ERROR: 2026/01/14 06:33:02 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host INFO: 2026/01/14 06:33:07 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config ERROR: 2026/01/14 06:33:07 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host ERROR: 2026/01/14 06:33:07 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host INFO: 2026/01/14 06:33:12 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config ERROR: 2026/01/14 06:33:12 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host ERROR: 2026/01/14 06:33:12 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host INFO: 2026/01/14 06:33:17 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config ERROR: 2026/01/14 06:33:17 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host ERROR: 2026/01/14 06:33:17 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host INFO: 2026/01/14 06:33:22 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config ERROR: 2026/01/14 06:33:22 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host ERROR: 2026/01/14 06:33:22 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host INFO: 2026/01/14 06:33:27 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config ERROR: 2026/01/14 06:33:27 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host ERROR: 2026/01/14 06:33:27 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host INFO: 2026/01/14 06:33:32 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config ERROR: 2026/01/14 06:33:32 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host ERROR: 2026/01/14 06:33:32 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host INFO: 2026/01/14 06:33:37 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config ERROR: 2026/01/14 06:33:37 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host ERROR: 2026/01/14 06:33:37 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp: lookup pangolin on 127.0.0.11:53: no such host INFO: 2026/01/14 06:33:42 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config ERROR: 2026/01/14 06:33:42 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp 172.19.0.4:3001: connect: connection refused ERROR: 2026/01/14 06:33:42 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp 172.19.0.4:3001: connect: connection refused INFO: 2026/01/14 06:33:47 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config INFO: 2026/01/14 06:33:50 Created WireGuard interface wg0 INFO: 2026/01/14 06:33:50 Assigned IP address 100.89.128.1/24 to interface wg0 INFO: 2026/01/14 06:33:50 Attempting to delete existing MSS clamping rule for chain INPUT INFO: 2026/01/14 06:33:50 Attempting to delete existing MSS clamping rule for chain OUTPUT INFO: 2026/01/14 06:33:50 Attempting to delete existing MSS clamping rule for chain FORWARD INFO: 2026/01/14 06:33:50 Adding MSS clamping rule for chain INPUT INFO: 2026/01/14 06:33:50 Successfully added and verified MSS clamping rule for chain INPUT INFO: 2026/01/14 06:33:50 Adding MSS clamping rule for chain OUTPUT INFO: 2026/01/14 06:33:50 Successfully added and verified MSS clamping rule for chain OUTPUT INFO: 2026/01/14 06:33:50 Adding MSS clamping rule for chain FORWARD INFO: 2026/01/14 06:33:50 Successfully added and verified MSS clamping rule for chain FORWARD INFO: 2026/01/14 06:33:50 WireGuard interface wg0 created and configured INFO: 2026/01/14 06:33:50 Received initial mappings: {"mappings":{}} INFO: 2026/01/14 06:33:50 Loaded 0 initial proxy mappings INFO: 2026/01/14 06:33:50 UDP server listening on :21820 INFO: 2026/01/14 06:33:50 Starting HTTP server on :3004 INFO: 2026/01/15 05:27:18 Clearing connections for added peer with WG IP: 100.89.128.4 INFO: 2026/01/15 05:27:18 Cleared 0 connections for WG IP: 100.89.128.4 INFO: 2026/01/15 05:27:18 Peer xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx= added successfully INFO: 2026/01/15 05:27:18 Cleared 0 sessions for WG IP: **PUB.LIC.IP.ADDRESS** ``` ### Environment - OS Type & Version: Ubuntu 24.04.3 LTS - Pangolin Version: fosrl/pangolin:1.14.1 - Gerbil Version: fosrl/gerbil:1.3.0 - Traefik Version: traefik:v3.6 - Newt Version: 1.8.1 **[running on remote server]** ### To Reproduce I typically notice this behavior occurring when restarting the stack related to pangolin. I restart the container stack directly using the terminal and `sudo docker compose` commands. Attempted to pull the latest container also to no avail, the error was still present. ### Expected Behavior i believe this one is pretty clear, the should be no panic.
GiteaMirror added the bug label 2026-04-27 18:09:27 -05:00
GiteaMirror commented 2026-04-27 18:09:30 -05:00
Author
Owner
Copy Link

@wittypixel commented on GitHub (Jan 19, 2026):

possibly important, I also just noticed the newt network connection looks to be having a hard time according to the logs on the remote server

That said I don't believe the 2 issues are mutually related and may just be coincidentally happening at the same time

2026-01-19T08:18:47.374846767Z WARN: 2026/01/19 08:18:47 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded
2026-01-19T08:19:22.376101257Z WARN: 2026/01/19 08:19:22 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded
2026-01-19T08:19:57.378019215Z WARN: 2026/01/19 08:19:57 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded
2026-01-19T08:20:32.378783197Z WARN: 2026/01/19 08:20:32 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded
2026-01-19T08:20:37.695300296Z ERROR: 2026/01/19 08:20:37 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:20:38.410936972Z INFO: 2026/01/19 08:20:38 Failed to report peer bandwidth: failed to send bandwidth data: not connected
2026-01-19T08:20:40.719468985Z ERROR: 2026/01/19 08:20:40 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:20:43.744977048Z ERROR: 2026/01/19 08:20:43 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:20:46.767069046Z ERROR: 2026/01/19 08:20:46 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:20:48.411549919Z INFO: 2026/01/19 08:20:48 Failed to report peer bandwidth: failed to send bandwidth data: not connected
2026-01-19T08:20:49.791344909Z ERROR: 2026/01/19 08:20:49 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:20:52.814890254Z ERROR: 2026/01/19 08:20:52 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:20:55.863906448Z ERROR: 2026/01/19 08:20:55 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:20:58.410607138Z INFO: 2026/01/19 08:20:58 Failed to report peer bandwidth: failed to send bandwidth data: not connected
2026-01-19T08:20:58.909164850Z ERROR: 2026/01/19 08:20:58 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:21:00.957069748Z WARN: 2026/01/19 08:21:00 Periodic ping failed (2 consecutive failures): all 2 ping attempts failed, last error: failed to read ICMP packet: i/o timeout
2026-01-19T08:21:01.932359426Z ERROR: 2026/01/19 08:21:01 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:21:04.955806497Z ERROR: 2026/01/19 08:21:04 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:21:07.380422606Z WARN: 2026/01/19 08:21:07 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded
2026-01-19T08:21:07.974919998Z ERROR: 2026/01/19 08:21:07 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:21:08.411259404Z INFO: 2026/01/19 08:21:08 Failed to report peer bandwidth: failed to send bandwidth data: not connected
2026-01-19T08:21:10.998842273Z ERROR: 2026/01/19 08:21:10 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:21:11.634791785Z WARN: 2026/01/19 08:21:11 Periodic ping failed (3 consecutive failures): all 2 ping attempts failed, last error: failed to read ICMP packet: i/o timeout
2026-01-19T08:21:14.022792067Z ERROR: 2026/01/19 08:21:14 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:21:17.042236727Z ERROR: 2026/01/19 08:21:17 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:21:18.410895474Z INFO: 2026/01/19 08:21:18 Failed to report peer bandwidth: failed to send bandwidth data: not connected
2026-01-19T08:21:20.071156149Z ERROR: 2026/01/19 08:21:20 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:21:22.321733499Z WARN: 2026/01/19 08:21:22 Periodic ping failed (4 consecutive failures): all 2 ping attempts failed, last error: failed to read ICMP packet: i/o timeout
2026-01-19T08:21:22.321758930Z WARN: 2026/01/19 08:21:22 Connection to server lost after 4 failures. Continuous reconnection attempts will be made.
2026-01-19T08:21:22.321764742Z ERROR: 2026/01/19 08:21:22 Failed to send registration message: not connected
2026-01-19T08:21:22.321795700Z ERROR: 2026/01/19 08:21:22 Failed to send initial message: not connected
2026-01-19T08:21:23.094873145Z ERROR: 2026/01/19 08:21:23 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:21:25.322466137Z ERROR: 2026/01/19 08:21:25 Failed to send message: not connected
2026-01-19T08:21:26.119002828Z ERROR: 2026/01/19 08:21:26 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:21:28.322228282Z ERROR: 2026/01/19 08:21:28 Failed to send message: not connected
2026-01-19T08:21:28.410611140Z INFO: 2026/01/19 08:21:28 Failed to report peer bandwidth: failed to send bandwidth data: not connected
2026-01-19T08:21:29.143125014Z ERROR: 2026/01/19 08:21:29 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:21:31.322789436Z ERROR: 2026/01/19 08:21:31 Failed to send message: not connected
2026-01-19T08:21:32.167416942Z ERROR: 2026/01/19 08:21:32 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:21:34.322961937Z ERROR: 2026/01/19 08:21:34 Failed to send message: not connected
2026-01-19T08:21:35.197596174Z ERROR: 2026/01/19 08:21:35 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:21:37.322609516Z ERROR: 2026/01/19 08:21:37 Failed to send message: not connected
2026-01-19T08:21:38.214773798Z ERROR: 2026/01/19 08:21:38 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:21:38.411415141Z INFO: 2026/01/19 08:21:38 Failed to report peer bandwidth: failed to send bandwidth data: not connected
2026-01-19T08:21:38.924095299Z WARN: 2026/01/19 08:21:38 Periodic ping failed (5 consecutive failures): all 2 ping attempts failed, last error: failed to read ICMP packet: i/o timeout
2026-01-19T08:21:40.322733207Z ERROR: 2026/01/19 08:21:40 Failed to send message: not connected
2026-01-19T08:21:41.233916078Z ERROR: 2026/01/19 08:21:41 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:21:42.381450798Z WARN: 2026/01/19 08:21:42 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded
2026-01-19T08:21:43.322681874Z ERROR: 2026/01/19 08:21:43 Failed to send message: not connected
2026-01-19T08:21:44.264402411Z ERROR: 2026/01/19 08:21:44 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:21:46.322703145Z ERROR: 2026/01/19 08:21:46 Failed to send message: not connected
2026-01-19T08:21:47.282343787Z ERROR: 2026/01/19 08:21:47 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:21:48.410605595Z INFO: 2026/01/19 08:21:48 Failed to report peer bandwidth: failed to send bandwidth data: not connected
2026-01-19T08:21:49.322604304Z ERROR: 2026/01/19 08:21:49 Failed to send message: not connected
2026-01-19T08:21:50.306469449Z ERROR: 2026/01/19 08:21:50 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:21:52.321988823Z INFO: 2026/01/19 08:21:52 SendMessageInterval timed out after 10 attempts for message type: newt/ping/request
2026-01-19T08:21:53.331125818Z ERROR: 2026/01/19 08:21:53 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:21:56.348950330Z ERROR: 2026/01/19 08:21:56 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:21:58.411471719Z INFO: 2026/01/19 08:21:58 Failed to report peer bandwidth: failed to send bandwidth data: not connected
2026-01-19T08:21:59.367857254Z ERROR: 2026/01/19 08:21:59 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:22:02.392006799Z ERROR: 2026/01/19 08:22:02 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:22:05.416057752Z ERROR: 2026/01/19 08:22:05 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:22:08.411354438Z INFO: 2026/01/19 08:22:08 Failed to report peer bandwidth: failed to send bandwidth data: not connected
2026-01-19T08:22:08.440315954Z ERROR: 2026/01/19 08:22:08 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:22:11.464145686Z ERROR: 2026/01/19 08:22:11 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:22:14.488300390Z ERROR: 2026/01/19 08:22:14 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:22:17.382465977Z WARN: 2026/01/19 08:22:17 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded
2026-01-19T08:22:17.512851876Z ERROR: 2026/01/19 08:22:17 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:22:18.411514742Z INFO: 2026/01/19 08:22:18 Failed to report peer bandwidth: failed to send bandwidth data: not connected
2026-01-19T08:22:18.912186430Z WARN: 2026/01/19 08:22:18 Periodic ping failed (6 consecutive failures): all 4 ping attempts failed, last error: failed to read ICMP packet: i/o timeout
2026-01-19T08:22:20.535943238Z ERROR: 2026/01/19 08:22:20 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:22:23.565702869Z ERROR: 2026/01/19 08:22:23 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:22:26.589559041Z ERROR: 2026/01/19 08:22:26 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:22:28.411489169Z INFO: 2026/01/19 08:22:28 Failed to report peer bandwidth: failed to send bandwidth data: not connected
2026-01-19T08:22:29.607627805Z ERROR: 2026/01/19 08:22:29 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:22:32.632349282Z ERROR: 2026/01/19 08:22:32 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:22:35.656124130Z ERROR: 2026/01/19 08:22:35 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:22:38.410614825Z INFO: 2026/01/19 08:22:38 Failed to report peer bandwidth: failed to send bandwidth data: not connected
2026-01-19T08:22:38.679928807Z ERROR: 2026/01/19 08:22:38 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:22:41.704350980Z ERROR: 2026/01/19 08:22:41 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:22:44.727896881Z ERROR: 2026/01/19 08:22:44 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:22:47.752955017Z ERROR: 2026/01/19 08:22:47 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s...
2026-01-19T08:22:48.411676059Z INFO: 2026/01/19 08:22:48 Failed to report peer bandwidth: failed to send bandwidth data: not connected
2026-01-19T08:22:50.805814197Z ERROR: 2026/01/19 08:22:50 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": tls: failed to verify certificate: x509: certificate is valid for xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.traefik.default, not subdomain.domain.com. Retrying in 3s...
2026-01-19T08:22:52.383491481Z WARN: 2026/01/19 08:22:52 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded
2026-01-19T08:22:54.087380976Z INFO: 2026/01/19 08:22:54 Server version: 1.14.1
2026-01-19T08:22:54.178556339Z INFO: 2026/01/19 08:22:54 Websocket connected
2026-01-19T08:22:57.449078521Z INFO: 2026/01/19 08:22:57 Connection to server restored after 6 failures!
2026-01-19T08:23:27.384423139Z WARN: 2026/01/19 08:23:27 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded
2026-01-19T08:24:02.385822911Z WARN: 2026/01/19 08:24:02 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded
2026-01-19T08:24:37.386416284Z WARN: 2026/01/19 08:24:37 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded
2026-01-19T08:25:12.388072138Z WARN: 2026/01/19 08:25:12 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded
2026-01-19T08:25:47.389372744Z WARN: 2026/01/19 08:25:47 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded
2026-01-19T08:26:22.391491664Z WARN: 2026/01/19 08:26:22 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded
2026-01-19T08:26:57.392505655Z WARN: 2026/01/19 08:26:57 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded
2026-01-19T08:27:32.394512058Z WARN: 2026/01/19 08:27:32 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded
2026-01-19T08:28:07.396001777Z WARN: 2026/01/19 08:28:07 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded
2026-01-19T08:28:42.397358840Z WARN: 2026/01/19 08:28:42 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded
2026-01-19T08:29:17.398991317Z WARN: 2026/01/19 08:29:17 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded
2026-01-19T08:29:52.399552209Z WARN: 2026/01/19 08:29:52 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded
2026-01-19T08:30:27.400009899Z WARN: 2026/01/19 08:30:27 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded
2026-01-19T08:31:02.401642318Z WARN: 2026/01/19 08:31:02 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded
2026-01-19T08:31:37.402527226Z WARN: 2026/01/19 08:31:37 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded
2026-01-19T08:32:12.404265370Z WARN: 2026/01/19 08:32:12 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded
[newt] 2026-01-19T08:32:47.404196049Z WARN: 2026/01/19 08:32:47 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded
<!-- gh-comment-id:3770579455 --> @wittypixel commented on GitHub (Jan 19, 2026): possibly important, I also just noticed the newt network connection looks to be having a hard time according to the logs on the remote server That said I don't believe the 2 issues are mutually related and may just be coincidentally happening at the same time ``` 2026-01-19T08:18:47.374846767Z WARN: 2026/01/19 08:18:47 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded 2026-01-19T08:19:22.376101257Z WARN: 2026/01/19 08:19:22 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded 2026-01-19T08:19:57.378019215Z WARN: 2026/01/19 08:19:57 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded 2026-01-19T08:20:32.378783197Z WARN: 2026/01/19 08:20:32 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded 2026-01-19T08:20:37.695300296Z ERROR: 2026/01/19 08:20:37 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:20:38.410936972Z INFO: 2026/01/19 08:20:38 Failed to report peer bandwidth: failed to send bandwidth data: not connected 2026-01-19T08:20:40.719468985Z ERROR: 2026/01/19 08:20:40 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:20:43.744977048Z ERROR: 2026/01/19 08:20:43 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:20:46.767069046Z ERROR: 2026/01/19 08:20:46 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:20:48.411549919Z INFO: 2026/01/19 08:20:48 Failed to report peer bandwidth: failed to send bandwidth data: not connected 2026-01-19T08:20:49.791344909Z ERROR: 2026/01/19 08:20:49 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:20:52.814890254Z ERROR: 2026/01/19 08:20:52 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:20:55.863906448Z ERROR: 2026/01/19 08:20:55 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:20:58.410607138Z INFO: 2026/01/19 08:20:58 Failed to report peer bandwidth: failed to send bandwidth data: not connected 2026-01-19T08:20:58.909164850Z ERROR: 2026/01/19 08:20:58 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:21:00.957069748Z WARN: 2026/01/19 08:21:00 Periodic ping failed (2 consecutive failures): all 2 ping attempts failed, last error: failed to read ICMP packet: i/o timeout 2026-01-19T08:21:01.932359426Z ERROR: 2026/01/19 08:21:01 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:21:04.955806497Z ERROR: 2026/01/19 08:21:04 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:21:07.380422606Z WARN: 2026/01/19 08:21:07 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded 2026-01-19T08:21:07.974919998Z ERROR: 2026/01/19 08:21:07 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:21:08.411259404Z INFO: 2026/01/19 08:21:08 Failed to report peer bandwidth: failed to send bandwidth data: not connected 2026-01-19T08:21:10.998842273Z ERROR: 2026/01/19 08:21:10 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:21:11.634791785Z WARN: 2026/01/19 08:21:11 Periodic ping failed (3 consecutive failures): all 2 ping attempts failed, last error: failed to read ICMP packet: i/o timeout 2026-01-19T08:21:14.022792067Z ERROR: 2026/01/19 08:21:14 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:21:17.042236727Z ERROR: 2026/01/19 08:21:17 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:21:18.410895474Z INFO: 2026/01/19 08:21:18 Failed to report peer bandwidth: failed to send bandwidth data: not connected 2026-01-19T08:21:20.071156149Z ERROR: 2026/01/19 08:21:20 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:21:22.321733499Z WARN: 2026/01/19 08:21:22 Periodic ping failed (4 consecutive failures): all 2 ping attempts failed, last error: failed to read ICMP packet: i/o timeout 2026-01-19T08:21:22.321758930Z WARN: 2026/01/19 08:21:22 Connection to server lost after 4 failures. Continuous reconnection attempts will be made. 2026-01-19T08:21:22.321764742Z ERROR: 2026/01/19 08:21:22 Failed to send registration message: not connected 2026-01-19T08:21:22.321795700Z ERROR: 2026/01/19 08:21:22 Failed to send initial message: not connected 2026-01-19T08:21:23.094873145Z ERROR: 2026/01/19 08:21:23 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:21:25.322466137Z ERROR: 2026/01/19 08:21:25 Failed to send message: not connected 2026-01-19T08:21:26.119002828Z ERROR: 2026/01/19 08:21:26 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:21:28.322228282Z ERROR: 2026/01/19 08:21:28 Failed to send message: not connected 2026-01-19T08:21:28.410611140Z INFO: 2026/01/19 08:21:28 Failed to report peer bandwidth: failed to send bandwidth data: not connected 2026-01-19T08:21:29.143125014Z ERROR: 2026/01/19 08:21:29 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:21:31.322789436Z ERROR: 2026/01/19 08:21:31 Failed to send message: not connected 2026-01-19T08:21:32.167416942Z ERROR: 2026/01/19 08:21:32 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:21:34.322961937Z ERROR: 2026/01/19 08:21:34 Failed to send message: not connected 2026-01-19T08:21:35.197596174Z ERROR: 2026/01/19 08:21:35 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:21:37.322609516Z ERROR: 2026/01/19 08:21:37 Failed to send message: not connected 2026-01-19T08:21:38.214773798Z ERROR: 2026/01/19 08:21:38 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:21:38.411415141Z INFO: 2026/01/19 08:21:38 Failed to report peer bandwidth: failed to send bandwidth data: not connected 2026-01-19T08:21:38.924095299Z WARN: 2026/01/19 08:21:38 Periodic ping failed (5 consecutive failures): all 2 ping attempts failed, last error: failed to read ICMP packet: i/o timeout 2026-01-19T08:21:40.322733207Z ERROR: 2026/01/19 08:21:40 Failed to send message: not connected 2026-01-19T08:21:41.233916078Z ERROR: 2026/01/19 08:21:41 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:21:42.381450798Z WARN: 2026/01/19 08:21:42 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded 2026-01-19T08:21:43.322681874Z ERROR: 2026/01/19 08:21:43 Failed to send message: not connected 2026-01-19T08:21:44.264402411Z ERROR: 2026/01/19 08:21:44 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:21:46.322703145Z ERROR: 2026/01/19 08:21:46 Failed to send message: not connected 2026-01-19T08:21:47.282343787Z ERROR: 2026/01/19 08:21:47 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:21:48.410605595Z INFO: 2026/01/19 08:21:48 Failed to report peer bandwidth: failed to send bandwidth data: not connected 2026-01-19T08:21:49.322604304Z ERROR: 2026/01/19 08:21:49 Failed to send message: not connected 2026-01-19T08:21:50.306469449Z ERROR: 2026/01/19 08:21:50 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:21:52.321988823Z INFO: 2026/01/19 08:21:52 SendMessageInterval timed out after 10 attempts for message type: newt/ping/request 2026-01-19T08:21:53.331125818Z ERROR: 2026/01/19 08:21:53 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:21:56.348950330Z ERROR: 2026/01/19 08:21:56 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:21:58.411471719Z INFO: 2026/01/19 08:21:58 Failed to report peer bandwidth: failed to send bandwidth data: not connected 2026-01-19T08:21:59.367857254Z ERROR: 2026/01/19 08:21:59 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:22:02.392006799Z ERROR: 2026/01/19 08:22:02 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:22:05.416057752Z ERROR: 2026/01/19 08:22:05 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:22:08.411354438Z INFO: 2026/01/19 08:22:08 Failed to report peer bandwidth: failed to send bandwidth data: not connected 2026-01-19T08:22:08.440315954Z ERROR: 2026/01/19 08:22:08 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:22:11.464145686Z ERROR: 2026/01/19 08:22:11 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:22:14.488300390Z ERROR: 2026/01/19 08:22:14 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:22:17.382465977Z WARN: 2026/01/19 08:22:17 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded 2026-01-19T08:22:17.512851876Z ERROR: 2026/01/19 08:22:17 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:22:18.411514742Z INFO: 2026/01/19 08:22:18 Failed to report peer bandwidth: failed to send bandwidth data: not connected 2026-01-19T08:22:18.912186430Z WARN: 2026/01/19 08:22:18 Periodic ping failed (6 consecutive failures): all 4 ping attempts failed, last error: failed to read ICMP packet: i/o timeout 2026-01-19T08:22:20.535943238Z ERROR: 2026/01/19 08:22:20 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:22:23.565702869Z ERROR: 2026/01/19 08:22:23 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:22:26.589559041Z ERROR: 2026/01/19 08:22:26 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:22:28.411489169Z INFO: 2026/01/19 08:22:28 Failed to report peer bandwidth: failed to send bandwidth data: not connected 2026-01-19T08:22:29.607627805Z ERROR: 2026/01/19 08:22:29 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:22:32.632349282Z ERROR: 2026/01/19 08:22:32 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:22:35.656124130Z ERROR: 2026/01/19 08:22:35 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:22:38.410614825Z INFO: 2026/01/19 08:22:38 Failed to report peer bandwidth: failed to send bandwidth data: not connected 2026-01-19T08:22:38.679928807Z ERROR: 2026/01/19 08:22:38 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:22:41.704350980Z ERROR: 2026/01/19 08:22:41 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:22:44.727896881Z ERROR: 2026/01/19 08:22:44 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:22:47.752955017Z ERROR: 2026/01/19 08:22:47 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": dial tcp IP.IP.IP.IP:443: connect: connection refused. Retrying in 3s... 2026-01-19T08:22:48.411676059Z INFO: 2026/01/19 08:22:48 Failed to report peer bandwidth: failed to send bandwidth data: not connected 2026-01-19T08:22:50.805814197Z ERROR: 2026/01/19 08:22:50 Failed to connect: failed to get token: failed to request new token: Post "https://subdomain.domain.com/api/v1/auth/newt/get-token": tls: failed to verify certificate: x509: certificate is valid for xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.traefik.default, not subdomain.domain.com. Retrying in 3s... 2026-01-19T08:22:52.383491481Z WARN: 2026/01/19 08:22:52 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded 2026-01-19T08:22:54.087380976Z INFO: 2026/01/19 08:22:54 Server version: 1.14.1 2026-01-19T08:22:54.178556339Z INFO: 2026/01/19 08:22:54 Websocket connected 2026-01-19T08:22:57.449078521Z INFO: 2026/01/19 08:22:57 Connection to server restored after 6 failures! 2026-01-19T08:23:27.384423139Z WARN: 2026/01/19 08:23:27 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded 2026-01-19T08:24:02.385822911Z WARN: 2026/01/19 08:24:02 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded 2026-01-19T08:24:37.386416284Z WARN: 2026/01/19 08:24:37 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded 2026-01-19T08:25:12.388072138Z WARN: 2026/01/19 08:25:12 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded 2026-01-19T08:25:47.389372744Z WARN: 2026/01/19 08:25:47 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded 2026-01-19T08:26:22.391491664Z WARN: 2026/01/19 08:26:22 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded 2026-01-19T08:26:57.392505655Z WARN: 2026/01/19 08:26:57 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded 2026-01-19T08:27:32.394512058Z WARN: 2026/01/19 08:27:32 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded 2026-01-19T08:28:07.396001777Z WARN: 2026/01/19 08:28:07 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded 2026-01-19T08:28:42.397358840Z WARN: 2026/01/19 08:28:42 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded 2026-01-19T08:29:17.398991317Z WARN: 2026/01/19 08:29:17 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded 2026-01-19T08:29:52.399552209Z WARN: 2026/01/19 08:29:52 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded 2026-01-19T08:30:27.400009899Z WARN: 2026/01/19 08:30:27 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded 2026-01-19T08:31:02.401642318Z WARN: 2026/01/19 08:31:02 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded 2026-01-19T08:31:37.402527226Z WARN: 2026/01/19 08:31:37 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded 2026-01-19T08:32:12.404265370Z WARN: 2026/01/19 08:32:12 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded [newt] 2026-01-19T08:32:47.404196049Z WARN: 2026/01/19 08:32:47 Target 1: health check failed: Get "https://100.90.128.0:9200/": context deadline exceeded
GiteaMirror commented 2026-04-27 18:09:32 -05:00
Author
Owner
Copy Link

@icezar commented on GitHub (Mar 19, 2026):

Running Pangolin 1.x with Gerbil on the same stack. Seeing the identical panic:

panic: close of closed channel
goroutine 1 [running]:
github.com/fosrl/gerbil/relay.(*UDPProxyServer).Stop(0xc0001ac000)
        /app/relay/relay.go:237 +0x99
main.main()
        /app/main.go:420 +0x217a

In my case the panic doesn't always coincide with a stack restart — it also occurs randomly during normal operation (no obvious external trigger). After Gerbil recovers and re-registers with Pangolin, some resources end up in a broken state: health checks fail and traffic returns "no server available" via Traefik. The resources still exist in the Pangolin UI but are no longer functional. Deleting and recreating the resource entry in Pangolin fixes it, which suggests Pangolin doesn't fully re-push all resource mappings to Gerbil after the crash recovery cycle.
Workaround I'm testing: restarting the Pangolin container after a Gerbil crash forces a clean re-registration of all resources. Still confirming whether this is reliable.
The core bug is in relay.go:237 — a channel is being closed twice, likely due to a race between the shutdown signal and the UDP proxy server's own cleanup. A nil-check or sync.Once guard on that close would likely fix it.

<!-- gh-comment-id:4090404123 --> @icezar commented on GitHub (Mar 19, 2026): Running Pangolin 1.x with Gerbil on the same stack. Seeing the identical panic: ``` panic: close of closed channel goroutine 1 [running]: github.com/fosrl/gerbil/relay.(*UDPProxyServer).Stop(0xc0001ac000) /app/relay/relay.go:237 +0x99 main.main() /app/main.go:420 +0x217a ``` In my case the panic doesn't always coincide with a stack restart — it also occurs randomly during normal operation (no obvious external trigger). After Gerbil recovers and re-registers with Pangolin, some resources end up in a broken state: health checks fail and traffic returns "no server available" via Traefik. The resources still exist in the Pangolin UI but are no longer functional. Deleting and recreating the resource entry in Pangolin fixes it, which suggests Pangolin doesn't fully re-push all resource mappings to Gerbil after the crash recovery cycle. Workaround I'm testing: restarting the Pangolin container after a Gerbil crash forces a clean re-registration of all resources. Still confirming whether this is reliable. The core bug is in relay.go:237 — a channel is being closed twice, likely due to a race between the shutdown signal and the UDP proxy server's own cleanup. A nil-check or sync.Once guard on that close would likely fix it.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
enhancement
pull-request
Mirrored from GitHub Pull Request
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/gerbil#232
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?