github-starred/newt
2
0
Fork 0
mirror of https://github.com/fosrl/newt.git synced 2026-11-11 16:33:24 -06:00
Code Issues 713 Packages Projects Releases 47 Wiki Activity

[GH-ISSUE #7] Newt doesn't automatically reconnect to Pangolin if Pangolin is restarted #8215

New Issue
Closed
opened 2026-07-13 14:34:54 -05:00 by GiteaMirror · 14 comments
GiteaMirror commented 2026-07-13 14:34:54 -05:00
Owner
Copy Link
Copy Source

Originally created by @wkulhanek on GitHub (Jan 23, 2025).
Original GitHub issue: https://github.com/fosrl/newt/issues/7

Originally assigned to: @oschwartz10612 on GitHub.

It appears that Newt doesn't automatically reconnect to Pangolin if Pangolin is restarted.

I have a Pangolin on a public cloud instance. That public cloud instance has a Newt instance as well. And then I have Newt instance on my private (homelab) machine. Whenever the public cloud instance gets restarted (or even if just the Pangolin service is restarted) I get a "Bad Gateway" when I try to access resources on my homelab. Restarting Newt fixes this.

It would be great if Newt would recognize that Pangolin got restarted and somehow reconnect to the Pangolin instance.

For reference here is my Newt system service (on Fedora Server 41):

[Unit]
Description=newt Container Service
Wants=network.target
After=pangolin.service
Requires=pangolin.service

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=always
ExecStartPre=-/usr/bin/podman rm -f newt
ExecStartPre=/bin/sleep 15
ExecStart=/usr/bin/podman run --name newt \
  --rm \
  --label "io.containers.autoupdate=image" \
  --user 1501:100 \
  --env NEWT_ID=<redacted> \
  --env NEWT_SECRET=<redacted> \
  --env PANGOLIN_ENDPOINT=<redacted> \
  docker.io/fosrl/newt:latest
ExecStop=/usr/bin/podman rm -f newt
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=newt

[Install]
WantedBy=multi-user.target default.target
Originally created by @wkulhanek on GitHub (Jan 23, 2025). Original GitHub issue: https://github.com/fosrl/newt/issues/7 Originally assigned to: @oschwartz10612 on GitHub. It appears that Newt doesn't automatically reconnect to Pangolin if Pangolin is restarted. I have a Pangolin on a public cloud instance. That public cloud instance has a Newt instance as well. And then I have Newt instance on my private (homelab) machine. Whenever the public cloud instance gets restarted (or even if just the Pangolin service is restarted) I get a "Bad Gateway" when I try to access resources on my homelab. Restarting Newt fixes this. It would be great if Newt would recognize that Pangolin got restarted and somehow reconnect to the Pangolin instance. For reference here is my Newt system service (on Fedora Server 41): ``` [Unit] Description=newt Container Service Wants=network.target After=pangolin.service Requires=pangolin.service [Service] Environment=PODMAN_SYSTEMD_UNIT=%n Restart=always ExecStartPre=-/usr/bin/podman rm -f newt ExecStartPre=/bin/sleep 15 ExecStart=/usr/bin/podman run --name newt \ --rm \ --label "io.containers.autoupdate=image" \ --user 1501:100 \ --env NEWT_ID=<redacted> \ --env NEWT_SECRET=<redacted> \ --env PANGOLIN_ENDPOINT=<redacted> \ docker.io/fosrl/newt:latest ExecStop=/usr/bin/podman rm -f newt StandardOutput=syslog StandardError=syslog SyslogIdentifier=newt [Install] WantedBy=multi-user.target default.target ```
GiteaMirror added the bug label 2026-07-13 14:34:55 -05:00
GiteaMirror commented 2026-07-13 14:35:04 -05:00
Author
Owner
Copy Link
Copy Source

@oschwartz10612 commented on GitHub (Jan 25, 2025):

Hi!

I think right now there is logic in Newt that should allow it to reconnect after Gerbil or Pangolin is restarted. I just tested and was able to see this working on my instance.

Could you please try to reproduce the issue and provide logs from Newt from when it can not reconnect? You can use the LOG_LEVEL env var to set to DEBUG to get more detail.

<!-- gh-comment-id:2614036503 --> @oschwartz10612 commented on GitHub (Jan 25, 2025): Hi! I think right now there is logic in Newt that should allow it to reconnect after Gerbil or Pangolin is restarted. I just tested and was able to see this working on my instance. Could you please try to reproduce the issue and provide logs from Newt from when it can not reconnect? You can use the LOG_LEVEL env var to set to DEBUG to get more detail.
GiteaMirror commented 2026-07-13 14:35:05 -05:00
Author
Owner
Copy Link
Copy Source

@wkulhanek commented on GitHub (Jan 26, 2025):

I did a few more tests. A systemctl restart pangolin on the cloud instance killed the newt instance on the cloud instance - which could be because the newt service has a dependency on the pangolin service. The newt instance in my homelab did in fact re-connect successfully.

So then I just tried rebooting the cloud instance. That brought pangolin and newt back up on the cloud instance - but the newt instance in my homelab never reconnected. The logs say it attempted to ping the pangolin server 5 times and then it gave up. I guess it needs to keep trying? Any attempt to access a web site behind the newt service in my homelab then ends with "Bad Gateway"

INFO: 2025/01/25 12:51:18 Started tcp proxy from 100.89.128.8:62587 to 192.168.2.210:9000
ERROR: 2025/01/26 13:36:18 Ping failed: use of closed network connection
INFO: 2025/01/26 13:36:18 Sent registration message
INFO: 2025/01/26 13:36:18 Received registration message
INFO: 2025/01/26 13:36:18 Already connected! But I will send a ping anyway...
INFO: 2025/01/26 13:36:18 Ping attempt 1 of 5
INFO: 2025/01/26 13:36:18 Pinging 100.89.128.1
WARN: 2025/01/26 13:36:28 Ping attempt 1 failed: failed to read ICMP packet: i/o timeout
INFO: 2025/01/26 13:36:30 Ping attempt 2 of 5
INFO: 2025/01/26 13:36:30 Pinging 100.89.128.1
INFO: 2025/01/26 13:36:30 Ping latency: 35.659279ms
ERROR: 2025/01/26 14:04:48 Ping failed: use of closed network connection
ERROR: 2025/01/26 14:05:08 Failed to connect: failed to get token: failed to request new token: Post "https://<redacted>/api/v1/auth/newt/get-token": dial tcp <redacted>:443: connect: connection refused. Retrying in 10s...
ERROR: 2025/01/26 14:05:18 Failed to connect: failed to get token: failed to request new token: Post "https://<redacted>/api/v1/auth/newt/get-token": dial tcp <redacted>:443: connect: connection refused. Retrying in 10s...
INFO: 2025/01/26 14:05:28 Sent registration message
INFO: 2025/01/26 14:05:29 Received registration message
INFO: 2025/01/26 14:05:29 Already connected! But I will send a ping anyway...
INFO: 2025/01/26 14:05:29 Ping attempt 1 of 5
INFO: 2025/01/26 14:05:29 Pinging 100.89.128.1
WARN: 2025/01/26 14:05:39 Ping attempt 1 failed: failed to read ICMP packet: i/o timeout
INFO: 2025/01/26 14:05:41 Ping attempt 2 of 5
INFO: 2025/01/26 14:05:41 Pinging 100.89.128.1
WARN: 2025/01/26 14:05:51 Ping attempt 2 failed: failed to read ICMP packet: i/o timeout
INFO: 2025/01/26 14:05:53 Ping attempt 3 of 5
INFO: 2025/01/26 14:05:53 Pinging 100.89.128.1
WARN: 2025/01/26 14:06:03 Ping attempt 3 failed: failed to read ICMP packet: i/o timeout
INFO: 2025/01/26 14:06:05 Ping attempt 4 of 5
INFO: 2025/01/26 14:06:05 Pinging 100.89.128.1
WARN: 2025/01/26 14:06:15 Ping attempt 4 failed: failed to read ICMP packet: i/o timeout
INFO: 2025/01/26 14:06:17 Ping attempt 5 of 5
INFO: 2025/01/26 14:06:17 Pinging 100.89.128.1
WARN: 2025/01/26 14:06:27 Ping attempt 5 failed: failed to read ICMP packet: i/o timeout
WARN: 2025/01/26 14:06:27 Failed to ping 100.89.128.1: all ping attempts failed after 5 tries, last error: failed to read ICMP packet: i/o timeout
WARN: 2025/01/26 14:06:27 HINT: Do you have UDP port 51280 (or the port in config.yml) open on your Pangolin server?
<!-- gh-comment-id:2614442235 --> @wkulhanek commented on GitHub (Jan 26, 2025): I did a few more tests. A `systemctl restart pangolin` on the cloud instance killed the newt instance on the cloud instance - which could be because the newt service has a dependency on the pangolin service. The newt instance in my homelab did in fact re-connect successfully. So then I just tried rebooting the cloud instance. That brought pangolin and newt back up on the cloud instance - but the newt instance in my homelab never reconnected. The logs say it attempted to ping the pangolin server 5 times and then it gave up. I guess it needs to keep trying? Any attempt to access a web site behind the newt service in my homelab then ends with "Bad Gateway" ``` INFO: 2025/01/25 12:51:18 Started tcp proxy from 100.89.128.8:62587 to 192.168.2.210:9000 ERROR: 2025/01/26 13:36:18 Ping failed: use of closed network connection INFO: 2025/01/26 13:36:18 Sent registration message INFO: 2025/01/26 13:36:18 Received registration message INFO: 2025/01/26 13:36:18 Already connected! But I will send a ping anyway... INFO: 2025/01/26 13:36:18 Ping attempt 1 of 5 INFO: 2025/01/26 13:36:18 Pinging 100.89.128.1 WARN: 2025/01/26 13:36:28 Ping attempt 1 failed: failed to read ICMP packet: i/o timeout INFO: 2025/01/26 13:36:30 Ping attempt 2 of 5 INFO: 2025/01/26 13:36:30 Pinging 100.89.128.1 INFO: 2025/01/26 13:36:30 Ping latency: 35.659279ms ERROR: 2025/01/26 14:04:48 Ping failed: use of closed network connection ERROR: 2025/01/26 14:05:08 Failed to connect: failed to get token: failed to request new token: Post "https://<redacted>/api/v1/auth/newt/get-token": dial tcp <redacted>:443: connect: connection refused. Retrying in 10s... ERROR: 2025/01/26 14:05:18 Failed to connect: failed to get token: failed to request new token: Post "https://<redacted>/api/v1/auth/newt/get-token": dial tcp <redacted>:443: connect: connection refused. Retrying in 10s... INFO: 2025/01/26 14:05:28 Sent registration message INFO: 2025/01/26 14:05:29 Received registration message INFO: 2025/01/26 14:05:29 Already connected! But I will send a ping anyway... INFO: 2025/01/26 14:05:29 Ping attempt 1 of 5 INFO: 2025/01/26 14:05:29 Pinging 100.89.128.1 WARN: 2025/01/26 14:05:39 Ping attempt 1 failed: failed to read ICMP packet: i/o timeout INFO: 2025/01/26 14:05:41 Ping attempt 2 of 5 INFO: 2025/01/26 14:05:41 Pinging 100.89.128.1 WARN: 2025/01/26 14:05:51 Ping attempt 2 failed: failed to read ICMP packet: i/o timeout INFO: 2025/01/26 14:05:53 Ping attempt 3 of 5 INFO: 2025/01/26 14:05:53 Pinging 100.89.128.1 WARN: 2025/01/26 14:06:03 Ping attempt 3 failed: failed to read ICMP packet: i/o timeout INFO: 2025/01/26 14:06:05 Ping attempt 4 of 5 INFO: 2025/01/26 14:06:05 Pinging 100.89.128.1 WARN: 2025/01/26 14:06:15 Ping attempt 4 failed: failed to read ICMP packet: i/o timeout INFO: 2025/01/26 14:06:17 Ping attempt 5 of 5 INFO: 2025/01/26 14:06:17 Pinging 100.89.128.1 WARN: 2025/01/26 14:06:27 Ping attempt 5 failed: failed to read ICMP packet: i/o timeout WARN: 2025/01/26 14:06:27 Failed to ping 100.89.128.1: all ping attempts failed after 5 tries, last error: failed to read ICMP packet: i/o timeout WARN: 2025/01/26 14:06:27 HINT: Do you have UDP port 51280 (or the port in config.yml) open on your Pangolin server? ```
GiteaMirror commented 2026-07-13 14:35:05 -05:00
Author
Owner
Copy Link
Copy Source

@oschwartz10612 commented on GitHub (Jan 26, 2025):

on the cloud instance killed the newt instance on the cloud instance

I am just curious why you have Newt on the cloud instance with Pangolin? Are you trying to proxy to stuff on the VPS itself? If so, you can use "local" sites.

The logs say it attempted to ping the pangolin server 5 times and then it gave up. I guess it needs to keep trying?

Yep that seems like it! It can handle a short downtime but a longer one means it stops pinging. I will make this change in the next Newt release!

<!-- gh-comment-id:2614459101 --> @oschwartz10612 commented on GitHub (Jan 26, 2025): > on the cloud instance killed the newt instance on the cloud instance I am just curious why you have Newt on the cloud instance with Pangolin? Are you trying to proxy to stuff on the VPS itself? If so, you can use ["local" sites](https://docs.fossorial.io/Pangolin/without-tunneling). > The logs say it attempted to ping the pangolin server 5 times and then it gave up. I guess it needs to keep trying? Yep that seems like it! It can handle a short downtime but a longer one means it stops pinging. I will make this change in the next Newt release!
GiteaMirror commented 2026-07-13 14:35:06 -05:00
Author
Owner
Copy Link
Copy Source

@wkulhanek commented on GitHub (Jan 26, 2025):

My cloud instance is doing double duty as a Pangolin server - but it also has Nextcloud and a few other services on it. I had used Cloudflare tunnels before switching to Pangolin to access these sites. I didn't realize that I don't need Newt for that. :-)

<!-- gh-comment-id:2614460278 --> @wkulhanek commented on GitHub (Jan 26, 2025): My cloud instance is doing double duty as a Pangolin server - but it also has Nextcloud and a few other services on it. I had used Cloudflare tunnels before switching to Pangolin to access these sites. I didn't realize that I don't need Newt for that. :-)
GiteaMirror commented 2026-07-13 14:35:07 -05:00
Author
Owner
Copy Link
Copy Source

@jibsaram commented on GitHub (Feb 4, 2025):

I'm seeing the same problem with a Newt docker container on one device being unable to reconnect to Pangolin running on another machine (and network), after I updated-and-restarted the Pangolin container from beta 10 to 11.

The Newt container is just showing the same log message over and over:

newt  | ERROR: 2025/02/04 01:42:31 Failed to connect: failed to get token: failed to request new token: Post "https://<<domain>>/api/v1/auth/newt/get-token": dial tcp <<ip>>:443: connect: connection refused. Retrying in 10s...
newt  | ERROR: 2025/02/04 01:42:41 Failed to connect: failed to get token: failed to request new token: Post "https://<<domain>>/api/v1/auth/newt/get-token": dial tcp <<ip>>:443: connect: connection refused. Retrying in 10s...
newt  | ERROR: 2025/02/04 01:42:51 Failed to connect: failed to get token: failed to request new token: Post "https://<<domain>>/api/v1/auth/newt/get-token": dial tcp <<ip>>:443: connect: connection refused. Retrying in 10s...

I am able to ping <<domain>> just fine from the host machine that's running the Newt container, and can curl the URL in question too (I just did a basic GET request that returned a JSON object with "unauthorized" error message, as expected). I could not test this from within the Newt container as it does not have ping/curl/wget.

Restarting the Newt container on its own did not help, though when I made the entire docker compose stack go down then up again, it did successfully reconnect to Pangolin. I am not sure if it was due to the down/up step or something else, but thought I'd mention it just in case.

Just to see if I could reproduce this I just tried restarting the Pangolin service twice (sans updating, as it's already up-to-date), but the Newt instance was able to successfully reconnect both times. So I'm not quite sure yet what the pattern might be. I did notice that both times it reconnected fine, the first two ping attempts consistently failed. I don't know if this is at all relevant/useful, but I thought I'd mention it just in case.

(...snip...)
newt  | ERROR: 2025/02/04 06:00:27 Ping failed: use of closed network connection
newt  | INFO: 2025/02/04 06:00:27 Sent registration message
newt  | INFO: 2025/02/04 06:00:27 Received registration message
newt  | INFO: 2025/02/04 06:00:27 Already connected! But I will send a ping anyway...
newt  | INFO: 2025/02/04 06:00:27 Ping attempt 1 of 5
newt  | INFO: 2025/02/04 06:00:27 Pinging 100.89.128.1
newt  | WARN: 2025/02/04 06:00:37 Ping attempt 1 failed: failed to read ICMP packet: i/o timeout
newt  | INFO: 2025/02/04 06:00:39 Ping attempt 2 of 5
newt  | INFO: 2025/02/04 06:00:39 Pinging 100.89.128.1
newt  | WARN: 2025/02/04 06:00:49 Ping attempt 2 failed: failed to read ICMP packet: i/o timeout
newt  | INFO: 2025/02/04 06:00:51 Ping attempt 3 of 5
newt  | INFO: 2025/02/04 06:00:51 Pinging 100.89.128.1
newt  | INFO: 2025/02/04 06:00:51 Ping latency: 4.604098ms

Please let me know if there is anything more I can provide. Thanks!

Update: Updated from beta 11 to 12 just now whilst following logs of the newt instance and pangolin both, it worked fine and the newt instance automatically reconnected just fine. At least from my end it doesn't appear to be post-upgrade specific behavior. I hope this helps!

<!-- gh-comment-id:2632958472 --> @jibsaram commented on GitHub (Feb 4, 2025): I'm seeing the same problem with a Newt docker container on one device being unable to reconnect to Pangolin running on another machine (and network), after I updated-and-restarted the Pangolin container from beta 10 to 11. The Newt container is just showing the same log message over and over: ```log newt | ERROR: 2025/02/04 01:42:31 Failed to connect: failed to get token: failed to request new token: Post "https://<<domain>>/api/v1/auth/newt/get-token": dial tcp <<ip>>:443: connect: connection refused. Retrying in 10s... newt | ERROR: 2025/02/04 01:42:41 Failed to connect: failed to get token: failed to request new token: Post "https://<<domain>>/api/v1/auth/newt/get-token": dial tcp <<ip>>:443: connect: connection refused. Retrying in 10s... newt | ERROR: 2025/02/04 01:42:51 Failed to connect: failed to get token: failed to request new token: Post "https://<<domain>>/api/v1/auth/newt/get-token": dial tcp <<ip>>:443: connect: connection refused. Retrying in 10s... ``` I am able to ping `<<domain>>` just fine from the host machine that's running the Newt container, and can `curl` the URL in question too (I just did a basic GET request that returned a JSON object with "unauthorized" error message, as expected). I could not test this from within the Newt container as it does not have `ping`/`curl`/`wget`. Restarting the Newt container on its own did not help, though when I made the entire docker compose stack go `down` then `up` again, it did successfully reconnect to Pangolin. I am not sure if it was due to the down/up step or something else, but thought I'd mention it just in case. --- Just to see if I could reproduce this I just tried restarting the Pangolin service twice (sans updating, as it's already up-to-date), but the Newt instance was able to successfully reconnect both times. So I'm not quite sure yet what the pattern might be. I did notice that both times it reconnected fine, the first two ping attempts consistently failed. I don't know if this is at all relevant/useful, but I thought I'd mention it just in case. ```log (...snip...) newt | ERROR: 2025/02/04 06:00:27 Ping failed: use of closed network connection newt | INFO: 2025/02/04 06:00:27 Sent registration message newt | INFO: 2025/02/04 06:00:27 Received registration message newt | INFO: 2025/02/04 06:00:27 Already connected! But I will send a ping anyway... newt | INFO: 2025/02/04 06:00:27 Ping attempt 1 of 5 newt | INFO: 2025/02/04 06:00:27 Pinging 100.89.128.1 newt | WARN: 2025/02/04 06:00:37 Ping attempt 1 failed: failed to read ICMP packet: i/o timeout newt | INFO: 2025/02/04 06:00:39 Ping attempt 2 of 5 newt | INFO: 2025/02/04 06:00:39 Pinging 100.89.128.1 newt | WARN: 2025/02/04 06:00:49 Ping attempt 2 failed: failed to read ICMP packet: i/o timeout newt | INFO: 2025/02/04 06:00:51 Ping attempt 3 of 5 newt | INFO: 2025/02/04 06:00:51 Pinging 100.89.128.1 newt | INFO: 2025/02/04 06:00:51 Ping latency: 4.604098ms ``` Please let me know if there is anything more I can provide. Thanks! **Update:** Updated from beta 11 to 12 just now whilst following logs of the newt instance and pangolin both, it worked fine and the newt instance automatically reconnected just fine. At least from my end it doesn't _appear_ to be post-upgrade specific behavior. I hope this helps!
GiteaMirror commented 2026-07-13 14:35:08 -05:00
Author
Owner
Copy Link
Copy Source

@miloschwartz commented on GitHub (Mar 16, 2025):

Closing because we cannot reproduce. Please add to this issue in the future if it's still a problem.

<!-- gh-comment-id:2727643194 --> @miloschwartz commented on GitHub (Mar 16, 2025): Closing because we cannot reproduce. Please add to this issue in the future if it's still a problem.
GiteaMirror commented 2026-07-13 14:35:08 -05:00
Author
Owner
Copy Link
Copy Source

@wkulhanek commented on GitHub (Mar 21, 2025):

@miloschwartz I don't understand - I just had this again. Reboot my cloud instance, local newt tries to reconnect 5 times and then gives up. This is not hard to reproduce.

INFO: 2025/03/20 15:12:26 Started tcp proxy from 100.89.128.12:41618 to 192.168.2.101:8040
ERROR: 2025/03/21 09:53:56 Ping failed: use of closed network connection
ERROR: 2025/03/21 09:54:07 Failed to connect: failed to get token: failed to request new token: Post "https://<redacted>/api/v1/auth/newt/get-token": dial tcp <redacted>:443: connect: connection refused. Retrying in 10s...
ERROR: 2025/03/21 09:54:17 Failed to connect: failed to get token: failed to request new token: Post "https://<redacted>/api/v1/auth/newt/get-token": dial tcp <redacted>:443: connect: connection refused. Retrying in 10s...
ERROR: 2025/03/21 09:54:27 Failed to connect: failed to get token: failed to request new token: Post "https://<redacted>/api/v1/auth/newt/get-token": dial tcp <redacted>:443: connect: connection refused. Retrying in 10s...
ERROR: 2025/03/21 09:54:39 Failed to save config: open /.config/newt-client/config.json: no such file or directory
2025/03/21 09:54:39 Failed to create config directory: mkdir /.config: permission denied
INFO: 2025/03/21 09:54:39 Sent registration message
INFO: 2025/03/21 09:54:39 Received registration message
INFO: 2025/03/21 09:54:39 Already connected! But I will send a ping anyway...
INFO: 2025/03/21 09:54:39 Ping attempt 1 of 5
INFO: 2025/03/21 09:54:39 Pinging 100.89.128.1
WARN: 2025/03/21 09:54:49 Ping attempt 1 failed: failed to read ICMP packet: i/o timeout
INFO: 2025/03/21 09:54:51 Ping attempt 2 of 5
INFO: 2025/03/21 09:54:51 Pinging 100.89.128.1
WARN: 2025/03/21 09:55:01 Ping attempt 2 failed: failed to read ICMP packet: i/o timeout
INFO: 2025/03/21 09:55:03 Ping attempt 3 of 5
INFO: 2025/03/21 09:55:03 Pinging 100.89.128.1
WARN: 2025/03/21 09:55:13 Ping attempt 3 failed: failed to read ICMP packet: i/o timeout
INFO: 2025/03/21 09:55:15 Ping attempt 4 of 5
INFO: 2025/03/21 09:55:15 Pinging 100.89.128.1
WARN: 2025/03/21 09:55:25 Ping attempt 4 failed: failed to read ICMP packet: i/o timeout
INFO: 2025/03/21 09:55:27 Ping attempt 5 of 5
INFO: 2025/03/21 09:55:27 Pinging 100.89.128.1
WARN: 2025/03/21 09:55:37 Ping attempt 5 failed: failed to read ICMP packet: i/o timeout
WARN: 2025/03/21 09:55:37 Failed to ping 100.89.128.1: all ping attempts failed after 5 tries, last error: failed to read ICMP packet: i/o timeout
WARN: 2025/03/21 09:55:37 HINT: Do you have UDP port 51820 (or the port in config.yml) open on your Pangolin server?
<!-- gh-comment-id:2742884977 --> @wkulhanek commented on GitHub (Mar 21, 2025): @miloschwartz I don't understand - I just had this again. Reboot my cloud instance, local newt tries to reconnect 5 times and then gives up. This is not hard to reproduce. ``` INFO: 2025/03/20 15:12:26 Started tcp proxy from 100.89.128.12:41618 to 192.168.2.101:8040 ERROR: 2025/03/21 09:53:56 Ping failed: use of closed network connection ERROR: 2025/03/21 09:54:07 Failed to connect: failed to get token: failed to request new token: Post "https://<redacted>/api/v1/auth/newt/get-token": dial tcp <redacted>:443: connect: connection refused. Retrying in 10s... ERROR: 2025/03/21 09:54:17 Failed to connect: failed to get token: failed to request new token: Post "https://<redacted>/api/v1/auth/newt/get-token": dial tcp <redacted>:443: connect: connection refused. Retrying in 10s... ERROR: 2025/03/21 09:54:27 Failed to connect: failed to get token: failed to request new token: Post "https://<redacted>/api/v1/auth/newt/get-token": dial tcp <redacted>:443: connect: connection refused. Retrying in 10s... ERROR: 2025/03/21 09:54:39 Failed to save config: open /.config/newt-client/config.json: no such file or directory 2025/03/21 09:54:39 Failed to create config directory: mkdir /.config: permission denied INFO: 2025/03/21 09:54:39 Sent registration message INFO: 2025/03/21 09:54:39 Received registration message INFO: 2025/03/21 09:54:39 Already connected! But I will send a ping anyway... INFO: 2025/03/21 09:54:39 Ping attempt 1 of 5 INFO: 2025/03/21 09:54:39 Pinging 100.89.128.1 WARN: 2025/03/21 09:54:49 Ping attempt 1 failed: failed to read ICMP packet: i/o timeout INFO: 2025/03/21 09:54:51 Ping attempt 2 of 5 INFO: 2025/03/21 09:54:51 Pinging 100.89.128.1 WARN: 2025/03/21 09:55:01 Ping attempt 2 failed: failed to read ICMP packet: i/o timeout INFO: 2025/03/21 09:55:03 Ping attempt 3 of 5 INFO: 2025/03/21 09:55:03 Pinging 100.89.128.1 WARN: 2025/03/21 09:55:13 Ping attempt 3 failed: failed to read ICMP packet: i/o timeout INFO: 2025/03/21 09:55:15 Ping attempt 4 of 5 INFO: 2025/03/21 09:55:15 Pinging 100.89.128.1 WARN: 2025/03/21 09:55:25 Ping attempt 4 failed: failed to read ICMP packet: i/o timeout INFO: 2025/03/21 09:55:27 Ping attempt 5 of 5 INFO: 2025/03/21 09:55:27 Pinging 100.89.128.1 WARN: 2025/03/21 09:55:37 Ping attempt 5 failed: failed to read ICMP packet: i/o timeout WARN: 2025/03/21 09:55:37 Failed to ping 100.89.128.1: all ping attempts failed after 5 tries, last error: failed to read ICMP packet: i/o timeout WARN: 2025/03/21 09:55:37 HINT: Do you have UDP port 51820 (or the port in config.yml) open on your Pangolin server? ```
GiteaMirror commented 2026-07-13 14:35:09 -05:00
Author
Owner
Copy Link
Copy Source

@oschwartz10612 commented on GitHub (Mar 21, 2025):

Increased the ping to 15 in 9d80161. Will release soon - might help. Let us know.

<!-- gh-comment-id:2744476347 --> @oschwartz10612 commented on GitHub (Mar 21, 2025): Increased the ping to 15 in [9d80161](https://github.com/fosrl/newt/commit/9d80161ab7bc86c4be1ab1a4fc1f5f0671dc0360). Will release soon - might help. Let us know.
GiteaMirror commented 2026-07-13 14:35:10 -05:00
Author
Owner
Copy Link
Copy Source

@Snocember commented on GitHub (May 10, 2026):

I also have this issue for a while now.

Reproducible on:

Pangolin host:

linux/arm64
Rocky Linux 10.1 (6.12.0-124.52.1.el10_1.aarch64)
podman version 5.6.0
fosrl/pangolin:latest (1.18.3, sha256:94e04ae48eec8a2f2c9925f1f12b8016abe5093f41f6f9829a0fca98e06b7fe0)

Newt client:

linux/amd64
Debian 12.3 (6.1.0-45-amd64) and Rocky Linux 9.7 (5.14.0-611.54.1.el9_7.x86_64)
Docker version 29.4.3, build 055a478
fosrl/newt:latest (sha256:881cd45911cc8a8e5f347edf4a7a278fb93f337af8e588b94f9042465d81c891)

Steps to reproduce:

  1. Start pangolin server and newt clients
  2. Stop and restart pangolin
    podman compose down and up -d ist used
    All routes to the sites connected with newt now show "Bad Gateway".

When restarting the newt client, the problem resolves.

Logs

Newt client:

newt  | DEBUG: 2026/05/10 14:54:04 Increased hole punch interval to 32s
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:54:08 peer(FoT7…i6ng) - Sending keepalive packet
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:54:17 peer(FoT7…i6ng) - Sending keepalive packet
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:54:23 peer(FoT7…i6ng) - Sending keepalive packet
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:54:28 peer(FoT7…i6ng) - Sending keepalive packet

After pangolin server restart:

newt  | DEBUG: 2026/05/10 14:54:36 Sent UDP hole punch to <ip>:21820: {"ephemeralPublicKey":"...","nonce":"...","ciphertext":"..."}
newt  | DEBUG: 2026/05/10 14:54:36 Increased hole punch interval to 1m0s
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:54:38 peer(FoT7…i6ng) - Sending keepalive packet
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:54:43 peer(FoT7…i6ng) - Sending keepalive packet
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:54:53 peer(FoT7…i6ng) - Sending keepalive packet
newt  | DEBUG: 2026/05/10 14:54:55 Ping attempt 1/2 failed: failed to read ICMP packet: i/o timeout
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:55:00 peer(FoT7…i6ng) - Sending keepalive packet
newt  | DEBUG: 2026/05/10 14:55:02 Ping attempt 2/2 failed: failed to read ICMP packet: i/o timeout
newt  | DEBUG: 2026/05/10 14:55:02 Periodic ping failed (1 consecutive failures): all 2 ping attempts failed, last error: failed to read ICMP packet: i/o timeout
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:55:03 peer(FoT7…i6ng) - Retrying handshake because we stopped hearing back after 15 seconds
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:55:03 peer(FoT7…i6ng) - Sending handshake initiation
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:55:08 peer(FoT7…i6ng) - Sending keepalive packet
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:55:08 peer(FoT7…i6ng) - Handshake did not complete after 5 seconds, retrying (try 2)
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:55:08 peer(FoT7…i6ng) - Sending handshake initiation
newt  | DEBUG: 2026/05/10 14:55:10 Ping attempt 1/2 failed: failed to read ICMP packet: i/o timeout
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:55:13 peer(FoT7…i6ng) - Handshake did not complete after 5 seconds, retrying (try 3)
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:55:13 peer(FoT7…i6ng) - Sending handshake initiation
newt  | DEBUG: 2026/05/10 14:55:17 Ping attempt 2/2 failed: failed to read ICMP packet: i/o timeout
newt  | WARN: 2026/05/10 14:55:17 Periodic ping failed (2 consecutive failures): all 2 ping attempts failed, last error: failed to read ICMP packet: i/o timeout
...
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:56:15 peer(FoT7…i6ng) - Handshake did not complete after 5 seconds, retrying (try 2)
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:56:15 peer(FoT7…i6ng) - Sending handshake initiation
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:56:20 peer(FoT7…i6ng) - Sending keepalive packet
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:56:20 peer(FoT7…i6ng) - Sending handshake initiation
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:56:25 peer(FoT7…i6ng) - Sending keepalive packet
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:56:25 peer(FoT7…i6ng) - Sending handshake initiation
newt  | DEBUG: 2026/05/10 14:56:29 Ping attempt 1/4 failed: failed to read ICMP packet: i/o timeout
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:56:30 peer(FoT7…i6ng) - Retrying handshake because we stopped hearing back after 15 seconds
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:56:30 peer(FoT7…i6ng) - Handshake did not complete after 5 seconds, retrying (try 2)
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:56:30 peer(FoT7…i6ng) - Sending handshake initiation
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:56:35 peer(FoT7…i6ng) - Sending keepalive packet
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:56:35 peer(FoT7…i6ng) - Sending handshake initiation
newt  | DEBUG: 2026/05/10 14:56:36 Sent UDP hole punch to <ip>:21820: {"ephemeralPublicKey":"...","nonce":"...","ciphertext":"..."}
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:56:40 peer(FoT7…i6ng) - Sending handshake initiation
newt  | DEBUG: 2026/05/10 14:56:43 Ping attempt 2/4 failed: failed to read ICMP packet: i/o timeout
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:56:45 peer(FoT7…i6ng) - Sending handshake initiation
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:56:50 peer(FoT7…i6ng) - Sending handshake initiation
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:56:55 peer(FoT7…i6ng) - Sending handshake initiation
newt  | DEBUG: 2026/05/10 14:56:58 Ping attempt 3/4 failed: failed to read ICMP packet: i/o timeout
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:57:00 peer(FoT7…i6ng) - Sending handshake initiation
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:57:05 peer(FoT7…i6ng) - Sending handshake initiation
newt  | DEBUG: gerbil-wireguard: 2026/05/10 14:57:10 peer(FoT7…i6ng) - Sending handshake initiation
newt  | DEBUG: 2026/05/10 14:57:14 Ping attempt 4/4 failed: failed to read ICMP packet: i/o timeout
newt  | WARN: 2026/05/10 14:57:14 Periodic ping failed (6 consecutive failures): all 4 ping attempts failed, last error: failed to read ICMP packet: i/o timeout

Same time with only INFO enabled I encountered also an error:

newt  | INFO: 2026/05/10 14:45:37 Failed to report peer bandwidth: failed to send bandwidth data: not connected
newt  | INFO: 2026/05/10 14:45:47 Failed to report peer bandwidth: failed to send bandwidth data: not connected
newt  | INFO: 2026/05/10 14:45:57 Failed to report peer bandwidth: failed to send bandwidth data: not connected
newt  | ERROR: 2026/05/10 14:46:02 Failed to connect: failed to get token: failed to request new token: Post "https://<domain>/api/v1/auth/newt/get-token": dial tcp <ip>:443: i/o timeout. Retrying in 3s...
newt  | INFO: 2026/05/10 14:46:07 Failed to report peer bandwidth: failed to send bandwidth data: not connected
newt  | WARN: 2026/05/10 14:46:12 Periodic ping failed (2 consecutive failures): all 2 ping attempts failed, last error: failed to read ICMP packet: i/o timeout
newt  | INFO: 2026/05/10 14:46:17 Failed to report peer bandwidth: failed to send bandwidth data: not connected
newt  | WARN: 2026/05/10 14:46:27 Periodic ping failed (3 consecutive failures): all 2 ping attempts failed, last error: failed to read ICMP packet: i/o timeout
newt  | INFO: 2026/05/10 14:46:27 Failed to report peer bandwidth: failed to send bandwidth data: not connected
newt  | ERROR: 2026/05/10 14:46:35 Failed to connect: failed to get token: failed to request new token: Post "https://<domain>/api/v1/auth/newt/get-token": dial tcp <ip>:443: i/o timeout. Retrying in 3s...
newt  | INFO: 2026/05/10 14:46:37 Failed to report peer bandwidth: failed to send bandwidth data: not connected
newt  | INFO: 2026/05/10 14:46:38 Server version: 1.18.3
newt  | INFO: 2026/05/10 14:46:38 Websocket connected
newt  | WARN: 2026/05/10 14:46:46 Periodic ping failed (4 consecutive failures): all 2 ping attempts failed, last error: failed to read ICMP packet: i/o timeout

Pangolin host:

c24505f94a47 > @fosrl/pangolin@0.0.0 start
c24505f94a47 > ENVIRONMENT=prod node dist/migrations.mjs && ENVIRONMENT=prod NODE_ENV=development node --enable-source-maps dist/server.mjs
c24505f94a47 
c24505f94a47 Starting migrations from version 1.18.3
c24505f94a47 Migrations to run: 
c24505f94a47 All migrations completed successfully
...
d5a492b45dea INFO: 2026/05/10 14:55:05 WireGuard firewall rules successfully configured for interface wg0
d5a492b45dea INFO: 2026/05/10 14:55:05 WireGuard interface wg0 created and configured
d5a492b45dea INFO: 2026/05/10 14:55:05 Peer 4f0...s3c= added successfully
d5a492b45dea INFO: 2026/05/10 14:55:05 Peer Pep...aWM= added successfully
d5a492b45dea INFO: 2026/05/10 14:55:05 Received initial mappings: {"mappings":{"<ip>:52136":{"destinations":[{"destinationIP":"100.89.128.8","destinationPort":53585}]},"<ip>:53585":{"destinations":[{"destinationIP":"100.89.128.4","destinationPort":52136}]}}}
d5a492b45dea INFO: 2026/05/10 14:55:05 Loaded 2 initial proxy mappings
d5a492b45dea INFO: 2026/05/10 14:55:05 UDP server listening on :21820
d5a492b45dea INFO: 2026/05/10 14:55:05 Starting 20 packet workers (CPUs: 1)
d5a492b45dea INFO: 2026/05/10 14:55:05 Metrics endpoint enabled at /metrics
d5a492b45dea INFO: 2026/05/10 14:55:05 Starting HTTP server on :3004
c24505f94a47 2026-05-10T14:56:29+00:00 [info]: Marking site 1 offline: newt cmhmqw0lpvfp5sr has no recent ping and no active WebSocket connection
c24505f94a47 2026-05-10T14:56:29+00:00 [info]: Marking site 2 offline: newt 6tssbbzooylnp31 has no recent ping and no active WebSocket connection

When newt is restarted:

d5a492b45dea INFO: 2026/05/10 14:57:42 Clearing connections for removed peer with WG IP: 100.89.128.8
d5a492b45dea INFO: 2026/05/10 14:57:42 Cleared 0 connections for WG IP: 100.89.128.8
d5a492b45dea INFO: 2026/05/10 14:57:42 Peer Pep...aWM= removed successfully
d5a492b45dea INFO: 2026/05/10 14:57:42 Clearing connections for added peer with WG IP: 100.89.128.8
d5a492b45dea INFO: 2026/05/10 14:57:42 Cleared 0 connections for WG IP: 100.89.128.8
d5a492b45dea INFO: 2026/05/10 14:57:42 Peer Jgy...EWk= added successfully
c24505f94a47 2026-05-10T14:57:42+00:00 [info]: Establishing websocket connection
c24505f94a47 2026-05-10T14:57:42+00:00 [info]: Client added to tracking - NEWT ID: 6tssbbzooylnp31, Connection ID: cacd6082-8db9-47ef-ad6c-67aac15d5644, Total connections: 1
c24505f94a47 2026-05-10T14:57:42+00:00 [info]: WebSocket connection established - NEWT ID: 6tssbbzooylnp31
c24505f94a47 2026-05-10T14:57:42+00:00 [info]: Handling ping request newt message!
c24505f94a47 2026-05-10T14:57:42+00:00 [warn]: Site last hole punch is too old; skipping this register. The site is failing to hole punch and identify its network address with the server. Can the site reach the server on UDP port 21820?
c24505f94a47 2026-05-10T14:57:42+00:00 [info]: Public key mismatch. Deleting old peer...
c24505f94a47 2026-05-10T14:57:42+00:00 [info]: Deleting peer with public key Pep...aWM= from exit node 1
c24505f94a47 2026-05-10T14:57:42+00:00 [info]: Exit node request successful: {"method":"DELETE","url":"http://gerbil:3004/peer?public_key=PepJ...aWM%3D","status":"Peer removed successfully"}
c24505f94a47 2026-05-10T14:57:42+00:00 [info]: Adding peer with public key Jgy...EWk= to exit node 1
c24505f94a47 2026-05-10T14:57:42+00:00 [info]: Exit node request successful: {"method":"POST","url":"http://gerbil:3004/peer","status":"Peer added successfully"}
c24505f94a47 2026-05-10T14:57:44+00:00 [info]: Exit node request successful: {"method":"POST","url":"http://gerbil:3004/update-proxy-mapping","status":"Proxy mappings updated successfully"}
d5a492b45dea INFO: 2026/05/10 14:57:44 Updated 0 proxy mappings: 100.89.128.8:53585 -> 100.89.128.8:52776
<!-- gh-comment-id:4415869333 --> @Snocember commented on GitHub (May 10, 2026): I also have this issue for a while now. ## Reproducible on: #### Pangolin host: linux/arm64 Rocky Linux 10.1 (6.12.0-124.52.1.el10_1.aarch64) podman version 5.6.0 fosrl/pangolin:latest (1.18.3, sha256:94e04ae48eec8a2f2c9925f1f12b8016abe5093f41f6f9829a0fca98e06b7fe0) #### Newt client: linux/amd64 Debian 12.3 (6.1.0-45-amd64) and Rocky Linux 9.7 (5.14.0-611.54.1.el9_7.x86_64) Docker version 29.4.3, build 055a478 fosrl/newt:latest (sha256:881cd45911cc8a8e5f347edf4a7a278fb93f337af8e588b94f9042465d81c891) ## Steps to reproduce: 1. Start pangolin server and newt clients 2. Stop and restart pangolin `podman compose down` and `up -d` ist used All routes to the sites connected with newt now show "Bad Gateway". When restarting the newt client, the problem resolves. ## Logs #### Newt client: ``` newt | DEBUG: 2026/05/10 14:54:04 Increased hole punch interval to 32s newt | DEBUG: gerbil-wireguard: 2026/05/10 14:54:08 peer(FoT7…i6ng) - Sending keepalive packet newt | DEBUG: gerbil-wireguard: 2026/05/10 14:54:17 peer(FoT7…i6ng) - Sending keepalive packet newt | DEBUG: gerbil-wireguard: 2026/05/10 14:54:23 peer(FoT7…i6ng) - Sending keepalive packet newt | DEBUG: gerbil-wireguard: 2026/05/10 14:54:28 peer(FoT7…i6ng) - Sending keepalive packet ``` After pangolin server restart: ``` newt | DEBUG: 2026/05/10 14:54:36 Sent UDP hole punch to <ip>:21820: {"ephemeralPublicKey":"...","nonce":"...","ciphertext":"..."} newt | DEBUG: 2026/05/10 14:54:36 Increased hole punch interval to 1m0s newt | DEBUG: gerbil-wireguard: 2026/05/10 14:54:38 peer(FoT7…i6ng) - Sending keepalive packet newt | DEBUG: gerbil-wireguard: 2026/05/10 14:54:43 peer(FoT7…i6ng) - Sending keepalive packet newt | DEBUG: gerbil-wireguard: 2026/05/10 14:54:53 peer(FoT7…i6ng) - Sending keepalive packet newt | DEBUG: 2026/05/10 14:54:55 Ping attempt 1/2 failed: failed to read ICMP packet: i/o timeout newt | DEBUG: gerbil-wireguard: 2026/05/10 14:55:00 peer(FoT7…i6ng) - Sending keepalive packet newt | DEBUG: 2026/05/10 14:55:02 Ping attempt 2/2 failed: failed to read ICMP packet: i/o timeout newt | DEBUG: 2026/05/10 14:55:02 Periodic ping failed (1 consecutive failures): all 2 ping attempts failed, last error: failed to read ICMP packet: i/o timeout newt | DEBUG: gerbil-wireguard: 2026/05/10 14:55:03 peer(FoT7…i6ng) - Retrying handshake because we stopped hearing back after 15 seconds newt | DEBUG: gerbil-wireguard: 2026/05/10 14:55:03 peer(FoT7…i6ng) - Sending handshake initiation newt | DEBUG: gerbil-wireguard: 2026/05/10 14:55:08 peer(FoT7…i6ng) - Sending keepalive packet newt | DEBUG: gerbil-wireguard: 2026/05/10 14:55:08 peer(FoT7…i6ng) - Handshake did not complete after 5 seconds, retrying (try 2) newt | DEBUG: gerbil-wireguard: 2026/05/10 14:55:08 peer(FoT7…i6ng) - Sending handshake initiation newt | DEBUG: 2026/05/10 14:55:10 Ping attempt 1/2 failed: failed to read ICMP packet: i/o timeout newt | DEBUG: gerbil-wireguard: 2026/05/10 14:55:13 peer(FoT7…i6ng) - Handshake did not complete after 5 seconds, retrying (try 3) newt | DEBUG: gerbil-wireguard: 2026/05/10 14:55:13 peer(FoT7…i6ng) - Sending handshake initiation newt | DEBUG: 2026/05/10 14:55:17 Ping attempt 2/2 failed: failed to read ICMP packet: i/o timeout newt | WARN: 2026/05/10 14:55:17 Periodic ping failed (2 consecutive failures): all 2 ping attempts failed, last error: failed to read ICMP packet: i/o timeout ... newt | DEBUG: gerbil-wireguard: 2026/05/10 14:56:15 peer(FoT7…i6ng) - Handshake did not complete after 5 seconds, retrying (try 2) newt | DEBUG: gerbil-wireguard: 2026/05/10 14:56:15 peer(FoT7…i6ng) - Sending handshake initiation newt | DEBUG: gerbil-wireguard: 2026/05/10 14:56:20 peer(FoT7…i6ng) - Sending keepalive packet newt | DEBUG: gerbil-wireguard: 2026/05/10 14:56:20 peer(FoT7…i6ng) - Sending handshake initiation newt | DEBUG: gerbil-wireguard: 2026/05/10 14:56:25 peer(FoT7…i6ng) - Sending keepalive packet newt | DEBUG: gerbil-wireguard: 2026/05/10 14:56:25 peer(FoT7…i6ng) - Sending handshake initiation newt | DEBUG: 2026/05/10 14:56:29 Ping attempt 1/4 failed: failed to read ICMP packet: i/o timeout newt | DEBUG: gerbil-wireguard: 2026/05/10 14:56:30 peer(FoT7…i6ng) - Retrying handshake because we stopped hearing back after 15 seconds newt | DEBUG: gerbil-wireguard: 2026/05/10 14:56:30 peer(FoT7…i6ng) - Handshake did not complete after 5 seconds, retrying (try 2) newt | DEBUG: gerbil-wireguard: 2026/05/10 14:56:30 peer(FoT7…i6ng) - Sending handshake initiation newt | DEBUG: gerbil-wireguard: 2026/05/10 14:56:35 peer(FoT7…i6ng) - Sending keepalive packet newt | DEBUG: gerbil-wireguard: 2026/05/10 14:56:35 peer(FoT7…i6ng) - Sending handshake initiation newt | DEBUG: 2026/05/10 14:56:36 Sent UDP hole punch to <ip>:21820: {"ephemeralPublicKey":"...","nonce":"...","ciphertext":"..."} newt | DEBUG: gerbil-wireguard: 2026/05/10 14:56:40 peer(FoT7…i6ng) - Sending handshake initiation newt | DEBUG: 2026/05/10 14:56:43 Ping attempt 2/4 failed: failed to read ICMP packet: i/o timeout newt | DEBUG: gerbil-wireguard: 2026/05/10 14:56:45 peer(FoT7…i6ng) - Sending handshake initiation newt | DEBUG: gerbil-wireguard: 2026/05/10 14:56:50 peer(FoT7…i6ng) - Sending handshake initiation newt | DEBUG: gerbil-wireguard: 2026/05/10 14:56:55 peer(FoT7…i6ng) - Sending handshake initiation newt | DEBUG: 2026/05/10 14:56:58 Ping attempt 3/4 failed: failed to read ICMP packet: i/o timeout newt | DEBUG: gerbil-wireguard: 2026/05/10 14:57:00 peer(FoT7…i6ng) - Sending handshake initiation newt | DEBUG: gerbil-wireguard: 2026/05/10 14:57:05 peer(FoT7…i6ng) - Sending handshake initiation newt | DEBUG: gerbil-wireguard: 2026/05/10 14:57:10 peer(FoT7…i6ng) - Sending handshake initiation newt | DEBUG: 2026/05/10 14:57:14 Ping attempt 4/4 failed: failed to read ICMP packet: i/o timeout newt | WARN: 2026/05/10 14:57:14 Periodic ping failed (6 consecutive failures): all 4 ping attempts failed, last error: failed to read ICMP packet: i/o timeout ``` Same time with only INFO enabled I encountered also an error: ``` newt | INFO: 2026/05/10 14:45:37 Failed to report peer bandwidth: failed to send bandwidth data: not connected newt | INFO: 2026/05/10 14:45:47 Failed to report peer bandwidth: failed to send bandwidth data: not connected newt | INFO: 2026/05/10 14:45:57 Failed to report peer bandwidth: failed to send bandwidth data: not connected newt | ERROR: 2026/05/10 14:46:02 Failed to connect: failed to get token: failed to request new token: Post "https://<domain>/api/v1/auth/newt/get-token": dial tcp <ip>:443: i/o timeout. Retrying in 3s... newt | INFO: 2026/05/10 14:46:07 Failed to report peer bandwidth: failed to send bandwidth data: not connected newt | WARN: 2026/05/10 14:46:12 Periodic ping failed (2 consecutive failures): all 2 ping attempts failed, last error: failed to read ICMP packet: i/o timeout newt | INFO: 2026/05/10 14:46:17 Failed to report peer bandwidth: failed to send bandwidth data: not connected newt | WARN: 2026/05/10 14:46:27 Periodic ping failed (3 consecutive failures): all 2 ping attempts failed, last error: failed to read ICMP packet: i/o timeout newt | INFO: 2026/05/10 14:46:27 Failed to report peer bandwidth: failed to send bandwidth data: not connected newt | ERROR: 2026/05/10 14:46:35 Failed to connect: failed to get token: failed to request new token: Post "https://<domain>/api/v1/auth/newt/get-token": dial tcp <ip>:443: i/o timeout. Retrying in 3s... newt | INFO: 2026/05/10 14:46:37 Failed to report peer bandwidth: failed to send bandwidth data: not connected newt | INFO: 2026/05/10 14:46:38 Server version: 1.18.3 newt | INFO: 2026/05/10 14:46:38 Websocket connected newt | WARN: 2026/05/10 14:46:46 Periodic ping failed (4 consecutive failures): all 2 ping attempts failed, last error: failed to read ICMP packet: i/o timeout ``` #### Pangolin host: ``` c24505f94a47 > @fosrl/pangolin@0.0.0 start c24505f94a47 > ENVIRONMENT=prod node dist/migrations.mjs && ENVIRONMENT=prod NODE_ENV=development node --enable-source-maps dist/server.mjs c24505f94a47 c24505f94a47 Starting migrations from version 1.18.3 c24505f94a47 Migrations to run: c24505f94a47 All migrations completed successfully ... d5a492b45dea INFO: 2026/05/10 14:55:05 WireGuard firewall rules successfully configured for interface wg0 d5a492b45dea INFO: 2026/05/10 14:55:05 WireGuard interface wg0 created and configured d5a492b45dea INFO: 2026/05/10 14:55:05 Peer 4f0...s3c= added successfully d5a492b45dea INFO: 2026/05/10 14:55:05 Peer Pep...aWM= added successfully d5a492b45dea INFO: 2026/05/10 14:55:05 Received initial mappings: {"mappings":{"<ip>:52136":{"destinations":[{"destinationIP":"100.89.128.8","destinationPort":53585}]},"<ip>:53585":{"destinations":[{"destinationIP":"100.89.128.4","destinationPort":52136}]}}} d5a492b45dea INFO: 2026/05/10 14:55:05 Loaded 2 initial proxy mappings d5a492b45dea INFO: 2026/05/10 14:55:05 UDP server listening on :21820 d5a492b45dea INFO: 2026/05/10 14:55:05 Starting 20 packet workers (CPUs: 1) d5a492b45dea INFO: 2026/05/10 14:55:05 Metrics endpoint enabled at /metrics d5a492b45dea INFO: 2026/05/10 14:55:05 Starting HTTP server on :3004 c24505f94a47 2026-05-10T14:56:29+00:00 [info]: Marking site 1 offline: newt cmhmqw0lpvfp5sr has no recent ping and no active WebSocket connection c24505f94a47 2026-05-10T14:56:29+00:00 [info]: Marking site 2 offline: newt 6tssbbzooylnp31 has no recent ping and no active WebSocket connection ``` When newt is restarted: ``` d5a492b45dea INFO: 2026/05/10 14:57:42 Clearing connections for removed peer with WG IP: 100.89.128.8 d5a492b45dea INFO: 2026/05/10 14:57:42 Cleared 0 connections for WG IP: 100.89.128.8 d5a492b45dea INFO: 2026/05/10 14:57:42 Peer Pep...aWM= removed successfully d5a492b45dea INFO: 2026/05/10 14:57:42 Clearing connections for added peer with WG IP: 100.89.128.8 d5a492b45dea INFO: 2026/05/10 14:57:42 Cleared 0 connections for WG IP: 100.89.128.8 d5a492b45dea INFO: 2026/05/10 14:57:42 Peer Jgy...EWk= added successfully c24505f94a47 2026-05-10T14:57:42+00:00 [info]: Establishing websocket connection c24505f94a47 2026-05-10T14:57:42+00:00 [info]: Client added to tracking - NEWT ID: 6tssbbzooylnp31, Connection ID: cacd6082-8db9-47ef-ad6c-67aac15d5644, Total connections: 1 c24505f94a47 2026-05-10T14:57:42+00:00 [info]: WebSocket connection established - NEWT ID: 6tssbbzooylnp31 c24505f94a47 2026-05-10T14:57:42+00:00 [info]: Handling ping request newt message! c24505f94a47 2026-05-10T14:57:42+00:00 [warn]: Site last hole punch is too old; skipping this register. The site is failing to hole punch and identify its network address with the server. Can the site reach the server on UDP port 21820? c24505f94a47 2026-05-10T14:57:42+00:00 [info]: Public key mismatch. Deleting old peer... c24505f94a47 2026-05-10T14:57:42+00:00 [info]: Deleting peer with public key Pep...aWM= from exit node 1 c24505f94a47 2026-05-10T14:57:42+00:00 [info]: Exit node request successful: {"method":"DELETE","url":"http://gerbil:3004/peer?public_key=PepJ...aWM%3D","status":"Peer removed successfully"} c24505f94a47 2026-05-10T14:57:42+00:00 [info]: Adding peer with public key Jgy...EWk= to exit node 1 c24505f94a47 2026-05-10T14:57:42+00:00 [info]: Exit node request successful: {"method":"POST","url":"http://gerbil:3004/peer","status":"Peer added successfully"} c24505f94a47 2026-05-10T14:57:44+00:00 [info]: Exit node request successful: {"method":"POST","url":"http://gerbil:3004/update-proxy-mapping","status":"Proxy mappings updated successfully"} d5a492b45dea INFO: 2026/05/10 14:57:44 Updated 0 proxy mappings: 100.89.128.8:53585 -> 100.89.128.8:52776 ```
GiteaMirror commented 2026-07-13 14:35:10 -05:00
Author
Owner
Copy Link
Copy Source

@oschwartz10612 commented on GitHub (May 11, 2026):

Could you try the latest newt? Some improvements were made.

<!-- gh-comment-id:4422957715 --> @oschwartz10612 commented on GitHub (May 11, 2026): Could you try the latest newt? Some improvements were made.
GiteaMirror commented 2026-07-13 14:35:11 -05:00
Author
Owner
Copy Link
Copy Source

@wkulhanek commented on GitHub (May 11, 2026):

Unless they were in the very latest tag I still see this behavior if my pangolin is offline for too long.

<!-- gh-comment-id:4423185665 --> @wkulhanek commented on GitHub (May 11, 2026): Unless they were in the very latest tag I still see this behavior if my pangolin is offline for too long.
GiteaMirror commented 2026-07-13 14:35:11 -05:00
Author
Owner
Copy Link
Copy Source

@Snocember commented on GitHub (May 12, 2026):

Could you try the latest newt? Some improvements were made.

Yup, seems to be resolved on pangolin 1.18.3 and newt 1.12.5.
I tested it with a downtime of like 30 seconds.
Great work, keep it up. (:

<!-- gh-comment-id:4434055106 --> @Snocember commented on GitHub (May 12, 2026): > Could you try the latest newt? Some improvements were made. Yup, seems to be resolved on pangolin 1.18.3 and newt 1.12.5. I tested it with a downtime of like 30 seconds. Great work, keep it up. (:
GiteaMirror commented 2026-07-13 14:35:12 -05:00
Author
Owner
Copy Link
Copy Source

@Snocember commented on GitHub (May 29, 2026):

I upgraded from pangolin 1.18.3 (with weeks uptime) to 1.18.4 with compose down and up and newt 1.12.5
The issue resurfaced and newt doesn't catch up.

newt  | WARN: 2026/05/28 23:38:39 Periodic ping failed (2 consecutive failures): all 2 ping attempts failed, last error: failed to read ICMP packet: i/o timeout
newt  | WARN: 2026/05/28 23:38:54 Periodic ping failed (3 consecutive failures): all 2 ping attempts failed, last error: failed to read ICMP packet: i/o timeout
newt  | WARN: 2026/05/28 23:39:09 Periodic ping failed (4 consecutive failures): all 2 ping attempts failed, last error: failed to read ICMP packet: i/o timeout
newt  | WARN: 2026/05/28 23:39:09 Connection to server lost after 4 failures. Continuous reconnection attempts will be made.
newt  | WARN: 2026/05/28 23:39:27 Periodic ping failed (5 consecutive failures): all 2 ping attempts failed, last error: failed to read ICMP packet: i/o timeout
newt  | INFO: 2026/05/28 23:39:57 SendMessageInterval timed out after 16 attempts for message type: newt/ping/request
newt  | WARN: 2026/05/28 23:40:14 Periodic ping failed (6 consecutive failures): all 4 ping attempts failed, last error: failed to read ICMP packet: i/o timeout
newt  | WARN: 2026/05/28 23:41:15 Periodic ping failed (7 consecutive failures): all 4 ping attempts failed, last error: failed to read ICMP packet: i/o timeout
newt  | WARN: 2026/05/28 23:42:18 Periodic ping failed (8 consecutive failures): all 4 ping attempts failed, last error: failed to read ICMP packet: i/o timeout

DEBUG similar as the log last post.

however after that with fresh started pangolin and restarted newt, when starting and stopping pangolin from the same version it reconnects

newt  | WARN: 2026/05/28 23:49:10 Periodic ping failed (2 consecutive failures): all 2 ping attempts failed, last error: failed to read ICMP packet: i/o timeout
newt  | WARN: 2026/05/28 23:49:25 Periodic ping failed (3 consecutive failures): all 2 ping attempts failed, last error: failed to read ICMP packet: i/o timeout
newt  | WARN: 2026/05/28 23:49:40 Periodic ping failed (4 consecutive failures): all 2 ping attempts failed, last error: failed to read ICMP packet: i/o timeout
newt  | WARN: 2026/05/28 23:49:40 Connection to server lost after 4 failures. Continuous reconnection attempts will be made.
newt  | INFO: 2026/05/28 23:49:40 Stopping ping check
newt  | INFO: 2026/05/28 23:49:41 Connecting to endpoint: ###.domain
newt  | WARN: 2026/05/28 23:49:41 Failed to start hole punch: hole punch already running
newt  | INFO: 2026/05/28 23:49:41 Tunnel connection to server established successfully!
newt  | INFO: 2026/05/28 23:49:41 Started tcp proxy to 192.168.40.203:443
<!-- gh-comment-id:4569411228 --> @Snocember commented on GitHub (May 29, 2026): I upgraded from pangolin 1.18.3 (with weeks uptime) to 1.18.4 with compose down and up and newt 1.12.5 The issue resurfaced and newt doesn't catch up. ``` newt | WARN: 2026/05/28 23:38:39 Periodic ping failed (2 consecutive failures): all 2 ping attempts failed, last error: failed to read ICMP packet: i/o timeout newt | WARN: 2026/05/28 23:38:54 Periodic ping failed (3 consecutive failures): all 2 ping attempts failed, last error: failed to read ICMP packet: i/o timeout newt | WARN: 2026/05/28 23:39:09 Periodic ping failed (4 consecutive failures): all 2 ping attempts failed, last error: failed to read ICMP packet: i/o timeout newt | WARN: 2026/05/28 23:39:09 Connection to server lost after 4 failures. Continuous reconnection attempts will be made. newt | WARN: 2026/05/28 23:39:27 Periodic ping failed (5 consecutive failures): all 2 ping attempts failed, last error: failed to read ICMP packet: i/o timeout newt | INFO: 2026/05/28 23:39:57 SendMessageInterval timed out after 16 attempts for message type: newt/ping/request newt | WARN: 2026/05/28 23:40:14 Periodic ping failed (6 consecutive failures): all 4 ping attempts failed, last error: failed to read ICMP packet: i/o timeout newt | WARN: 2026/05/28 23:41:15 Periodic ping failed (7 consecutive failures): all 4 ping attempts failed, last error: failed to read ICMP packet: i/o timeout newt | WARN: 2026/05/28 23:42:18 Periodic ping failed (8 consecutive failures): all 4 ping attempts failed, last error: failed to read ICMP packet: i/o timeout ``` DEBUG similar as the log last post. however after that with fresh started pangolin and restarted newt, when starting and stopping pangolin from the same version it reconnects ``` newt | WARN: 2026/05/28 23:49:10 Periodic ping failed (2 consecutive failures): all 2 ping attempts failed, last error: failed to read ICMP packet: i/o timeout newt | WARN: 2026/05/28 23:49:25 Periodic ping failed (3 consecutive failures): all 2 ping attempts failed, last error: failed to read ICMP packet: i/o timeout newt | WARN: 2026/05/28 23:49:40 Periodic ping failed (4 consecutive failures): all 2 ping attempts failed, last error: failed to read ICMP packet: i/o timeout newt | WARN: 2026/05/28 23:49:40 Connection to server lost after 4 failures. Continuous reconnection attempts will be made. newt | INFO: 2026/05/28 23:49:40 Stopping ping check newt | INFO: 2026/05/28 23:49:41 Connecting to endpoint: ###.domain newt | WARN: 2026/05/28 23:49:41 Failed to start hole punch: hole punch already running newt | INFO: 2026/05/28 23:49:41 Tunnel connection to server established successfully! newt | INFO: 2026/05/28 23:49:41 Started tcp proxy to 192.168.40.203:443 ```
GiteaMirror commented 2026-07-13 14:35:12 -05:00
Author
Owner
Copy Link
Copy Source

@oschwartz10612 commented on GitHub (May 29, 2026):

Ahh yeah what I think is happening here is it cant talk to the server so
the SendMessageInterval for newt/ping/request times out because the
server does not respond.

This was brought up on the original thread when we discussed this but I
did not have a good solution. Let me revisit this. Basically because the
server is down it does not recover.

<!-- gh-comment-id:4569569474 --> @oschwartz10612 commented on GitHub (May 29, 2026): Ahh yeah what I think is happening here is it cant talk to the server so the SendMessageInterval for newt/ping/request times out because the server does not respond. This was brought up on the original thread when we discussed this but I did not have a good solution. Let me revisit this. Basically because the server is down it does not recover.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
dependencies
docker
enhancement
go
good first issue
help wanted
Improvement
needs investigating
pull-request
Mirrored from GitHub Pull Request
 stale
wontfix
No labels bug
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
GiteaMirror ninjasurge (Kaehlen Rog)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/newt#8215