github-starred/pangolin
2
0
Fork 0
mirror of https://github.com/fosrl/pangolin.git synced 2026-07-16 14:47:04 -05:00
Code Issues 1.6k Packages Projects Releases 79 Wiki Activity

[GH-ISSUE #1004] Newt loses connection to server and will only reconnect on service restart #29551

New Issue
Closed
opened 2026-06-13 09:27:37 -05:00 by GiteaMirror · 7 comments
GiteaMirror commented 2026-06-13 09:27:37 -05:00
Owner
Copy Link

Originally created by @Ejmathewp on GitHub (Jul 3, 2025).
Original GitHub issue: https://github.com/fosrl/pangolin/issues/1004

I am running Newt on a TrueNAS server, I have tried both through the app catalog as well as regular docker deployment and encounter this issue with either approach. At least once a day Newt will report:

newt  | INFO: 2025/07/02 14:27:32 Pinging [ip redacted]
newt  | INFO: 2025/07/02 14:27:33 Ping latency: 142.385859ms
newt  | INFO: 2025/07/02 14:28:02 Pinging [ip redacted]
newt  | WARN: 2025/07/02 14:28:12 Connection to server lost. Continuous reconnection attempts will be made.
newt  | WARN: 2025/07/02 14:28:12 Please check your internet connection and ensure the Pangolin server is online.
newt  | WARN: 2025/07/02 14:28:12 Newt will continue reconnection attempts automatically when connectivity is restored.
newt  | INFO: 2025/07/02 14:28:32 Pinging [ip redacted]
newt  | INFO: 2025/07/02 14:29:02 Pinging [ip redacted]
newt  | INFO: 2025/07/02 14:29:32 Pinging [ip redacted]
newt  | INFO: 2025/07/02 14:30:02 Pinging [ip redacted]
newt  | INFO: 2025/07/02 14:30:32 Pinging [ip redacted]

A connection is never re-established, but if I restart the container there is zero issue reconnecting.

The drop is not at a consistent time and there are no connection issues with any other services.

Originally created by @Ejmathewp on GitHub (Jul 3, 2025). Original GitHub issue: https://github.com/fosrl/pangolin/issues/1004 I am running Newt on a TrueNAS server, I have tried both through the app catalog as well as regular docker deployment and encounter this issue with either approach. At least once a day Newt will report: ``` newt | INFO: 2025/07/02 14:27:32 Pinging [ip redacted] newt | INFO: 2025/07/02 14:27:33 Ping latency: 142.385859ms newt | INFO: 2025/07/02 14:28:02 Pinging [ip redacted] newt | WARN: 2025/07/02 14:28:12 Connection to server lost. Continuous reconnection attempts will be made. newt | WARN: 2025/07/02 14:28:12 Please check your internet connection and ensure the Pangolin server is online. newt | WARN: 2025/07/02 14:28:12 Newt will continue reconnection attempts automatically when connectivity is restored. newt | INFO: 2025/07/02 14:28:32 Pinging [ip redacted] newt | INFO: 2025/07/02 14:29:02 Pinging [ip redacted] newt | INFO: 2025/07/02 14:29:32 Pinging [ip redacted] newt | INFO: 2025/07/02 14:30:02 Pinging [ip redacted] newt | INFO: 2025/07/02 14:30:32 Pinging [ip redacted] ``` A connection is never re-established, but if I restart the container there is zero issue reconnecting. The drop is not at a consistent time and there are no connection issues with any other services.
GiteaMirror added the stale label 2026-06-13 09:27:37 -05:00
GiteaMirror commented 2026-06-13 09:27:39 -05:00
Author
Owner
Copy Link

@oschwartz10612 commented on GitHub (Jul 3, 2025):

Should be fixed in the next newt release!

<!-- gh-comment-id:3032809915 --> @oschwartz10612 commented on GitHub (Jul 3, 2025): Should be fixed in the next newt release!
GiteaMirror commented 2026-06-13 09:27:39 -05:00
Author
Owner
Copy Link

@Ejmathewp commented on GitHub (Jul 3, 2025):

Perfect, thanks!

<!-- gh-comment-id:3032822401 --> @Ejmathewp commented on GitHub (Jul 3, 2025): Perfect, thanks!
GiteaMirror commented 2026-06-13 09:27:39 -05:00
Author
Owner
Copy Link

@github-actions[bot] commented on GitHub (Jul 18, 2025):

This issue has been automatically marked as stale due to 14 days of inactivity. It will be closed in 14 days if no further activity occurs.

<!-- gh-comment-id:3086063465 --> @github-actions[bot] commented on GitHub (Jul 18, 2025): This issue has been automatically marked as stale due to 14 days of inactivity. It will be closed in 14 days if no further activity occurs.
GiteaMirror commented 2026-06-13 09:27:40 -05:00
Author
Owner
Copy Link

@oschwartz10612 commented on GitHub (Jul 18, 2025):

Try the latest newt release and Pangolin!

<!-- gh-comment-id:3086443541 --> @oschwartz10612 commented on GitHub (Jul 18, 2025): Try the latest newt release and Pangolin!
GiteaMirror commented 2026-06-13 09:27:41 -05:00
Author
Owner
Copy Link

@sandroshu commented on GitHub (Mar 26, 2026):

I have a similar issue. If I loose WAN connectivity or WAN IP changes I need to restart newt manually. After newt restart everything start working instantly.

Mar 26 11:30:39 access newt[358]: WARN: 2026/03/26 11:30:39 Periodic ping failed (169 consecutive failures): all 4 ping attempts failed, last error: failed to read ICMP packet: i/o timeout
Mar 26 11:31:42 access newt[358]: WARN: 2026/03/26 11:31:42 Periodic ping failed (170 consecutive failures): all 4 ping attempts failed, last error: failed to read ICMP packet: i/o timeout
Mar 26 11:32:46 access newt[358]: WARN: 2026/03/26 11:32:46 Periodic ping failed (171 consecutive failures): all 4 ping attempts failed, last error: failed to read ICMP packet: i/o timeout
Mar 26 11:33:49 access newt[358]: WARN: 2026/03/26 11:33:49 Periodic ping failed (172 consecutive failures): all 4 ping attempts failed, last error: failed to read ICMP packet: i/o timeout
Mar 26 11:34:53 access newt[358]: WARN: 2026/03/26 11:34:53 Periodic ping failed (173 consecutive failures): all 4 ping attempts failed, last error: failed to read ICMP packet: i/o timeout
Mar 26 11:35:56 access newt[358]: WARN: 2026/03/26 11:35:56 Periodic ping failed (174 consecutive failures): all 4 ping attempts failed, last error: failed to read ICMP packet: i/o timeout
Mar 26 11:37:00 access newt[358]: WARN: 2026/03/26 11:37:00 Periodic ping failed (175 consecutive failures): all 4 ping attempts failed, last error: failed to read ICMP packet: i/o timeout
Mar 26 11:38:03 access newt[358]: WARN: 2026/03/26 11:38:03 Periodic ping failed (176 consecutive failures): all 4 ping attempts failed, last error: failed to read ICMP packet: i/o timeout
Mar 26 11:39:07 access newt[358]: WARN: 2026/03/26 11:39:07 Periodic ping failed (177 consecutive failures): all 4 ping attempts failed, last error: failed to read ICMP packet: i/o timeout
Mar 26 11:40:10 access newt[358]: WARN: 2026/03/26 11:40:10 Periodic ping failed (178 consecutive failures): all 4 ping attempts failed, last error: failed to read ICMP packet: i/o timeout
<!-- gh-comment-id:4133941133 --> @sandroshu commented on GitHub (Mar 26, 2026): I have a similar issue. If I loose WAN connectivity or WAN IP changes I need to restart newt manually. After newt restart everything start working instantly. ``` Mar 26 11:30:39 access newt[358]: WARN: 2026/03/26 11:30:39 Periodic ping failed (169 consecutive failures): all 4 ping attempts failed, last error: failed to read ICMP packet: i/o timeout Mar 26 11:31:42 access newt[358]: WARN: 2026/03/26 11:31:42 Periodic ping failed (170 consecutive failures): all 4 ping attempts failed, last error: failed to read ICMP packet: i/o timeout Mar 26 11:32:46 access newt[358]: WARN: 2026/03/26 11:32:46 Periodic ping failed (171 consecutive failures): all 4 ping attempts failed, last error: failed to read ICMP packet: i/o timeout Mar 26 11:33:49 access newt[358]: WARN: 2026/03/26 11:33:49 Periodic ping failed (172 consecutive failures): all 4 ping attempts failed, last error: failed to read ICMP packet: i/o timeout Mar 26 11:34:53 access newt[358]: WARN: 2026/03/26 11:34:53 Periodic ping failed (173 consecutive failures): all 4 ping attempts failed, last error: failed to read ICMP packet: i/o timeout Mar 26 11:35:56 access newt[358]: WARN: 2026/03/26 11:35:56 Periodic ping failed (174 consecutive failures): all 4 ping attempts failed, last error: failed to read ICMP packet: i/o timeout Mar 26 11:37:00 access newt[358]: WARN: 2026/03/26 11:37:00 Periodic ping failed (175 consecutive failures): all 4 ping attempts failed, last error: failed to read ICMP packet: i/o timeout Mar 26 11:38:03 access newt[358]: WARN: 2026/03/26 11:38:03 Periodic ping failed (176 consecutive failures): all 4 ping attempts failed, last error: failed to read ICMP packet: i/o timeout Mar 26 11:39:07 access newt[358]: WARN: 2026/03/26 11:39:07 Periodic ping failed (177 consecutive failures): all 4 ping attempts failed, last error: failed to read ICMP packet: i/o timeout Mar 26 11:40:10 access newt[358]: WARN: 2026/03/26 11:40:10 Periodic ping failed (178 consecutive failures): all 4 ping attempts failed, last error: failed to read ICMP packet: i/o timeout ```
GiteaMirror commented 2026-06-13 09:27:43 -05:00
Author
Owner
Copy Link

@bajo commented on GitHub (Apr 29, 2026):

@sandroshu I faced a similar problem when my cheap VPS ran out of memory and OOM killer did its job on pangolin. After the service restarted, newt failed to automatically reconnect. That is on newt version 1.10.4 on NixOS.

Thus, I added a simple systemd watchdog service and timer on the host on which newt is running.

systemctl cat newt-watchdog.timer
[Unit]
Description=Run newt-watchdog every 5 minutes

[Timer]
OnBootSec=2min
OnUnitActiveSec=5min
Unit=newt-watchdog.service

[Install]
WantedBy=timers.target

and

systemctl cat newt-watchdog.service
[Unit]
Description=Restarts Newt if the tunnel is dead
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
# We use /bin/bash -c to allow for the 'if' logic and curl pipe
ExecStart=/bin/bash -c 'if ! /usr/bin/curl -fs --max-time 10 --resolve your-app.example.com:443:1.2.3.4 https://your-app.example.com > /dev/null; then echo "Tunnel check failed. Restarting Newt..."; /usr/bin/systemctl restart newt; else echo "Tunnel is healthy."; fi'

[Install]
WantedBy=multi-user.target

The --resolve your-app.example.com:443:1.2.3.4 part is only useful for me because I'm using the same fqdn internally as I do on the public Internet, so I need to make sure curl actually checks the app in the Internet and not on my LAN.

<!-- gh-comment-id:4341019881 --> @bajo commented on GitHub (Apr 29, 2026): @sandroshu I faced a similar problem when my cheap VPS ran out of memory and OOM killer did its job on pangolin. After the service restarted, newt failed to automatically reconnect. That is on newt version 1.10.4 on NixOS. Thus, I added a simple systemd watchdog service and timer on the host on which newt is running. ``` systemctl cat newt-watchdog.timer [Unit] Description=Run newt-watchdog every 5 minutes [Timer] OnBootSec=2min OnUnitActiveSec=5min Unit=newt-watchdog.service [Install] WantedBy=timers.target ``` and ``` systemctl cat newt-watchdog.service [Unit] Description=Restarts Newt if the tunnel is dead After=network-online.target Wants=network-online.target [Service] Type=oneshot # We use /bin/bash -c to allow for the 'if' logic and curl pipe ExecStart=/bin/bash -c 'if ! /usr/bin/curl -fs --max-time 10 --resolve your-app.example.com:443:1.2.3.4 https://your-app.example.com > /dev/null; then echo "Tunnel check failed. Restarting Newt..."; /usr/bin/systemctl restart newt; else echo "Tunnel is healthy."; fi' [Install] WantedBy=multi-user.target ``` The `--resolve your-app.example.com:443:1.2.3.4` part is only useful for me because I'm using the same fqdn internally as I do on the public Internet, so I need to make sure curl actually checks the app in the Internet and not on my LAN.
GiteaMirror commented 2026-06-13 09:27:43 -05:00
Author
Owner
Copy Link

@sandroshu commented on GitHub (Apr 29, 2026):

@bajo I have ""solved"" this issue by montoring the newt systemd services log.
There is an open issue here: https://github.com/fosrl/newt/issues/284

<!-- gh-comment-id:4341950961 --> @sandroshu commented on GitHub (Apr 29, 2026): @bajo I have ""solved"" this issue by montoring the newt systemd services log. There is an open issue here: https://github.com/fosrl/newt/issues/284
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
api
authentication
bug
config
dependencies
docker
documentation
enhancement
good first issue
help wanted
Improvement
Look Into
needs investigating
networking
new feature
non-critical bug
potential bug
pull-request
Mirrored from GitHub Pull Request
 question
reverse proxy
Security
stale
ui
wontfix
No Label stale
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/pangolin#29551
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?