[GH-ISSUE #3096] Newt site stays online but WireGuard data plane stops carrying traffic #30143

New Issue

GiteaMirror · 2026-06-13T10:39:21-05:00

GiteaMirror commented

2026-06-13 10:39:21 -05:00

Originally created by @yusuf-madkour on GitHub (May 17, 2026).
Original GitHub issue: https://github.com/fosrl/pangolin/issues/3096

Describe the Bug

A Newt site that previously worked stopped carrying public-resource traffic. Pangolin still showed the site online / reconnected, but resources behind that Newt site returned 502.

The site had historical non-zero data in/out counters, so this was not a fresh setup that never worked. The same backend resources were restored immediately by moving them from the Newt site to a local Pangolin site on the VPS, reaching the same backend IPs/ports over an existing NetBird route.

Environment

Pangolin/Gerbil/Traefik host: Ubuntu 24.04.3 LTS, aarch64, Oracle VPS
Original Newt host: Proxmox VE 9.1.4, unprivileged Ubuntu LXC 100, amd64
Reproduction Newt host: Debian 13.2 (trixie), aarch64, bare-metal madpi
Pangolin Version: 1.18.4
Gerbil Version: 1.4.0
Traefik Version: 3.5
Newt Version: 1.12.5

To Reproduce

I do not have a clean minimal reproducer yet. The failure was observed in this sequence:

Run Pangolin/Gerbil/Traefik on a VPS with a Newt site connected from a home network behind CGNAT.
Use the Newt site successfully for public resources for some time; the site had non-zero data in/out counters.
After the failure starts, keep the same Newt site connected. Pangolin still shows it online / reconnected.
Access public resources routed through that Newt site.
The resources return 502, while Gerbil wg0 shows no useful RX / handshake traffic.

Additional checks:

The issue reproduced with a separate bare-metal Newt test on a raspberry pi device in the same subnet.
A plain UDP probe to Gerbil 51820 reached the VPS.

Expected Behavior

If the Newt site is online/reconnected, the WireGuard data plane should recover or report a clear failed state. Previously working public resources behind the Newt site should not remain stuck at 502 with no useful wg0 RX/handshake traffic.

Originally created by @yusuf-madkour on GitHub (May 17, 2026). Original GitHub issue: https://github.com/fosrl/pangolin/issues/3096 ### Describe the Bug A Newt site that previously worked stopped carrying public-resource traffic. Pangolin still showed the site online / reconnected, but resources behind that Newt site returned 502. The site had historical non-zero data in/out counters, so this was not a fresh setup that never worked. The same backend resources were restored immediately by moving them from the Newt site to a local Pangolin site on the VPS, reaching the same backend IPs/ports over an existing NetBird route. ### Environment - Pangolin/Gerbil/Traefik host: Ubuntu 24.04.3 LTS, aarch64, Oracle VPS - Original Newt host: Proxmox VE 9.1.4, unprivileged Ubuntu LXC 100, amd64 - Reproduction Newt host: Debian 13.2 (trixie), aarch64, bare-metal `madpi` - Pangolin Version: 1.18.4 - Gerbil Version: 1.4.0 - Traefik Version: 3.5 - Newt Version: 1.12.5 ### To Reproduce I do not have a clean minimal reproducer yet. The failure was observed in this sequence: 1. Run Pangolin/Gerbil/Traefik on a VPS with a Newt site connected from a home network behind CGNAT. 2. Use the Newt site successfully for public resources for some time; the site had non-zero data in/out counters. 3. After the failure starts, keep the same Newt site connected. Pangolin still shows it online / reconnected. 4. Access public resources routed through that Newt site. 5. The resources return 502, while Gerbil `wg0` shows no useful RX / handshake traffic. Additional checks: - The issue reproduced with a separate bare-metal Newt test on a raspberry pi device in the same subnet. - A plain UDP probe to Gerbil `51820` reached the VPS. ### Expected Behavior If the Newt site is online/reconnected, the WireGuard data plane should recover or report a clear failed state. Previously working public resources behind the Newt site should not remain stuck at 502 with no useful `wg0` RX/handshake traffic.

GiteaMirror added the stale label 2026-06-13 10:39:21 -05:00

GiteaMirror commented

2026-06-13 10:39:23 -05:00

@LaurenceJJones commented on GitHub (May 25, 2026):

After the failure starts, ...

What is the "failure starts" like dashboard goes down or what happens?

@LaurenceJJones commented on GitHub (May 25, 2026): > After the failure starts, ... What is the "failure starts" like dashboard goes down or what happens?

GiteaMirror commented

2026-06-13 10:39:25 -05:00

@AstralDestiny commented on GitHub (May 25, 2026):

What's a "madpi" ?

@AstralDestiny commented on GitHub (May 25, 2026): What's a "madpi" ?

GiteaMirror commented

2026-06-13 10:39:26 -05:00

@jeffonthemove commented on GitHub (May 25, 2026):

Could this be related to https://github.com/fosrl/pangolin/issues/2815?
Updated to more recent issue

@jeffonthemove commented on GitHub (May 25, 2026): Could this be related to https://github.com/fosrl/pangolin/issues/2815? _Updated to more recent issue_

GiteaMirror commented

2026-06-13 10:39:28 -05:00

@AstralDestiny commented on GitHub (May 25, 2026):

That was patched in 1.12.4, what version are you right now?

@AstralDestiny commented on GitHub (May 25, 2026): That was patched in 1.12.4, what version are you right now?

GiteaMirror commented

2026-06-13 10:39:29 -05:00

@yusuf-madkour commented on GitHub (May 25, 2026):

Hello @AstralDestiny

As reported above, newt is on v1.12.5.

Please ignore the "madpi" thing, this is just my hostname for the other device on which I tried to install newt to test if the issue would happen on a new site.

The failure is that, pangolin can't route to the resources from that site. The site looks online on the dashboard but I could never reach the resources and the Data In/out never chages.

@yusuf-madkour commented on GitHub (May 25, 2026): Hello @AstralDestiny As reported above, newt is on v1.12.5. Please ignore the "madpi" thing, this is just my hostname for the other device on which I tried to install newt to test if the issue would happen on a new site. The failure is that, pangolin can't route to the resources from that site. The site looks online on the dashboard but I could never reach the resources and the Data In/out never chages.

GiteaMirror commented

2026-06-13 10:39:29 -05:00

@yusuf-madkour commented on GitHub (May 25, 2026):

Here are the logs when I deleted the site and tried to add it again.

> newt --id smie... --secret <redacted> --endpoint https://pangolin.domain.tld
INFO: 2026/05/25 22:40:03 Newt version 1.12.5
INFO: 2026/05/25 22:40:04 Server version: 1.18.4
INFO: 2026/05/25 22:40:04 Websocket connected
INFO: 2026/05/25 22:40:05 Connecting to endpoint: pangolin.domain.tld
INFO: 2026/05/25 22:40:36 SendMessageInterval timed out after 16 attempts for message type: newt/wg/get-config
WARN: 2026/05/25 22:40:46 Initial reliable ping failed, but continuing: all 5 ping attempts failed, last error: failed to read ICMP packet: i/o timeout
WARN: 2026/05/25 22:40:53 Ping attempt 1 failed: failed to read ICMP packet: i/o timeout
INFO: 2026/05/25 22:40:53 Client connectivity setup. Ready to accept connections from clients!

@yusuf-madkour commented on GitHub (May 25, 2026): Here are the logs when I deleted the site and tried to add it again. ``` > newt --id smie... --secret <redacted> --endpoint https://pangolin.domain.tld INFO: 2026/05/25 22:40:03 Newt version 1.12.5 INFO: 2026/05/25 22:40:04 Server version: 1.18.4 INFO: 2026/05/25 22:40:04 Websocket connected INFO: 2026/05/25 22:40:05 Connecting to endpoint: pangolin.domain.tld INFO: 2026/05/25 22:40:36 SendMessageInterval timed out after 16 attempts for message type: newt/wg/get-config WARN: 2026/05/25 22:40:46 Initial reliable ping failed, but continuing: all 5 ping attempts failed, last error: failed to read ICMP packet: i/o timeout WARN: 2026/05/25 22:40:53 Ping attempt 1 failed: failed to read ICMP packet: i/o timeout INFO: 2026/05/25 22:40:53 Client connectivity setup. Ready to accept connections from clients! ```

GiteaMirror commented

2026-06-13 10:39:30 -05:00

@AstralDestiny commented on GitHub (May 25, 2026):

So failing to do the ping over the tunnel where's newt in relation to pangolin same host or different?

@AstralDestiny commented on GitHub (May 25, 2026): So failing to do the ping over the tunnel where's newt in relation to pangolin same host or different?

GiteaMirror commented

2026-06-13 10:39:31 -05:00

@AstralDestiny commented on GitHub (May 25, 2026):

Mmm mind throwing me a site config in dm's by chance?

@AstralDestiny commented on GitHub (May 25, 2026): Mmm mind throwing me a site config in dm's by chance?

GiteaMirror commented

2026-06-13 10:39:32 -05:00

@oschwartz10612 commented on GitHub (May 26, 2026):

Also if you could follow up with debug logs (--log-level debug) from a natural failure. Deleting and recreating the site will cause the wireguard to disconnect like that because its removed from gerbil.

@oschwartz10612 commented on GitHub (May 26, 2026): Also if you could follow up with debug logs (--log-level debug) from a natural failure. Deleting and recreating the site will cause the wireguard to disconnect like that because its removed from gerbil.

GiteaMirror commented

2026-06-13 10:39:33 -05:00

@yusuf-madkour commented on GitHub (May 27, 2026):

So failing to do the ping over the tunnel where's newt in relation to pangolin same host or different?

Pangolin is on a remote VPS, newt is on a host at home behind NAT.

Mmm mind throwing me a site config in dm's by chance?

If by "site config" you mean the newt-client config, it is really basic:

{
  "id": "ou6...0l5",
  "secret": "...",
  "endpoint": "https://redacted.domain.tld",
  "tlsClientCert": ""
}

Also if you could follow up with debug logs (--log-level debug) from a natural failure. Deleting and recreating the site will cause the wireguard to disconnect like that because its removed from gerbil.

Logs attached.

newt-debug.log

I have tried other hosts too but still getting the same symptoms. The only failure I see is ping attempts failing, as shared in the earlier logs. Happy to assist with more information in discord DMs.

@yusuf-madkour commented on GitHub (May 27, 2026): > So failing to do the ping over the tunnel where's newt in relation to pangolin same host or different? Pangolin is on a remote VPS, newt is on a host at home behind NAT. > Mmm mind throwing me a site config in dm's by chance? If by "site config" you mean the newt-client config, it is really basic: ``` { "id": "ou6...0l5", "secret": "...", "endpoint": "https://redacted.domain.tld", "tlsClientCert": "" } ``` > Also if you could follow up with debug logs (--log-level debug) from a natural failure. Deleting and recreating the site will cause the wireguard to disconnect like that because its removed from gerbil. Logs attached. [newt-debug.log](https://github.com/user-attachments/files/28295282/newt-debug-newer.compact.redacted.log) I have tried other hosts too but still getting the same symptoms. The only failure I see is ping attempts failing, as shared in the earlier logs. Happy to assist with more information in discord DMs.

GiteaMirror commented

2026-06-13 10:39:34 -05:00

@yusuf-madkour commented on GitHub (May 27, 2026):

Just tried deploying newt on a remote VPS (not in my country) and it worked.

So full disclosure, this problem can be rescoped to "most likely some DPI inline in my country", not a general newt failure. But it is worth mentioning that newt used to work in the same conditions before. I had GBs of data in/out from that site according to Pangolin's web UI.

If my theory about DPI is right, newt used to be more resistant to DPI/censorship before.

It is also noteworthy to mention that I could get around this by installing a netbird peer on both the vps and my device at home, this allowed pangolin to piggyback that connection and allowed me access to my resources at home. Only mentioning this because my understanding is that both gerbil <--> newt and netbird use wireguard under the hood so if netbird could get through, gerbil should also get through to newt.

Now:
remote device -> pangolin vps (local site) < --- netbird connection ---> resource on host at home

Previously it used to be:
remote device -> pangolin gerbil <-- newt --> resource on host at home

Happy to assist with more information in discord DMs if you are interested in more debugging. Otherwise, please close the issue if this is a very rare case that you are not interested in fixing.

@yusuf-madkour commented on GitHub (May 27, 2026): Just tried deploying newt on a remote VPS (not in my country) and it worked. So full disclosure, this problem can be rescoped to **"most likely some DPI inline in my country", not a general newt failure**. But it is worth mentioning that **newt used to work in the same conditions before**. I had GBs of data in/out from that site according to Pangolin's web UI. If my theory about DPI is right, `newt` used to be more resistant to DPI/censorship before. It is also noteworthy to mention that I could get around this by installing a netbird peer on both the vps and my device at home, this allowed pangolin to piggyback that connection and allowed me access to my resources at home. Only mentioning this because my understanding is that both gerbil <--> newt and netbird use wireguard under the hood so if netbird could get through, gerbil should also get through to newt. Now: `remote device -> pangolin vps (local site) < --- netbird connection ---> resource on host at home` Previously it used to be: `remote device -> pangolin gerbil <-- newt --> resource on host at home` Happy to assist with more information in discord DMs if you are interested in more debugging. Otherwise, please close the issue if this is a very rare case that you are not interested in fixing.

GiteaMirror commented

2026-06-13 10:39:35 -05:00

@github-actions[bot] commented on GitHub (Jun 13, 2026):

This issue has been automatically marked as stale due to 14 days of inactivity. It will be closed in 14 days if no further activity occurs.

@github-actions[bot] commented on GitHub (Jun 13, 2026): This issue has been automatically marked as stale due to 14 days of inactivity. It will be closed in 14 days if no further activity occurs.

Sign in to join this conversation.

Branches Tags

main

dev

dependabot/npm_and_yarn/npm-dependencies-8b2d4a9f3a

local-connection

dependabot/go_modules/install/go-install-dependencies-3804ca7238

dependabot/docker/docker-dependencies-4faa477378

dependabot/github_actions/github-actions-dependencies-6d79802a48

api-improvements

feat/remember-last-idp-on-smart-login-form

refactor/batch-status-requests

dependabot/npm_and_yarn/multi-5f1280885e

fix/non-semver-version-error

private-resource-page

resource-launcher

backhaul

exit-node-reconnect

feat/command-palette

ssh

delete-account

msg-delivery

org-only-idp

cicd

patch

site-targets-auto-login

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/pangolin#30143