A Newt site that previously worked stopped carrying public-resource traffic. Pangolin still showed the site online / reconnected, but resources behind that Newt site returned 502.
The site had historical non-zero data in/out counters, so this was not a fresh setup that never worked. The same backend resources were restored immediately by moving them from the Newt site to a local Pangolin site on the VPS, reaching the same backend IPs/ports over an existing NetBird route.
I do not have a clean minimal reproducer yet. The failure was observed in this sequence:
Run Pangolin/Gerbil/Traefik on a VPS with a Newt site connected from a home network behind CGNAT.
Use the Newt site successfully for public resources for some time; the site had non-zero data in/out counters.
After the failure starts, keep the same Newt site connected. Pangolin still shows it online / reconnected.
Access public resources routed through that Newt site.
The resources return 502, while Gerbil wg0 shows no useful RX / handshake traffic.
Additional checks:
The issue reproduced with a separate bare-metal Newt test on a raspberry pi device in the same subnet.
A plain UDP probe to Gerbil 51820 reached the VPS.
Expected Behavior
If the Newt site is online/reconnected, the WireGuard data plane should recover or report a clear failed state. Previously working public resources behind the Newt site should not remain stuck at 502 with no useful wg0 RX/handshake traffic.
Originally created by @yusuf-madkour on GitHub (May 17, 2026).
Original GitHub issue: https://github.com/fosrl/pangolin/issues/3096
### Describe the Bug
A Newt site that previously worked stopped carrying public-resource traffic. Pangolin still showed the site online / reconnected, but resources behind that Newt site returned 502.
The site had historical non-zero data in/out counters, so this was not a fresh setup that never worked. The same backend resources were restored immediately by moving them from the Newt site to a local Pangolin site on the VPS, reaching the same backend IPs/ports over an existing NetBird route.
### Environment
- Pangolin/Gerbil/Traefik host: Ubuntu 24.04.3 LTS, aarch64, Oracle VPS
- Original Newt host: Proxmox VE 9.1.4, unprivileged Ubuntu LXC 100, amd64
- Reproduction Newt host: Debian 13.2 (trixie), aarch64, bare-metal `madpi`
- Pangolin Version: 1.18.4
- Gerbil Version: 1.4.0
- Traefik Version: 3.5
- Newt Version: 1.12.5
### To Reproduce
I do not have a clean minimal reproducer yet. The failure was observed in this sequence:
1. Run Pangolin/Gerbil/Traefik on a VPS with a Newt site connected from a home network behind CGNAT.
2. Use the Newt site successfully for public resources for some time; the site had non-zero data in/out counters.
3. After the failure starts, keep the same Newt site connected. Pangolin still shows it online / reconnected.
4. Access public resources routed through that Newt site.
5. The resources return 502, while Gerbil `wg0` shows no useful RX / handshake traffic.
Additional checks:
- The issue reproduced with a separate bare-metal Newt test on a raspberry pi device in the same subnet.
- A plain UDP probe to Gerbil `51820` reached the VPS.
### Expected Behavior
If the Newt site is online/reconnected, the WireGuard data plane should recover or report a clear failed state. Previously working public resources behind the Newt site should not remain stuck at 502 with no useful `wg0` RX/handshake traffic.
@LaurenceJJones commented on GitHub (May 25, 2026):
After the failure starts, ...
What is the "failure starts" like dashboard goes down or what happens?
<!-- gh-comment-id:4533789003 -->
@LaurenceJJones commented on GitHub (May 25, 2026):
> After the failure starts, ...
What is the "failure starts" like dashboard goes down or what happens?
<!-- gh-comment-id:4537630123 -->
@jeffonthemove commented on GitHub (May 25, 2026):
Could this be related to https://github.com/fosrl/pangolin/issues/2815?
_Updated to more recent issue_
@yusuf-madkour commented on GitHub (May 25, 2026):
Hello @AstralDestiny
As reported above, newt is on v1.12.5.
Please ignore the "madpi" thing, this is just my hostname for the other device on which I tried to install newt to test if the issue would happen on a new site.
The failure is that, pangolin can't route to the resources from that site. The site looks online on the dashboard but I could never reach the resources and the Data In/out never chages.
<!-- gh-comment-id:4537719483 -->
@yusuf-madkour commented on GitHub (May 25, 2026):
Hello @AstralDestiny
As reported above, newt is on v1.12.5.
Please ignore the "madpi" thing, this is just my hostname for the other device on which I tried to install newt to test if the issue would happen on a new site.
The failure is that, pangolin can't route to the resources from that site. The site looks online on the dashboard but I could never reach the resources and the Data In/out never chages.
@AstralDestiny commented on GitHub (May 25, 2026):
So failing to do the ping over the tunnel where's newt in relation to pangolin same host or different?
<!-- gh-comment-id:4538143763 -->
@AstralDestiny commented on GitHub (May 25, 2026):
So failing to do the ping over the tunnel where's newt in relation to pangolin same host or different?
@oschwartz10612 commented on GitHub (May 26, 2026):
Also if you could follow up with debug logs (--log-level debug) from a natural failure. Deleting and recreating the site will cause the wireguard to disconnect like that because its removed from gerbil.
<!-- gh-comment-id:4547617448 -->
@oschwartz10612 commented on GitHub (May 26, 2026):
Also if you could follow up with debug logs (--log-level debug) from a natural failure. Deleting and recreating the site will cause the wireguard to disconnect like that because its removed from gerbil.
Also if you could follow up with debug logs (--log-level debug) from a natural failure. Deleting and recreating the site will cause the wireguard to disconnect like that because its removed from gerbil.
I have tried other hosts too but still getting the same symptoms. The only failure I see is ping attempts failing, as shared in the earlier logs. Happy to assist with more information in discord DMs.
<!-- gh-comment-id:4551822599 -->
@yusuf-madkour commented on GitHub (May 27, 2026):
> So failing to do the ping over the tunnel where's newt in relation to pangolin same host or different?
Pangolin is on a remote VPS, newt is on a host at home behind NAT.
> Mmm mind throwing me a site config in dm's by chance?
If by "site config" you mean the newt-client config, it is really basic:
```
{
"id": "ou6...0l5",
"secret": "...",
"endpoint": "https://redacted.domain.tld",
"tlsClientCert": ""
}
```
> Also if you could follow up with debug logs (--log-level debug) from a natural failure. Deleting and recreating the site will cause the wireguard to disconnect like that because its removed from gerbil.
Logs attached.
[newt-debug.log](https://github.com/user-attachments/files/28295282/newt-debug-newer.compact.redacted.log)
I have tried other hosts too but still getting the same symptoms. The only failure I see is ping attempts failing, as shared in the earlier logs. Happy to assist with more information in discord DMs.
@yusuf-madkour commented on GitHub (May 27, 2026):
Just tried deploying newt on a remote VPS (not in my country) and it worked.
So full disclosure, this problem can be rescoped to "most likely some DPI inline in my country", not a general newt failure. But it is worth mentioning that newt used to work in the same conditions before. I had GBs of data in/out from that site according to Pangolin's web UI.
If my theory about DPI is right, newt used to be more resistant to DPI/censorship before.
It is also noteworthy to mention that I could get around this by installing a netbird peer on both the vps and my device at home, this allowed pangolin to piggyback that connection and allowed me access to my resources at home. Only mentioning this because my understanding is that both gerbil <--> newt and netbird use wireguard under the hood so if netbird could get through, gerbil should also get through to newt.
Now: remote device -> pangolin vps (local site) < --- netbird connection ---> resource on host at home
Previously it used to be: remote device -> pangolin gerbil <-- newt --> resource on host at home
Happy to assist with more information in discord DMs if you are interested in more debugging. Otherwise, please close the issue if this is a very rare case that you are not interested in fixing.
<!-- gh-comment-id:4551903560 -->
@yusuf-madkour commented on GitHub (May 27, 2026):
Just tried deploying newt on a remote VPS (not in my country) and it worked.
So full disclosure, this problem can be rescoped to **"most likely some DPI inline in my country", not a general newt failure**. But it is worth mentioning that **newt used to work in the same conditions before**. I had GBs of data in/out from that site according to Pangolin's web UI.
If my theory about DPI is right, `newt` used to be more resistant to DPI/censorship before.
It is also noteworthy to mention that I could get around this by installing a netbird peer on both the vps and my device at home, this allowed pangolin to piggyback that connection and allowed me access to my resources at home. Only mentioning this because my understanding is that both gerbil <--> newt and netbird use wireguard under the hood so if netbird could get through, gerbil should also get through to newt.
Now:
`remote device -> pangolin vps (local site) < --- netbird connection ---> resource on host at home`
Previously it used to be:
`remote device -> pangolin gerbil <-- newt --> resource on host at home`
Happy to assist with more information in discord DMs if you are interested in more debugging. Otherwise, please close the issue if this is a very rare case that you are not interested in fixing.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @yusuf-madkour on GitHub (May 17, 2026).
Original GitHub issue: https://github.com/fosrl/pangolin/issues/3096
Describe the Bug
A Newt site that previously worked stopped carrying public-resource traffic. Pangolin still showed the site online / reconnected, but resources behind that Newt site returned 502.
The site had historical non-zero data in/out counters, so this was not a fresh setup that never worked. The same backend resources were restored immediately by moving them from the Newt site to a local Pangolin site on the VPS, reaching the same backend IPs/ports over an existing NetBird route.
Environment
madpiTo Reproduce
I do not have a clean minimal reproducer yet. The failure was observed in this sequence:
wg0shows no useful RX / handshake traffic.Additional checks:
51820reached the VPS.Expected Behavior
If the Newt site is online/reconnected, the WireGuard data plane should recover or report a clear failed state. Previously working public resources behind the Newt site should not remain stuck at 502 with no useful
wg0RX/handshake traffic.@LaurenceJJones commented on GitHub (May 25, 2026):
What is the "failure starts" like dashboard goes down or what happens?
@AstralDestiny commented on GitHub (May 25, 2026):
What's a "madpi" ?
@jeffonthemove commented on GitHub (May 25, 2026):
Could this be related to https://github.com/fosrl/pangolin/issues/2815?
Updated to more recent issue
@AstralDestiny commented on GitHub (May 25, 2026):
That was patched in 1.12.4, what version are you right now?
@yusuf-madkour commented on GitHub (May 25, 2026):
Hello @AstralDestiny
As reported above, newt is on v1.12.5.
Please ignore the "madpi" thing, this is just my hostname for the other device on which I tried to install newt to test if the issue would happen on a new site.
The failure is that, pangolin can't route to the resources from that site. The site looks online on the dashboard but I could never reach the resources and the Data In/out never chages.
@yusuf-madkour commented on GitHub (May 25, 2026):
Here are the logs when I deleted the site and tried to add it again.
@AstralDestiny commented on GitHub (May 25, 2026):
So failing to do the ping over the tunnel where's newt in relation to pangolin same host or different?
@AstralDestiny commented on GitHub (May 25, 2026):
Mmm mind throwing me a site config in dm's by chance?
@oschwartz10612 commented on GitHub (May 26, 2026):
Also if you could follow up with debug logs (--log-level debug) from a natural failure. Deleting and recreating the site will cause the wireguard to disconnect like that because its removed from gerbil.
@yusuf-madkour commented on GitHub (May 27, 2026):
Pangolin is on a remote VPS, newt is on a host at home behind NAT.
If by "site config" you mean the newt-client config, it is really basic:
Logs attached.
newt-debug.log
I have tried other hosts too but still getting the same symptoms. The only failure I see is ping attempts failing, as shared in the earlier logs. Happy to assist with more information in discord DMs.
@yusuf-madkour commented on GitHub (May 27, 2026):
Just tried deploying newt on a remote VPS (not in my country) and it worked.
So full disclosure, this problem can be rescoped to "most likely some DPI inline in my country", not a general newt failure. But it is worth mentioning that newt used to work in the same conditions before. I had GBs of data in/out from that site according to Pangolin's web UI.
If my theory about DPI is right,
newtused to be more resistant to DPI/censorship before.It is also noteworthy to mention that I could get around this by installing a netbird peer on both the vps and my device at home, this allowed pangolin to piggyback that connection and allowed me access to my resources at home. Only mentioning this because my understanding is that both gerbil <--> newt and netbird use wireguard under the hood so if netbird could get through, gerbil should also get through to newt.
Now:
remote device -> pangolin vps (local site) < --- netbird connection ---> resource on host at homePreviously it used to be:
remote device -> pangolin gerbil <-- newt --> resource on host at homeHappy to assist with more information in discord DMs if you are interested in more debugging. Otherwise, please close the issue if this is a very rare case that you are not interested in fixing.