Originally assigned to: @oschwartz10612 on GitHub.
I have pangolin on a vps and newt on a proxmox machine in my homelab and in my fathers homelab.
I don't have any issues with my fathers connection but from time to time the connection to my homelab is broken.
I can not reproduce this scenario... :(
Newt lost the connection
docker compose down and up did not solved the connection problem, so i rebooted the newt machine (LXC)
connection was succesfull but with "invalid target format null:null" so my UDP stream and HTTPS still not worked.
disabled and enabled the targets in pangolin to resync with newt and then it worked again.
It is the secound time i have this problem.
Newt Log
INFO: 2025/02/14 23:34:35 Already connected! But I will send a ping anyway...
INFO: 2025/02/14 23:34:35 Ping attempt 1 of 5
INFO: 2025/02/14 23:34:35 Pinging 100.89.128.1
WARN: 2025/02/14 23:34:45 Ping attempt 1 failed: failed to read ICMP packet: i/o timeout
INFO: 2025/02/14 23:34:47 Ping attempt 2 of 5
INFO: 2025/02/14 23:34:47 Pinging 100.89.128.1
WARN: 2025/02/14 23:34:57 Ping attempt 2 failed: failed to read ICMP packet: i/o timeout
INFO: 2025/02/14 23:34:59 Ping attempt 3 of 5
INFO: 2025/02/14 23:34:59 Pinging 100.89.128.1
WARN: 2025/02/14 23:35:09 Ping attempt 3 failed: failed to read ICMP packet: i/o timeout
INFO: 2025/02/14 23:35:11 Ping attempt 4 of 5
INFO: 2025/02/14 23:35:11 Pinging 100.89.128.1
WARN: 2025/02/14 23:35:21 Ping attempt 4 failed: failed to read ICMP packet: i/o timeout
INFO: 2025/02/14 23:35:23 Ping attempt 5 of 5
INFO: 2025/02/14 23:35:23 Pinging 100.89.128.1
WARN: 2025/02/14 23:35:33 Ping attempt 5 failed: failed to read ICMP packet: i/o timeout
WARN: 2025/02/14 23:35:33 Failed to ping 100.89.128.1: all ping attempts failed after 5 tries, last error: failed to read ICMP packet: i/o timeout
WARN: 2025/02/14 23:35:33 HINT: Do you have UDP port 51280 (or the port in config.yml) open on your Pangolin server?
INFO: 2025/02/15 07:56:34 Sent registration message
INFO: 2025/02/15 07:56:34 Received registration message
INFO: 2025/02/15 07:56:34 Received: {Type:newt/wg/connect Data:map[endpoint:my.domain:44400 publicKey:---key---- serverIP:100.89.128.1 targets:map[tcp:[null:null] udp:[null:null 45964:192.168.10.32:10090]] tunnelIP:100.89.128.4]}
INFO: 2025/02/15 07:56:34 WireGuard device created. Lets ping the server now...
INFO: 2025/02/15 07:56:34 Ping attempt 1 of 5
INFO: 2025/02/15 07:56:34 Pinging 100.89.128.1
INFO: 2025/02/15 07:56:34 Ping latency: 57.496664ms
INFO: 2025/02/15 07:56:34 Starting ping check
INFO: 2025/02/15 07:56:34 Invalid target format: null:null
INFO: 2025/02/15 07:56:34 Invalid target format: null:null
INFO: 2025/02/15 07:56:34 Not adding target because not running
INFO: 2025/02/15 07:56:34 Started udp proxy from 100.89.128.4:45964 to 192.168.10.32:10090
INFO: 2025/02/15 08:00:29 Received: {Type:newt/udp/add Data:map[targets:[55851:192.168.10.5:10091]]}
INFO: 2025/02/15 08:00:29 Started udp proxy from 100.89.128.4:55851 to 192.168.10.5:10091
INFO: 2025/02/15 08:00:46 Received: {Type:newt/udp/add Data:map[targets:[54486:192.168.10.5:10091]]}
Originally created by @Lokowitz on GitHub (Feb 15, 2025).
Original GitHub issue: https://github.com/fosrl/pangolin/issues/204
Originally assigned to: @oschwartz10612 on GitHub.
I have pangolin on a vps and newt on a proxmox machine in my homelab and in my fathers homelab.
I don't have any issues with my fathers connection but from time to time the connection to my homelab is broken.
I can not reproduce this scenario... :(
1. Newt lost the connection
2. docker compose down and up did not solved the connection problem, so i rebooted the newt machine (LXC)
3. connection was succesfull but with "invalid target format null:null" so my UDP stream and HTTPS still not worked.
4. disabled and enabled the targets in pangolin to resync with newt and then it worked again.
It is the secound time i have this problem.
## Newt Log
```
INFO: 2025/02/14 23:34:35 Already connected! But I will send a ping anyway...
INFO: 2025/02/14 23:34:35 Ping attempt 1 of 5
INFO: 2025/02/14 23:34:35 Pinging 100.89.128.1
WARN: 2025/02/14 23:34:45 Ping attempt 1 failed: failed to read ICMP packet: i/o timeout
INFO: 2025/02/14 23:34:47 Ping attempt 2 of 5
INFO: 2025/02/14 23:34:47 Pinging 100.89.128.1
WARN: 2025/02/14 23:34:57 Ping attempt 2 failed: failed to read ICMP packet: i/o timeout
INFO: 2025/02/14 23:34:59 Ping attempt 3 of 5
INFO: 2025/02/14 23:34:59 Pinging 100.89.128.1
WARN: 2025/02/14 23:35:09 Ping attempt 3 failed: failed to read ICMP packet: i/o timeout
INFO: 2025/02/14 23:35:11 Ping attempt 4 of 5
INFO: 2025/02/14 23:35:11 Pinging 100.89.128.1
WARN: 2025/02/14 23:35:21 Ping attempt 4 failed: failed to read ICMP packet: i/o timeout
INFO: 2025/02/14 23:35:23 Ping attempt 5 of 5
INFO: 2025/02/14 23:35:23 Pinging 100.89.128.1
WARN: 2025/02/14 23:35:33 Ping attempt 5 failed: failed to read ICMP packet: i/o timeout
WARN: 2025/02/14 23:35:33 Failed to ping 100.89.128.1: all ping attempts failed after 5 tries, last error: failed to read ICMP packet: i/o timeout
WARN: 2025/02/14 23:35:33 HINT: Do you have UDP port 51280 (or the port in config.yml) open on your Pangolin server?
INFO: 2025/02/15 07:56:34 Sent registration message
INFO: 2025/02/15 07:56:34 Received registration message
INFO: 2025/02/15 07:56:34 Received: {Type:newt/wg/connect Data:map[endpoint:my.domain:44400 publicKey:---key---- serverIP:100.89.128.1 targets:map[tcp:[null:null] udp:[null:null 45964:192.168.10.32:10090]] tunnelIP:100.89.128.4]}
INFO: 2025/02/15 07:56:34 WireGuard device created. Lets ping the server now...
INFO: 2025/02/15 07:56:34 Ping attempt 1 of 5
INFO: 2025/02/15 07:56:34 Pinging 100.89.128.1
INFO: 2025/02/15 07:56:34 Ping latency: 57.496664ms
INFO: 2025/02/15 07:56:34 Starting ping check
INFO: 2025/02/15 07:56:34 Invalid target format: null:null
INFO: 2025/02/15 07:56:34 Invalid target format: null:null
INFO: 2025/02/15 07:56:34 Not adding target because not running
INFO: 2025/02/15 07:56:34 Started udp proxy from 100.89.128.4:45964 to 192.168.10.32:10090
INFO: 2025/02/15 08:00:29 Received: {Type:newt/udp/add Data:map[targets:[55851:192.168.10.5:10091]]}
INFO: 2025/02/15 08:00:29 Started udp proxy from 100.89.128.4:55851 to 192.168.10.5:10091
INFO: 2025/02/15 08:00:46 Received: {Type:newt/udp/add Data:map[targets:[54486:192.168.10.5:10091]]}
```
## Traefik Log
```
traefik | 2025-02-15T09:00:11+01:00 ERR Error while handling UDP stream error="read udp 100.89.128.1:56889->100.89.128.4:46533: read: connection refused"
traefik | 2025-02-15T09:00:17+01:00 ERR Error while handling UDP stream error="read udp 100.89.128.1:38761->100.89.128.4:46533: read: connection refused"
traefik | 2025-02-15T09:00:23+01:00 ERR Error while handling UDP stream error="read udp 100.89.128.1:45171->100.89.128.4:46533: read: connection refused"
traefik | 2025-02-15T09:00:28+01:00 ERR Error while handling UDP stream error="read udp 100.89.128.1:59707->100.89.128.4:46533: read: connection refused"
traefik | 2025-02-15T09:00:33+01:00 ERR Error while handling UDP stream error="read udp 100.89.128.1:52478->100.89.128.4:46533: read: connection refused"
```
@oschwartz10612 commented on GitHub (Feb 15, 2025):
There is an open issue to make newt reconnect a little more reliability so I am looking into that. If your ISP is flaky then it might be dropping and the pings are not enough to get it to reconnect. I will let you know if I get a solution shortly.
About the null:null targets: are you on the latest version of Pangolin beta.14? We fixed a couple of issues with Newt and targets a little while back.
<!-- gh-comment-id:2660985129 -->
@oschwartz10612 commented on GitHub (Feb 15, 2025):
There is an open issue to make newt reconnect a little more reliability so I am looking into that. If your ISP is flaky then it might be dropping and the pings are not enough to get it to reconnect. I will let you know if I get a solution shortly.
About the null:null targets: are you on the latest version of Pangolin `beta.14`? We fixed a couple of issues with Newt and targets a little while back.
Yes I am at Beta.14. But I am not sure if this problem was before or after the update. Will have an eye on it the next days and let you know. Maybe the update (short downtime/connection loss) causes this problem.
<!-- gh-comment-id:2661097971 -->
@Lokowitz commented on GitHub (Feb 15, 2025):
Yes I am at Beta.14. But I am not sure if this problem was before or after the update. Will have an eye on it the next days and let you know. Maybe the update (short downtime/connection loss) causes this problem.
just want to let you know that it happend again.
No changes were made from my side this time, so maybe its my ISP as you mentioned.
But it looks like that 1 of 3 resources got back online (port 10090).
newt log
INFO: 2025/02/19 18:31:47 Sent registration message
INFO: 2025/02/19 18:31:47 Received registration message
INFO: 2025/02/19 18:31:47 Received: {Type:newt/wg/connect Data:map[endpointmy.domain:44400 publicKey:--key--= serverIP:100.89.128.1 targets:map[tcp:[null:null] udp:[null:null 45964:192.168.10.32:10090]] tunnelIP:100.89.128.4]}
INFO: 2025/02/19 18:31:47 WireGuard device created. Lets ping the server now...
INFO: 2025/02/19 18:31:47 Ping attempt 1 of 5
INFO: 2025/02/19 18:31:47 Pinging 100.89.128.1
INFO: 2025/02/19 18:31:47 Ping latency: 52.74737ms
INFO: 2025/02/19 18:31:47 Starting ping check
INFO: 2025/02/19 18:31:47 Invalid target format: null:null
INFO: 2025/02/19 18:31:47 Invalid target format: null:null
INFO: 2025/02/19 18:31:47 Not adding target because not running
INFO: 2025/02/19 18:31:47 Started udp proxy from 100.89.128.4:45964 to 192.168.10.32:10090
<!-- gh-comment-id:2670978939 -->
@Lokowitz commented on GitHub (Feb 20, 2025):
Hi guys,
just want to let you know that it happend again.
No changes were made from my side this time, so maybe its my ISP as you mentioned.
But it looks like that 1 of 3 resources got back online (port 10090).
## newt log
```
INFO: 2025/02/19 18:31:47 Sent registration message
INFO: 2025/02/19 18:31:47 Received registration message
INFO: 2025/02/19 18:31:47 Received: {Type:newt/wg/connect Data:map[endpointmy.domain:44400 publicKey:--key--= serverIP:100.89.128.1 targets:map[tcp:[null:null] udp:[null:null 45964:192.168.10.32:10090]] tunnelIP:100.89.128.4]}
INFO: 2025/02/19 18:31:47 WireGuard device created. Lets ping the server now...
INFO: 2025/02/19 18:31:47 Ping attempt 1 of 5
INFO: 2025/02/19 18:31:47 Pinging 100.89.128.1
INFO: 2025/02/19 18:31:47 Ping latency: 52.74737ms
INFO: 2025/02/19 18:31:47 Starting ping check
INFO: 2025/02/19 18:31:47 Invalid target format: null:null
INFO: 2025/02/19 18:31:47 Invalid target format: null:null
INFO: 2025/02/19 18:31:47 Not adding target because not running
INFO: 2025/02/19 18:31:47 Started udp proxy from 100.89.128.4:45964 to 192.168.10.32:10090
```
@oschwartz10612 commented on GitHub (Feb 20, 2025):
Thanks for confirming you saw this again and providing the logs. If you
are definitely on beta.14 (see bottom of the screen when logged into
Pangolin dashboard) and we are having this null:null target issue still
then there may be a problem (in addition to the reconnecting).
I will try to take a look into this as soon as possible and see if I can
reproduce.
<!-- gh-comment-id:2671708290 -->
@oschwartz10612 commented on GitHub (Feb 20, 2025):
Thanks for confirming you saw this again and providing the logs. If you
are definitely on beta.14 (see bottom of the screen when logged into
Pangolin dashboard) and we are having this null:null target issue still
then there may be a problem (in addition to the reconnecting).
I will try to take a look into this as soon as possible and see if I can
reproduce.
For my understanding the reconnection works but not the sync of the tagets (only partially).
It was enough to just disable and enable the target to get it back working.
<!-- gh-comment-id:2673791757 -->
@Lokowitz commented on GitHub (Feb 21, 2025):
Yes, everything is running on latest tag.
For my understanding the reconnection works but not the sync of the tagets (only partially).
It was enough to just disable and enable the target to get it back working.

@oschwartz10612 commented on GitHub (Feb 21, 2025):
Ahh okay are the two resources that dont come back TCP/UDP resources and not HTTP resources? That helps me narrow this down!
<!-- gh-comment-id:2674695080 -->
@oschwartz10612 commented on GitHub (Feb 21, 2025):
Ahh okay are the two resources that dont come back TCP/UDP resources and not HTTP resources? That helps me narrow this down!
One was http but disabled target and the other enabled target udp.
The one working is also enabled target udp.
<!-- gh-comment-id:2674739744 -->
@Lokowitz commented on GitHub (Feb 21, 2025):
One was http but disabled target and the other enabled target udp.
The one working is also enabled target udp.
Hello, I have the same problem also. Please check it thanks.
<!-- gh-comment-id:2704249501 -->
@cevinov commented on GitHub (Mar 6, 2025):
Hello, I have the same problem also. Please check it thanks.


@oschwartz10612 commented on GitHub (Mar 6, 2025):
Hum thanks for the info. Will try to prioritize investigating this ASAP.
<!-- gh-comment-id:2704484869 -->
@oschwartz10612 commented on GitHub (Mar 6, 2025):
Hum thanks for the info. Will try to prioritize investigating this ASAP.
When I try I get this error:
Failed to connect: failed to get token: failed to request new token.
Please check it.
Thanks.
<!-- gh-comment-id:2712202429 -->
@cevinov commented on GitHub (Mar 11, 2025):
Hello Oschwartz,
When I try I get this error:
Failed to connect: failed to get token: failed to request new token.

Please check it.
Thanks.
@oschwartz10612 commented on GitHub (Mar 11, 2025):
@cevinov I think your command is wrong. You have an extra newt after ./newt_windows_amd64.exe
<!-- gh-comment-id:2712213203 -->
@oschwartz10612 commented on GitHub (Mar 11, 2025):
@cevinov I think your command is wrong. You have an extra newt after `./newt_windows_amd64.exe`
<!-- gh-comment-id:2712295559 -->
@cevinov commented on GitHub (Mar 11, 2025):
But why did it give me the error can't be reached? The site is already online and the configuration of the resources (http) has been done.
https://go.xxxxx.xxxxx.id

Please check it thanks.
I would come ask this question on Discord or maybe open a new issue because I dont think this is the same issue.
<!-- gh-comment-id:2712301270 -->
@oschwartz10612 commented on GitHub (Mar 11, 2025):
I think that would be caused higher up in the stack and is not related to this issue. Could you make sure that your DNS is working correctly and pointed to your VPS etc? https://docs.fossorial.io/Getting%20Started/dns-networking
I would come ask this question on Discord or maybe open a new issue because I dont think this is the same issue.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @Lokowitz on GitHub (Feb 15, 2025).
Original GitHub issue: https://github.com/fosrl/pangolin/issues/204
Originally assigned to: @oschwartz10612 on GitHub.
I have pangolin on a vps and newt on a proxmox machine in my homelab and in my fathers homelab.
I don't have any issues with my fathers connection but from time to time the connection to my homelab is broken.
I can not reproduce this scenario... :(
It is the secound time i have this problem.
Newt Log
Traefik Log
@oschwartz10612 commented on GitHub (Feb 15, 2025):
There is an open issue to make newt reconnect a little more reliability so I am looking into that. If your ISP is flaky then it might be dropping and the pings are not enough to get it to reconnect. I will let you know if I get a solution shortly.
About the null:null targets: are you on the latest version of Pangolin
beta.14? We fixed a couple of issues with Newt and targets a little while back.@Lokowitz commented on GitHub (Feb 15, 2025):
Yes I am at Beta.14. But I am not sure if this problem was before or after the update. Will have an eye on it the next days and let you know. Maybe the update (short downtime/connection loss) causes this problem.
@Lokowitz commented on GitHub (Feb 20, 2025):
Hi guys,
just want to let you know that it happend again.
No changes were made from my side this time, so maybe its my ISP as you mentioned.
But it looks like that 1 of 3 resources got back online (port 10090).
newt log
@oschwartz10612 commented on GitHub (Feb 20, 2025):
Thanks for confirming you saw this again and providing the logs. If you
are definitely on beta.14 (see bottom of the screen when logged into
Pangolin dashboard) and we are having this null:null target issue still
then there may be a problem (in addition to the reconnecting).
I will try to take a look into this as soon as possible and see if I can
reproduce.
@Lokowitz commented on GitHub (Feb 21, 2025):
Yes, everything is running on latest tag.
For my understanding the reconnection works but not the sync of the tagets (only partially).
It was enough to just disable and enable the target to get it back working.
@oschwartz10612 commented on GitHub (Feb 21, 2025):
Ahh okay are the two resources that dont come back TCP/UDP resources and not HTTP resources? That helps me narrow this down!
@Lokowitz commented on GitHub (Feb 21, 2025):
One was http but disabled target and the other enabled target udp.
The one working is also enabled target udp.
@cevinov commented on GitHub (Mar 6, 2025):
Hello, I have the same problem also. Please check it thanks.
@oschwartz10612 commented on GitHub (Mar 6, 2025):
Hum thanks for the info. Will try to prioritize investigating this ASAP.
@cevinov commented on GitHub (Mar 7, 2025):
Thanks, I will wait
@oschwartz10612 commented on GitHub (Mar 10, 2025):
I think #307 resolves this. Please reopen if there is an issue.
@cevinov commented on GitHub (Mar 11, 2025):
Hello Oschwartz,
When I try I get this error:
Failed to connect: failed to get token: failed to request new token.
Please check it.
Thanks.
@oschwartz10612 commented on GitHub (Mar 11, 2025):
@cevinov I think your command is wrong. You have an extra newt after
./newt_windows_amd64.exe@cevinov commented on GitHub (Mar 11, 2025):
Thanks sir, it works now.
@cevinov commented on GitHub (Mar 11, 2025):
But why did it give me the error can't be reached? The site is already online and the configuration of the resources (http) has been done.
https://go.xxxxx.xxxxx.id
Please check it thanks.
@oschwartz10612 commented on GitHub (Mar 11, 2025):
I think that would be caused higher up in the stack and is not related to this issue. Could you make sure that your DNS is working correctly and pointed to your VPS etc? https://docs.fossorial.io/Getting%20Started/dns-networking
I would come ask this question on Discord or maybe open a new issue because I dont think this is the same issue.
@cevinov commented on GitHub (Mar 11, 2025):
I see thanks sir.