github-starred/pangolin
2
0
Fork 0
mirror of https://github.com/fosrl/pangolin.git synced 2026-05-07 05:09:18 -05:00
Code Issues 564 Packages Projects Releases 72 Wiki Activity

[GH-ISSUE #204] Error while handling UDP stream #3282

New Issue
Closed
opened 2026-04-20 07:12:35 -05:00 by GiteaMirror · 17 comments
GiteaMirror commented 2026-04-20 07:12:35 -05:00
Owner
Copy Link

Originally created by @Lokowitz on GitHub (Feb 15, 2025).
Original GitHub issue: https://github.com/fosrl/pangolin/issues/204

Originally assigned to: @oschwartz10612 on GitHub.

I have pangolin on a vps and newt on a proxmox machine in my homelab and in my fathers homelab.
I don't have any issues with my fathers connection but from time to time the connection to my homelab is broken.
I can not reproduce this scenario... :(

  1. Newt lost the connection
  2. docker compose down and up did not solved the connection problem, so i rebooted the newt machine (LXC)
  3. connection was succesfull but with "invalid target format null:null" so my UDP stream and HTTPS still not worked.
  4. disabled and enabled the targets in pangolin to resync with newt and then it worked again.

It is the secound time i have this problem.

Newt Log

INFO: 2025/02/14 23:34:35 Already connected! But I will send a ping anyway...
INFO: 2025/02/14 23:34:35 Ping attempt 1 of 5
INFO: 2025/02/14 23:34:35 Pinging 100.89.128.1
WARN: 2025/02/14 23:34:45 Ping attempt 1 failed: failed to read ICMP packet: i/o timeout
INFO: 2025/02/14 23:34:47 Ping attempt 2 of 5
INFO: 2025/02/14 23:34:47 Pinging 100.89.128.1
WARN: 2025/02/14 23:34:57 Ping attempt 2 failed: failed to read ICMP packet: i/o timeout
INFO: 2025/02/14 23:34:59 Ping attempt 3 of 5
INFO: 2025/02/14 23:34:59 Pinging 100.89.128.1
WARN: 2025/02/14 23:35:09 Ping attempt 3 failed: failed to read ICMP packet: i/o timeout
INFO: 2025/02/14 23:35:11 Ping attempt 4 of 5
INFO: 2025/02/14 23:35:11 Pinging 100.89.128.1
WARN: 2025/02/14 23:35:21 Ping attempt 4 failed: failed to read ICMP packet: i/o timeout
INFO: 2025/02/14 23:35:23 Ping attempt 5 of 5
INFO: 2025/02/14 23:35:23 Pinging 100.89.128.1
WARN: 2025/02/14 23:35:33 Ping attempt 5 failed: failed to read ICMP packet: i/o timeout
WARN: 2025/02/14 23:35:33 Failed to ping 100.89.128.1: all ping attempts failed after 5 tries, last error: failed to read ICMP packet: i/o timeout
WARN: 2025/02/14 23:35:33 HINT: Do you have UDP port 51280 (or the port in config.yml) open on your Pangolin server?
INFO: 2025/02/15 07:56:34 Sent registration message
INFO: 2025/02/15 07:56:34 Received registration message
INFO: 2025/02/15 07:56:34 Received: {Type:newt/wg/connect Data:map[endpoint:my.domain:44400 publicKey:---key---- serverIP:100.89.128.1 targets:map[tcp:[null:null] udp:[null:null 45964:192.168.10.32:10090]] tunnelIP:100.89.128.4]}
INFO: 2025/02/15 07:56:34 WireGuard device created. Lets ping the server now...
INFO: 2025/02/15 07:56:34 Ping attempt 1 of 5
INFO: 2025/02/15 07:56:34 Pinging 100.89.128.1
INFO: 2025/02/15 07:56:34 Ping latency: 57.496664ms
INFO: 2025/02/15 07:56:34 Starting ping check
INFO: 2025/02/15 07:56:34 Invalid target format: null:null
INFO: 2025/02/15 07:56:34 Invalid target format: null:null
INFO: 2025/02/15 07:56:34 Not adding target because not running
INFO: 2025/02/15 07:56:34 Started udp proxy from 100.89.128.4:45964 to 192.168.10.32:10090
INFO: 2025/02/15 08:00:29 Received: {Type:newt/udp/add Data:map[targets:[55851:192.168.10.5:10091]]}
INFO: 2025/02/15 08:00:29 Started udp proxy from 100.89.128.4:55851 to 192.168.10.5:10091
INFO: 2025/02/15 08:00:46 Received: {Type:newt/udp/add Data:map[targets:[54486:192.168.10.5:10091]]}

Traefik Log

traefik   | 2025-02-15T09:00:11+01:00 ERR Error while handling UDP stream error="read udp 100.89.128.1:56889->100.89.128.4:46533: read: connection refused"
traefik   | 2025-02-15T09:00:17+01:00 ERR Error while handling UDP stream error="read udp 100.89.128.1:38761->100.89.128.4:46533: read: connection refused"
traefik   | 2025-02-15T09:00:23+01:00 ERR Error while handling UDP stream error="read udp 100.89.128.1:45171->100.89.128.4:46533: read: connection refused"
traefik   | 2025-02-15T09:00:28+01:00 ERR Error while handling UDP stream error="read udp 100.89.128.1:59707->100.89.128.4:46533: read: connection refused"
traefik   | 2025-02-15T09:00:33+01:00 ERR Error while handling UDP stream error="read udp 100.89.128.1:52478->100.89.128.4:46533: read: connection refused"
Originally created by @Lokowitz on GitHub (Feb 15, 2025). Original GitHub issue: https://github.com/fosrl/pangolin/issues/204 Originally assigned to: @oschwartz10612 on GitHub. I have pangolin on a vps and newt on a proxmox machine in my homelab and in my fathers homelab. I don't have any issues with my fathers connection but from time to time the connection to my homelab is broken. I can not reproduce this scenario... :( 1. Newt lost the connection 2. docker compose down and up did not solved the connection problem, so i rebooted the newt machine (LXC) 3. connection was succesfull but with "invalid target format null:null" so my UDP stream and HTTPS still not worked. 4. disabled and enabled the targets in pangolin to resync with newt and then it worked again. It is the secound time i have this problem. ## Newt Log ``` INFO: 2025/02/14 23:34:35 Already connected! But I will send a ping anyway... INFO: 2025/02/14 23:34:35 Ping attempt 1 of 5 INFO: 2025/02/14 23:34:35 Pinging 100.89.128.1 WARN: 2025/02/14 23:34:45 Ping attempt 1 failed: failed to read ICMP packet: i/o timeout INFO: 2025/02/14 23:34:47 Ping attempt 2 of 5 INFO: 2025/02/14 23:34:47 Pinging 100.89.128.1 WARN: 2025/02/14 23:34:57 Ping attempt 2 failed: failed to read ICMP packet: i/o timeout INFO: 2025/02/14 23:34:59 Ping attempt 3 of 5 INFO: 2025/02/14 23:34:59 Pinging 100.89.128.1 WARN: 2025/02/14 23:35:09 Ping attempt 3 failed: failed to read ICMP packet: i/o timeout INFO: 2025/02/14 23:35:11 Ping attempt 4 of 5 INFO: 2025/02/14 23:35:11 Pinging 100.89.128.1 WARN: 2025/02/14 23:35:21 Ping attempt 4 failed: failed to read ICMP packet: i/o timeout INFO: 2025/02/14 23:35:23 Ping attempt 5 of 5 INFO: 2025/02/14 23:35:23 Pinging 100.89.128.1 WARN: 2025/02/14 23:35:33 Ping attempt 5 failed: failed to read ICMP packet: i/o timeout WARN: 2025/02/14 23:35:33 Failed to ping 100.89.128.1: all ping attempts failed after 5 tries, last error: failed to read ICMP packet: i/o timeout WARN: 2025/02/14 23:35:33 HINT: Do you have UDP port 51280 (or the port in config.yml) open on your Pangolin server? INFO: 2025/02/15 07:56:34 Sent registration message INFO: 2025/02/15 07:56:34 Received registration message INFO: 2025/02/15 07:56:34 Received: {Type:newt/wg/connect Data:map[endpoint:my.domain:44400 publicKey:---key---- serverIP:100.89.128.1 targets:map[tcp:[null:null] udp:[null:null 45964:192.168.10.32:10090]] tunnelIP:100.89.128.4]} INFO: 2025/02/15 07:56:34 WireGuard device created. Lets ping the server now... INFO: 2025/02/15 07:56:34 Ping attempt 1 of 5 INFO: 2025/02/15 07:56:34 Pinging 100.89.128.1 INFO: 2025/02/15 07:56:34 Ping latency: 57.496664ms INFO: 2025/02/15 07:56:34 Starting ping check INFO: 2025/02/15 07:56:34 Invalid target format: null:null INFO: 2025/02/15 07:56:34 Invalid target format: null:null INFO: 2025/02/15 07:56:34 Not adding target because not running INFO: 2025/02/15 07:56:34 Started udp proxy from 100.89.128.4:45964 to 192.168.10.32:10090 INFO: 2025/02/15 08:00:29 Received: {Type:newt/udp/add Data:map[targets:[55851:192.168.10.5:10091]]} INFO: 2025/02/15 08:00:29 Started udp proxy from 100.89.128.4:55851 to 192.168.10.5:10091 INFO: 2025/02/15 08:00:46 Received: {Type:newt/udp/add Data:map[targets:[54486:192.168.10.5:10091]]} ``` ## Traefik Log ``` traefik | 2025-02-15T09:00:11+01:00 ERR Error while handling UDP stream error="read udp 100.89.128.1:56889->100.89.128.4:46533: read: connection refused" traefik | 2025-02-15T09:00:17+01:00 ERR Error while handling UDP stream error="read udp 100.89.128.1:38761->100.89.128.4:46533: read: connection refused" traefik | 2025-02-15T09:00:23+01:00 ERR Error while handling UDP stream error="read udp 100.89.128.1:45171->100.89.128.4:46533: read: connection refused" traefik | 2025-02-15T09:00:28+01:00 ERR Error while handling UDP stream error="read udp 100.89.128.1:59707->100.89.128.4:46533: read: connection refused" traefik | 2025-02-15T09:00:33+01:00 ERR Error while handling UDP stream error="read udp 100.89.128.1:52478->100.89.128.4:46533: read: connection refused" ```
GiteaMirror added the needs investigatingbug labels 2026-04-20 07:12:35 -05:00
GiteaMirror commented 2026-04-20 07:12:37 -05:00
Author
Owner
Copy Link

@oschwartz10612 commented on GitHub (Feb 15, 2025):

There is an open issue to make newt reconnect a little more reliability so I am looking into that. If your ISP is flaky then it might be dropping and the pings are not enough to get it to reconnect. I will let you know if I get a solution shortly.

About the null:null targets: are you on the latest version of Pangolin beta.14? We fixed a couple of issues with Newt and targets a little while back.

<!-- gh-comment-id:2660985129 --> @oschwartz10612 commented on GitHub (Feb 15, 2025): There is an open issue to make newt reconnect a little more reliability so I am looking into that. If your ISP is flaky then it might be dropping and the pings are not enough to get it to reconnect. I will let you know if I get a solution shortly. About the null:null targets: are you on the latest version of Pangolin `beta.14`? We fixed a couple of issues with Newt and targets a little while back.
GiteaMirror commented 2026-04-20 07:12:38 -05:00
Author
Owner
Copy Link

@Lokowitz commented on GitHub (Feb 15, 2025):

Yes I am at Beta.14. But I am not sure if this problem was before or after the update. Will have an eye on it the next days and let you know. Maybe the update (short downtime/connection loss) causes this problem.

<!-- gh-comment-id:2661097971 --> @Lokowitz commented on GitHub (Feb 15, 2025): Yes I am at Beta.14. But I am not sure if this problem was before or after the update. Will have an eye on it the next days and let you know. Maybe the update (short downtime/connection loss) causes this problem.
GiteaMirror commented 2026-04-20 07:12:38 -05:00
Author
Owner
Copy Link

@Lokowitz commented on GitHub (Feb 20, 2025):

Hi guys,

just want to let you know that it happend again.
No changes were made from my side this time, so maybe its my ISP as you mentioned.
But it looks like that 1 of 3 resources got back online (port 10090).

newt log

INFO: 2025/02/19 18:31:47 Sent registration message
INFO: 2025/02/19 18:31:47 Received registration message
INFO: 2025/02/19 18:31:47 Received: {Type:newt/wg/connect Data:map[endpointmy.domain:44400 publicKey:--key--= serverIP:100.89.128.1 targets:map[tcp:[null:null] udp:[null:null 45964:192.168.10.32:10090]] tunnelIP:100.89.128.4]}
INFO: 2025/02/19 18:31:47 WireGuard device created. Lets ping the server now...
INFO: 2025/02/19 18:31:47 Ping attempt 1 of 5
INFO: 2025/02/19 18:31:47 Pinging 100.89.128.1
INFO: 2025/02/19 18:31:47 Ping latency: 52.74737ms
INFO: 2025/02/19 18:31:47 Starting ping check
INFO: 2025/02/19 18:31:47 Invalid target format: null:null
INFO: 2025/02/19 18:31:47 Invalid target format: null:null
INFO: 2025/02/19 18:31:47 Not adding target because not running
INFO: 2025/02/19 18:31:47 Started udp proxy from 100.89.128.4:45964 to 192.168.10.32:10090
<!-- gh-comment-id:2670978939 --> @Lokowitz commented on GitHub (Feb 20, 2025): Hi guys, just want to let you know that it happend again. No changes were made from my side this time, so maybe its my ISP as you mentioned. But it looks like that 1 of 3 resources got back online (port 10090). ## newt log ``` INFO: 2025/02/19 18:31:47 Sent registration message INFO: 2025/02/19 18:31:47 Received registration message INFO: 2025/02/19 18:31:47 Received: {Type:newt/wg/connect Data:map[endpointmy.domain:44400 publicKey:--key--= serverIP:100.89.128.1 targets:map[tcp:[null:null] udp:[null:null 45964:192.168.10.32:10090]] tunnelIP:100.89.128.4]} INFO: 2025/02/19 18:31:47 WireGuard device created. Lets ping the server now... INFO: 2025/02/19 18:31:47 Ping attempt 1 of 5 INFO: 2025/02/19 18:31:47 Pinging 100.89.128.1 INFO: 2025/02/19 18:31:47 Ping latency: 52.74737ms INFO: 2025/02/19 18:31:47 Starting ping check INFO: 2025/02/19 18:31:47 Invalid target format: null:null INFO: 2025/02/19 18:31:47 Invalid target format: null:null INFO: 2025/02/19 18:31:47 Not adding target because not running INFO: 2025/02/19 18:31:47 Started udp proxy from 100.89.128.4:45964 to 192.168.10.32:10090 ```
GiteaMirror commented 2026-04-20 07:12:39 -05:00
Author
Owner
Copy Link

@oschwartz10612 commented on GitHub (Feb 20, 2025):

Thanks for confirming you saw this again and providing the logs. If you
are definitely on beta.14 (see bottom of the screen when logged into
Pangolin dashboard) and we are having this null:null target issue still
then there may be a problem (in addition to the reconnecting).

I will try to take a look into this as soon as possible and see if I can
reproduce.

<!-- gh-comment-id:2671708290 --> @oschwartz10612 commented on GitHub (Feb 20, 2025): Thanks for confirming you saw this again and providing the logs. If you are definitely on beta.14 (see bottom of the screen when logged into Pangolin dashboard) and we are having this null:null target issue still then there may be a problem (in addition to the reconnecting). I will try to take a look into this as soon as possible and see if I can reproduce.
GiteaMirror commented 2026-04-20 07:12:39 -05:00
Author
Owner
Copy Link

@Lokowitz commented on GitHub (Feb 21, 2025):

Yes, everything is running on latest tag.

For my understanding the reconnection works but not the sync of the tagets (only partially).
It was enough to just disable and enable the target to get it back working.

Image

<!-- gh-comment-id:2673791757 --> @Lokowitz commented on GitHub (Feb 21, 2025): Yes, everything is running on latest tag. For my understanding the reconnection works but not the sync of the tagets (only partially). It was enough to just disable and enable the target to get it back working. ![Image](https://github.com/user-attachments/assets/9e2d6848-69f6-4313-ba50-b9ead666a04b)
GiteaMirror commented 2026-04-20 07:12:40 -05:00
Author
Owner
Copy Link

@oschwartz10612 commented on GitHub (Feb 21, 2025):

Ahh okay are the two resources that dont come back TCP/UDP resources and not HTTP resources? That helps me narrow this down!

<!-- gh-comment-id:2674695080 --> @oschwartz10612 commented on GitHub (Feb 21, 2025): Ahh okay are the two resources that dont come back TCP/UDP resources and not HTTP resources? That helps me narrow this down!
GiteaMirror commented 2026-04-20 07:12:40 -05:00
Author
Owner
Copy Link

@Lokowitz commented on GitHub (Feb 21, 2025):

One was http but disabled target and the other enabled target udp.
The one working is also enabled target udp.

<!-- gh-comment-id:2674739744 --> @Lokowitz commented on GitHub (Feb 21, 2025): One was http but disabled target and the other enabled target udp. The one working is also enabled target udp.
GiteaMirror commented 2026-04-20 07:12:41 -05:00
Author
Owner
Copy Link

@cevinov commented on GitHub (Mar 6, 2025):

Hello, I have the same problem also. Please check it thanks.

Image

Image

<!-- gh-comment-id:2704249501 --> @cevinov commented on GitHub (Mar 6, 2025): Hello, I have the same problem also. Please check it thanks. ![Image](https://github.com/user-attachments/assets/07e2ebb0-b1c1-4bc7-a0a4-8b0de6837b30) ![Image](https://github.com/user-attachments/assets/0b618b81-5b32-41c6-be50-78d9c6890cad)
GiteaMirror commented 2026-04-20 07:12:41 -05:00
Author
Owner
Copy Link

@oschwartz10612 commented on GitHub (Mar 6, 2025):

Hum thanks for the info. Will try to prioritize investigating this ASAP.

<!-- gh-comment-id:2704484869 --> @oschwartz10612 commented on GitHub (Mar 6, 2025): Hum thanks for the info. Will try to prioritize investigating this ASAP.
GiteaMirror commented 2026-04-20 07:12:42 -05:00
Author
Owner
Copy Link

@cevinov commented on GitHub (Mar 7, 2025):

Thanks, I will wait

<!-- gh-comment-id:2707165246 --> @cevinov commented on GitHub (Mar 7, 2025): Thanks, I will wait
GiteaMirror commented 2026-04-20 07:12:42 -05:00
Author
Owner
Copy Link

@oschwartz10612 commented on GitHub (Mar 10, 2025):

I think #307 resolves this. Please reopen if there is an issue.

<!-- gh-comment-id:2710940982 --> @oschwartz10612 commented on GitHub (Mar 10, 2025): I think #307 resolves this. Please reopen if there is an issue.
GiteaMirror commented 2026-04-20 07:12:43 -05:00
Author
Owner
Copy Link

@cevinov commented on GitHub (Mar 11, 2025):

Hello Oschwartz,

When I try I get this error:
Failed to connect: failed to get token: failed to request new token.

Image

Please check it.

Thanks.

<!-- gh-comment-id:2712202429 --> @cevinov commented on GitHub (Mar 11, 2025): Hello Oschwartz, When I try I get this error: Failed to connect: failed to get token: failed to request new token. ![Image](https://github.com/user-attachments/assets/93e1c0cc-2e57-4b32-b0ba-e9f8bc34d181) Please check it. Thanks.
GiteaMirror commented 2026-04-20 07:12:44 -05:00
Author
Owner
Copy Link

@oschwartz10612 commented on GitHub (Mar 11, 2025):

@cevinov I think your command is wrong. You have an extra newt after ./newt_windows_amd64.exe

<!-- gh-comment-id:2712213203 --> @oschwartz10612 commented on GitHub (Mar 11, 2025): @cevinov I think your command is wrong. You have an extra newt after `./newt_windows_amd64.exe`
GiteaMirror commented 2026-04-20 07:12:45 -05:00
Author
Owner
Copy Link

@cevinov commented on GitHub (Mar 11, 2025):

Thanks sir, it works now.

Image

<!-- gh-comment-id:2712267183 --> @cevinov commented on GitHub (Mar 11, 2025): Thanks sir, it works now. ![Image](https://github.com/user-attachments/assets/e06e97af-d426-4fe0-895d-d2cd1fb75122)
GiteaMirror commented 2026-04-20 07:12:45 -05:00
Author
Owner
Copy Link

@cevinov commented on GitHub (Mar 11, 2025):

But why did it give me the error can't be reached? The site is already online and the configuration of the resources (http) has been done.

https://go.xxxxx.xxxxx.id

Image

Please check it thanks.

<!-- gh-comment-id:2712295559 --> @cevinov commented on GitHub (Mar 11, 2025): But why did it give me the error can't be reached? The site is already online and the configuration of the resources (http) has been done. https://go.xxxxx.xxxxx.id ![Image](https://github.com/user-attachments/assets/f6fb5073-c415-4f08-b098-2b80304524a7) Please check it thanks.
GiteaMirror commented 2026-04-20 07:12:47 -05:00
Author
Owner
Copy Link

@oschwartz10612 commented on GitHub (Mar 11, 2025):

I think that would be caused higher up in the stack and is not related to this issue. Could you make sure that your DNS is working correctly and pointed to your VPS etc? https://docs.fossorial.io/Getting%20Started/dns-networking

I would come ask this question on Discord or maybe open a new issue because I dont think this is the same issue.

<!-- gh-comment-id:2712301270 --> @oschwartz10612 commented on GitHub (Mar 11, 2025): I think that would be caused higher up in the stack and is not related to this issue. Could you make sure that your DNS is working correctly and pointed to your VPS etc? https://docs.fossorial.io/Getting%20Started/dns-networking I would come ask this question on Discord or maybe open a new issue because I dont think this is the same issue.
GiteaMirror commented 2026-04-20 07:12:47 -05:00
Author
Owner
Copy Link

@cevinov commented on GitHub (Mar 11, 2025):

I see thanks sir.

<!-- gh-comment-id:2712317636 --> @cevinov commented on GitHub (Mar 11, 2025): I see thanks sir.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
api
authentication
bug
config
dependencies
docker
documentation
enhancement
good first issue
help wanted
Improvement
Look Into
needs investigating
networking
new feature
non-critical bug
potential bug
pull-request
Mirrored from GitHub Pull Request
 question
reverse proxy
Security
stale
ui
wontfix
No Label bug needs investigating
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/pangolin#3282
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?