github-starred/newt
2
0
Fork 0
mirror of https://github.com/fosrl/newt.git synced 2026-05-06 07:59:04 -05:00
Code Issues 227 Packages Projects Releases 40 Wiki Activity

[GH-ISSUE #220] TCP socket leak when healthcheck enabled with 204 response code #837

New Issue
Closed
opened 2026-04-19 14:19:30 -05:00 by GiteaMirror · 1 comment
GiteaMirror commented 2026-04-19 14:19:30 -05:00
Owner
Copy Link

Originally created by @mprokopiev on GitHub (Dec 24, 2025).
Original GitHub issue: https://github.com/fosrl/newt/issues/220

Originally assigned to: @oschwartz10612 on GitHub.

Describe the Bug

I've been using a latest newt (problem also happened in 1.7.0 - upgraded for this issue purpose) and observed that among all resources with healthchecks configured, newt has a tcp socket leak for a single resource, which responds with 204 (which is configured as successful response code). This problem eventually caused the error when connecting to the resource either for healthchecks or any other valid communication:

health check failed: Get "http://172.19.7.10:8008/healthz": dial tcp 172.19.7.10:8008: connect: cannot assign requested address

Before newt restart, I found newt has 23k of opened sockets with above address - resource address in docker container.

I've made a simple script:

#!/bin/bash
SERVICE_NAME="newt"
PID=$(systemctl show -p MainPID --value "$SERVICE_NAME")
COUNT=$(sudo lsof -p "$PID" -Pn | grep 8008 | wc -l)

echo "========================================"
echo "Timestamp: $(LC_TIME=en_US.UTF-8 date '+%Y-%m-%d %H:%M:%S')"
echo "Service: $SERVICE_NAME (PID: $PID)"
echo "Port 8008 connections: $COUNT"
echo "========================================"
echo ""

And called it every 10 min:

  1. post restart of newt
Dec 23 22:11:31 container.ro.internal newt-port-checker.sh[2534639]: ========================================
Dec 23 22:11:31 container.ro.internal newt-port-checker.sh[2534639]: Timestamp: 2025-12-23 22:11:31
Dec 23 22:11:31 container.ro.internal newt-port-checker.sh[2534639]: Service: newt (PID: 2532760)
Dec 23 22:11:31 container.ro.internal newt-port-checker.sh[2534639]: Port 8008 connections: 7
Dec 23 22:11:31 container.ro.internal newt-port-checker.sh[2534639]: ========================================
  1. one hour after:
Dec 23 23:10:01 container.ro.internal newt-port-checker.sh[2715914]: ========================================
Dec 23 23:10:01 container.ro.internal newt-port-checker.sh[2715914]: Timestamp: 2025-12-23 23:10:01
Dec 23 23:10:01 container.ro.internal newt-port-checker.sh[2715914]: Service: newt (PID: 2532760)
Dec 23 23:10:01 container.ro.internal newt-port-checker.sh[2715914]: Port 8008 connections: 709
Dec 23 23:10:01 container.ro.internal newt-port-checker.sh[2715914]: ========================================
  1. a night after:
Dec 24 09:00:01 container.ro.internal newt-port-checker.sh[372028]: ========================================
Dec 24 09:00:01 container.ro.internal newt-port-checker.sh[372028]: Timestamp: 2025-12-24 09:00:01
Dec 24 09:00:01 container.ro.internal newt-port-checker.sh[372028]: Service: newt (PID: 2532760)
Dec 24 09:00:01 container.ro.internal newt-port-checker.sh[372028]: Port 8008 connections: 4994
Dec 24 09:00:01 container.ro.internal newt-port-checker.sh[372028]: ========================================

All of the opened sockets are related to a single resource:

❯ sudo lsof -p $(systemctl show -p MainPID --value newt) -Pn | grep 172.19.7.10:8008 | wc -l
5122
❯ sudo lsof -p $(systemctl show -p MainPID --value newt) -Pn | grep 172.19.7.10:8008 -v | wc -l
21

All of the opened sockets has ESTABLISHED state:

❯ sudo lsof -p $(systemctl show -p MainPID --value newt) -Pn | grep 172.19.7.10:8008 | grep ESTABLISHED | wc -l
5134

The config is quite generic:
Image

I run curl for that IP and there is no unusual headers which may cause newt to keep the connection opened:

❯ curl "http://172.19.7.10:8008/healthz" -v
*   Trying 172.19.7.10:8008...
* Connected to 172.19.7.10 (172.19.7.10) port 8008
* using HTTP/1.x
> GET /healthz HTTP/1.1
> Host: 172.19.7.10:8008
> User-Agent: curl/8.15.0
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 204 No Content
< Cache-Control: private, no-store
< Content-Security-Policy: default-src 'self'; base-uri 'self'; object-src 'none'; frame-ancestors 'none'; form-action 'self'; img-src * blob:;font-src 'self'; style-src 'self' 'unsafe-inline'; script-src 'self' 'nonce-tDp9tj2nn8-C6APb0ntQRA'
< Date: Wed, 24 Dec 2025 07:12:45 GMT
<
* Connection #0 to host 172.19.7.10 left intact

Disabling healthcheck for that resource stops this problem - no sockets are being added up.

I think it should be easy to reproduce, however I saved newt debug journal and can attach if needed.

Environment

  • Fedora 43
  • Pangolin Version: v1.13.1
  • Gerbil Version: 1.3.0
  • Traefik Version: v3.6.5
  • Newt Version: 1.8.0

To Reproduce

I believe it has to do to 204 HTTP code cause the rest of resources are working fine. Assuming so, configuring any resource to respond to healthcheck with 204 should do the trick. I am using Pocket ID which responds with 204 to the healthcheck.

Expected Behavior

Connection close.

Originally created by @mprokopiev on GitHub (Dec 24, 2025). Original GitHub issue: https://github.com/fosrl/newt/issues/220 Originally assigned to: @oschwartz10612 on GitHub. ### Describe the Bug I've been using a latest newt (problem also happened in 1.7.0 - upgraded for this issue purpose) and observed that among all resources with healthchecks configured, newt has a tcp socket leak for a single resource, which responds with 204 (which is configured as successful response code). This problem eventually caused the error when connecting to the resource either for healthchecks or any other valid communication: ``` health check failed: Get "http://172.19.7.10:8008/healthz": dial tcp 172.19.7.10:8008: connect: cannot assign requested address ``` Before newt restart, I found newt has 23k of opened sockets with above address - resource address in docker container. I've made a simple script: ``` #!/bin/bash SERVICE_NAME="newt" PID=$(systemctl show -p MainPID --value "$SERVICE_NAME") COUNT=$(sudo lsof -p "$PID" -Pn | grep 8008 | wc -l) echo "========================================" echo "Timestamp: $(LC_TIME=en_US.UTF-8 date '+%Y-%m-%d %H:%M:%S')" echo "Service: $SERVICE_NAME (PID: $PID)" echo "Port 8008 connections: $COUNT" echo "========================================" echo "" ``` And called it every 10 min: 1. post restart of newt ``` Dec 23 22:11:31 container.ro.internal newt-port-checker.sh[2534639]: ======================================== Dec 23 22:11:31 container.ro.internal newt-port-checker.sh[2534639]: Timestamp: 2025-12-23 22:11:31 Dec 23 22:11:31 container.ro.internal newt-port-checker.sh[2534639]: Service: newt (PID: 2532760) Dec 23 22:11:31 container.ro.internal newt-port-checker.sh[2534639]: Port 8008 connections: 7 Dec 23 22:11:31 container.ro.internal newt-port-checker.sh[2534639]: ======================================== ``` 2. one hour after: ``` Dec 23 23:10:01 container.ro.internal newt-port-checker.sh[2715914]: ======================================== Dec 23 23:10:01 container.ro.internal newt-port-checker.sh[2715914]: Timestamp: 2025-12-23 23:10:01 Dec 23 23:10:01 container.ro.internal newt-port-checker.sh[2715914]: Service: newt (PID: 2532760) Dec 23 23:10:01 container.ro.internal newt-port-checker.sh[2715914]: Port 8008 connections: 709 Dec 23 23:10:01 container.ro.internal newt-port-checker.sh[2715914]: ======================================== ``` 3. a night after: ``` Dec 24 09:00:01 container.ro.internal newt-port-checker.sh[372028]: ======================================== Dec 24 09:00:01 container.ro.internal newt-port-checker.sh[372028]: Timestamp: 2025-12-24 09:00:01 Dec 24 09:00:01 container.ro.internal newt-port-checker.sh[372028]: Service: newt (PID: 2532760) Dec 24 09:00:01 container.ro.internal newt-port-checker.sh[372028]: Port 8008 connections: 4994 Dec 24 09:00:01 container.ro.internal newt-port-checker.sh[372028]: ======================================== ``` All of the opened sockets are related to a single resource: ``` ❯ sudo lsof -p $(systemctl show -p MainPID --value newt) -Pn | grep 172.19.7.10:8008 | wc -l 5122 ❯ sudo lsof -p $(systemctl show -p MainPID --value newt) -Pn | grep 172.19.7.10:8008 -v | wc -l 21 ``` All of the opened sockets has ESTABLISHED state: ``` ❯ sudo lsof -p $(systemctl show -p MainPID --value newt) -Pn | grep 172.19.7.10:8008 | grep ESTABLISHED | wc -l 5134 ``` The config is quite generic: <img width="626" height="828" alt="Image" src="https://github.com/user-attachments/assets/14c79b38-03c1-440e-acfd-b7e05d088b1d" /> I run curl for that IP and there is no unusual headers which may cause newt to keep the connection opened: ``` ❯ curl "http://172.19.7.10:8008/healthz" -v * Trying 172.19.7.10:8008... * Connected to 172.19.7.10 (172.19.7.10) port 8008 * using HTTP/1.x > GET /healthz HTTP/1.1 > Host: 172.19.7.10:8008 > User-Agent: curl/8.15.0 > Accept: */* > * Request completely sent off < HTTP/1.1 204 No Content < Cache-Control: private, no-store < Content-Security-Policy: default-src 'self'; base-uri 'self'; object-src 'none'; frame-ancestors 'none'; form-action 'self'; img-src * blob:;font-src 'self'; style-src 'self' 'unsafe-inline'; script-src 'self' 'nonce-tDp9tj2nn8-C6APb0ntQRA' < Date: Wed, 24 Dec 2025 07:12:45 GMT < * Connection #0 to host 172.19.7.10 left intact ``` Disabling healthcheck for that resource stops this problem - no sockets are being added up. I think it should be easy to reproduce, however I saved newt debug journal and can attach if needed. ### Environment - Fedora 43 - Pangolin Version: v1.13.1 - Gerbil Version: 1.3.0 - Traefik Version: v3.6.5 - Newt Version: 1.8.0 ### To Reproduce I believe it has to do to 204 HTTP code cause the rest of resources are working fine. Assuming so, configuring any resource to respond to healthcheck with 204 should do the trick. I am using Pocket ID which responds with 204 to the healthcheck. ### Expected Behavior Connection close.
GiteaMirror commented 2026-04-19 14:19:31 -05:00
Author
Owner
Copy Link

@oschwartz10612 commented on GitHub (Dec 24, 2025):

This will be fixed in the next patch release by a701add824

<!-- gh-comment-id:3690156095 --> @oschwartz10612 commented on GitHub (Dec 24, 2025): This will be fixed in the next patch release by a701add8249811c01540797be08eb6f2c94ddb78
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
dependencies
enhancement
good first issue
help wanted
Improvement
needs investigating
pull-request
Mirrored from GitHub Pull Request
 stale
wontfix
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/newt#837
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?