github-starred/newt
2
0
Fork 0
mirror of https://github.com/fosrl/newt.git synced 2026-11-11 16:33:24 -06:00
Code Issues 713 Packages Projects Releases 47 Wiki Activity

[GH-ISSUE #240] Newt health check expects 2xx response code after server restart #7548

New Issue
Closed
opened 2026-06-21 23:59:47 -05:00 by GiteaMirror · 3 comments
GiteaMirror commented 2026-06-21 23:59:47 -05:00
Owner
Copy Link
Copy Source

Originally created by @xupefei on GitHub (Feb 18, 2026).
Original GitHub issue: https://github.com/fosrl/newt/issues/240

Originally assigned to: @LaurenceJJones on GitHub.

Describe the Bug

When a Pangolin server got restarted and Newt reconnected, existing health checks with custom Response Codes are reverted to 2xx.

Environment

  • OS Type & Version: Debian 13
  • Pangolin Version: 1.15.4
  • Gerbil Version: -
  • Traefik Version: -
  • Newt Version: 1.9.0
  • Olm Version: (if applicable)

To Reproduce

  1. Create a REST endpoint that returns HTTP code 409.
  2. Create a service, add a health check with that endpoint, fill the Expected Response Code to 409.
  3. Look at Newt log, it should say Target 1 initial status: healthy.
  4. Now restart the Pangolin server stack.
  5. Look at Newt log again. It should reconnect in a moment, but the health check remains FAIL forever.
  6. Look at Newt log again, find health check failed with status code 409. Note that we are actually expecting 409 to be a good response.

Looking at the code, it seems Newt is now expecting 200-300 codes instead, because if otherwise the log will be the following:

health check failed with status code NNN (expected: NNN)

instead of (when checking for 2xx range)

health check failed with status code NNN

The logic of printing different logs is at

https://github.com/fosrl/newt/blob/b7af49d7598d651f522c1358b59af8759b9a6e26/healthcheck/healthcheck.go#L428-L449

This suggests that Newt is expecting the response code to be the default 2xx instead of the custom one.

Attached screenshots:

My health check settings with a custom response code:
Image

Newt log after server restart:
Image

Expected Behavior

The service should work. However now with a failed health check, the service is always down.

Originally created by @xupefei on GitHub (Feb 18, 2026). Original GitHub issue: https://github.com/fosrl/newt/issues/240 Originally assigned to: @LaurenceJJones on GitHub. ### Describe the Bug When a Pangolin server got restarted and Newt reconnected, existing health checks with custom Response Codes are reverted to 2xx. ### Environment - OS Type & Version: Debian 13 - Pangolin Version: 1.15.4 - Gerbil Version: - - Traefik Version: - - Newt Version: 1.9.0 - Olm Version: (if applicable) ### To Reproduce 1. Create a REST endpoint that returns HTTP code 409. 2. Create a service, add a health check with that endpoint, fill the Expected Response Code to 409. 3. Look at Newt log, it should say `Target 1 initial status: healthy`. 4. Now restart the Pangolin server stack. 5. Look at Newt log again. It should reconnect in a moment, but the health check remains FAIL forever. 6. Look at Newt log again, find `health check failed with status code 409`. Note that we are actually expecting 409 to be a good response. Looking at the code, it seems Newt is now expecting 200-300 codes instead, because if otherwise the log will be the following: ``` health check failed with status code NNN (expected: NNN) ``` instead of (when checking for 2xx range) ``` health check failed with status code NNN ``` The logic of printing different logs is at https://github.com/fosrl/newt/blob/b7af49d7598d651f522c1358b59af8759b9a6e26/healthcheck/healthcheck.go#L428-L449 This suggests that Newt is expecting the response code to be the default 2xx instead of the custom one. Attached screenshots: My health check settings with a custom response code: <img width="717" height="557" alt="Image" src="https://github.com/user-attachments/assets/ef1d7a6e-50ab-4dc1-9593-e92608d41dec" /> Newt log after server restart: <img width="715" height="455" alt="Image" src="https://github.com/user-attachments/assets/d52d4c29-34fb-4d13-b5c9-711506af11ab" /> ### Expected Behavior The service should work. However now with a failed health check, the service is always down.
GiteaMirror added the bug label 2026-06-21 23:59:48 -05:00
GiteaMirror commented 2026-06-21 23:59:55 -05:00
Author
Owner
Copy Link
Copy Source

@DevNinja90 commented on GitHub (Mar 5, 2026):

We can confirm the bug. The issue is witin newt restarts.
Our hcStatusCode is expected to be 401 (Unauthorized). Newt resets this to 200 after a restart. Only reconfiguring it in pangolin pushes the correct hcStatus to newt.

For debugging purposes the hcStatusCode is not shown in newt debug logs on receiving the registration message data. This made me think. Maybe the Code is not send to newt in the first place? It turns out that is correct.
⭕ Pangolin bug.

DEBUG: 2026/03/05 07:34:13 Received registration message data: map[endpoint:XXX:51820 healthCheckTargets:[map[hcEnabled:true hcHeaders:map[] hcHostname:172.28.0.3 hcInterval:300 hcMethod:GET hcMode:http hcPath:/ hcPort:443 hcScheme:https hcTimeout:15 hcTlsServerName: hcUnhealthyInterval:600 id:28] [...]

❓ no hcStatus

the function buildTargetConfigurationForNewtClient is missing the hcStatus.
in here https://github.com/fosrl/pangolin/blob/1.16.2/server/routers/newt/buildConfiguration.ts#L168

        return {
            id: target.targetId,
            hcEnabled: target.hcEnabled,
            hcPath: target.hcPath,
            hcScheme: target.hcScheme,
            hcMode: target.hcMode,
            hcHostname: target.hcHostname,
            hcPort: target.hcPort,
            hcInterval: target.hcInterval, // in seconds
            hcUnhealthyInterval: target.hcUnhealthyInterval, // in seconds
            hcTimeout: target.hcTimeout, // in seconds
            hcHeaders: hcHeadersSend,
            hcMethod: target.hcMethod,
            ❓MISSING HERE❓
            hcTlsServerName: target.hcTlsServerName
        };

When changing a config in pangolin the function addTargets sends the hcStatus
https://github.com/fosrl/pangolin/blob/1.16.2/server/routers/newt/targets.ts

        return {
            id: target.targetId,
            hcEnabled: hc.hcEnabled,
            hcPath: hc.hcPath,
            hcScheme: hc.hcScheme,
            hcMode: hc.hcMode,
            hcHostname: hc.hcHostname,
            hcPort: hc.hcPort,
            hcInterval: hc.hcInterval, // in seconds
            hcUnhealthyInterval: hc.hcUnhealthyInterval, // in seconds
            hcTimeout: hc.hcTimeout, // in seconds
            hcHeaders: hcHeadersSend,
            hcMethod: hc.hcMethod,
            👍🏼 hcStatus: hcStatus,👍🏼 
            hcTlsServerName: hc.hcTlsServerName
        };

I think a simple fix 👯

<!-- gh-comment-id:4003173100 --> @DevNinja90 commented on GitHub (Mar 5, 2026): We can confirm the bug. The issue is witin newt restarts. Our hcStatusCode is expected to be 401 (Unauthorized). Newt resets this to 200 after a restart. Only reconfiguring it in pangolin pushes the correct hcStatus to newt. For debugging purposes the hcStatusCode is not shown in newt debug logs on receiving the registration message data. This made me think. Maybe the Code is not send to newt in the first place? It turns out that is correct. ⭕ Pangolin bug. > DEBUG: 2026/03/05 07:34:13 Received registration message data: map[endpoint:XXX:51820 healthCheckTargets:[map[hcEnabled:true hcHeaders:map[] hcHostname:172.28.0.3 hcInterval:300 hcMethod:GET hcMode:http hcPath:/ hcPort:443 hcScheme:https hcTimeout:15 hcTlsServerName: hcUnhealthyInterval:600 id:28] [...] ❓ no hcStatus the function buildTargetConfigurationForNewtClient is missing the hcStatus. in here https://github.com/fosrl/pangolin/blob/1.16.2/server/routers/newt/buildConfiguration.ts#L168 ``` return { id: target.targetId, hcEnabled: target.hcEnabled, hcPath: target.hcPath, hcScheme: target.hcScheme, hcMode: target.hcMode, hcHostname: target.hcHostname, hcPort: target.hcPort, hcInterval: target.hcInterval, // in seconds hcUnhealthyInterval: target.hcUnhealthyInterval, // in seconds hcTimeout: target.hcTimeout, // in seconds hcHeaders: hcHeadersSend, hcMethod: target.hcMethod, ❓MISSING HERE❓ hcTlsServerName: target.hcTlsServerName }; ``` When changing a config in pangolin the function addTargets sends the hcStatus https://github.com/fosrl/pangolin/blob/1.16.2/server/routers/newt/targets.ts ``` return { id: target.targetId, hcEnabled: hc.hcEnabled, hcPath: hc.hcPath, hcScheme: hc.hcScheme, hcMode: hc.hcMode, hcHostname: hc.hcHostname, hcPort: hc.hcPort, hcInterval: hc.hcInterval, // in seconds hcUnhealthyInterval: hc.hcUnhealthyInterval, // in seconds hcTimeout: hc.hcTimeout, // in seconds hcHeaders: hcHeadersSend, hcMethod: hc.hcMethod, 👍🏼 hcStatus: hcStatus,👍🏼 hcTlsServerName: hc.hcTlsServerName }; ``` I think a simple fix 👯
GiteaMirror commented 2026-06-21 23:59:58 -05:00
Author
Owner
Copy Link
Copy Source

@LaurenceJJones commented on GitHub (Mar 7, 2026):

Thanks for the debugging @DevNinja90 your correct, the initial healthcheck that is sent when a resource is created includes the custom status, when newt reconnects and get sent the connect websocket this was missing the custom status. Opened a PR to get this resolved.

<!-- gh-comment-id:4015738291 --> @LaurenceJJones commented on GitHub (Mar 7, 2026): Thanks for the debugging @DevNinja90 your correct, the initial healthcheck that is sent when a resource is created includes the custom status, when newt reconnects and get sent the connect websocket this was missing the custom status. Opened a PR to get this resolved.
GiteaMirror commented 2026-06-22 00:00:01 -05:00
Author
Owner
Copy Link
Copy Source

@LaurenceJJones commented on GitHub (Mar 10, 2026):

will be resolved in next pangolin release, closing as completed as development fix has been issued.

<!-- gh-comment-id:4031592814 --> @LaurenceJJones commented on GitHub (Mar 10, 2026): will be resolved in next pangolin release, closing as completed as development fix has been issued.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
dependencies
docker
enhancement
go
good first issue
help wanted
Improvement
needs investigating
pull-request
Mirrored from GitHub Pull Request
 stale
wontfix
No labels bug
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
GiteaMirror ninjasurge (Kaehlen Rog)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/newt#7548