[GH-ISSUE #240] Newt health check expects 2xx response code after server restart #1748

New Issue

GiteaMirror · 2026-04-27T22:28:34-05:00

GiteaMirror commented

2026-04-27 22:28:34 -05:00

Originally created by @xupefei on GitHub (Feb 18, 2026).
Original GitHub issue: https://github.com/fosrl/newt/issues/240

Originally assigned to: @LaurenceJJones on GitHub.

Describe the Bug

When a Pangolin server got restarted and Newt reconnected, existing health checks with custom Response Codes are reverted to 2xx.

Environment

OS Type & Version: Debian 13
Pangolin Version: 1.15.4
Gerbil Version: -
Traefik Version: -
Newt Version: 1.9.0
Olm Version: (if applicable)

To Reproduce

Create a REST endpoint that returns HTTP code 409.
Create a service, add a health check with that endpoint, fill the Expected Response Code to 409.
Look at Newt log, it should say Target 1 initial status: healthy.
Now restart the Pangolin server stack.
Look at Newt log again. It should reconnect in a moment, but the health check remains FAIL forever.
Look at Newt log again, find health check failed with status code 409. Note that we are actually expecting 409 to be a good response.

Looking at the code, it seems Newt is now expecting 200-300 codes instead, because if otherwise the log will be the following:

health check failed with status code NNN (expected: NNN)

instead of (when checking for 2xx range)

health check failed with status code NNN

The logic of printing different logs is at

b7af49d759/healthcheck/healthcheck.go (L428-L449)

This suggests that Newt is expecting the response code to be the default 2xx instead of the custom one.

Attached screenshots:

My health check settings with a custom response code:

Newt log after server restart:

Expected Behavior

The service should work. However now with a failed health check, the service is always down.

Originally created by @xupefei on GitHub (Feb 18, 2026). Original GitHub issue: https://github.com/fosrl/newt/issues/240 Originally assigned to: @LaurenceJJones on GitHub. ### Describe the Bug When a Pangolin server got restarted and Newt reconnected, existing health checks with custom Response Codes are reverted to 2xx. ### Environment - OS Type & Version: Debian 13 - Pangolin Version: 1.15.4 - Gerbil Version: - - Traefik Version: - - Newt Version: 1.9.0 - Olm Version: (if applicable) ### To Reproduce 1. Create a REST endpoint that returns HTTP code 409. 2. Create a service, add a health check with that endpoint, fill the Expected Response Code to 409. 3. Look at Newt log, it should say `Target 1 initial status: healthy`. 4. Now restart the Pangolin server stack. 5. Look at Newt log again. It should reconnect in a moment, but the health check remains FAIL forever. 6. Look at Newt log again, find `health check failed with status code 409`. Note that we are actually expecting 409 to be a good response. Looking at the code, it seems Newt is now expecting 200-300 codes instead, because if otherwise the log will be the following: ``` health check failed with status code NNN (expected: NNN) ``` instead of (when checking for 2xx range) ``` health check failed with status code NNN ``` The logic of printing different logs is at https://github.com/fosrl/newt/blob/b7af49d7598d651f522c1358b59af8759b9a6e26/healthcheck/healthcheck.go#L428-L449 This suggests that Newt is expecting the response code to be the default 2xx instead of the custom one. Attached screenshots: My health check settings with a custom response code: <img width="717" height="557" alt="Image" src="https://github.com/user-attachments/assets/ef1d7a6e-50ab-4dc1-9593-e92608d41dec" /> Newt log after server restart: <img width="715" height="455" alt="Image" src="https://github.com/user-attachments/assets/d52d4c29-34fb-4d13-b5c9-711506af11ab" /> ### Expected Behavior The service should work. However now with a failed health check, the service is always down.

GiteaMirror added the bug label 2026-04-27 22:28:34 -05:00

GiteaMirror closed this issue

2026-04-27 22:28:35 -05:00

GiteaMirror commented

2026-04-27 22:28:37 -05:00

@DevNinja90 commented on GitHub (Mar 5, 2026):

We can confirm the bug. The issue is witin newt restarts.
Our hcStatusCode is expected to be 401 (Unauthorized). Newt resets this to 200 after a restart. Only reconfiguring it in pangolin pushes the correct hcStatus to newt.

For debugging purposes the hcStatusCode is not shown in newt debug logs on receiving the registration message data. This made me think. Maybe the Code is not send to newt in the first place? It turns out that is correct.
⭕ Pangolin bug.

DEBUG: 2026/03/05 07:34:13 Received registration message data: map[endpoint:XXX:51820 healthCheckTargets:[map[hcEnabled:true hcHeaders:map[] hcHostname:172.28.0.3 hcInterval:300 hcMethod:GET hcMode:http hcPath:/ hcPort:443 hcScheme:https hcTimeout:15 hcTlsServerName: hcUnhealthyInterval:600 id:28] [...]

❓ no hcStatus

the function buildTargetConfigurationForNewtClient is missing the hcStatus.
in here https://github.com/fosrl/pangolin/blob/1.16.2/server/routers/newt/buildConfiguration.ts#L168

        return {
            id: target.targetId,
            hcEnabled: target.hcEnabled,
            hcPath: target.hcPath,
            hcScheme: target.hcScheme,
            hcMode: target.hcMode,
            hcHostname: target.hcHostname,
            hcPort: target.hcPort,
            hcInterval: target.hcInterval, // in seconds
            hcUnhealthyInterval: target.hcUnhealthyInterval, // in seconds
            hcTimeout: target.hcTimeout, // in seconds
            hcHeaders: hcHeadersSend,
            hcMethod: target.hcMethod,
            ❓MISSING HERE❓
            hcTlsServerName: target.hcTlsServerName
        };

When changing a config in pangolin the function addTargets sends the hcStatus
https://github.com/fosrl/pangolin/blob/1.16.2/server/routers/newt/targets.ts

        return {
            id: target.targetId,
            hcEnabled: hc.hcEnabled,
            hcPath: hc.hcPath,
            hcScheme: hc.hcScheme,
            hcMode: hc.hcMode,
            hcHostname: hc.hcHostname,
            hcPort: hc.hcPort,
            hcInterval: hc.hcInterval, // in seconds
            hcUnhealthyInterval: hc.hcUnhealthyInterval, // in seconds
            hcTimeout: hc.hcTimeout, // in seconds
            hcHeaders: hcHeadersSend,
            hcMethod: hc.hcMethod,
            👍🏼 hcStatus: hcStatus,👍🏼 
            hcTlsServerName: hc.hcTlsServerName
        };

I think a simple fix 👯

@DevNinja90 commented on GitHub (Mar 5, 2026): We can confirm the bug. The issue is witin newt restarts. Our hcStatusCode is expected to be 401 (Unauthorized). Newt resets this to 200 after a restart. Only reconfiguring it in pangolin pushes the correct hcStatus to newt. For debugging purposes the hcStatusCode is not shown in newt debug logs on receiving the registration message data. This made me think. Maybe the Code is not send to newt in the first place? It turns out that is correct. ⭕ Pangolin bug. > DEBUG: 2026/03/05 07:34:13 Received registration message data: map[endpoint:XXX:51820 healthCheckTargets:[map[hcEnabled:true hcHeaders:map[] hcHostname:172.28.0.3 hcInterval:300 hcMethod:GET hcMode:http hcPath:/ hcPort:443 hcScheme:https hcTimeout:15 hcTlsServerName: hcUnhealthyInterval:600 id:28] [...] ❓ no hcStatus the function buildTargetConfigurationForNewtClient is missing the hcStatus. in here https://github.com/fosrl/pangolin/blob/1.16.2/server/routers/newt/buildConfiguration.ts#L168 ``` return { id: target.targetId, hcEnabled: target.hcEnabled, hcPath: target.hcPath, hcScheme: target.hcScheme, hcMode: target.hcMode, hcHostname: target.hcHostname, hcPort: target.hcPort, hcInterval: target.hcInterval, // in seconds hcUnhealthyInterval: target.hcUnhealthyInterval, // in seconds hcTimeout: target.hcTimeout, // in seconds hcHeaders: hcHeadersSend, hcMethod: target.hcMethod, ❓MISSING HERE❓ hcTlsServerName: target.hcTlsServerName }; ``` When changing a config in pangolin the function addTargets sends the hcStatus https://github.com/fosrl/pangolin/blob/1.16.2/server/routers/newt/targets.ts ``` return { id: target.targetId, hcEnabled: hc.hcEnabled, hcPath: hc.hcPath, hcScheme: hc.hcScheme, hcMode: hc.hcMode, hcHostname: hc.hcHostname, hcPort: hc.hcPort, hcInterval: hc.hcInterval, // in seconds hcUnhealthyInterval: hc.hcUnhealthyInterval, // in seconds hcTimeout: hc.hcTimeout, // in seconds hcHeaders: hcHeadersSend, hcMethod: hc.hcMethod, 👍🏼 hcStatus: hcStatus,👍🏼 hcTlsServerName: hc.hcTlsServerName }; ``` I think a simple fix 👯

GiteaMirror commented

2026-04-27 22:28:40 -05:00

@LaurenceJJones commented on GitHub (Mar 7, 2026):

Thanks for the debugging @DevNinja90 your correct, the initial healthcheck that is sent when a resource is created includes the custom status, when newt reconnects and get sent the connect websocket this was missing the custom status. Opened a PR to get this resolved.

@LaurenceJJones commented on GitHub (Mar 7, 2026): Thanks for the debugging @DevNinja90 your correct, the initial healthcheck that is sent when a resource is created includes the custom status, when newt reconnects and get sent the connect websocket this was missing the custom status. Opened a PR to get this resolved.

GiteaMirror commented

2026-04-27 22:28:45 -05:00

@LaurenceJJones commented on GitHub (Mar 10, 2026):

will be resolved in next pangolin release, closing as completed as development fix has been issued.

@LaurenceJJones commented on GitHub (Mar 10, 2026): will be resolved in next pangolin release, closing as completed as development fix has been issued.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/newt#1748