mirror of
https://github.com/fosrl/pangolin.git
synced 2026-07-16 14:47:04 -05:00
[GH-ISSUE #3111] Traefik error handling intercepts backend HTTP 4xx JSON responses, breaking applications that rely on semantic error codes (e.g. Vaultwarden 2FA) #35739
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: github-starred/pangolin#35739
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @StefanSa on GitHub (May 19, 2026).
Original GitHub issue: https://github.com/fosrl/pangolin/issues/3111
Describe the Bug
When a backend application behind Pangolin returns an HTTP 400 (or other 4xx) response with a JSON body, Pangolin's Traefik layer intercepts the response and replaces the JSON body with its own error page (HTML). This breaks any application that uses 4xx status codes as part of its normal API flow.
The most prominent example is Vaultwarden (Bitwarden-compatible password manager). Its 2FA login is a two-step process:
POST /identity/connect/token400 Bad Requestwith a JSON body containing the 2FA provider information and challenge dataWhen Pangolin/Traefik intercepts step 2 and replaces the JSON body with an HTML error page, the client cannot parse the 2FA challenge and login fails completely.
Actual Behavior
The JSON body of the 400 response is replaced with an HTML error page. The Bitwarden client cannot parse the response and fails with an error. The Vaultwarden server log shows:
This can be confirmed by inspecting the browser DevTools (Network tab): the response to the
POST /identity/connect/tokenrequest contains HTML instead of the expected JSON.Scope of Impact
This is not specific to Vaultwarden. Any backend application that uses HTTP 4xx status codes with meaningful response bodies as part of its API contract will be affected. Common examples include:
400with JSON error details400,401,403,409,422with structured error responsesRelated Upstream Reports
This issue has been independently reported and confirmed by multiple users across the Vaultwarden community, with Pangolin specifically identified as the cause:
Suggested Solution
Ideally, Pangolin should not replace response bodies from healthy backend connections. Error pages should only be served when the backend is unreachable or the connection itself fails — not when the backend intentionally returns a 4xx status code with a valid response body.
If that is not feasible as a default, a per-resource option to disable error page interception (e.g. "Pass through backend error responses") would resolve the issue.
Environment
To Reproduce
Expected Behavior
The
400 Bad Requestresponse from Vaultwarden should be passed through to the client unmodified, including the JSON body. The client then prompts for the 2FA code and completes the login.@AstralDestiny commented on GitHub (May 19, 2026):
Are you using pangolin auth or vaultwarden natively?
@StefanSa commented on GitHub (May 19, 2026):
@AstralDestiny
Vaultwarden handles authentication natively — I am not using Pangolin’s built-in auth for this resource.
To rule out any interaction, I tested both configurations:
The result is identical in both cases: the 400 Bad Request JSON response body from Vaultwarden’s 2FA challenge is replaced with HTML. The issue occurs at the Traefik error response handling level, independent of whether Pangolin auth is involved or not.
@AstralDestiny commented on GitHub (May 19, 2026):
If it's traefik then it would have to be posted within the traefik issue github honestly.
Though I will spin up a test vaultwarden in a moment give me a bit.
@AstralDestiny commented on GitHub (May 19, 2026):
Which method? TOTP or security key?
@StefanSa commented on GitHub (May 19, 2026):
security key
@AstralDestiny commented on GitHub (May 19, 2026):
There was a problem reading the security key. Try again.
Fun.. give me a few. (Forgot to set basedomain within vw)
@AstralDestiny commented on GitHub (May 19, 2026):
I'm able to login just fine with a pass key and a physical security key.
Are you differing at all from base deployment?
@StefanSa commented on GitHub (May 19, 2026):
That's interesting. What are you doing differently from me and from the other cases i mentioned?
@AstralDestiny commented on GitHub (May 19, 2026):
Mine differs a bit but it was posted to https://discord.com/channels/1325658630518865980/1438910182372540536/1438910182372540536
Though I hope to get some time soon to publish the page on pangolin.net docs, But I doubt any of that should cause any breakage.
@StefanSa commented on GitHub (May 19, 2026):
I actually tested WebAuthn with a physical security key as well. I was able to register the key via the Web UI without issues, but login with WebAuthn also failed due to the same 400 response body being replaced. Login was only possible again after disabling 2FA entirely.
So this affects all 2FA methods (TOTP and WebAuthn) — which makes sense, since both rely on the same POST /identity/connect/token → 400 Bad Request with JSON body flow for the initial challenge-response step.
This is a base Pangolin deployment with no custom Traefik middleware or error page configuration. Could you share your Pangolin version and check in DevTools (Network tab) whether your 400 response body on POST /identity/connect/token returns JSON or HTML?
@AstralDestiny commented on GitHub (May 19, 2026):
I'm on EE-1.18.4, No error pages here as I think it's a waste of resources and has traefik respond to stuff it should otherwise not bother with.
My headers are changed drastically to align with security that most modern sites don't even bother with.
Request URL https://vw.domain4.com/identity/connect/token
Request Method POST
Status Code 400 Bad Request
Remote Address XX.XX.XX.XX:443
Referrer Policy same-origin
JSON format
Request URL https://vw.domain4.com/identity/connect/token
Request Method POST
Status Code 200 OK
Remote Address XX.XX.XX.XX:443
Referrer Policy same-origin
JSON format
Tried posting portion of the json it was just broken when previewed within github. But looks proper within webtools.
@AstralDestiny commented on GitHub (May 19, 2026):
If you want if you can poke me on the discord or slack, see if it's a web browser issue or somewhere like I'll have you try against an instance here.
What's your traefik version by chance?
@StefanSa commented on GitHub (May 19, 2026):
Thanks, that’s very helpful. So the key difference is clear: no error pages on your side → JSON passes through correctly. On my side → JSON gets replaced with HTML.
I did not explicitly configure any custom error pages either, so this might be a default middleware in the community edition base deployment. I’ll dig into my Traefik dynamic config and check for any errors middleware that might be intercepting 4xx responses by default.
By the way — you mentioned you’ve drastically changed your headers and optimized your Traefik config for security. If you’re open to sharing the relevant parts, that would be a great reference point for comparison. No pressure of course, but it would definitely help narrow this down.
@AstralDestiny commented on GitHub (May 19, 2026):
Oh all of it is in that discord url I posted it's on the official discord. Well minus some of the dnssec and such stuff. and the read only and the apparmor/selinux profiles.
https://discord.gg/MZtgvEfNCc If it helps, I plan to add the stuff to the docs just been overly busy to sit down finish it.
@StefanSa commented on GitHub (May 20, 2026):
Update — Correction after further testing:
After more thorough testing I need to narrow down the scope of this issue significantly.
Web UI works correctly: Login with 2FA (WebAuthn via Token2 security key) through the Vaultwarden Web Vault works without issues. The
400 Bad RequestJSON response containing the 2FA challenge is passed through correctly by Pangolin/Traefik/badger, and the 2FA flow completes as expected.The issue is limited to the native Bitwarden app. The app fails to complete the 2FA login flow against Vaultwarden behind Pangolin. The Vaultwarden server log still shows:
This suggests the native app either handles the 2FA challenge-response flow differently than the Web UI, or sends different headers that cause a different interaction with the middleware chain.
The Traefik access log confirms that the backend response has
downstream_Content-Type: application/jsonwith aDownstreamContentSizeof 351 bytes, which appears to be the correct JSON response from Vaultwarden. The question is whether the native app receives and parses this response correctly, or whether something in the middleware chain (badger) treats the native app's requests differently based on headers or client identification.I apologize for the initial broad scope of this report. The core issue remains — 2FA login fails for native Bitwarden clients behind Pangolin — but it is not a general error page interception problem as originally described.
@AstralDestiny commented on GitHub (May 20, 2026):
Mm mind testing against one of my instances for that by chance?
@StefanSa commented on GitHub (May 20, 2026):
Sure, happy to test. Quick question first — are you also running the Bitwarden Windows Desktop App successfully with 2FA enabled against your Vaultwarden instance? That would help me understand if it's specific to my setup or a general Desktop App issue.
@AstralDestiny commented on GitHub (May 20, 2026):
I don't have vaultwarden deployed past the web side part of things. Seems I'm getting rate limited by github again.
@AstralDestiny commented on GitHub (May 20, 2026):
Mostly asking for you to test as the desktop app just seems to not be happy at all with me.
@LaurenceJJones commented on GitHub (May 20, 2026):
Are you using the cloud or self hosted?
If it's the cloud, we setup error middleware but we tried to minimize the scope as much as possible so it didn't intercept application responses.
If you are self hosted then you need to check if you setup the error pages middleware and if so you need to tune them as Pangolin has no control over that.
@StefanSa commented on GitHub (May 20, 2026):
@LaurenceJJones @AstralDestiny
Self-hosted. And to clarify — after further testing, the issue turned out to be not about error pages intercepting
400responses. The Web UI with 2FA (WebAuthn) works correctly through Pangolin. The problem is specifically with the Bitwarden Windows Desktop App failing to complete the 2FA flow.The Vaultwarden admin diagnostic page reveals the actual cause — response header conflicts:
These headers are injected by a Traefik security-headers middleware on the entrypoint level. The Web UI tolerates the mismatch, but the Desktop App (Electron) does not.
The challenge with Pangolin is that there's currently no clean way to solve this per-resource:
A per-resource "Custom Response Headers" option in Pangolin (similar to the existing Request Headers field) would solve this cleanly. Would that be feasible?
@AstralDestiny commented on GitHub (May 20, 2026):
Did you attempt with my header changes found on the discord?
Also no the "Custom response" only applies between traefik and vaultwarden and not vaultwarden and user which those headers should be existing on though don't recall pangolin pushing headers.. be curious to know what the github is pushing right now as I've attempted to modify it a few times in the past but keeps being overwritten later.
@StefanSa commented on GitHub (May 20, 2026):
Here's what Pangolin currently generates via the HTTP provider (http://pangolin:3001/api/v1/traefik-config) for the Vaultwarden route:
So Pangolin itself only injects badger — no header middleware, no error pages. The problematic headers in my case were coming from my own security-headers middleware applied at the Traefik entrypoint level. After aligning those values with your config from Discord
(referrerPolicy: "same-origin",browserXssFilter: false,customBrowserXSSValue: "0"), the API diagnostic errors are resolved.Still testing whether this also fixes the Desktop App 2FA issue — will report back.
@AstralDestiny commented on GitHub (May 20, 2026):
So happy? or?
@StefanSa commented on GitHub (May 20, 2026):
Resolved — Root cause identified and fix documented.
After extensive debugging, this turned out to be a combination of issues rather than a single error page interception problem. Here's the full breakdown for anyone running Vaultwarden behind Pangolin:
The Problem
The Bitwarden Windows Desktop App (Electron) could not complete 2FA (WebAuthn) login against Vaultwarden behind Pangolin, while the Web UI worked fine.
Root Causes
1.
X-Frame-Options: SAMEORIGINblocking the WebAuthn connectorThe Desktop App loads
webauthn-connector.htmlfrom Vaultwarden in an iframe. The parent page is a localfile://URL inside the Electron app. A TraefikframeDeny: truemiddleware on the entrypoint addedX-Frame-Options: SAMEORIGIN, which blocked the iframe becausefile://≠https://. This causedERR_BLOCKED_BY_RESPONSE.2. Vaultwarden CSP
frame-ancestorsmissingfile://*Even after removing
X-Frame-Options, the Electron iframe was blocked because Vaultwarden's own CSPframe-ancestorsdid not includefile://*.3. Security headers overriding Vaultwarden's expected headers
Global Traefik security headers (
referrer-policy,x-xss-protection,content-security-policy) conflicted with what Vaultwarden expects, causing diagnostic failures.4. Missing Pangolin rule for
/connectors/*The WebAuthn connector path needed an explicit allow rule in Pangolin.
Solution
Vaultwarden environment variable:
Pangolin:
/connectors/*Traefik security-headers middleware (entrypoint level):
What was NOT the cause
Recommendation
A per-resource Custom Response Headers option in Pangolin would make this much easier to solve. Currently, disabling
frameDenyor adjusting headers requires changing the global Traefik entrypoint config, which affects all resources. Being able to override response headers per resource (similar to the existing Custom Request Headers field) would allow targeted fixes without weakening security for other services.Thanks to @AstralDestiny for helping narrow this down — the comparison with his hardened Traefik config was key to identifying the header conflicts.
@AstralDestiny commented on GitHub (May 20, 2026):
I don't go for legacy support, Also the CSP is fine, If an app provides it traefik will opt for vaultwarden's csp even if it's different from traefik.
Also would just use customFrameOptionsValue: SAMEORIGIN at that point.
Also if it's using file: for a reference that shouldn't break it.. I'll deep dive this latter, If it's really using that then it's violating some spec likely.
Also part of this reads like aI.
Also adding a rule doesn't do anything unless you have pangolin fronting the auth.
@StefanSa commented on GitHub (May 20, 2026):
Closing this — figured out what was going on.
The actual issue had nothing to do with error page interception. The 400 JSON responses from Vaultwarden were being passed through correctly by Pangolin/badger the whole time.
What actually broke the Desktop App (Electron) was a combination of things:
My Traefik security-headers middleware had
frameDeny: trueon the entrypoint, which addedX-Frame-Options: SAMEORIGINglobally. The Desktop App loadswebauthn-connector.htmlin an iframe, but the parent is a localfile://path inside the Electron app — so SAMEORIGIN blocks it becausefile://≠https://. That gave meERR_BLOCKED_BY_RESPONSEin the Electron console.After fixing that, the iframe loaded but the WebAuthn ceremony still didn't work. Turned out I also needed
ALLOWED_IFRAME_ANCESTORS=file://*in my Vaultwarden config so the CSPframe-ancestorsincludesfile://for the Electron context. Plus a Pangolin rule for/connectors/*.With all three changes the Desktop App works with WebAuthn 2FA now. Web UI was never affected since it doesn't rely on cross-origin iframes.
For anyone else running Vaultwarden behind Pangolin with the Desktop App and WebAuthn — don't set
X-Frame-Optionsvia your reverse proxy, let Vaultwarden handle framing via its own CSP, and setALLOWED_IFRAME_ANCESTORS=file://*.Thanks @AstralDestiny for the help debugging and the Traefik config comparison — that pointed me in the right direction with the header conflicts.
@AstralDestiny commented on GitHub (May 20, 2026):
Yeah but file:// is a violation for Fido2.. if anything it's a hack.. which bitwarden themselves shouldn't be using..
@AstralDestiny commented on GitHub (May 20, 2026):
Also if you can, tell bitwarden about it. As that behavior is not fun if it does any loading via file:// as someone could tamper with that over it requesting from the server. Could lead to non noisy issues in security of the desktop application, putting that in the iframe also allows some abuse to a point too.
@StefanSa commented on GitHub (May 20, 2026):
Agreed — loading a security-critical WebAuthn ceremony via file:// and then requiring frame-ancestors file://* to make it work is questionable architecture. There are still open issues around Desktop App + WebAuthn, e.g. bitwarden/clients#18013. Might be worth raising the file:// framing concern there since Bitwarden's security team would be the right audience for that.
@StefanSa commented on GitHub (May 20, 2026):
One more follow-up — I've opened an issue on the Bitwarden side about the underlying architectural problem that caused all of this: the Desktop App uses a file:// iframe to load webauthn-connector.html, which forces server admins to weaken their security headers to make WebAuthn 2FA work.
bitwarden/clients#20745
Hopefully they'll move to native Electron WebAuthn calls at some point, which would make the ALLOWED_IFRAME_ANCESTORS workaround and the X-Frame-Options removal unnecessary.
Thanks again @AstralDestiny for the help and the security observation that led to this.
@AstralDestiny commented on GitHub (May 20, 2026):
No worries.
@github-actions[bot] commented on GitHub (Jun 4, 2026):
This issue has been automatically marked as stale due to 14 days of inactivity. It will be closed in 14 days if no further activity occurs.
@github-actions[bot] commented on GitHub (Jun 19, 2026):
This issue has been automatically closed due to inactivity. If you believe this is still relevant, please open a new issue with up-to-date information.