[GH-ISSUE #3262] Connection pool exhaustion in 1.19.0/1.19.1 (ee-postgresql) #30181

New Issue

GiteaMirror · 2026-06-13T10:43:11-05:00

GiteaMirror commented

2026-06-13 10:43:11 -05:00

Originally created by @cendenta on GitHub (Jun 12, 2026).
Original GitHub issue: https://github.com/fosrl/pangolin/issues/3262

Originally assigned to: @oschwartz10612 on GitHub.

Summary

After updating the ee-postgresql image from 1.18.4 → 1.19.1, Pangolin became unable to acquire database connections within minutes of startup. Every query — both getResourceByDomain (per-request session verification) and getTraefikConfig (the Traefik config provider) — fails with timeout exceeded when trying to connect. This takes down all proxied resources at once.

The PostgreSQL server is healthy and idle (33/100 connections), yet Pangolin's pool holds ~20 connections that PostgreSQL reports as idle while Pangolin simultaneously reports it cannot obtain a connection. This is the signature of pooled clients being acquired but never released (or the event loop being starved), saturating the pool at its max.

A container restart restores service for a few minutes, then it re-degrades as the pool re-saturates. Downgrading to ee-postgresql-1.18.4 resolves it completely. 1.19.2 does not touch the relevant code and is also affected.

Logs

Both the per-request path and the config provider fail the same way:

[cause]: Error: timeout exceeded when trying to connect
    at async getResourceByDomain (server/db/queries/verifySessionQueries.ts:102:30)
    at async verifyResourceSession (server/routers/badger/verifySession.ts:156:28)

[cause]: Error: timeout exceeded when trying to connect
    at async getTraefikConfig (drizzle-orm/node-postgres/session.js:117)
    at async traefikConfigProvider

Evidence it is pool saturation, not the database

PostgreSQL is healthy and far from any limit, but Pangolin's connections are all parked idle while it still can't get one:

-- total connections vs limit (Postgres is fine)
 total | max
-------+-----
    33 | 100

-- pangolin DB: ~20 connections, effectively all idle, while every new query times out
 state  | count | oldest
--------+-------+----------
 active |     3 |
 idle   |    17 | 00:00:16

A request that does get a connection completes in ~67 ms, so the database and query are fast — the failure is purely connection acquisition. PostgreSQL's own log shows no too many clients/slot exhaustion.

Suspected cause

The regression window points squarely at the 1.19.0 "improve performance in hot paths / API endpoints with thousands of sites and users" rework:

Introduced in 1.19.0; 1.19.1 (hotfix) and 1.19.2 do not modify the DB session/query layer.
1.18.4 is unaffected.

This suggests a code path in the reworked getResourceByDomain / Traefik config provider that checks out a pool client and does not release it on all branches (e.g., on early return or error), so the default pool (max ≈ 20) saturates. Pinning to 1.18.4 is a clean workaround.

Ask

Could you review connection acquire/release in the 1.19.0 hot-path changes — specifically whether pooled clients are released on every path (including error/early-return) in verifySessionQueries/the Traefik config provider — and whether a default pool max of ~20 is expected to be held indefinitely under normal traffic?

Happy to provide additional pg_stat_activity snapshots, full logs, or test a patched build.

Environment

OS Type & Version: Ubuntu 24.04.4 LTS (kernel 6.8.0-124-generic)
Pangolin Version: 1.19.1 (ee-postgresql) — bug present; also reproduces on 1.19.0; NOT present on 1.18.4
Edition: Enterprise (ee-postgresql image)
Gerbil Version: 1.4.1 (:latest)
Traefik Version: v3.6.10
Newt Version: 1.13.0 (:latest)
Client Version: N/A

To Reproduce

Run ee-postgresql-1.19.1 against a PostgreSQL backend under normal traffic.
Within a few minutes, all resources become unreachable and the logs fill with the error below.
Restart the Pangolin container → service recovers briefly, then degrades again as the pool re-saturates.

Expected Behavior

Expected: each query checks out a pooled connection and returns it; steady-state pool stays well below max.
Actual: the pool grows to ~20 and stays there; every new acquisition times out.

Originally created by @cendenta on GitHub (Jun 12, 2026). Original GitHub issue: https://github.com/fosrl/pangolin/issues/3262 Originally assigned to: @oschwartz10612 on GitHub. **Summary** After updating the ee-postgresql image from 1.18.4 → 1.19.1, Pangolin became unable to acquire database connections within minutes of startup. Every query — both getResourceByDomain (per-request session verification) and getTraefikConfig (the Traefik config provider) — fails with timeout exceeded when trying to connect. This takes down all proxied resources at once. The PostgreSQL server is healthy and idle (33/100 connections), yet Pangolin's pool holds ~20 connections that PostgreSQL reports as idle while Pangolin simultaneously reports it cannot obtain a connection. This is the signature of pooled clients being acquired but never released (or the event loop being starved), saturating the pool at its max. A container restart restores service for a few minutes, then it re-degrades as the pool re-saturates. Downgrading to ee-postgresql-1.18.4 resolves it completely. 1.19.2 does not touch the relevant code and is also affected. **Logs** Both the per-request path and the config provider fail the same way: ``` [cause]: Error: timeout exceeded when trying to connect at async getResourceByDomain (server/db/queries/verifySessionQueries.ts:102:30) at async verifyResourceSession (server/routers/badger/verifySession.ts:156:28) [cause]: Error: timeout exceeded when trying to connect at async getTraefikConfig (drizzle-orm/node-postgres/session.js:117) at async traefikConfigProvider ``` **Evidence it is pool saturation, not the database** PostgreSQL is healthy and far from any limit, but Pangolin's connections are all parked idle while it still can't get one: ``` -- total connections vs limit (Postgres is fine) total | max -------+----- 33 | 100 -- pangolin DB: ~20 connections, effectively all idle, while every new query times out state | count | oldest --------+-------+---------- active | 3 | idle | 17 | 00:00:16 ``` A request that does get a connection completes in ~67 ms, so the database and query are fast — the failure is purely connection acquisition. PostgreSQL's own log shows no too many clients/slot exhaustion. **Suspected cause** The regression window points squarely at the 1.19.0 "improve performance in hot paths / API endpoints with thousands of sites and users" rework: - Introduced in 1.19.0; 1.19.1 (hotfix) and 1.19.2 do not modify the DB session/query layer. - 1.18.4 is unaffected. This suggests a code path in the reworked getResourceByDomain / Traefik config provider that checks out a pool client and does not release it on all branches (e.g., on early return or error), so the default pool (max ≈ 20) saturates. Pinning to 1.18.4 is a clean workaround. **Ask** Could you review connection acquire/release in the 1.19.0 hot-path changes — specifically whether pooled clients are released on every path (including error/early-return) in verifySessionQueries/the Traefik config provider — and whether a default pool max of ~20 is expected to be held indefinitely under normal traffic? Happy to provide additional pg_stat_activity snapshots, full logs, or test a patched build. ### Environment - OS Type & Version: Ubuntu 24.04.4 LTS (kernel 6.8.0-124-generic) - Pangolin Version: 1.19.1 (ee-postgresql) — bug present; also reproduces on 1.19.0; NOT present on 1.18.4 - Edition: Enterprise (ee-postgresql image) - Gerbil Version: 1.4.1 (:latest) - Traefik Version: v3.6.10 - Newt Version: 1.13.0 (:latest) - Client Version: N/A ### To Reproduce 1. Run ee-postgresql-1.19.1 against a PostgreSQL backend under normal traffic. 2. Within a few minutes, all resources become unreachable and the logs fill with the error below. 3. Restart the Pangolin container → service recovers briefly, then degrades again as the pool re-saturates. ### Expected Behavior - Expected: each query checks out a pooled connection and returns it; steady-state pool stays well below max. - Actual: the pool grows to ~20 and stays there; every new acquisition times out.

GiteaMirror added the potential bug label 2026-06-13 10:43:11 -05:00

GiteaMirror commented

2026-06-13 10:43:12 -05:00

@oschwartz10612 commented on GitHub (Jun 12, 2026):

Hum on our cloud platform we are not experiencing this but something could be amiss. Could you try tweaking these values in your config and let me know if you are able to settle in a sweat spot? We are looking for query problems around the application but have not found a glaring issue yet. There were some updates done to the dependencies so perhaps there is something there as well.

postgres:
    pool:
        max_connections: 80
        max_replica_connections: 40
        idle_timeout_ms: 30000
        connection_timeout_ms: 10000

@oschwartz10612 commented on GitHub (Jun 12, 2026): Hum on our cloud platform we are not experiencing this but something could be amiss. Could you try tweaking these values in your config and let me know if you are able to settle in a sweat spot? We are looking for query problems around the application but have not found a glaring issue yet. There were some updates done to the dependencies so perhaps there is something there as well. ``` postgres: pool: max_connections: 80 max_replica_connections: 40 idle_timeout_ms: 30000 connection_timeout_ms: 10000 ```

GiteaMirror commented

2026-06-13 10:43:12 -05:00

@mprokopiev commented on GitHub (Jun 13, 2026):

Got the same timeouts right after upgrade to 1.19.1 (or at least I noticed that after upgrade to 1.19.1) on community edition. When idle - works fine. When accessing resources - they are very slow or failed. Log is full of postgres timeouts. Added pool config - slowness and exceptions are gone and solution works much smoother.

@mprokopiev commented on GitHub (Jun 13, 2026): Got the same timeouts right after upgrade to 1.19.1 (or at least I noticed that after upgrade to 1.19.1) on community edition. When idle - works fine. When accessing resources - they are very slow or failed. Log is full of postgres timeouts. Added pool config - slowness and exceptions are gone and solution works much smoother.

Sign in to join this conversation.

Branches Tags

main

crowdin_dev

dev

dependabot/npm_and_yarn/npm-dependencies-338850a190

dependabot/go_modules/install/go-install-dependencies-3804ca7238

dependabot/docker/docker-dependencies-4faa477378

dependabot/github_actions/github-actions-dependencies-6d79802a48

api-improvements

feat/remember-last-idp-on-smart-login-form

refactor/batch-status-requests

dependabot/npm_and_yarn/multi-5f1280885e

fix/non-semver-version-error

private-resource-page

resource-launcher

backhaul

exit-node-reconnect

feat/command-palette

ssh

delete-account

msg-delivery

org-only-idp

cicd

patch

site-targets-auto-login

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/pangolin#30181