Originally assigned to: @oschwartz10612 on GitHub.
Summary
After updating the ee-postgresql image from 1.18.4 → 1.19.1, Pangolin became unable to acquire database connections within minutes of startup. Every query — both getResourceByDomain (per-request session verification) and getTraefikConfig (the Traefik config provider) — fails with timeout exceeded when trying to connect. This takes down all proxied resources at once.
The PostgreSQL server is healthy and idle (33/100 connections), yet Pangolin's pool holds ~20 connections that PostgreSQL reports as idle while Pangolin simultaneously reports it cannot obtain a connection. This is the signature of pooled clients being acquired but never released (or the event loop being starved), saturating the pool at its max.
A container restart restores service for a few minutes, then it re-degrades as the pool re-saturates. Downgrading to ee-postgresql-1.18.4 resolves it completely. 1.19.2 does not touch the relevant code and is also affected.
Logs
Both the per-request path and the config provider fail the same way:
[cause]: Error: timeout exceeded when trying to connect
at async getResourceByDomain (server/db/queries/verifySessionQueries.ts:102:30)
at async verifyResourceSession (server/routers/badger/verifySession.ts:156:28)
[cause]: Error: timeout exceeded when trying to connect
at async getTraefikConfig (drizzle-orm/node-postgres/session.js:117)
at async traefikConfigProvider
Evidence it is pool saturation, not the database
PostgreSQL is healthy and far from any limit, but Pangolin's connections are all parked idle while it still can't get one:
-- total connections vs limit (Postgres is fine)
total | max
-------+-----
33 | 100
-- pangolin DB: ~20 connections, effectively all idle, while every new query times out
state | count | oldest
--------+-------+----------
active | 3 |
idle | 17 | 00:00:16
A request that does get a connection completes in ~67 ms, so the database and query are fast — the failure is purely connection acquisition. PostgreSQL's own log shows no too many clients/slot exhaustion.
Suspected cause
The regression window points squarely at the 1.19.0 "improve performance in hot paths / API endpoints with thousands of sites and users" rework:
Introduced in 1.19.0; 1.19.1 (hotfix) and 1.19.2 do not modify the DB session/query layer.
1.18.4 is unaffected.
This suggests a code path in the reworked getResourceByDomain / Traefik config provider that checks out a pool client and does not release it on all branches (e.g., on early return or error), so the default pool (max ≈ 20) saturates. Pinning to 1.18.4 is a clean workaround.
Ask
Could you review connection acquire/release in the 1.19.0 hot-path changes — specifically whether pooled clients are released on every path (including error/early-return) in verifySessionQueries/the Traefik config provider — and whether a default pool max of ~20 is expected to be held indefinitely under normal traffic?
Happy to provide additional pg_stat_activity snapshots, full logs, or test a patched build.
Environment
OS Type & Version: Ubuntu 24.04.4 LTS (kernel 6.8.0-124-generic)
Pangolin Version: 1.19.1 (ee-postgresql) — bug present; also reproduces on 1.19.0; NOT present on 1.18.4
Edition: Enterprise (ee-postgresql image)
Gerbil Version: 1.4.1 (:latest)
Traefik Version: v3.6.10
Newt Version: 1.13.0 (:latest)
Client Version: N/A
To Reproduce
Run ee-postgresql-1.19.1 against a PostgreSQL backend under normal traffic.
Within a few minutes, all resources become unreachable and the logs fill with the error below.
Restart the Pangolin container → service recovers briefly, then degrades again as the pool re-saturates.
Expected Behavior
Expected: each query checks out a pooled connection and returns it; steady-state pool stays well below max.
Actual: the pool grows to ~20 and stays there; every new acquisition times out.
Originally created by @cendenta on GitHub (Jun 12, 2026).
Original GitHub issue: https://github.com/fosrl/pangolin/issues/3262
Originally assigned to: @oschwartz10612 on GitHub.
**Summary**
After updating the ee-postgresql image from 1.18.4 → 1.19.1, Pangolin became unable to acquire database connections within minutes of startup. Every query — both getResourceByDomain (per-request session verification) and getTraefikConfig (the Traefik config provider) — fails with timeout exceeded when trying to connect. This takes down all proxied resources at once.
The PostgreSQL server is healthy and idle (33/100 connections), yet Pangolin's pool holds ~20 connections that PostgreSQL reports as idle while Pangolin simultaneously reports it cannot obtain a connection. This is the signature of pooled clients being acquired but never released (or the event loop being starved), saturating the pool at its max.
A container restart restores service for a few minutes, then it re-degrades as the pool re-saturates. Downgrading to ee-postgresql-1.18.4 resolves it completely. 1.19.2 does not touch the relevant code and is also affected.
**Logs**
Both the per-request path and the config provider fail the same way:
```
[cause]: Error: timeout exceeded when trying to connect
at async getResourceByDomain (server/db/queries/verifySessionQueries.ts:102:30)
at async verifyResourceSession (server/routers/badger/verifySession.ts:156:28)
[cause]: Error: timeout exceeded when trying to connect
at async getTraefikConfig (drizzle-orm/node-postgres/session.js:117)
at async traefikConfigProvider
```
**Evidence it is pool saturation, not the database**
PostgreSQL is healthy and far from any limit, but Pangolin's connections are all parked idle while it still can't get one:
```
-- total connections vs limit (Postgres is fine)
total | max
-------+-----
33 | 100
-- pangolin DB: ~20 connections, effectively all idle, while every new query times out
state | count | oldest
--------+-------+----------
active | 3 |
idle | 17 | 00:00:16
```
A request that does get a connection completes in ~67 ms, so the database and query are fast — the failure is purely connection acquisition. PostgreSQL's own log shows no too many clients/slot exhaustion.
**Suspected cause**
The regression window points squarely at the 1.19.0 "improve performance in hot paths / API endpoints with thousands of sites and users" rework:
- Introduced in 1.19.0; 1.19.1 (hotfix) and 1.19.2 do not modify the DB session/query layer.
- 1.18.4 is unaffected.
This suggests a code path in the reworked getResourceByDomain / Traefik config provider that checks out a pool client and does not release it on all branches (e.g., on early return or error), so the default pool (max ≈ 20) saturates. Pinning to 1.18.4 is a clean workaround.
**Ask**
Could you review connection acquire/release in the 1.19.0 hot-path changes — specifically whether pooled clients are released on every path (including error/early-return) in verifySessionQueries/the Traefik config provider — and whether a default pool max of ~20 is expected to be held indefinitely under normal traffic?
Happy to provide additional pg_stat_activity snapshots, full logs, or test a patched build.
### Environment
- OS Type & Version: Ubuntu 24.04.4 LTS (kernel 6.8.0-124-generic)
- Pangolin Version: 1.19.1 (ee-postgresql) — bug present; also reproduces on 1.19.0; NOT present on 1.18.4
- Edition: Enterprise (ee-postgresql image)
- Gerbil Version: 1.4.1 (:latest)
- Traefik Version: v3.6.10
- Newt Version: 1.13.0 (:latest)
- Client Version: N/A
### To Reproduce
1. Run ee-postgresql-1.19.1 against a PostgreSQL backend under normal traffic.
2. Within a few minutes, all resources become unreachable and the logs fill with the error below.
3. Restart the Pangolin container → service recovers briefly, then degrades again as the pool re-saturates.
### Expected Behavior
- Expected: each query checks out a pooled connection and returns it; steady-state pool stays well below max.
- Actual: the pool grows to ~20 and stays there; every new acquisition times out.
@oschwartz10612 commented on GitHub (Jun 12, 2026):
Hum on our cloud platform we are not experiencing this but something could be amiss. Could you try tweaking these values in your config and let me know if you are able to settle in a sweat spot? We are looking for query problems around the application but have not found a glaring issue yet. There were some updates done to the dependencies so perhaps there is something there as well.
<!-- gh-comment-id:4696217713 -->
@oschwartz10612 commented on GitHub (Jun 12, 2026):
Hum on our cloud platform we are not experiencing this but something could be amiss. Could you try tweaking these values in your config and let me know if you are able to settle in a sweat spot? We are looking for query problems around the application but have not found a glaring issue yet. There were some updates done to the dependencies so perhaps there is something there as well.
```
postgres:
pool:
max_connections: 80
max_replica_connections: 40
idle_timeout_ms: 30000
connection_timeout_ms: 10000
```
Got the same timeouts right after upgrade to 1.19.1 (or at least I noticed that after upgrade to 1.19.1) on community edition. When idle - works fine. When accessing resources - they are very slow or failed. Log is full of postgres timeouts. Added pool config - slowness and exceptions are gone and solution works much smoother.
<!-- gh-comment-id:4698214787 -->
@mprokopiev commented on GitHub (Jun 13, 2026):
Got the same timeouts right after upgrade to 1.19.1 (or at least I noticed that after upgrade to 1.19.1) on community edition. When idle - works fine. When accessing resources - they are very slow or failed. Log is full of postgres timeouts. Added pool config - slowness and exceptions are gone and solution works much smoother.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @cendenta on GitHub (Jun 12, 2026).
Original GitHub issue: https://github.com/fosrl/pangolin/issues/3262
Originally assigned to: @oschwartz10612 on GitHub.
Summary
After updating the ee-postgresql image from 1.18.4 → 1.19.1, Pangolin became unable to acquire database connections within minutes of startup. Every query — both getResourceByDomain (per-request session verification) and getTraefikConfig (the Traefik config provider) — fails with timeout exceeded when trying to connect. This takes down all proxied resources at once.
The PostgreSQL server is healthy and idle (33/100 connections), yet Pangolin's pool holds ~20 connections that PostgreSQL reports as idle while Pangolin simultaneously reports it cannot obtain a connection. This is the signature of pooled clients being acquired but never released (or the event loop being starved), saturating the pool at its max.
A container restart restores service for a few minutes, then it re-degrades as the pool re-saturates. Downgrading to ee-postgresql-1.18.4 resolves it completely. 1.19.2 does not touch the relevant code and is also affected.
Logs
Both the per-request path and the config provider fail the same way:
Evidence it is pool saturation, not the database
PostgreSQL is healthy and far from any limit, but Pangolin's connections are all parked idle while it still can't get one:
A request that does get a connection completes in ~67 ms, so the database and query are fast — the failure is purely connection acquisition. PostgreSQL's own log shows no too many clients/slot exhaustion.
Suspected cause
The regression window points squarely at the 1.19.0 "improve performance in hot paths / API endpoints with thousands of sites and users" rework:
This suggests a code path in the reworked getResourceByDomain / Traefik config provider that checks out a pool client and does not release it on all branches (e.g., on early return or error), so the default pool (max ≈ 20) saturates. Pinning to 1.18.4 is a clean workaround.
Ask
Could you review connection acquire/release in the 1.19.0 hot-path changes — specifically whether pooled clients are released on every path (including error/early-return) in verifySessionQueries/the Traefik config provider — and whether a default pool max of ~20 is expected to be held indefinitely under normal traffic?
Happy to provide additional pg_stat_activity snapshots, full logs, or test a patched build.
Environment
To Reproduce
Expected Behavior
@oschwartz10612 commented on GitHub (Jun 12, 2026):
Hum on our cloud platform we are not experiencing this but something could be amiss. Could you try tweaking these values in your config and let me know if you are able to settle in a sweat spot? We are looking for query problems around the application but have not found a glaring issue yet. There were some updates done to the dependencies so perhaps there is something there as well.
@mprokopiev commented on GitHub (Jun 13, 2026):
Got the same timeouts right after upgrade to 1.19.1 (or at least I noticed that after upgrade to 1.19.1) on community edition. When idle - works fine. When accessing resources - they are very slow or failed. Log is full of postgres timeouts. Added pool config - slowness and exceptions are gone and solution works much smoother.