mirror of
https://github.com/fosrl/pangolin.git
synced 2026-07-16 06:08:33 -05:00
[GH-ISSUE #3262] Connection pool exhaustion in 1.19.0/1.19.1 (ee-postgresql) #35771
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: github-starred/pangolin#35771
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @cendenta on GitHub (Jun 12, 2026).
Original GitHub issue: https://github.com/fosrl/pangolin/issues/3262
Originally assigned to: @oschwartz10612 on GitHub.
Summary
After updating the ee-postgresql image from 1.18.4 → 1.19.1, Pangolin became unable to acquire database connections within minutes of startup. Every query — both getResourceByDomain (per-request session verification) and getTraefikConfig (the Traefik config provider) — fails with timeout exceeded when trying to connect. This takes down all proxied resources at once.
The PostgreSQL server is healthy and idle (33/100 connections), yet Pangolin's pool holds ~20 connections that PostgreSQL reports as idle while Pangolin simultaneously reports it cannot obtain a connection. This is the signature of pooled clients being acquired but never released (or the event loop being starved), saturating the pool at its max.
A container restart restores service for a few minutes, then it re-degrades as the pool re-saturates. Downgrading to ee-postgresql-1.18.4 resolves it completely. 1.19.2 does not touch the relevant code and is also affected.
Logs
Both the per-request path and the config provider fail the same way:
Evidence it is pool saturation, not the database
PostgreSQL is healthy and far from any limit, but Pangolin's connections are all parked idle while it still can't get one:
A request that does get a connection completes in ~67 ms, so the database and query are fast — the failure is purely connection acquisition. PostgreSQL's own log shows no too many clients/slot exhaustion.
Suspected cause
The regression window points squarely at the 1.19.0 "improve performance in hot paths / API endpoints with thousands of sites and users" rework:
This suggests a code path in the reworked getResourceByDomain / Traefik config provider that checks out a pool client and does not release it on all branches (e.g., on early return or error), so the default pool (max ≈ 20) saturates. Pinning to 1.18.4 is a clean workaround.
Ask
Could you review connection acquire/release in the 1.19.0 hot-path changes — specifically whether pooled clients are released on every path (including error/early-return) in verifySessionQueries/the Traefik config provider — and whether a default pool max of ~20 is expected to be held indefinitely under normal traffic?
Happy to provide additional pg_stat_activity snapshots, full logs, or test a patched build.
Environment
To Reproduce
Expected Behavior
@oschwartz10612 commented on GitHub (Jun 12, 2026):
Hum on our cloud platform we are not experiencing this but something could be amiss. Could you try tweaking these values in your config and let me know if you are able to settle in a sweat spot? We are looking for query problems around the application but have not found a glaring issue yet. There were some updates done to the dependencies so perhaps there is something there as well.
@mprokopiev commented on GitHub (Jun 13, 2026):
Got the same timeouts right after upgrade to 1.19.1 (or at least I noticed that after upgrade to 1.19.1) on community edition. When idle - works fine. When accessing resources - they are very slow or failed. Log is full of postgres timeouts. Added pool config - slowness and exceptions are gone and solution works much smoother.
@Blacks-Army commented on GitHub (Jun 13, 2026):
I have the same experience
@dx911xd commented on GitHub (Jun 13, 2026):
I have the same exp after upgrade from v1.18.4; high CPU usage observed with the app and postgresql containers. similar behavior also observed in 1.19.2 but resolved with manual db vaccum.
@oschwartz10612 commented on GitHub (Jun 13, 2026):
@Blacks-Army and @dx911xd could you try the same pool config above?
If that helps ease the issue then I will incorporate that as the
default. Perhaps the dependency changed some defaults or something that
is causing this.
@mprokopiev commented on GitHub (Jun 13, 2026):
I am not sure if mine was actually the same pool exchaust as for the reporter but after some time with suggested pool config - the problem came back. Docker stats shows frequent spikes in CPU usage (>100%) most of the time. I downgraded back to 1.18.4. I'll monitor if the problem will came back.
@Blacks-Army commented on GitHub (Jun 13, 2026):
@oschwartz10612 the config did not help. But VACCUM did.
@mprokopiev commented on GitHub (Jun 14, 2026):
System works normally on 1.18.4. CPU usage is way lesser after downgrade and overall performance is better.
Maybe some complex/heavy queries were added in 1.19.x? I see the most timed out quer in the log is this one (formatted):
I am on Racknerd VPS - 2CPU, 2GB RAM.
@mprokopiev commented on GitHub (Jun 14, 2026):
A query plan for the reference:
Rarely plan execition time was ~5s. If this is running very often - I guess it could overload DB quickly.
@ex-aequo-et-bono commented on GitHub (Jun 14, 2026):
Confirming VACUUM worked to fix this problem for me. CE Postgres version.
@AstralDestiny commented on GitHub (Jun 14, 2026):
What's Vacuum sorry?
@ex-aequo-et-bono commented on GitHub (Jun 14, 2026):
It's a postgres command to reclaim dead tuples.
docker exec -it <container_name> psql -U <username> -d <database_name> -c "VACUUM ANALYZE;"@ooCloudXoo commented on GitHub (Jun 15, 2026):
Same here
@mprokopiev commented on GitHub (Jun 15, 2026):
Upgraded. Vacuum helped too. Same query execution time now took lesser than second.
@ooCloudXoo commented on GitHub (Jun 16, 2026):
This also worked for me
@Hutch79 commented on GitHub (Jun 16, 2026):
Had the same problem and
VACUUM ANALYZE;fixed it for me. Although I guess this is only a temporary fix since the DB will get fragmented again.Thx for the workaround!!! 💜