github-starred/pangolin
2
0
Fork 0
mirror of https://github.com/fosrl/pangolin.git synced 2026-05-06 20:59:07 -05:00
Code Issues 564 Packages Projects Releases 72 Wiki Activity

[PR #2998] [MERGED] fix: deterministically finalize SQLite prepared statements to prevent native memory leak (#2120) #12051

New Issue
Closed
opened 2026-05-06 16:38:16 -05:00 by GiteaMirror · 0 comments
GiteaMirror commented 2026-05-06 16:38:16 -05:00
Owner
Copy Link

📋 Pull Request Information

Original PR: https://github.com/fosrl/pangolin/pull/2998
Author: @Josh-Voyles
Created: 5/4/2026
Status: ✅ Merged
Merged: 5/5/2026
Merged by: @oschwartz10612

Base: dev ← Head: mem-fix-2

📝 Commits (5)

  • d6abe83 fix: memory improvements
  • 2c85bcd fix(db): deterministically finalize prepared statements after execution
  • 0655ba9 fix: revert investigative changes, keep root cause fixes only
  • 9bd3307 cleaned comments - more concise
  • 2154811 removed possible introduced HA Redis bug; improved comment

📊 Changes

3 files changed (+94 additions, -7 deletions)

View changed files

📝 server/db/sqlite/driver.ts (+62 -1)
📝 server/private/routers/ws/ws.ts (+9 -3)
📝 server/routers/ws/ws.ts (+23 -3)

📄 Description

Community Contribution License Agreement

By creating this pull request, I grant the project maintainers an unlimited,
perpetual license to use, modify, and redistribute these contributions under any terms they
choose, including both the AGPLv3 and the Fossorial Commercial license terms. I
represent that I have the right to grant this license for all contributed content.

Description

Preface

I'm a cloud engineer with experience in Python and Bash; I do not have much experience with JS/TS. This fix relied heavily on OpenCode and Claude Opus 4.6 and several iterations testing 'confident' bug fixes. However, this implementation has been very solid and performant, seems logical, and is relatively minimal. No code or comments are left over from previous attempts to fix the problem.

Summary

  • Wraps better-sqlite3 prepare() to call finalize() immediately after statement execution, preventing unbounded native sqlite3_stmt accumulation under sustained load
  • Adds SQLite PRAGMA tuning (WAL mode, busy_timeout, cache_size, mmap_size) to reduce I/O contention and event loop blocking
  • Cleans up clientConfigVersions Map entries on WebSocket disconnect to prevent unbounded Map growth

Context

Drizzle-orm creates a new native sqlite3_stmt (8-32KB) per query and never calls finalize(). Under the increased DB load introduced in v1.13.0 (OLM per-user-per-org, ping accumulators, bandwidth tracking), statement creation outpaced GC, causing steady off-heap memory growth until OOM. PostgreSQL was unaffected because prepared statements live server-side and connection pool recycling cleans them up.

Changes

  • server/db/sqlite/driver.ts — autoFinalizeStatement() wrapper + PRAGMA tuning
  • server/routers/ws/ws.ts — clientConfigVersions.delete() on disconnect
  • server/private/routers/ws/ws.ts — same Map cleanup + eager cleanup in disconnectClient()

Verification

Tested under normal usage, 7 day requests retention, and Uptime Kuma checks for 48 hours. Pangolin container memory usually hovers around 335MB. I do not have any memory limits set in my compose file. No memory issues or pegged CPU causing a zombie instance.

How to test?

Build docker container community version using sqlite database.
1GB 2vCPU AWS t3a.micro instance

Fixes #2120

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/fosrl/pangolin/pull/2998 **Author:** [@Josh-Voyles](https://github.com/Josh-Voyles) **Created:** 5/4/2026 **Status:** ✅ Merged **Merged:** 5/5/2026 **Merged by:** [@oschwartz10612](https://github.com/oschwartz10612) **Base:** `dev` ← **Head:** `mem-fix-2` --- ### 📝 Commits (5) - [`d6abe83`](https://github.com/fosrl/pangolin/commit/d6abe83fdcabc1c22ddf9c211cb82bf33815413c) fix: memory improvements - [`2c85bcd`](https://github.com/fosrl/pangolin/commit/2c85bcd06b3f9caf44ddf1e03e3a0d30f8d57d24) fix(db): deterministically finalize prepared statements after execution - [`0655ba9`](https://github.com/fosrl/pangolin/commit/0655ba9423c3a1501e907e5781542a7aeb9097e6) fix: revert investigative changes, keep root cause fixes only - [`9bd3307`](https://github.com/fosrl/pangolin/commit/9bd33072f477a1fa14b3ac010c9813207f61d279) cleaned comments - more concise - [`2154811`](https://github.com/fosrl/pangolin/commit/2154811ffb140ae0b656f93d5e02775ae88d8915) removed possible introduced HA Redis bug; improved comment ### 📊 Changes **3 files changed** (+94 additions, -7 deletions) <details> <summary>View changed files</summary> 📝 `server/db/sqlite/driver.ts` (+62 -1) 📝 `server/private/routers/ws/ws.ts` (+9 -3) 📝 `server/routers/ws/ws.ts` (+23 -3) </details> ### 📄 Description ## Community Contribution License Agreement By creating this pull request, I grant the project maintainers an unlimited, perpetual license to use, modify, and redistribute these contributions under any terms they choose, including both the AGPLv3 and the Fossorial Commercial license terms. I represent that I have the right to grant this license for all contributed content. ## Description ### Preface I'm a cloud engineer with experience in Python and Bash; I do not have much experience with JS/TS. This fix relied heavily on OpenCode and Claude Opus 4.6 and several iterations testing 'confident' bug fixes. However, this implementation has been very solid and performant, seems logical, and is relatively minimal. No code or comments are left over from previous attempts to fix the problem. ### Summary - Wraps better-sqlite3 prepare() to call finalize() immediately after statement execution, preventing unbounded native sqlite3_stmt accumulation under sustained load - Adds SQLite PRAGMA tuning (WAL mode, busy_timeout, cache_size, mmap_size) to reduce I/O contention and event loop blocking - Cleans up clientConfigVersions Map entries on WebSocket disconnect to prevent unbounded Map growth ### Context Drizzle-orm creates a new native sqlite3_stmt (8-32KB) per query and never calls finalize(). Under the increased DB load introduced in v1.13.0 (OLM per-user-per-org, ping accumulators, bandwidth tracking), statement creation outpaced GC, causing steady off-heap memory growth until OOM. PostgreSQL was unaffected because prepared statements live server-side and connection pool recycling cleans them up. ### Changes - server/db/sqlite/driver.ts — autoFinalizeStatement() wrapper + PRAGMA tuning - server/routers/ws/ws.ts — clientConfigVersions.delete() on disconnect - server/private/routers/ws/ws.ts — same Map cleanup + eager cleanup in disconnectClient() ### Verification Tested under normal usage, 7 day requests retention, and Uptime Kuma checks for 48 hours. Pangolin container memory usually hovers around 335MB. I do not have any memory limits set in my compose file. No memory issues or pegged CPU causing a zombie instance. ## How to test? Build docker container community version using sqlite database. 1GB 2vCPU AWS t3a.micro instance Fixes #2120 --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
GiteaMirror added the pull-request label 2026-05-06 16:38:16 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
api
authentication
bug
config
dependencies
docker
documentation
enhancement
good first issue
help wanted
Improvement
Look Into
needs investigating
networking
new feature
non-critical bug
potential bug
pull-request
Mirrored from GitHub Pull Request
 question
reverse proxy
Security
stale
ui
wontfix
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/pangolin#12051
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?