[PR #304] fix: chunk large Docker container lists to prevent WebSocket message loss #6712

New Issue

GiteaMirror · 2026-06-17T02:34:32-05:00

GiteaMirror commented

2026-06-17 02:34:32 -05:00

📋 Pull Request Information

Original PR: https://github.com/fosrl/newt/pull/304
Author: @jaydeep-pipaliya
Created: 4/9/2026
Status: 🔄 Open

Base: main ← Head: fix/chunk-docker-container-messages

📝 Commits (10+)

458912e Merge pull request #174 from fosrl/dependabot/github_actions/docker/metadata-action-5.9.0
7148574 Merge pull request #175 from fosrl/dependabot/github_actions/docker/setup-qemu-action-3.7.0
78dc39e Merge pull request #181 from fosrl/dependabot/github_actions/github/codeql-action-4.31.5
37c96d0 Merge pull request #182 from fosrl/dependabot/github_actions/actions/setup-go-6.1.0
f417ee3 Merge pull request #186 from fosrl/dependabot/docker/minor-updates-60be0b6e22
e5e7331 Merge branch 'main' into dev
5ce3f45 Fix adding new exit nodes to hp not sending interval
87e2eb3 Update readme
3bcafbf Handle server version and prevent backward issues with clients
6948066 Merge pull request #192 from fosrl/dev

📊 Changes

102 files changed (+21682 additions, -1115 deletions)

View changed files

➕ .env.example (+5 -0)
➕ .github/DISCUSSION_TEMPLATE/feature-requests.yml (+47 -0)
➕ .github/ISSUE_TEMPLATE/1.bug_report.yml (+51 -0)
➕ .github/ISSUE_TEMPLATE/config.yml (+8 -0)
📝 .github/dependabot.yml (+5 -0)
📝 .github/workflows/cicd.yml (+932 -57)
➕ .github/workflows/mirror.yaml (+132 -0)
➕ .github/workflows/nix-build.yml (+23 -0)
➕ .github/workflows/nix-dependabot-update-hash.yml (+48 -0)
➕ .github/workflows/stale-bot.yml (+37 -0)
📝 .github/workflows/test.yml (+24 -13)
📝 .gitignore (+6 -2)
📝 .go-version (+1 -1)
📝 CONTRIBUTING.md (+2 -6)
📝 Dockerfile (+13 -5)
📝 Makefile (+64 -28)
📝 README.md (+7 -157)
📝 SECURITY.md (+1 -1)
➕ authdaemon.go (+150 -0)
➕ authdaemon/connection.go (+27 -0)

...and 80 more files

📄 Description

What does this PR do?

Companion PR for fosrl/pangolin#2117 — Docker Container View not displaying when >20 containers are running.

Problem

gorilla/websocket.WriteJSON serializes the full container list into a single WebSocket text frame. With 55+ containers (each 1-5KB of JSON metadata), the resulting frame can be 55-275KB. Intermediary proxies (Traefik, nginx, Cloudflare tunnels) can silently drop or truncate frames this large, causing the pangolin server to never receive the container data.

Solution

Added sendContainerList() that automatically chunks large container lists:

// ���15 containers: single message (backward compatible, zero behavior change)
{"containers": [...]}

// >15 containers: chunked with batch metadata
{"containers": [...], "chunkIndex": 0, "totalChunks": 4, "batchId": "a1b2c3d4"}
{"containers": [...], "chunkIndex": 1, "totalChunks": 4, "batchId": "a1b2c3d4"}
{"containers": [...], "chunkIndex": 2, "totalChunks": 4, "batchId": "a1b2c3d4"}
{"containers": [...], "chunkIndex": 3, "totalChunks": 4, "batchId": "a1b2c3d4"}

Why `batchId`?

Two concurrent container sends can happen when a manual fetch request and a Docker event fire at the same time. Without batchId, their chunks would interleave and corrupt the accumulated data on the server. Each batch gets a unique ID (reusing the existing generateChainId helper) so the server can track and supersede batches correctly.

Why chunk size of 15?

Each container with full metadata (labels, ports, networks) serializes to 1-5KB. 15 containers ≈ 15-75KB per WebSocket frame — safely under common proxy defaults (Traefik default buffer: 64KB, nginx: 64KB). The threshold was chosen to balance message count vs. frame size.

Changes

main.go:

Added sendContainerList() — chunks large lists, passes small lists through unchanged
Updated both call sites: initial fetch handler + Docker event monitor callback
Reuses existing generateChainId() for batch IDs — no new dependencies

Companion PR

Pangolin server: https://github.com/fosrl/pangolin/pull/2817

Chunk reassembly with batchId tracking, input validation, typed accumulator, 120s TTL on partial chunks

Testing

go build passes
≤15 containers: identical to current behavior (single message, no metadata)
>15 containers: splits into chunks of 15 with chunkIndex/totalChunks/batchId
Concurrent sends: each gets unique batchId, server handles superseding

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/fosrl/newt/pull/304 **Author:** [@jaydeep-pipaliya](https://github.com/jaydeep-pipaliya) **Created:** 4/9/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `fix/chunk-docker-container-messages` --- ### 📝 Commits (10+) - [`458912e`](https://github.com/fosrl/newt/commit/458912e5be581334f87871307aa0ccaa1cb5de5c) Merge pull request #174 from fosrl/dependabot/github_actions/docker/metadata-action-5.9.0 - [`7148574`](https://github.com/fosrl/newt/commit/71485743add185dd395deb304da1eb92b04b2386) Merge pull request #175 from fosrl/dependabot/github_actions/docker/setup-qemu-action-3.7.0 - [`78dc39e`](https://github.com/fosrl/newt/commit/78dc39e1530b7a750c8c8dc7a4bcb052de4f55d1) Merge pull request #181 from fosrl/dependabot/github_actions/github/codeql-action-4.31.5 - [`37c96d0`](https://github.com/fosrl/newt/commit/37c96d0b3ef20155c8a59aa0e8fa1901b127fbd5) Merge pull request #182 from fosrl/dependabot/github_actions/actions/setup-go-6.1.0 - [`f417ee3`](https://github.com/fosrl/newt/commit/f417ee32fb3ac60e0c0f140f4f40caed95cc3c41) Merge pull request #186 from fosrl/dependabot/docker/minor-updates-60be0b6e22 - [`e5e7331`](https://github.com/fosrl/newt/commit/e5e733123b63cbc2333cda7495f132a3caf2ae9f) Merge branch 'main' into dev - [`5ce3f45`](https://github.com/fosrl/newt/commit/5ce3f4502de4078846a4c9352e93453c5ab6a837) Fix adding new exit nodes to hp not sending interval - [`87e2eb3`](https://github.com/fosrl/newt/commit/87e2eb33dba9b10494cb260761446a92177493d8) Update readme - [`3bcafbf`](https://github.com/fosrl/newt/commit/3bcafbf07a3b435de9c0bf393b78c6624b4e6fd8) Handle server version and prevent backward issues with clients - [`6948066`](https://github.com/fosrl/newt/commit/6948066ae4ec89d2339ed9e7c1db092cb57a74c9) Merge pull request #192 from fosrl/dev ### 📊 Changes **102 files changed** (+21682 additions, -1115 deletions) <details> <summary>View changed files</summary> ➕ `.env.example` (+5 -0) ➕ `.github/DISCUSSION_TEMPLATE/feature-requests.yml` (+47 -0) ➕ `.github/ISSUE_TEMPLATE/1.bug_report.yml` (+51 -0) ➕ `.github/ISSUE_TEMPLATE/config.yml` (+8 -0) 📝 `.github/dependabot.yml` (+5 -0) 📝 `.github/workflows/cicd.yml` (+932 -57) ➕ `.github/workflows/mirror.yaml` (+132 -0) ➕ `.github/workflows/nix-build.yml` (+23 -0) ➕ `.github/workflows/nix-dependabot-update-hash.yml` (+48 -0) ➕ `.github/workflows/stale-bot.yml` (+37 -0) 📝 `.github/workflows/test.yml` (+24 -13) 📝 `.gitignore` (+6 -2) 📝 `.go-version` (+1 -1) 📝 `CONTRIBUTING.md` (+2 -6) 📝 `Dockerfile` (+13 -5) 📝 `Makefile` (+64 -28) 📝 `README.md` (+7 -157) 📝 `SECURITY.md` (+1 -1) ➕ `authdaemon.go` (+150 -0) ➕ `authdaemon/connection.go` (+27 -0) _...and 80 more files_ </details> ### 📄 Description ## What does this PR do? Companion PR for fosrl/pangolin#2117 — Docker Container View not displaying when >20 containers are running. ## Problem `gorilla/websocket.WriteJSON` serializes the full container list into a single WebSocket text frame. With 55+ containers (each 1-5KB of JSON metadata), the resulting frame can be 55-275KB. Intermediary proxies (Traefik, nginx, Cloudflare tunnels) can silently drop or truncate frames this large, causing the pangolin server to never receive the container data. ## Solution Added `sendContainerList()` that automatically chunks large container lists: ```go // ��15 containers: single message (backward compatible, zero behavior change) {"containers": [...]} // >15 containers: chunked with batch metadata {"containers": [...], "chunkIndex": 0, "totalChunks": 4, "batchId": "a1b2c3d4"} {"containers": [...], "chunkIndex": 1, "totalChunks": 4, "batchId": "a1b2c3d4"} {"containers": [...], "chunkIndex": 2, "totalChunks": 4, "batchId": "a1b2c3d4"} {"containers": [...], "chunkIndex": 3, "totalChunks": 4, "batchId": "a1b2c3d4"} ``` ### Why `batchId`? Two concurrent container sends can happen when a manual fetch request and a Docker event fire at the same time. Without `batchId`, their chunks would interleave and corrupt the accumulated data on the server. Each batch gets a unique ID (reusing the existing `generateChainId` helper) so the server can track and supersede batches correctly. ### Why chunk size of 15? Each container with full metadata (labels, ports, networks) serializes to 1-5KB. 15 containers ≈ 15-75KB per WebSocket frame — safely under common proxy defaults (Traefik default buffer: 64KB, nginx: 64KB). The threshold was chosen to balance message count vs. frame size. ## Changes **`main.go`**: - Added `sendContainerList()` — chunks large lists, passes small lists through unchanged - Updated both call sites: initial fetch handler + Docker event monitor callback - Reuses existing `generateChainId()` for batch IDs — no new dependencies ## Companion PR **Pangolin server:** https://github.com/fosrl/pangolin/pull/2817 - Chunk reassembly with `batchId` tracking, input validation, typed accumulator, 120s TTL on partial chunks ## Testing - `go build` passes - ≤15 containers: identical to current behavior (single message, no metadata) - \>15 containers: splits into chunks of 15 with `chunkIndex`/`totalChunks`/`batchId` - Concurrent sends: each gets unique `batchId`, server handles superseding --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

GiteaMirror added the pull-request label 2026-06-17 02:34:32 -05:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/newt#6712

[PR #304] fix: chunk large Docker container lists to prevent WebSocket message loss #6712