[PR #3294] Batch Redis websocket fan-out and deduplicate queued rebuild jobs #34670

New Issue

GiteaMirror · 2026-06-17T04:19:06-05:00

GiteaMirror commented

2026-06-17 04:19:06 -05:00

📋 Pull Request Information

Original PR: https://github.com/fosrl/pangolin/pull/3294
Author: @Copilot
Created: 6/16/2026
Status: 🔄 Open

Base: queue ← Head: copilot/add-batching-mechanism-to-rebuild-client-associati

📝 Commits (2)

45158ff feat: batch redis ws direct messages and dedupe rebuild queue jobs
b66140b refactor: tighten ws batch typing and queue cleanup logging

📊 Changes

3 files changed (+170 additions, -27 deletions)

View changed files

📝 server/private/lib/rebuildQueue.ts (+29 -0)
📝 server/private/routers/ws/ws.ts (+118 -18)
📝 server/routers/ws/types.ts (+23 -9)

📄 Description

Rebuild association paths can emit thousands of websocket sends in bursts, which was creating Redis/websocket backpressure. This update introduces batched cross-node direct delivery and queue-level deduplication for rebuild jobs so duplicate queued work is dropped while preserving re-queue behavior for in-progress jobs.

Websocket cross-node batching (direct sends)
- Added Redis direct-batch message support and typed it in the shared WS message contract.
- Replaced per-message Redis publish for remote direct sends with buffered batching (size 250, short flush interval).
- Added subscriber-side bulk handling to fan out batched direct messages efficiently to local connected clients.
Queue deduplication semantics for rebuilds
- Added queued-job dedupe using Redis set membership keyed by type:id.
- Skip enqueue when an identical job is already queued.
- Remove dedupe key at dequeue time (not completion), so if a job is currently in progress, a new same-type/id job can still be queued.
Type/contract tightening
- Refined RedisMessage into a discriminated union for direct, direct-batch, and broadcast, making batch payload shape explicit.

// queued dedupe behavior
const dedupeKey = `${job.type}:${job.id}`;
const added = await redis.sadd(QUEUED_SET_KEY, dedupeKey);
if (added === 0) return; // already queued
await redis.rpush(QUEUE_KEY, JSON.stringify(job));

// allow requeue while in-progress
const payload = await redis.lpop(QUEUE_KEY);
await redis.srem(QUEUED_SET_KEY, `${job.type}:${job.id}`);

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/fosrl/pangolin/pull/3294 **Author:** [@Copilot](https://github.com/apps/copilot-swe-agent) **Created:** 6/16/2026 **Status:** 🔄 Open **Base:** `queue` ← **Head:** `copilot/add-batching-mechanism-to-rebuild-client-associati` --- ### 📝 Commits (2) - [`45158ff`](https://github.com/fosrl/pangolin/commit/45158ff45bbc7e70aa9174056345c9b91066b9f9) feat: batch redis ws direct messages and dedupe rebuild queue jobs - [`b66140b`](https://github.com/fosrl/pangolin/commit/b66140bfd2c61dce94b16ff5f237239493cbd835) refactor: tighten ws batch typing and queue cleanup logging ### 📊 Changes **3 files changed** (+170 additions, -27 deletions) <details> <summary>View changed files</summary> 📝 `server/private/lib/rebuildQueue.ts` (+29 -0) 📝 `server/private/routers/ws/ws.ts` (+118 -18) 📝 `server/routers/ws/types.ts` (+23 -9) </details> ### 📄 Description Rebuild association paths can emit thousands of websocket sends in bursts, which was creating Redis/websocket backpressure. This update introduces batched cross-node direct delivery and queue-level deduplication for rebuild jobs so duplicate queued work is dropped while preserving re-queue behavior for in-progress jobs. - **Websocket cross-node batching (direct sends)** - Added Redis `direct-batch` message support and typed it in the shared WS message contract. - Replaced per-message Redis publish for remote direct sends with buffered batching (size `250`, short flush interval). - Added subscriber-side bulk handling to fan out batched direct messages efficiently to local connected clients. - **Queue deduplication semantics for rebuilds** - Added queued-job dedupe using Redis set membership keyed by `type:id`. - Skip enqueue when an identical job is already queued. - Remove dedupe key at dequeue time (not completion), so if a job is currently in progress, a new same-type/id job can still be queued. - **Type/contract tightening** - Refined `RedisMessage` into a discriminated union for `direct`, `direct-batch`, and `broadcast`, making batch payload shape explicit. ```ts // queued dedupe behavior const dedupeKey = `${job.type}:${job.id}`; const added = await redis.sadd(QUEUED_SET_KEY, dedupeKey); if (added === 0) return; // already queued await redis.rpush(QUEUE_KEY, JSON.stringify(job)); // allow requeue while in-progress const payload = await redis.lpop(QUEUE_KEY); await redis.srem(QUEUED_SET_KEY, `${job.type}:${job.id}`); ``` --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

GiteaMirror added the pull-request label 2026-06-17 04:19:06 -05:00

Sign in to join this conversation.

Branches Tags

main

crowdin_dev

dev

dependabot/npm_and_yarn/npm-dependencies-338850a190

dependabot/go_modules/install/go-install-dependencies-3804ca7238

dependabot/docker/docker-dependencies-4faa477378

dependabot/github_actions/github-actions-dependencies-6d79802a48

api-improvements

feat/remember-last-idp-on-smart-login-form

refactor/batch-status-requests

dependabot/npm_and_yarn/multi-5f1280885e

fix/non-semver-version-error

private-resource-page

resource-launcher

backhaul

exit-node-reconnect

feat/command-palette

ssh

delete-account

msg-delivery

org-only-idp

cicd

patch

site-targets-auto-login

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/pangolin#34670