[PR #879] [MERGED] multi-process concurrent cobalt instances #1276

New Issue

GiteaMirror · 2025-11-09T10:09:38-06:00

GiteaMirror commented

2025-11-09 10:09:38 -06:00

📋 Pull Request Information

Original PR: https://github.com/imputnet/cobalt/pull/879
Author: @dumbmoron
Created: 11/1/2024
Status: ✅ Merged
Merged: 11/2/2024
Merged by: @github-actions[bot]

Base: develop ← Head: reuseport

📝 Commits (10+)

5e92b64 api: add API_REDIS_URL env
18acad1 api: implement redis/memory store for cache
11314fb api/store: implement has() method
132255b api/stream/manage: use cobalt Store for stream cache
93e6344 api/stream/manage: make itunnel port configurable
2d6d406 api/crypto: use buffers for salt directly instead of hex strings
40d6a02 api: cluster support
66cb8d3 api: move hmac secrets to single file
693204b api/store: use basic strings instead of hashes for keys
d466f8a api: upgrade express-rate-limit to v7, reuse key generator

📊 Changes

25 files changed (+712 additions, -117 deletions)

View changed files

📝 Dockerfile (+2 -2)
📝 api/package.json (+5 -2)
📝 api/src/cobalt.js (+9 -2)
📝 api/src/config.js (+17 -1)
📝 api/src/core/api.js (+64 -36)
➕ api/src/misc/cluster.js (+71 -0)
📝 api/src/misc/crypto.js (+1 -9)
📝 api/src/processing/cookie/cookie.js (+16 -5)
📝 api/src/processing/cookie/manager.js (+65 -17)
📝 api/src/processing/services/bluesky.js (+1 -1)
📝 api/src/processing/services/instagram.js (+4 -4)
📝 api/src/processing/services/snapchat.js (+2 -2)
📝 api/src/processing/services/tiktok.js (+1 -1)
📝 api/src/processing/services/twitter.js (+3 -3)
📝 api/src/security/api-keys.js (+20 -4)
➕ api/src/security/secrets.js (+62 -0)
➕ api/src/store/base-store.js (+48 -0)
➕ api/src/store/memory-store.js (+76 -0)
➕ api/src/store/redis-ratelimit.js (+19 -0)
➕ api/src/store/redis-store.js (+64 -0)

...and 5 more files

📄 Description

adds support for storing cache data in redis, configurable with API_REDIS_URL env (redis://localhost:6379)
adds ability to spawn multiple cross-communicating cobalt processes using node:cluster, configured with API_INSTANCE_COUNT env
- requires API_REDIS_URL to be configured
- ratelimit and stream cache exclusively stored in redis
- istream cache stored locally on each instance, istreams are bound to specific instance that is serving the stream
- main instance fetches cookies/api keys, distributes them amongst newly spawned processes
- works on linux, node >= 23.1.0

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/imputnet/cobalt/pull/879 **Author:** [@dumbmoron](https://github.com/dumbmoron) **Created:** 11/1/2024 **Status:** ✅ Merged **Merged:** 11/2/2024 **Merged by:** [@github-actions[bot]](https://github.com/apps/github-actions) **Base:** `develop` ← **Head:** `reuseport` --- ### 📝 Commits (10+) - [`5e92b64`](https://github.com/imputnet/cobalt/commit/5e92b649a33dcff0553dd03f61eacf304adc9da5) api: add `API_REDIS_URL` env - [`18acad1`](https://github.com/imputnet/cobalt/commit/18acad19b931418c343a14cf88656684827b0e02) api: implement redis/memory store for cache - [`11314fb`](https://github.com/imputnet/cobalt/commit/11314fb8d15ef7b2b90da51a43360e1aa8e02732) api/store: implement has() method - [`132255b`](https://github.com/imputnet/cobalt/commit/132255b00412f16cd12ffb825ab7d80a821692b8) api/stream/manage: use cobalt Store for stream cache - [`93e6344`](https://github.com/imputnet/cobalt/commit/93e6344fc7ea73cf7e489bc0c5b8de71d424030a) api/stream/manage: make itunnel port configurable - [`2d6d406`](https://github.com/imputnet/cobalt/commit/2d6d406f4834ffe0f44a38e743468e4a4017c12d) api/crypto: use buffers for salt directly instead of hex strings - [`40d6a02`](https://github.com/imputnet/cobalt/commit/40d6a02b61f061a8ef65702aa4012bb6b045daec) api: cluster support - [`66cb8d3`](https://github.com/imputnet/cobalt/commit/66cb8d360dd88b3bd137e4fb1e3155e156d3f550) api: move hmac secrets to single file - [`693204b`](https://github.com/imputnet/cobalt/commit/693204b799d9d673bd3dd31e737e65c92fa41772) api/store: use basic strings instead of hashes for keys - [`d466f8a`](https://github.com/imputnet/cobalt/commit/d466f8a4af41deda5b2f0616b94c9464db6336ea) api: upgrade express-rate-limit to v7, reuse key generator ### 📊 Changes **25 files changed** (+712 additions, -117 deletions) <details> <summary>View changed files</summary> 📝 `Dockerfile` (+2 -2) 📝 `api/package.json` (+5 -2) 📝 `api/src/cobalt.js` (+9 -2) 📝 `api/src/config.js` (+17 -1) 📝 `api/src/core/api.js` (+64 -36) ➕ `api/src/misc/cluster.js` (+71 -0) 📝 `api/src/misc/crypto.js` (+1 -9) 📝 `api/src/processing/cookie/cookie.js` (+16 -5) 📝 `api/src/processing/cookie/manager.js` (+65 -17) 📝 `api/src/processing/services/bluesky.js` (+1 -1) 📝 `api/src/processing/services/instagram.js` (+4 -4) 📝 `api/src/processing/services/snapchat.js` (+2 -2) 📝 `api/src/processing/services/tiktok.js` (+1 -1) 📝 `api/src/processing/services/twitter.js` (+3 -3) 📝 `api/src/security/api-keys.js` (+20 -4) ➕ `api/src/security/secrets.js` (+62 -0) ➕ `api/src/store/base-store.js` (+48 -0) ➕ `api/src/store/memory-store.js` (+76 -0) ➕ `api/src/store/redis-ratelimit.js` (+19 -0) ➕ `api/src/store/redis-store.js` (+64 -0) _...and 5 more files_ </details> ### 📄 Description - adds support for storing cache data in redis, configurable with `API_REDIS_URL` env (`redis://localhost:6379`) - adds ability to spawn multiple cross-communicating cobalt processes using `node:cluster`, configured with `API_INSTANCE_COUNT` env - requires `API_REDIS_URL` to be configured - ratelimit and stream cache exclusively stored in redis - istream cache stored locally on each instance, istreams are bound to specific instance that is serving the stream - main instance fetches cookies/api keys, distributes them amongst newly spawned processes - works on linux, node >= 23.1.0 --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

GiteaMirror added the pull-request label 2025-11-09 10:09:38 -06:00

GiteaMirror closed this issue

2025-11-09 10:09:38 -06:00

GiteaMirror referenced this issue

2025-11-09 10:11:32 -06:00

[PR #1276] [MERGED] api/create-filename: relax sanitizeString and use fullwidth replacements #1366

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/cobalt#1276