[PR #356] [MERGED] feat(github): reuse ETags across syncs via conditional requests #7584

Closed
opened 2026-08-21 21:05:04 -05:00 by GiteaMirror · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/RayLabsHQ/gitea-mirror/pull/356
Author: @joshfree
Created: 8/5/2026
Status: ✅ Merged
Merged: 8/6/2026
Merged by: @arunavo4

Base: main ← Head: feat/conditional-requests-etag


📝 Commits (3)

  • 79ccd0d feat(github): reuse ETags across syncs via conditional requests
  • 81b5f03 test(github): cover conditional-request ETag cache
  • e427500 fix: key conditional-request cache by expanded URL

📊 Changes

3 files changed (+322 additions, -0 deletions)

View changed files

➕ src/lib/github-conditional-requests.test.ts (+177 -0)
➕ src/lib/github-conditional-requests.ts (+140 -0)
📝 src/lib/github.ts (+5 -0)

📄 Description

Summary

Fixes #355.

On each scheduled sync the mirror re-lists a repository's pull requests (octokit.paginate(octokit.rest.pulls.list, { state: "all", per_page: 100, ... })) and then fetches per-PR details. The Octokit client in createGitHubClient was built with the throttling plugin but no conditional-request/ETag layer, so unchanged pull request data was downloaded in full on every run and consumed full rate-limit budget.

This adds ETag-based conditional requests. For every GET the client now replays the previously stored ETag as If-None-Match; GitHub then returns 304 Not Modified when nothing changed, which does not count against the token's primary rate limit, and the cached body is returned in its place. Sync results are unchanged — only redundant full downloads are removed.

What changed

  • New src/lib/github-conditional-requests.ts: an octokit.hook.wrap("request", ...) that attaches If-None-Match on GETs, caches 200 bodies keyed by user scope + method + URL, and serves the cached body (as a 200) when GitHub replies 304. The paginate link header is preserved on a cache hit so pagination still follows subsequent pages. Non-GET requests are passed through untouched.
  • The cache store is an interface with a bounded in-memory implementation, wired into createGitHubClient. Gitea Mirror runs as a long-lived server whose scheduler re-syncs on an interval, so a process-lifetime store is enough to turn each repeated poll into a cheap 304. No new dependencies and no schema/migration changes. The store is injectable, so a persistent (e.g. SQLite-backed) implementation could be added later for cross-restart reuse.
  • Entries are scoped per user so one token never reads another's cached data.

Testing

src/lib/github-conditional-requests.test.ts drives a real Octokit instance with a stubbed fetch and verifies: the second GET replays If-None-Match; a 304 is transparently served from cache as a 200 with the same body; responses without an ETag are re-fetched; non-GET requests are never made conditional; and cache entries are isolated by scope.

Notes

  • Value to self-hosted users: fewer full downloads and lower pressure on the token's hourly rate limit, so large mirror sets or short sync intervals are less likely to hit throttling.
  • Optional follow-up (not included here to keep this focused): narrowing the list to state: "open" or an incremental since query where appropriate would further reduce redundant work.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/RayLabsHQ/gitea-mirror/pull/356 **Author:** [@joshfree](https://github.com/joshfree) **Created:** 8/5/2026 **Status:** ✅ Merged **Merged:** 8/6/2026 **Merged by:** [@arunavo4](https://github.com/arunavo4) **Base:** `main` ← **Head:** `feat/conditional-requests-etag` --- ### 📝 Commits (3) - [`79ccd0d`](https://github.com/RayLabsHQ/gitea-mirror/commit/79ccd0d026685f7901967d397d30eb1922d7dc82) feat(github): reuse ETags across syncs via conditional requests - [`81b5f03`](https://github.com/RayLabsHQ/gitea-mirror/commit/81b5f030b227edfca94f685afc570452321bba9d) test(github): cover conditional-request ETag cache - [`e427500`](https://github.com/RayLabsHQ/gitea-mirror/commit/e427500412ff307bf39a3e8518760d39aac7311a) fix: key conditional-request cache by expanded URL ### 📊 Changes **3 files changed** (+322 additions, -0 deletions) <details> <summary>View changed files</summary> ➕ `src/lib/github-conditional-requests.test.ts` (+177 -0) ➕ `src/lib/github-conditional-requests.ts` (+140 -0) 📝 `src/lib/github.ts` (+5 -0) </details> ### 📄 Description ## Summary Fixes #355. On each scheduled sync the mirror re-lists a repository's pull requests (`octokit.paginate(octokit.rest.pulls.list, { state: "all", per_page: 100, ... })`) and then fetches per-PR details. The Octokit client in `createGitHubClient` was built with the throttling plugin but no conditional-request/ETag layer, so unchanged pull request data was downloaded in full on every run and consumed full rate-limit budget. This adds ETag-based conditional requests. For every GET the client now replays the previously stored `ETag` as `If-None-Match`; GitHub then returns `304 Not Modified` when nothing changed, which does **not** count against the token's primary rate limit, and the cached body is returned in its place. Sync results are unchanged — only redundant full downloads are removed. ## What changed - New `src/lib/github-conditional-requests.ts`: an `octokit.hook.wrap("request", ...)` that attaches `If-None-Match` on GETs, caches `200` bodies keyed by user scope + method + URL, and serves the cached body (as a `200`) when GitHub replies `304`. The paginate `link` header is preserved on a cache hit so pagination still follows subsequent pages. Non-GET requests are passed through untouched. - The cache store is an interface with a bounded in-memory implementation, wired into `createGitHubClient`. Gitea Mirror runs as a long-lived server whose scheduler re-syncs on an interval, so a process-lifetime store is enough to turn each repeated poll into a cheap `304`. No new dependencies and no schema/migration changes. The store is injectable, so a persistent (e.g. SQLite-backed) implementation could be added later for cross-restart reuse. - Entries are scoped per user so one token never reads another's cached data. ## Testing `src/lib/github-conditional-requests.test.ts` drives a real `Octokit` instance with a stubbed `fetch` and verifies: the second GET replays `If-None-Match`; a `304` is transparently served from cache as a `200` with the same body; responses without an `ETag` are re-fetched; non-GET requests are never made conditional; and cache entries are isolated by scope. ## Notes - Value to self-hosted users: fewer full downloads and lower pressure on the token's hourly rate limit, so large mirror sets or short sync intervals are less likely to hit throttling. - Optional follow-up (not included here to keep this focused): narrowing the list to `state: "open"` or an incremental `since` query where appropriate would further reduce redundant work. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
GiteaMirror added the pull-request label 2026-08-21 21:05:04 -05:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/gitea-mirror#7584