[PR #296] [MERGED] fix: prevent duplicate issues on retry-after-deadlock + pre-fetch pagination #5968

New Issue

GiteaMirror · 2026-06-21T21:52:52-05:00

GiteaMirror commented

2026-06-21 21:52:52 -05:00

📋 Pull Request Information

Original PR: https://github.com/RayLabsHQ/gitea-mirror/pull/296
Author: @seanmousseau
Created: 5/22/2026
Status: ✅ Merged
Merged: 5/23/2026
Merged by: @arunavo4

Base: main ← Head: fix/dedup-create-on-deadlock-retry

📝 Commits (1)

b76073b fix: prevent duplicate issues & PRs on retry-after-deadlock + Link-header pagination

📊 Changes

2 files changed (+446 additions, -28 deletions)

View changed files

➕ src/lib/gitea-issue-dedup-on-retry.test.ts (+282 -0)
📝 src/lib/gitea.ts (+164 -28)

📄 Description

Summary

Fixes two compounding bugs in mirrorGitRepoIssuesToGitea and mirrorGitRepoPullRequestsToGitea that produce duplicate Gitea issue/PR-as-issue rows on every sync against any non-SQLite backend. In a real-world test (Gitea 1.26.2 on PostgreSQL 17.10), an unpatched sync produced +1,878 dups in 15 minutes during initial repro and +3,345 dups overnight with hourly cron firing the bug repeatedly. With the patch, the same workload produces 0 new duplicates.

Bug #1 — pagination on existing-issues / existing-PRs / per-issue-comments

Three paginated pre-fetches in gitea.ts paginated with limit=100 and broke on pageX.length < itemsPerPage. Gitea caps response size at server-side [api].MAX_RESPONSE_ITEMS (default 50), so the very first page already looks "short" and pagination terminated after one page. The existing-issue, existing-PR, and existing-comment maps were built from only ~50 items per repo, so every entry past page 1 was misclassified as new on every sync and re-created via the CREATE branch.

Naive removal of the short-page break doesn't work either: for some Gitea endpoints (e.g. issue comments past the actual end) Gitea returns the same data on every page instead of returning [], so the loop runs forever.

Fix: use the Link header (RFC 5988). If rel="next" is absent in the response, terminate pagination. Applied to: existing-issues pre-fetch, existing-PRs pre-fetch, and per-issue comments fetch. The empty-page check is kept as a defensive early-out.

Bug #2 — retry-after-deadlock duplicates issues and PRs

Even when the dedup map is complete, a second class of duplication fires when Gitea's CreateIssue handler commits the issue insert in one transaction and then deadlocks on the subsequent addLabel / UPDATE repository transaction. The row is committed and visible, but the in-memory dedup map is never refreshed between retries. processWithRetry (default maxRetries: 2) re-invokes the per-item callback, sees existingItem === undefined from the stale map, and creates a fresh duplicate via httpPost.

Reproduces deterministically on MySQL (Error 1213 (40001)) and PostgreSQL (pq: deadlock detected (40P01)). SQLite escapes because writes serialize globally.

Fix: two guards inside each create branch:

Defensive recheck before create. Query Gitea by [GH-ISSUE #N] (issues) or [PR #N] (PRs) before httpPost. If found, switch to PATCH and cache the hit.
Cache successful creates immediately. Write the new item into the dedup map right after a successful httpPost.

Applied to: the issues create path, the PR mirror's enriched create path, and the PR mirror's basic-fallback create path. Logs a Recovered orphan from prior failed attempt for #N line so operators can spot recovery in practice.

Real-world evidence

Same test environment, same gitea-mirror cron schedule:

Run	Retrying (deadlock)	Recovered orphan	Duplicates created (10h cron)
Stock v3.16.0	27 in initial repro	n/a	+3,345 dups overnight (PRs duplicated once per cron tick)
Patched (all fixes)	0	0 (none needed!)	0

With the pagination fix the pre-fetch correctly enumerates all existing items via Link: rel="next". Every GitHub iteration finds its match → routes to PATCH → no httpPost → no addLabel deadlock → no retry → no orphan to recover. The recheck-before-create is a defense-in-depth layer for genuinely-new items whose first create attempt deadlocks; on a re-sync it's essentially never invoked.

Tests

Adds src/lib/gitea-issue-dedup-on-retry.test.ts using the structural-source test pattern from gitea-mirror-failure-recovery.test.ts (issue #268's regression test). 8 assertions covering:

Per-item lookup against the dedup map (sanity guards against refactors)
Create-issue httpPost call still exists
Defensive recheck via httpGet using [GH-ISSUE #N] runs before the create
Recheck hits cache into the dedup map and route to PATCH with a recognisable log line
Existing-issues pagination loop uses Link header (rel=\"next\")
Per-issue comments pagination loop uses the Link header
Successful creates cache the new item into the dedup map
PR mirror function has the same guarantees: Link-header pagination + ≥2 Recovered orphan log lines (one each for the enriched + basic-fallback create paths)

bun test — 255 pass / 4 skip / 0 fail (up from 247 baseline).

#262 — "Issue's doubles when sync" (closed by #266). #266's metadata guard prevents re-mirroring already-mirrored issues across syncs, but doesn't address either of the bugs in this PR.
#157 — @emrebasarannn raised a similar "issues are duplicate" symptom that wasn't fully resolved.
#268 — "Imported repos stuck at mirroring" (partial fix in v3.15.7 via #280). Different code path; not addressed here.

Caveat / honest disclosure

The test sync occasionally stalls partway through — sync produces some successful Mirrored issue log lines then goes quiet, while the job row sits at in_progress=true, last_checkpoint=never and the recovery detector logs "Found 1 interrupted jobs" repeatedly without ever invoking the recovery handler. This stall is independent of this PR's changes — reproduces identically on stock ghcr.io/raylabshq/gitea-mirror:latest, matches reports in #268's open comments, and lives entirely in orchestration code (helpers.ts, recovery.ts, scheduler-service.ts) that this PR does not touch. On the items that do process, the dedup guarantee in this PR holds: zero new duplicates. Worth a follow-up.

Test plan

bun install && bun test — all green (255/4/0)
docker buildx build --platform linux/amd64 succeeds
Run patched image against real Gitea 1.26.2 / PostgreSQL 17.10 backend with 6 repos
Click Sync on a repo with 325 issues — observe 0 duplicates created (vs +602 dups on stock)
Click Sync on a repo with 1,381 issues + many PRs — observe 0 new duplicates created (vs ~1 dup/PR/sync on stock cron)
Confirm stock v3.16.0 still exhibits the deadlock+duplication on the same backend
Confirm Link-header pagination iterates all 9 pages of a 413-issue repo (vs page-1-only on stock)

🤖 Generated with Claude Code

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/RayLabsHQ/gitea-mirror/pull/296 **Author:** [@seanmousseau](https://github.com/seanmousseau) **Created:** 5/22/2026 **Status:** ✅ Merged **Merged:** 5/23/2026 **Merged by:** [@arunavo4](https://github.com/arunavo4) **Base:** `main` ← **Head:** `fix/dedup-create-on-deadlock-retry` --- ### 📝 Commits (1) - [`b76073b`](https://github.com/RayLabsHQ/gitea-mirror/commit/b76073bcf1baf0baa42492d58ee143121c4a7aa6) fix: prevent duplicate issues & PRs on retry-after-deadlock + Link-header pagination ### 📊 Changes **2 files changed** (+446 additions, -28 deletions) <details> <summary>View changed files</summary> ➕ `src/lib/gitea-issue-dedup-on-retry.test.ts` (+282 -0) 📝 `src/lib/gitea.ts` (+164 -28) </details> ### 📄 Description ## Summary Fixes two compounding bugs in `mirrorGitRepoIssuesToGitea` **and** `mirrorGitRepoPullRequestsToGitea` that produce duplicate Gitea issue/PR-as-issue rows on every sync against any non-SQLite backend. In a real-world test (Gitea 1.26.2 on PostgreSQL 17.10), an unpatched sync produced **+1,878 dups in 15 minutes** during initial repro and **+3,345 dups overnight** with hourly cron firing the bug repeatedly. With the patch, the same workload produces **0 new duplicates**. ## Bug #1 — pagination on existing-issues / existing-PRs / per-issue-comments Three paginated pre-fetches in `gitea.ts` paginated with `limit=100` and broke on `pageX.length < itemsPerPage`. Gitea caps response size at server-side `[api].MAX_RESPONSE_ITEMS` (default **50**), so the very first page already looks "short" and pagination terminated after one page. The existing-issue, existing-PR, and existing-comment maps were built from only ~50 items per repo, so every entry past page 1 was misclassified as new on every sync and re-created via the CREATE branch. Naive removal of the short-page break doesn't work either: for some Gitea endpoints (e.g. issue comments past the actual end) Gitea returns the **same data on every page** instead of returning `[]`, so the loop runs forever. **Fix:** use the `Link` header (RFC 5988). If `rel="next"` is absent in the response, terminate pagination. Applied to: existing-issues pre-fetch, existing-PRs pre-fetch, and per-issue comments fetch. The empty-page check is kept as a defensive early-out. ## Bug #2 — retry-after-deadlock duplicates issues and PRs Even when the dedup map is complete, a second class of duplication fires when Gitea's `CreateIssue` handler commits the issue insert in one transaction and then deadlocks on the subsequent `addLabel` / `UPDATE repository` transaction. The row is committed and visible, but the in-memory dedup map is never refreshed between retries. `processWithRetry` (default `maxRetries: 2`) re-invokes the per-item callback, sees `existingItem === undefined` from the stale map, and creates a fresh duplicate via `httpPost`. Reproduces deterministically on **MySQL** (`Error 1213 (40001)`) and **PostgreSQL** (`pq: deadlock detected (40P01)`). SQLite escapes because writes serialize globally. **Fix:** two guards inside each create branch: 1. **Defensive recheck before create.** Query Gitea by `[GH-ISSUE #N]` (issues) or `[PR #N]` (PRs) before `httpPost`. If found, switch to PATCH and cache the hit. 2. **Cache successful creates immediately.** Write the new item into the dedup map right after a successful `httpPost`. Applied to: the issues create path, the PR mirror's enriched create path, and the PR mirror's basic-fallback create path. Logs a `Recovered orphan from prior failed attempt for #N` line so operators can spot recovery in practice. ## Real-world evidence Same test environment, same gitea-mirror cron schedule: | Run | Retrying (deadlock) | Recovered orphan | Duplicates created (10h cron) | |---|---|---|---| | **Stock v3.16.0** | 27 in initial repro | n/a | **+3,345 dups overnight** (PRs duplicated once per cron tick) | | **Patched (all fixes)** | 0 | 0 (none needed!) | **0** | With the pagination fix the pre-fetch correctly enumerates all existing items via `Link: rel="next"`. Every GitHub iteration finds its match → routes to PATCH → no `httpPost` → no addLabel deadlock → no retry → no orphan to recover. The recheck-before-create is a defense-in-depth layer for genuinely-new items whose first create attempt deadlocks; on a re-sync it's essentially never invoked. ## Tests Adds `src/lib/gitea-issue-dedup-on-retry.test.ts` using the structural-source test pattern from `gitea-mirror-failure-recovery.test.ts` (issue #268's regression test). 8 assertions covering: - Per-item lookup against the dedup map (sanity guards against refactors) - Create-issue `httpPost` call still exists - Defensive recheck via `httpGet` using `[GH-ISSUE #N]` runs **before** the create - Recheck hits cache into the dedup map and route to PATCH with a recognisable log line - Existing-issues pagination loop uses **`Link` header (`rel=\"next\"`)** - Per-issue comments pagination loop uses the `Link` header - Successful creates cache the new item into the dedup map - **PR mirror function has the same guarantees**: Link-header pagination + ≥2 `Recovered orphan` log lines (one each for the enriched + basic-fallback create paths) `bun test` — 255 pass / 4 skip / 0 fail (up from 247 baseline). ## Related issues - **#262** — \"Issue's doubles when sync\" (closed by #266). #266's metadata guard prevents re-mirroring already-mirrored issues *across* syncs, but doesn't address either of the bugs in this PR. - **#157** — @emrebasarannn raised a similar \"issues are duplicate\" symptom that wasn't fully resolved. - **#268** — \"Imported repos stuck at mirroring\" (partial fix in v3.15.7 via #280). Different code path; not addressed here. ## Caveat / honest disclosure The test sync occasionally stalls partway through — sync produces some successful `Mirrored issue` log lines then goes quiet, while the job row sits at `in_progress=true, last_checkpoint=never` and the recovery detector logs \"Found 1 interrupted jobs\" repeatedly without ever invoking the recovery handler. **This stall is independent of this PR's changes** — reproduces identically on stock `ghcr.io/raylabshq/gitea-mirror:latest`, matches reports in #268's open comments, and lives entirely in orchestration code (`helpers.ts`, `recovery.ts`, `scheduler-service.ts`) that this PR does not touch. On the items that *do* process, the dedup guarantee in this PR holds: zero new duplicates. Worth a follow-up. ## Test plan - [x] `bun install && bun test` — all green (255/4/0) - [x] `docker buildx build --platform linux/amd64` succeeds - [x] Run patched image against real Gitea 1.26.2 / PostgreSQL 17.10 backend with 6 repos - [x] Click Sync on a repo with 325 issues — observe 0 duplicates created (vs +602 dups on stock) - [x] Click Sync on a repo with 1,381 issues + many PRs — observe 0 new duplicates created (vs ~1 dup/PR/sync on stock cron) - [x] Confirm stock v3.16.0 still exhibits the deadlock+duplication on the same backend - [x] Confirm Link-header pagination iterates all 9 pages of a 413-issue repo (vs page-1-only on stock) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

GiteaMirror added the pull-request label 2026-06-21 21:52:52 -05:00

GiteaMirror closed this issue

2026-06-21 21:52:53 -05:00

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/gitea-mirror#5968