[PR #347] [MERGED] fix: recover repositories stuck in syncing/mirroring after crashes (#339) #8264

Closed
opened 2026-08-24 04:01:27 -05:00 by GiteaMirror · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/RayLabsHQ/gitea-mirror/pull/347
Author: @arunavo4
Created: 7/16/2026
Status: ✅ Merged
Merged: 7/16/2026
Merged by: @arunavo4

Base: main ← Head: fix/339-stuck-syncing-status


📝 Commits (1)

  • 6c86c90 fix: reset repositories stuck in 'syncing'/'mirroring' after interrupted runs (#339)

📊 Changes

5 files changed (+491 additions, -2 deletions)

View changed files

📝 scripts/startup-recovery.ts (+21 -1)
📝 src/lib/recovery.ts (+8 -0)
📝 src/lib/scheduler-service.ts (+14 -1)
➕ src/lib/stuck-status-recovery.test.ts (+197 -0)
➕ src/lib/stuck-status-recovery.ts (+251 -0)

📄 Description

Summary

Fixes #339 — repositories permanently stuck at "Syncing" after the process dies mid-sync (container restart, OOM kill, redeploy, host reboot). Reproduced exactly with a SIGKILL mid-sync.

Root cause

  1. Status is written to syncing/mirroring before the work starts (src/lib/gitea-enhanced.ts:343, src/lib/gitea.ts:765, :1595). In-process failures reset it in catch blocks — a hard kill never reaches them, so the row stays syncing in SQLite.
  2. Nothing ever reconciles it:
    • the scheduler only selects mirrored|synced|failed|pending repos — in-flight statuses are invisible forever
    • startup recovery only reconciles mirrorJobs rows, and scheduled syncs create no job records, so it early-exits on "No jobs need recovery"
    • cleanupStaleJobs marks stale jobs failed but never touches repositories.status
  3. The UI disables Sync/Mirror buttons while the repo shows an in-flight status — no way out.

Fix

New src/lib/stuck-status-recovery.ts — resetStuckMirrorStatuses() resets repos stuck at mirroring/syncing (and orgs stuck at mirroring) to failed with an explanatory errorMessage, plus an activity entry and SSE event per row. A row counts as stuck only past max(process start, now − 2h) — the same 2h window isRepoCurrentlyMirroring already uses — so live in-process work is never clobbered. Never throws.

Wired in three places:

  • scheduler tick (every minute, before enabled-config filtering, so it heals even without scheduling enabled)
  • initializeRecovery() after cleanupStaleJobs()
  • scripts/startup-recovery.ts before the "no jobs need recovery" early exit (which previously skipped exactly the crashed-scheduled-sync case) — so stuck repos self-heal on container start

After reset, repos show failed + Retry in the UI and the scheduler's existing failed-repo handling picks them up. No schema change, no job-system redesign.

Testing

  • bun test: 398 pass, 0 fail (17 new tests: cutoff computation, stuck detection incl. live-work protection, reset payloads, never-throws, wiring regression pins)
  • Real-DB e2e (fresh SQLite + real migrations, no mocks): only the 3 stuck rows reset; healthy + live rows untouched; second run is a no-op
  • Crash repro: mark repo syncing → SIGKILL → row stuck (bug reproduced) → run real scripts/startup-recovery.ts (the docker-entrypoint path) → healed to failed with explanation + SSE event

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/RayLabsHQ/gitea-mirror/pull/347 **Author:** [@arunavo4](https://github.com/arunavo4) **Created:** 7/16/2026 **Status:** ✅ Merged **Merged:** 7/16/2026 **Merged by:** [@arunavo4](https://github.com/arunavo4) **Base:** `main` ← **Head:** `fix/339-stuck-syncing-status` --- ### 📝 Commits (1) - [`6c86c90`](https://github.com/RayLabsHQ/gitea-mirror/commit/6c86c905fba2ece06050f44b64125f73156dc5db) fix: reset repositories stuck in 'syncing'/'mirroring' after interrupted runs (#339) ### 📊 Changes **5 files changed** (+491 additions, -2 deletions) <details> <summary>View changed files</summary> 📝 `scripts/startup-recovery.ts` (+21 -1) 📝 `src/lib/recovery.ts` (+8 -0) 📝 `src/lib/scheduler-service.ts` (+14 -1) ➕ `src/lib/stuck-status-recovery.test.ts` (+197 -0) ➕ `src/lib/stuck-status-recovery.ts` (+251 -0) </details> ### 📄 Description ## Summary Fixes #339 — repositories permanently stuck at "Syncing" after the process dies mid-sync (container restart, OOM kill, redeploy, host reboot). Reproduced exactly with a SIGKILL mid-sync. ## Root cause 1. Status is written to `syncing`/`mirroring` **before** the work starts (`src/lib/gitea-enhanced.ts:343`, `src/lib/gitea.ts:765`, `:1595`). In-process failures reset it in catch blocks — a hard kill never reaches them, so the row stays `syncing` in SQLite. 2. Nothing ever reconciles it: - the scheduler only selects `mirrored|synced|failed|pending` repos — in-flight statuses are invisible forever - startup recovery only reconciles `mirrorJobs` rows, and scheduled syncs create **no** job records, so it early-exits on "No jobs need recovery" - `cleanupStaleJobs` marks stale jobs failed but never touches `repositories.status` 3. The UI disables Sync/Mirror buttons while the repo shows an in-flight status — no way out. ## Fix New `src/lib/stuck-status-recovery.ts` — `resetStuckMirrorStatuses()` resets repos stuck at `mirroring`/`syncing` (and orgs stuck at `mirroring`) to `failed` with an explanatory `errorMessage`, plus an activity entry and SSE event per row. A row counts as stuck only past `max(process start, now − 2h)` — the same 2h window `isRepoCurrentlyMirroring` already uses — so live in-process work is never clobbered. Never throws. Wired in three places: - **scheduler tick** (every minute, before enabled-config filtering, so it heals even without scheduling enabled) - **`initializeRecovery()`** after `cleanupStaleJobs()` - **`scripts/startup-recovery.ts`** *before* the "no jobs need recovery" early exit (which previously skipped exactly the crashed-scheduled-sync case) — so stuck repos self-heal on container start After reset, repos show `failed` + Retry in the UI and the scheduler's existing failed-repo handling picks them up. No schema change, no job-system redesign. ## Testing - `bun test`: 398 pass, 0 fail (17 new tests: cutoff computation, stuck detection incl. live-work protection, reset payloads, never-throws, wiring regression pins) - Real-DB e2e (fresh SQLite + real migrations, no mocks): only the 3 stuck rows reset; healthy + live rows untouched; second run is a no-op - Crash repro: mark repo `syncing` → SIGKILL → row stuck (bug reproduced) → run real `scripts/startup-recovery.ts` (the docker-entrypoint path) → healed to `failed` with explanation + SSE event --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
GiteaMirror added the pull-request label 2026-08-24 04:01:27 -05:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/gitea-mirror#8264