Fixes#339 — repositories permanently stuck at "Syncing" after the process dies mid-sync (container restart, OOM kill, redeploy, host reboot). Reproduced exactly with a SIGKILL mid-sync.
Root cause
Status is written to syncing/mirroringbefore the work starts (src/lib/gitea-enhanced.ts:343, src/lib/gitea.ts:765, :1595). In-process failures reset it in catch blocks — a hard kill never reaches them, so the row stays syncing in SQLite.
Nothing ever reconciles it:
the scheduler only selects mirrored|synced|failed|pending repos — in-flight statuses are invisible forever
startup recovery only reconciles mirrorJobs rows, and scheduled syncs create no job records, so it early-exits on "No jobs need recovery"
cleanupStaleJobs marks stale jobs failed but never touches repositories.status
The UI disables Sync/Mirror buttons while the repo shows an in-flight status — no way out.
Fix
New src/lib/stuck-status-recovery.ts — resetStuckMirrorStatuses() resets repos stuck at mirroring/syncing (and orgs stuck at mirroring) to failed with an explanatory errorMessage, plus an activity entry and SSE event per row. A row counts as stuck only past max(process start, now − 2h) — the same 2h window isRepoCurrentlyMirroring already uses — so live in-process work is never clobbered. Never throws.
Wired in three places:
scheduler tick (every minute, before enabled-config filtering, so it heals even without scheduling enabled)
initializeRecovery() after cleanupStaleJobs()
scripts/startup-recovery.tsbefore the "no jobs need recovery" early exit (which previously skipped exactly the crashed-scheduled-sync case) — so stuck repos self-heal on container start
After reset, repos show failed + Retry in the UI and the scheduler's existing failed-repo handling picks them up. No schema change, no job-system redesign.
Real-DB e2e (fresh SQLite + real migrations, no mocks): only the 3 stuck rows reset; healthy + live rows untouched; second run is a no-op
Crash repro: mark repo syncing → SIGKILL → row stuck (bug reproduced) → run real scripts/startup-recovery.ts (the docker-entrypoint path) → healed to failed with explanation + SSE event
🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.
## 📋 Pull Request Information
**Original PR:** https://github.com/RayLabsHQ/gitea-mirror/pull/347
**Author:** [@arunavo4](https://github.com/arunavo4)
**Created:** 7/16/2026
**Status:** ✅ Merged
**Merged:** 7/16/2026
**Merged by:** [@arunavo4](https://github.com/arunavo4)
**Base:** `main` ← **Head:** `fix/339-stuck-syncing-status`
---
### 📝 Commits (1)
- [`6c86c90`](https://github.com/RayLabsHQ/gitea-mirror/commit/6c86c905fba2ece06050f44b64125f73156dc5db) fix: reset repositories stuck in 'syncing'/'mirroring' after interrupted runs (#339)
### 📊 Changes
**5 files changed** (+491 additions, -2 deletions)
<details>
<summary>View changed files</summary>
📝 `scripts/startup-recovery.ts` (+21 -1)
📝 `src/lib/recovery.ts` (+8 -0)
📝 `src/lib/scheduler-service.ts` (+14 -1)
➕ `src/lib/stuck-status-recovery.test.ts` (+197 -0)
➕ `src/lib/stuck-status-recovery.ts` (+251 -0)
</details>
### 📄 Description
## Summary
Fixes #339 — repositories permanently stuck at "Syncing" after the process dies mid-sync (container restart, OOM kill, redeploy, host reboot). Reproduced exactly with a SIGKILL mid-sync.
## Root cause
1. Status is written to `syncing`/`mirroring` **before** the work starts (`src/lib/gitea-enhanced.ts:343`, `src/lib/gitea.ts:765`, `:1595`). In-process failures reset it in catch blocks — a hard kill never reaches them, so the row stays `syncing` in SQLite.
2. Nothing ever reconciles it:
- the scheduler only selects `mirrored|synced|failed|pending` repos — in-flight statuses are invisible forever
- startup recovery only reconciles `mirrorJobs` rows, and scheduled syncs create **no** job records, so it early-exits on "No jobs need recovery"
- `cleanupStaleJobs` marks stale jobs failed but never touches `repositories.status`
3. The UI disables Sync/Mirror buttons while the repo shows an in-flight status — no way out.
## Fix
New `src/lib/stuck-status-recovery.ts` — `resetStuckMirrorStatuses()` resets repos stuck at `mirroring`/`syncing` (and orgs stuck at `mirroring`) to `failed` with an explanatory `errorMessage`, plus an activity entry and SSE event per row. A row counts as stuck only past `max(process start, now − 2h)` — the same 2h window `isRepoCurrentlyMirroring` already uses — so live in-process work is never clobbered. Never throws.
Wired in three places:
- **scheduler tick** (every minute, before enabled-config filtering, so it heals even without scheduling enabled)
- **`initializeRecovery()`** after `cleanupStaleJobs()`
- **`scripts/startup-recovery.ts`** *before* the "no jobs need recovery" early exit (which previously skipped exactly the crashed-scheduled-sync case) — so stuck repos self-heal on container start
After reset, repos show `failed` + Retry in the UI and the scheduler's existing failed-repo handling picks them up. No schema change, no job-system redesign.
## Testing
- `bun test`: 398 pass, 0 fail (17 new tests: cutoff computation, stuck detection incl. live-work protection, reset payloads, never-throws, wiring regression pins)
- Real-DB e2e (fresh SQLite + real migrations, no mocks): only the 3 stuck rows reset; healthy + live rows untouched; second run is a no-op
- Crash repro: mark repo `syncing` → SIGKILL → row stuck (bug reproduced) → run real `scripts/startup-recovery.ts` (the docker-entrypoint path) → healed to `failed` with explanation + SSE event
---
<sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
📋 Pull Request Information
Original PR: https://github.com/RayLabsHQ/gitea-mirror/pull/347
Author: @arunavo4
Created: 7/16/2026
Status: ✅ Merged
Merged: 7/16/2026
Merged by: @arunavo4
Base:
main← Head:fix/339-stuck-syncing-status📝 Commits (1)
6c86c90fix: reset repositories stuck in 'syncing'/'mirroring' after interrupted runs (#339)📊 Changes
5 files changed (+491 additions, -2 deletions)
View changed files
📝
scripts/startup-recovery.ts(+21 -1)📝
src/lib/recovery.ts(+8 -0)📝
src/lib/scheduler-service.ts(+14 -1)➕
src/lib/stuck-status-recovery.test.ts(+197 -0)➕
src/lib/stuck-status-recovery.ts(+251 -0)📄 Description
Summary
Fixes #339 — repositories permanently stuck at "Syncing" after the process dies mid-sync (container restart, OOM kill, redeploy, host reboot). Reproduced exactly with a SIGKILL mid-sync.
Root cause
syncing/mirroringbefore the work starts (src/lib/gitea-enhanced.ts:343,src/lib/gitea.ts:765,:1595). In-process failures reset it in catch blocks — a hard kill never reaches them, so the row stayssyncingin SQLite.mirrored|synced|failed|pendingrepos — in-flight statuses are invisible forevermirrorJobsrows, and scheduled syncs create no job records, so it early-exits on "No jobs need recovery"cleanupStaleJobsmarks stale jobs failed but never touchesrepositories.statusFix
New
src/lib/stuck-status-recovery.ts—resetStuckMirrorStatuses()resets repos stuck atmirroring/syncing(and orgs stuck atmirroring) tofailedwith an explanatoryerrorMessage, plus an activity entry and SSE event per row. A row counts as stuck only pastmax(process start, now − 2h)— the same 2h windowisRepoCurrentlyMirroringalready uses — so live in-process work is never clobbered. Never throws.Wired in three places:
initializeRecovery()aftercleanupStaleJobs()scripts/startup-recovery.tsbefore the "no jobs need recovery" early exit (which previously skipped exactly the crashed-scheduled-sync case) — so stuck repos self-heal on container startAfter reset, repos show
failed+ Retry in the UI and the scheduler's existing failed-repo handling picks them up. No schema change, no job-system redesign.Testing
bun test: 398 pass, 0 fail (17 new tests: cutoff computation, stuck detection incl. live-work protection, reset payloads, never-throws, wiring regression pins)syncing→ SIGKILL → row stuck (bug reproduced) → run realscripts/startup-recovery.ts(the docker-entrypoint path) → healed tofailedwith explanation + SSE event🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.