Stops mirrorJobs rows from getting permanently stuck at inProgress=true, lastCheckpoint=never when a sync fails mid-flight after server boot. Found while debugging a separate dedup issue (#296) and observing endless Found 1 interrupted jobs: log spam with no recovery action ever taking place.
The bug
The middleware-level recovery trigger looked like this:
letrecoveryInitialized=false;letrecoveryAttempted=false;// ...
if(!recoveryInitialized&&!recoveryAttempted){// ← once per process, ever
recoveryAttempted=true;if(needsRecovery)initializeRecovery(...);recoveryInitialized=true;}
Both flags get set on the first request. After that, recovery never re-engages — even if new interruptions appear later in the process lifetime.
So the failure mode is:
Server boots cleanly. Middleware runs initial recovery pass on the first request. No stuck jobs. Both flags flip to true.
User triggers a sync. Job row is created with inProgress=true.
The sync crashes mid-flight (deadlock retry hitting maxRetries, network blip, container restart of a dependency, bun runtime quirk — anything that exits the per-item callback without reaching the resume/finalize codepath).
findInterruptedJobs (called periodically from the health endpoint via hasJobsNeedingRecovery) detects it and logs Found 1 interrupted jobs: on every poll.
But resumeInterruptedJob is only invoked from inside initializeRecovery, which the middleware gate prevents from re-running. The job is stuck forever.
The detector kept finding the row but the resumer never re-fired, producing the symptom: a mirroring-state repo in the UI plus log spam on every health-check tick.
The fix
Middleware (primary fix)
Replace the one-shot gate with an in-flight latch released in a finally block. The actual "don't thrash" guarantee is delegated to the existing 5-minute skipIfRecentAttempt throttle inside initializeRecovery(), which is the right place for it:
recoveryInitialized is kept and only used to control whether the "first run" log lines fire (so the log doesn't say "startup script may not have run" on every request).
findInterruptedJobs logging (secondary fix)
It previously logged Found N interrupted jobs: unconditionally on every call, including from passive polls (hasJobsNeedingRecovery from the health endpoint and middleware). That produced one log line per poll per stuck job — minutes of log spam for a single stuck job. Made it opt-in:
The active recovery cycle in initializeRecovery() passes { logFound: true } so operators still see which jobs are being worked on.
Tests
Adds src/lib/orchestrator-resume-after-startup.test.ts using the structural-source test pattern from gitea-mirror-failure-recovery.test.ts (issue #268's regression test). 4 assertions:
Middleware no longer references the once-per-process recoveryAttempted flag
Middleware uses an in-flight latch (recoveryInFlight) released in a finally block (so an exception during recovery doesn't permanently jam the latch)
findInterruptedJobs accepts a { logFound } option that defaults to false
The active recovery path in initializeRecovery() explicitly opts in with { logFound: true }
bun test — 251 pass / 4 skip / 0 fail.
Related
#268 — partially addressed. The PR #280 / v3.15.7 fix was about a JS scoping bug that made the initial migrate call's catch path crash before transitioning the repo to failed. That was one cause of stuck mirroring state; this PR addresses the orthogonal issue that even when a job is correctly detectable as interrupted, the post-startup recovery path never re-engages.
#296 — companion PR fixing dedup-on-retry. Independent code paths; ship either independently. The duplication symptom in #296 is what made the "stuck job" log spam visible in our test environment, but the two bugs are unrelated.
Test plan
bun install && bun test — all green (251/4/0)
Confirmed the symptom on stock v3.16.0: Found 1 interrupted jobs: repeating with no recovery action
Confirmed the patched build re-engages recovery on subsequent requests when a runtime-interrupted job is detected
🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.
## 📋 Pull Request Information
**Original PR:** https://github.com/RayLabsHQ/gitea-mirror/pull/297
**Author:** [@seanmousseau](https://github.com/seanmousseau)
**Created:** 5/22/2026
**Status:** ✅ Merged
**Merged:** 5/23/2026
**Merged by:** [@arunavo4](https://github.com/arunavo4)
**Base:** `main` ← **Head:** `fix/orchestrator-resume-after-startup`
---
### 📝 Commits (1)
- [`a5563e1`](https://github.com/RayLabsHQ/gitea-mirror/commit/a5563e1300e13e51581bdc9fdddae78d7092ea29) fix: resume interrupted jobs after startup (not only at boot)
### 📊 Changes
**4 files changed** (+181 additions, -15 deletions)
<details>
<summary>View changed files</summary>
📝 `src/lib/helpers.ts` (+17 -4)
➕ `src/lib/orchestrator-resume-after-startup.test.ts` (+128 -0)
📝 `src/lib/recovery.ts` (+3 -2)
📝 `src/middleware.ts` (+33 -9)
</details>
### 📄 Description
## Summary
Stops `mirrorJobs` rows from getting permanently stuck at `inProgress=true, lastCheckpoint=never` when a sync fails mid-flight after server boot. Found while debugging a separate dedup issue (#296) and observing endless `Found 1 interrupted jobs:` log spam with no recovery action ever taking place.
## The bug
The middleware-level recovery trigger looked like this:
```ts
let recoveryInitialized = false;
let recoveryAttempted = false;
// ...
if (!recoveryInitialized && !recoveryAttempted) { // ← once per process, ever
recoveryAttempted = true;
if (needsRecovery) initializeRecovery(...);
recoveryInitialized = true;
}
```
Both flags get set on the first request. After that, recovery never re-engages — even if new interruptions appear later in the process lifetime.
So the failure mode is:
1. Server boots cleanly. Middleware runs initial recovery pass on the first request. No stuck jobs. Both flags flip to true.
2. User triggers a sync. Job row is created with `inProgress=true`.
3. The sync crashes mid-flight (deadlock retry hitting `maxRetries`, network blip, container restart of a dependency, `bun` runtime quirk — anything that exits the per-item callback without reaching the resume/finalize codepath).
4. `findInterruptedJobs` (called periodically from the health endpoint via `hasJobsNeedingRecovery`) detects it and logs `Found 1 interrupted jobs:` on every poll.
5. But `resumeInterruptedJob` is only invoked from inside `initializeRecovery`, which the middleware gate prevents from re-running. **The job is stuck forever.**
The detector kept finding the row but the resumer never re-fired, producing the symptom: a `mirroring`-state repo in the UI plus log spam on every health-check tick.
## The fix
### Middleware (primary fix)
Replace the one-shot gate with an in-flight latch released in a `finally` block. The actual "don't thrash" guarantee is delegated to the existing 5-minute `skipIfRecentAttempt` throttle inside `initializeRecovery()`, which is the right place for it:
```ts
let recoveryInFlight = false; // per-process mutex
// ...
if (!recoveryInFlight) {
recoveryInFlight = true;
try {
if (await hasJobsNeedingRecovery()) {
await initializeRecovery({ skipIfRecentAttempt: true, ... });
}
} finally {
recoveryInFlight = false;
}
}
```
`recoveryInitialized` is kept and only used to control whether the "first run" log lines fire (so the log doesn't say "startup script may not have run" on every request).
### `findInterruptedJobs` logging (secondary fix)
It previously logged `Found N interrupted jobs:` unconditionally on every call, including from passive polls (`hasJobsNeedingRecovery` from the health endpoint and middleware). That produced one log line per poll per stuck job — minutes of log spam for a single stuck job. Made it opt-in:
```ts
export async function findInterruptedJobs(options: { logFound?: boolean } = {}) {
const { logFound = false } = options;
// ... detect ...
if (logFound && interruptedJobs.length > 0) { /* log per-job details */ }
return interruptedJobs;
}
```
The active recovery cycle in `initializeRecovery()` passes `{ logFound: true }` so operators still see which jobs are being worked on.
## Tests
Adds `src/lib/orchestrator-resume-after-startup.test.ts` using the structural-source test pattern from `gitea-mirror-failure-recovery.test.ts` (issue #268's regression test). 4 assertions:
- Middleware no longer references the once-per-process `recoveryAttempted` flag
- Middleware uses an in-flight latch (`recoveryInFlight`) released in a `finally` block (so an exception during recovery doesn't permanently jam the latch)
- `findInterruptedJobs` accepts a `{ logFound }` option that defaults to `false`
- The active recovery path in `initializeRecovery()` explicitly opts in with `{ logFound: true }`
`bun test` — 251 pass / 4 skip / 0 fail.
## Related
- **#268** — partially addressed. The PR #280 / v3.15.7 fix was about a JS scoping bug that made the *initial* migrate call's catch path crash before transitioning the repo to `failed`. That was one cause of stuck `mirroring` state; this PR addresses the orthogonal issue that even when a job *is* correctly detectable as interrupted, the post-startup recovery path never re-engages.
- **#296** — companion PR fixing dedup-on-retry. Independent code paths; ship either independently. The duplication symptom in #296 is what made the "stuck job" log spam visible in our test environment, but the two bugs are unrelated.
## Test plan
- [x] `bun install && bun test` — all green (251/4/0)
- [x] Confirmed the symptom on stock v3.16.0: `Found 1 interrupted jobs:` repeating with no recovery action
- [x] Confirmed the patched build re-engages recovery on subsequent requests when a runtime-interrupted job is detected
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---
<sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
📋 Pull Request Information
Original PR: https://github.com/RayLabsHQ/gitea-mirror/pull/297
Author: @seanmousseau
Created: 5/22/2026
Status: ✅ Merged
Merged: 5/23/2026
Merged by: @arunavo4
Base:
main← Head:fix/orchestrator-resume-after-startup📝 Commits (1)
a5563e1fix: resume interrupted jobs after startup (not only at boot)📊 Changes
4 files changed (+181 additions, -15 deletions)
View changed files
📝
src/lib/helpers.ts(+17 -4)➕
src/lib/orchestrator-resume-after-startup.test.ts(+128 -0)📝
src/lib/recovery.ts(+3 -2)📝
src/middleware.ts(+33 -9)📄 Description
Summary
Stops
mirrorJobsrows from getting permanently stuck atinProgress=true, lastCheckpoint=neverwhen a sync fails mid-flight after server boot. Found while debugging a separate dedup issue (#296) and observing endlessFound 1 interrupted jobs:log spam with no recovery action ever taking place.The bug
The middleware-level recovery trigger looked like this:
Both flags get set on the first request. After that, recovery never re-engages — even if new interruptions appear later in the process lifetime.
So the failure mode is:
inProgress=true.maxRetries, network blip, container restart of a dependency,bunruntime quirk — anything that exits the per-item callback without reaching the resume/finalize codepath).findInterruptedJobs(called periodically from the health endpoint viahasJobsNeedingRecovery) detects it and logsFound 1 interrupted jobs:on every poll.resumeInterruptedJobis only invoked from insideinitializeRecovery, which the middleware gate prevents from re-running. The job is stuck forever.The detector kept finding the row but the resumer never re-fired, producing the symptom: a
mirroring-state repo in the UI plus log spam on every health-check tick.The fix
Middleware (primary fix)
Replace the one-shot gate with an in-flight latch released in a
finallyblock. The actual "don't thrash" guarantee is delegated to the existing 5-minuteskipIfRecentAttemptthrottle insideinitializeRecovery(), which is the right place for it:recoveryInitializedis kept and only used to control whether the "first run" log lines fire (so the log doesn't say "startup script may not have run" on every request).findInterruptedJobslogging (secondary fix)It previously logged
Found N interrupted jobs:unconditionally on every call, including from passive polls (hasJobsNeedingRecoveryfrom the health endpoint and middleware). That produced one log line per poll per stuck job — minutes of log spam for a single stuck job. Made it opt-in:The active recovery cycle in
initializeRecovery()passes{ logFound: true }so operators still see which jobs are being worked on.Tests
Adds
src/lib/orchestrator-resume-after-startup.test.tsusing the structural-source test pattern fromgitea-mirror-failure-recovery.test.ts(issue #268's regression test). 4 assertions:recoveryAttemptedflagrecoveryInFlight) released in afinallyblock (so an exception during recovery doesn't permanently jam the latch)findInterruptedJobsaccepts a{ logFound }option that defaults tofalseinitializeRecovery()explicitly opts in with{ logFound: true }bun test— 251 pass / 4 skip / 0 fail.Related
failed. That was one cause of stuckmirroringstate; this PR addresses the orthogonal issue that even when a job is correctly detectable as interrupted, the post-startup recovery path never re-engages.Test plan
bun install && bun test— all green (251/4/0)Found 1 interrupted jobs:repeating with no recovery action🤖 Generated with Claude Code
🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.