Broken queue causes various problems, need to report in self-check #14253

New Issue

GiteaMirror · 2025-11-02T11:07:37-06:00

GiteaMirror commented

2025-11-02 11:07:37 -06:00

Originally created by @joshrnoll on GitHub (Mar 14, 2025).

Description

Sometimes, the pull/push mirrors stopped synchronizing.

Gitea Version

All

(Edited by wxiaoguang: only keep the problem description, the workaround is in the following comment)

Originally created by @joshrnoll on GitHub (Mar 14, 2025). ### Description Sometimes, the pull/push mirrors stopped synchronizing. ### Gitea Version All ---- (Edited by wxiaoguang: only keep the problem description, the workaround is in the following comment)

GiteaMirror added the type/bug issue/workaround labels 2025-11-02 11:07:37 -06:00

GiteaMirror commented

2025-11-02 11:07:38 -06:00

@wxiaoguang commented on GitHub (Mar 14, 2025):

Maybe this could help your case: https://github.com/go-gitea/gitea/issues/33862#issuecomment-2717931247

Does the queue system work? Check it from admin panel -> monitor -> queue /-/admin/monitor/queue
If the queue system gets stuck, you need to click "Remove All" to clean the corrupted data

@wxiaoguang commented on GitHub (Mar 14, 2025): Maybe this could help your case: https://github.com/go-gitea/gitea/issues/33862#issuecomment-2717931247 > Does the queue system work? Check it from admin panel -> monitor -> queue `/-/admin/monitor/queue` > If the queue system gets stuck, you need to click "Remove All" to clean the corrupted data

GiteaMirror commented

2025-11-02 11:07:38 -06:00

@joshrnoll commented on GitHub (Mar 14, 2025):

This fixed it!

For those who stumble on this in the future:

Go to Site Administration --> Monitoring --> Queues

Click on Review / Add Workers next to the mirror queue, then click Remove all (I had 9 in the queue before clearing)

After doing this I initially received a permission denied error, but this was because I had deleted and recreated my remote repo without recreating the PAT:

Once I recreated my PAT, I got a successful sync again.

@wxiaoguang thanks for the help! Is this a bug? Should I leave this issue open or go ahead and close?

@joshrnoll commented on GitHub (Mar 14, 2025): This fixed it! For those who stumble on this in the future: Go to *Site Administration* --> *Monitoring* --> *Queues* Click on *Review / Add Workers* next to the **mirror** queue, then click *Remove all* (I had 9 in the queue before clearing) ![Image](https://github.com/user-attachments/assets/aaeeeb96-0bbf-4cea-ad48-4d786ef8818b) After doing this I initially received a permission denied error, but this was because I had deleted and recreated my remote repo without recreating the PAT: ![Image](https://github.com/user-attachments/assets/c04ba629-fb02-4f4a-affc-b941ca66226e) Once I recreated my PAT, I got a successful sync again. @wxiaoguang thanks for the help! Is this a bug? Should I leave this issue open or go ahead and close?

GiteaMirror commented

2025-11-02 11:07:39 -06:00

@wxiaoguang commented on GitHub (Mar 14, 2025):

It is a bug (explained here https://github.com/go-gitea/gitea/issues/33862#issuecomment-2717987150)

But it's difficult to fix.

The best approach we could do at the moment is to add the "queue check" to "self check" page to remind users .....

So I think we can leave this open until we have a good "self check" prompt

@wxiaoguang commented on GitHub (Mar 14, 2025): It is a bug (explained here https://github.com/go-gitea/gitea/issues/33862#issuecomment-2717987150) But it's difficult to fix. The best approach we could do at the moment is to add the "queue check" to "self check" page to remind users ..... So I think we can leave this open until we have a good "self check" prompt

GiteaMirror commented

2025-11-02 11:07:39 -06:00

@wxiaoguang commented on GitHub (Mar 14, 2025):

By the way, the corruption only occurs in "levelqueue". So you could switch to "redis" queue which won't get stuck 😆

@wxiaoguang commented on GitHub (Mar 14, 2025): By the way, the corruption only occurs in "levelqueue". So you could switch to "redis" queue which won't get stuck 😆

GiteaMirror commented

2025-11-02 11:07:39 -06:00

@TheFox0x7 commented on GitHub (Mar 14, 2025):

Would it be possible to replace levelqueue with something that's less prone to failing?

I've been eyeing nats but it's a bit harder to integrate it into queues (with persistence that is) than I originally thought, however it can be embedded into the binary which is why I find worth considering.

@TheFox0x7 commented on GitHub (Mar 14, 2025): Would it be possible to replace levelqueue with something that's less prone to failing? I've been eyeing nats but it's a bit harder to integrate it into queues (with persistence that is) than I originally thought, however it can be embedded into the binary which is why I find worth considering.

GiteaMirror commented

2025-11-02 11:07:39 -06:00

@wxiaoguang commented on GitHub (Mar 14, 2025):

Would it be possible to replace levelqueue with something that's less prone to failing?

No idea at the moment. Gitea's queue has some special behaviors like "unique queue". IMO making levelqueue could self-heal is a quick approach.

@wxiaoguang commented on GitHub (Mar 14, 2025): > Would it be possible to replace levelqueue with something that's less prone to failing? No idea at the moment. Gitea's queue has some special behaviors like "unique queue". IMO making levelqueue could self-heal is a quick approach.

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/gitea#14253