mirror of
https://github.com/go-gitea/gitea.git
synced 2026-07-15 20:09:18 -05:00
No worker instances after upgrade #8321
Closed
opened 2025-11-02 08:02:17 -06:00 by GiteaMirror
·
12 comments
No Branch/Tag Specified
main
release/v1.25
release/v1.24
release/v1.23
release/v1.22
release/v1.21
release/v1.20
release/v1.19
release/v1.18
release/v1.17
release/v1.16
release/v1.15
release/v1.14
release/v1.13
release/v1.12
release/v1.11
release/v1.10
release/v1.9
release/v1.8
v1.25.3
v1.25.2
v1.25.1
v1.25.0
v1.24.7
v1.25.0-rc0
v1.26.0-dev
v1.24.6
v1.24.5
v1.24.4
v1.24.3
v1.24.2
v1.24.1
v1.24.0
v1.23.8
v1.24.0-rc0
v1.25.0-dev
v1.23.7
v1.23.6
v1.23.5
v1.23.4
v1.23.3
v1.23.2
v1.23.1
v1.23.0
v1.23.0-rc0
v1.24.0-dev
v1.22.6
v1.22.5
v1.22.4
v1.22.3
v1.22.2
v1.22.1
v1.22.0
v1.23.0-dev
v1.22.0-rc1
v1.21.11
v1.22.0-rc0
v1.21.10
v1.21.9
v1.21.8
v1.21.7
v1.21.6
v1.21.5
v1.21.4
v1.21.3
v1.21.2
v1.20.6
v1.21.1
v1.21.0
v1.21.0-rc2
v1.21.0-rc1
v1.20.5
v1.22.0-dev
v1.21.0-rc0
v1.20.4
v1.20.3
v1.20.2
v1.20.1
v1.20.0
v1.19.4
v1.21.0-dev
v1.20.0-rc2
v1.20.0-rc1
v1.20.0-rc0
v1.19.3
v1.19.2
v1.19.1
v1.19.0
v1.19.0-rc1
v1.20.0-dev
v1.19.0-rc0
v1.18.5
v1.18.4
v1.18.3
v1.18.2
v1.18.1
v1.18.0
v1.17.4
v1.18.0-rc1
v1.19.0-dev
v1.18.0-rc0
v1.17.3
v1.17.2
v1.17.1
v1.17.0
v1.17.0-rc2
v1.16.9
v1.17.0-rc1
v1.18.0-dev
v1.16.8
v1.16.7
v1.16.6
v1.16.5
v1.16.4
v1.16.3
v1.16.2
v1.16.1
v1.16.0
v1.15.11
v1.17.0-dev
v1.16.0-rc1
v1.15.10
v1.15.9
v1.15.8
v1.15.7
v1.15.6
v1.15.5
v1.15.4
v1.15.3
v1.15.2
v1.15.1
v1.14.7
v1.15.0
v1.15.0-rc3
v1.14.6
v1.15.0-rc2
v1.14.5
v1.16.0-dev
v1.15.0-rc1
v1.14.4
v1.14.3
v1.14.2
v1.14.1
v1.14.0
v1.13.7
v1.14.0-rc2
v1.13.6
v1.13.5
v1.14.0-rc1
v1.15.0-dev
v1.13.4
v1.13.3
v1.13.2
v1.13.1
v1.13.0
v1.12.6
v1.13.0-rc2
v1.14.0-dev
v1.13.0-rc1
v1.12.5
v1.12.4
v1.12.3
v1.12.2
v1.12.1
v1.11.8
v1.12.0
v1.11.7
v1.12.0-rc2
v1.11.6
v1.12.0-rc1
v1.13.0-dev
v1.11.5
v1.11.4
v1.11.3
v1.10.6
v1.12.0-dev
v1.11.2
v1.10.5
v1.11.1
v1.10.4
v1.11.0
v1.11.0-rc2
v1.10.3
v1.11.0-rc1
v1.10.2
v1.10.1
v1.10.0
v1.9.6
v1.9.5
v1.10.0-rc2
v1.11.0-dev
v1.10.0-rc1
v1.9.4
v1.9.3
v1.9.2
v1.9.1
v1.9.0
v1.9.0-rc2
v1.10.0-dev
v1.9.0-rc1
v1.8.3
v1.8.2
v1.8.1
v1.8.0
v1.8.0-rc3
v1.7.6
v1.8.0-rc2
v1.7.5
v1.8.0-rc1
v1.9.0-dev
v1.7.4
v1.7.3
v1.7.2
v1.7.1
v1.7.0
v1.7.0-rc3
v1.6.4
v1.7.0-rc2
v1.6.3
v1.7.0-rc1
v1.7.0-dev
v1.6.2
v1.6.1
v1.6.0
v1.6.0-rc2
v1.5.3
v1.6.0-rc1
v1.6.0-dev
v1.5.2
v1.5.1
v1.5.0
v1.5.0-rc2
v1.5.0-rc1
v1.5.0-dev
v1.4.3
v1.4.2
v1.4.1
v1.4.0
v1.4.0-rc3
v1.4.0-rc2
v1.3.3
v1.4.0-rc1
v1.3.2
v1.3.1
v1.3.0
v1.3.0-rc2
v1.3.0-rc1
v1.2.3
v1.2.2
v1.2.1
v1.2.0
v1.2.0-rc3
v1.2.0-rc2
v1.1.4
v1.2.0-rc1
v1.1.3
v1.1.2
v1.1.1
v1.1.0
v1.0.2
v1.0.1
v1.0.0
v0.9.99
Labels
Clear labels
$20
$250
$50
$500
backport/done
💎 Bounty
docs-update-needed
good first issue
hacktoberfest
issue/bounty
issue/confirmed
issue/critical
issue/duplicate
issue/needs-feedback
issue/not-a-bug
issue/regression
issue/stale
issue/workaround
lgtm/need 2
modifies/api
modifies/translation
outdated/backport/v1.18
outdated/theme/markdown
outdated/theme/timetracker
performance/bigrepo
performance/cpu
performance/memory
performance/speed
pr/breaking
proposal/accepted
proposal/rejected
pr/wip
pull-request
reviewed/wontfix
💰 Rewarded
skip-changelog
status/blocked
topic/accessibility
topic/api
topic/authentication
topic/build
topic/code-linting
topic/commit-signing
topic/content-rendering
topic/deployment
topic/distribution
topic/federation
topic/gitea-actions
topic/issues
topic/lfs
topic/mobile
topic/moderation
topic/packages
topic/pr
topic/projects
topic/repo
topic/repo-migration
topic/security
topic/theme
topic/ui
topic/ui-interaction
topic/ux
topic/webhooks
topic/wiki
type/bug
type/deprecation
type/docs
type/enhancement
type/feature
type/miscellaneous
type/proposal
type/question
type/refactoring
type/summary
type/testing
type/upstream
Mirrored from GitHub Pull Request
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: github-starred/gitea#8321
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @penguineer on GitHub (Jan 5, 2022).
Gitea Version
1.15.9
Git Version
2.30.2 (from container)
Operating System
Alpine 3.13 (from container)
How are you running Gitea?
I'm using the proposed docker-compose configuraiton.
Initial version: 1.12 (.3?)
Upgrade to: 1.15.7
Database
SQLite
Can you reproduce the bug on the Gitea demo site?
No
Log Gist
No response
Description
After upgrading Gitea to 1.15.9 everything seemed well, but over time we experienced weird effects:
A restart of the container would now and then fix one of the issues, but only momentarily. I would see log messages like:
2022/01/04 19:10:48 ...ue/queue_bytefifo.go:224:doPop() [E] level: task-level Failed to unmarshal with error: readObjectStart: expect { or n, but found [, error found in #1 byte of ...|[{"PusherID|..., bigger context ...|[{"PusherID":1,"PusherName":"t","RepoUserName":"t|... 2022/01/04 19:10:48 ...ue/queue_bytefifo.go:224:doPop() [E] level: task-level Failed to unmarshal with error: readObjectStart: expect { or n, but found ", error found in #1 byte of ...|"429"|..., bigger context ...|"429"|...These, however, disappeared after this one instance.
All these problems do not occur on a fresh instance, but we want to keep our data. (...)
Unfortunately we could also not just go back, because we realized the problems too late for any of the Backups to be viable.
I did the following during debugging:
recreate-table) and fix warnings on all checksIn the end I realized that there are no workers (Site Administration → Monitoring). My initial guess was that these workers are instantiated automatically of there are tasks in the queue. Later I checked the queue configurations and clicked "Add Flush Workers" with three findings:
As a workaround I have added a group with one worker to each Queue. I could not find these worker groups in the database though.
There is still the problem described in #17204, but it has not happened on new PRs yet.
Is there a problem during upgrade that leads to a situation where no workers exist?
Screenshots
No response
@zeripath commented on GitHub (Jan 5, 2022):
I think you need to give us more logs.
From the tiny amount of logging you have given it looks like the problem lies with there being something invalid in your task level queue, now why that would be the case I am uncertain.
I also need to see your app.ini to check your queue configuration.
@penguineer commented on GitHub (Jan 5, 2022):
app.inilooks like this: (values in <> have been replaces)@penguineer commented on GitHub (Jan 5, 2022):
Unfortunately the logs contain sensitive information (Employee and Customer names are part of repositories). There are no warnings or errors in the logs. The error I've quoted above happened once and then vanished.
The logs from the upgrade are already gone through the rotation. :(
I'm happy to post logs for specific actions, if that helps.
@penguineer commented on GitHub (Jan 5, 2022):
After adding a worker manually the Configuration for the queue

push_update-channelnow looks like this:It looks like this for every queue.
@zeripath commented on GitHub (Jan 5, 2022):
Why have you set:
?
Does removing this help?
@zeripath commented on GitHub (Jan 5, 2022):
The reasoning as to why this might be the cause of the problem is that on 1.15 Gitea does not start with any workers by default:
With BLOCK_TIMEOUT=0 no blocking of the queue will be detected and thus no boosting will occur.
This is mentioned in the breaking notes for Gitea 1.15.0:
https://blog.gitea.io/2021/08/gitea-1.15.0-is-released/#-change-default-queue-settings-to-be-low-go-routines-15964httpsgithubcomgo-giteagiteapull15964
@penguineer commented on GitHub (Jan 5, 2022):
Most of the configuration has been generated by gitea. Could anything have triggered this setting?
It actually does.
From the documentation it seems that this inhibited the creation of new workers.
Maybe it makes sense to log a warning when this setting is detected?
@penguineer commented on GitHub (Jan 5, 2022):
Thanks for pointing this out!
However, I would not have realized this even when reading this change log.
@penguineer commented on GitHub (Jan 5, 2022):
@zeripath Thank you for the quick help! This means I can remove the work-around now.
@penguineer commented on GitHub (Jan 5, 2022):
Or it could be part of the doctor tool. I ran all the tests and they said things are fine - maybe one test could check that the configuration will lead to viable workers, so that an administrator doesn't need to know too much about the internal workings of Gitea.
@zeripath commented on GitHub (Jan 5, 2022):
Nope this will have been added by hand. It's really not the kind of setting we'd be setting to 0 by default as it turns the scaling worker pool into a non-scaling worker pool. I wouldn't have written the scaling pool in the first place if I thought that most people would not need it but the option exists because I know that some people will need to prevent scaling for certain deployments.
The breaking notice clearly refers to queues and definitely states that the default worker count is now 0. (The config cheat sheet similarly says this.) The only reason to set
BLOCK_TIMEOUT=0is to prevent worker scaling and should be clear that changing the defaultWORKERSto 0 would conflict with that. If I'd predicted that people were out there settingBLOCK_TIMEOUT=0without adjustingWORKERSI'd have mentioned it explicitly there but I didn't - however, in the end it's your configuration - you're responsible for looking at your configuration and checking how the breaking notices interact.I never expected that anyone would be setting this value who did not have a really deep understanding of what it was doing. Certainly not without overriding the default WORKERS number manually themselves.
There are a lot of way you can add settings that will cause a fundamentally broken configuration (for most circumstances) - it's difficult to detect or predict every way in which things will be broken. We try to provide a decent set of defaults to keep things reasonable and to predict how people will be affected by changes in those defaults but honestly I missed this one, setting
BLOCK_TIMEOUT=0alone without any other settings in queues is not something that was predicted.In the end though quite a lot of information was already provided to figure this out, you diagnosed that the issue was that there were no workers, there was a warning in the breaking notices about changes to the default number of workers, you looked at the monitoring page and discovered that there were no workers where it would also have said that the block timeout was 0, the cheat sheet tells you default number of workers has changed to 0 and you had your app.ini. While we could provide more information it's not as if there wasn't a lot of information already there.
Now, what can be done?
@penguineer commented on GitHub (Jan 16, 2022):
Thank you for the long post!
Please take my remarks not so much as a critique on this very nice project, but as a user/administrator feedback to help improve the experience.
We checked again and it is unclear why the option is there, but upgrades in three other instances didn't show any such issues. One thing came up during discussion: Upgrade from 1.12 to 1.15 was considered a minor upgrade, yet the meaning of a configuration variable changed, which should have triggered a major upgrade. I do not want to go to deep into that rabbit hole. A suggestion was that a new variable would have been better than changing the meaning of an existing variable. (It is another discussion weather configuration is part of the API.)
Please consider a user (or administrator) of this software who does not know about the Gitea or Go internals. Up until this issue I simply was not aware that queues are used, as it was not needed. It is an implementation detail that for our setup never needed to be tweaked.
I only figured this out because, luckily, I am able to read the source code (I have never developed with Go tho) and have enough experience with similar systems to understand the structure. The logs do not reveal that queues are used.
This was not achieved from the log files. The change log entry also would not have made sense to me before this analysis.
I agree, during normal operations this would not be noticed. When normal operations stopped, I checked the logs thoroughly and would definitely have seen this warning.
I also looked at the queue configuration in the UI, but they are formatted as JSON and I did not relate them to the ini-style configuration file.
If the log entry is easy to achieve, it would be a nice safeguard. Otherwise I believe that this issue and discussion might be enough, hoping that it will show up in search results for other administrators with a similar problem.
Please keep up the good work! :)