github-starred/gitea
2
0
Fork 0
mirror of https://github.com/go-gitea/gitea.git synced 2026-03-09 12:46:42 -05:00
Code Issues 2.6k Packages Projects Releases 100 Wiki Activity

Gitea Current Goroutines 1500+ and Spiking Slow Server Response Time #14130

New Issue
Closed
opened 2025-11-02 11:03:49 -06:00 by GiteaMirror · 6 comments
GiteaMirror commented 2025-11-02 11:03:49 -06:00
Owner
Copy Link

Originally created by @oDinZu on GitHub (Feb 13, 2025).

Description

The Gitea instance jumps around between 60 to 2000+ go routines and was wondering how to limit this? When it jumps to 800, the server load time significantly increases and sometimes the server receives a 502 error.

The server hardware I have Gitea running on has 1CPUs and 2GB of RAM.

Gitea Version

1.23.1 built with GNU Make 4.3, go1.23.4 : bindata, sqlite

Can you reproduce the bug on the Gitea demo site?

Yes

Log Gist

[W] router: slow GET..
[W] router: failed    GET..
err=runtime error: invalid memory address or nil pointer dereference
/usr/local/go/src/runtime/panic.go:785 (0x4776d1)
/source/modules/web/routing/logger_manager.go:116 (0x1fcac88)
/usr/local/go/src/runtime/panic.go:785 (0x4776d1)
/usr/local/go/src/runtime/panic.go:262 (0x479b38)
/usr/local/go/src/runtime/signal_unix.go:917 (0x479b08)

Screenshots

Image

Git Version

2.25.1

Operating System

Ubuntu 22

How are you running Gitea?

Running Gitea with a binary

Database

MySQL

Originally created by @oDinZu on GitHub (Feb 13, 2025). ### Description The Gitea instance jumps around between 60 to 2000+ go routines and was wondering how to limit this? When it jumps to 800, the server load time significantly increases and sometimes the server receives a 502 error. The server hardware I have Gitea running on has 1CPUs and 2GB of RAM. ### Gitea Version 1.23.1 built with GNU Make 4.3, go1.23.4 : bindata, sqlite ### Can you reproduce the bug on the Gitea demo site? Yes ### Log Gist ``` [W] router: slow GET.. [W] router: failed GET.. err=runtime error: invalid memory address or nil pointer dereference /usr/local/go/src/runtime/panic.go:785 (0x4776d1) /source/modules/web/routing/logger_manager.go:116 (0x1fcac88) /usr/local/go/src/runtime/panic.go:785 (0x4776d1) /usr/local/go/src/runtime/panic.go:262 (0x479b38) /usr/local/go/src/runtime/signal_unix.go:917 (0x479b08) ``` ### Screenshots ![Image](https://github.com/user-attachments/assets/e82dbc25-be23-423e-aa1f-4e71bbcc406a) ### Git Version 2.25.1 ### Operating System Ubuntu 22 ### How are you running Gitea? Running Gitea with a binary ### Database MySQL
GiteaMirror added the issue/needs-feedbacktype/bugissue/duplicate labels 2025-11-02 11:03:49 -06:00
GiteaMirror commented 2025-11-02 11:03:50 -06:00
Author
Owner
Copy Link

@wxiaoguang commented on GitHub (Feb 13, 2025):

When it "jump high", what are the requests (the logs could tell)? Do you run the instance publicly and some crawls are heavily accessing various URLs (if so, you can try to limit the public access)?

@wxiaoguang commented on GitHub (Feb 13, 2025): When it "jump high", what are the requests (the logs could tell)? Do you run the instance publicly and some crawls are heavily accessing various URLs (if so, you can try to limit the public access)?
GiteaMirror commented 2025-11-02 11:03:50 -06:00
Author
Owner
Copy Link

@oDinZu commented on GitHub (Feb 13, 2025):

The router slows, then fails and then the gitea binary fails/crashes/restarts. I scheduled a repository sync for a specific time each day.

The gitea server does the GET requests and then eventually has a err=runtime error: invalid memory address or nil pointer dereference.

GET /odinzugiteainstance/gitea/commits/commit/53cf46cae7475befa2dde554bbd9147e436072b9/modules/templates?page=1 for XX.XXX.5.21:0, panic in 59883.8ms @ repo/commit.go:44(repo.RefCommits), err=runtime error: invalid memory address or nil pointer dereference

The server is accessible by the public for cloning/mirroring repositories, but to add repositories and edit, you have to have a username. If what you mean is the gitea instance only accessible to specific set of IP's, then no.

I have fail2ban instance setup and there are no problems in the admin/self-check with the db, including system notices

this error happens first, then the router: slow begins to happen and the go routines begin to stack.

2025/02/12 23:55:29 ...ters/common/serve.go:29:func1() [E] ServeBlob: Close: io: read/write on closed pipe

2025/02/12 23:50:39 ...eb/routing/logger.go:68:func1() [W] router: slow      GET /odinzugiteainstance/NvChad/src/commit/f7180602609a17dc76353c99078a886a33729739/lua/core/default_config.lua for 57.141.5.9:0, elapsed 3642.9ms @ repo/view_home.go:314(repo.Home)
2025/02/12 23:50:39 ...eb/routing/logger.go:68:func1() [W] router: slow      GET /odinzugiteainstance/PostgreSQL/src/commit/2b7334d4877ba445003f96b0bb7eed4e7078a39b/src/bin/pg_config/nls.mk for 57.141.5.4:0, elapsed 3203.6ms @ repo/view_home.go:314(repo.Home)
2025/02/12 23:50:39 ...eb/routing/logger.go:68:func1() [W] router: slow      GET /odinzugiteainstance/gitea/blame/commit/9b2536b78fdcd3cf444a2f54857d9871e153858f/services/migrations/gitea_uploader_test.go for 57.141.5.15:0, elapsed 3491.0ms @ repo/blame.go:42(repo.RefBlame)
2025/02/12 23:50:39 ...eb/routing/logger.go:68:func1() [W] router: slow      GET /odinzugiteainstance/liboqs/src/commit/d704da0c5fd0e6d22d2a2f4212910cf383d8c345/scripts/validate_cbom.sh for 57.141.5.18:0, elapsed 3935.8ms @ repo/view_home.go:314(repo.Home)

Image

Maybe related to: https://github.com/go-gitea/gitea/issues/28215

@oDinZu commented on GitHub (Feb 13, 2025): The router slows, then fails and then the gitea binary fails/crashes/restarts. I scheduled a repository sync for a specific time each day. The gitea server does the GET requests and then eventually has a err=runtime error: invalid memory address or nil pointer dereference. ``` GET /odinzugiteainstance/gitea/commits/commit/53cf46cae7475befa2dde554bbd9147e436072b9/modules/templates?page=1 for XX.XXX.5.21:0, panic in 59883.8ms @ repo/commit.go:44(repo.RefCommits), err=runtime error: invalid memory address or nil pointer dereference ``` The server is accessible by the public for cloning/mirroring repositories, but to add repositories and edit, you have to have a username. If what you mean is the gitea instance only accessible to specific set of IP's, then no. I have fail2ban instance setup and there are no problems in the admin/self-check with the db, including system notices this error happens first, then the router: slow begins to happen and the go routines begin to stack. ``` 2025/02/12 23:55:29 ...ters/common/serve.go:29:func1() [E] ServeBlob: Close: io: read/write on closed pipe ``` ``` 2025/02/12 23:50:39 ...eb/routing/logger.go:68:func1() [W] router: slow GET /odinzugiteainstance/NvChad/src/commit/f7180602609a17dc76353c99078a886a33729739/lua/core/default_config.lua for 57.141.5.9:0, elapsed 3642.9ms @ repo/view_home.go:314(repo.Home) 2025/02/12 23:50:39 ...eb/routing/logger.go:68:func1() [W] router: slow GET /odinzugiteainstance/PostgreSQL/src/commit/2b7334d4877ba445003f96b0bb7eed4e7078a39b/src/bin/pg_config/nls.mk for 57.141.5.4:0, elapsed 3203.6ms @ repo/view_home.go:314(repo.Home) 2025/02/12 23:50:39 ...eb/routing/logger.go:68:func1() [W] router: slow GET /odinzugiteainstance/gitea/blame/commit/9b2536b78fdcd3cf444a2f54857d9871e153858f/services/migrations/gitea_uploader_test.go for 57.141.5.15:0, elapsed 3491.0ms @ repo/blame.go:42(repo.RefBlame) 2025/02/12 23:50:39 ...eb/routing/logger.go:68:func1() [W] router: slow GET /odinzugiteainstance/liboqs/src/commit/d704da0c5fd0e6d22d2a2f4212910cf383d8c345/scripts/validate_cbom.sh for 57.141.5.18:0, elapsed 3935.8ms @ repo/view_home.go:314(repo.Home) ``` ![Image](https://github.com/user-attachments/assets/89ed4094-c6ae-42ba-8176-cfe7df193cec) Maybe related to: https://github.com/go-gitea/gitea/issues/28215
GiteaMirror commented 2025-11-02 11:03:50 -06:00
Author
Owner
Copy Link

@wxiaoguang commented on GitHub (Feb 13, 2025):

It seems to be related to the crawls.

  • OOM caused by numerous crawls #33520

No idea what kind of crawl is the one you met.

Gitea might also try to optimize some code but I guess the most affects are caused by crawls.

@wxiaoguang commented on GitHub (Feb 13, 2025): It seems to be related to the crawls. * OOM caused by numerous crawls #33520 No idea what kind of crawl is the one you met. * If it respects "robots.txt", you could set one, or add use a custom template to add "meta": https://stackoverflow.com/questions/67192771/what-is-the-impact-of-a-meta-tag-with-content-noindex-nofollow * If it doesn't respect "robots.txt", either: * block them * OR provide more CPU/memory to satisfy them ---- Gitea might also try to optimize some code but I guess the most affects are caused by crawls.
GiteaMirror commented 2025-11-02 11:03:50 -06:00
Author
Owner
Copy Link

@oDinZu commented on GitHub (Feb 13, 2025):

That seems to be the case; I will monitor it and update this thread accordingly if more problems arise. At the moment, the gitea instance is running 60 Goroutines.

the solution I made use of is adding custom templates.

I added this line to custom/templates/base/head.tmpl:

<meta name="robots" content="noindex, nofollow, noarchive">

I would recommend this as an official addition to the default head.tmpl

@oDinZu commented on GitHub (Feb 13, 2025): That seems to be the case; I will monitor it and update this thread accordingly if more problems arise. At the moment, the gitea instance is running 60 Goroutines. the solution I made use of is adding custom templates. I added this line to custom/templates/base/head.tmpl: ``` <meta name="robots" content="noindex, nofollow, noarchive"> ``` I would recommend this as an official addition to the default head.tmpl
GiteaMirror commented 2025-11-02 11:03:50 -06:00
Author
Owner
Copy Link

@oDinZu commented on GitHub (Feb 13, 2025):

Today, I observed the server and the goroutines have dramatically reduced, however, I am unable to migrate data from a git server like GitHub.

I continue to receive similar gitea crash logs and warnings about slow routers from various mirrors.

I temporarily disabled the firewall thinking it may be blocking a server, yet doing so resulted in the same errors.

I haven't changed any nginx settings and have only updated the gitea binary. Yesterday, I was able to migrate a git repository from GH before adding the above fix to my head.tmpl.

I am also having a weird bug about deletion of a repository. I deleted it, yet testing a new migration with the same name results in The repository name is already used.

Update: I had to manually delete the repository from gitea-repositories/username/reponame

Update 2: The repository I was migrating was 3.5GB, so it took much longer than anticipated, however I continue to receive the same errors as previously stated, but everything seems to be speedy and working as intended.

@oDinZu commented on GitHub (Feb 13, 2025): Today, I observed the server and the goroutines have dramatically reduced, however, I am unable to migrate data from a git server like GitHub. I continue to receive similar gitea crash logs and warnings about slow routers from various mirrors. I temporarily disabled the firewall thinking it may be blocking a server, yet doing so resulted in the same errors. I haven't changed any nginx settings and have only updated the gitea binary. Yesterday, I was able to migrate a git repository from GH before adding the above fix to my head.tmpl. I am also having a weird bug about deletion of a repository. I deleted it, yet testing a new migration with the same name results in `The repository name is already used.` Update: I had to manually delete the repository from `gitea-repositories/username/reponame` Update 2: The repository I was migrating was 3.5GB, so it took much longer than anticipated, however I continue to receive the same errors as previously stated, but everything seems to be speedy and working as intended.
GiteaMirror commented 2025-11-02 11:03:50 -06:00
Author
Owner
Copy Link

@GiteaBot commented on GitHub (Mar 16, 2025):

We close issues that need feedback from the author if there were no new comments for a month. 🍵

@GiteaBot commented on GitHub (Mar 16, 2025): We close issues that need feedback from the author if there were no new comments for a month. :tea:
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
$20
$250
$50
$500
backport/done
💎 Bounty
docs-update-needed
good first issue
hacktoberfest
issue/bounty
issue/confirmed
issue/critical
issue/duplicate
issue/needs-feedback
issue/not-a-bug
issue/regression
issue/stale
issue/workaround
lgtm/need 2
modifies/api
modifies/translation
outdated/backport/v1.18
outdated/theme/markdown
outdated/theme/timetracker
performance/bigrepo
performance/cpu
performance/memory
performance/speed
pr/breaking
proposal/accepted
proposal/rejected
pr/wip
pull-request
Mirrored from GitHub Pull Request
 reviewed/wontfix
💰 Rewarded
skip-changelog
status/blocked
topic/accessibility
topic/api
topic/authentication
topic/build
topic/code-linting
topic/commit-signing
topic/content-rendering
topic/deployment
topic/distribution
topic/federation
topic/gitea-actions
topic/issues
topic/lfs
topic/mobile
topic/moderation
topic/packages
topic/pr
topic/projects
topic/repo
topic/repo-migration
topic/security
topic/theme
topic/ui
topic/ui-interaction
topic/ux
topic/webhooks
topic/wiki
type/bug
type/deprecation
type/docs
type/enhancement
type/feature
type/miscellaneous
type/proposal
type/question
type/refactoring
type/summary
type/testing
type/upstream
No Label issue/duplicate issue/needs-feedback type/bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/gitea#14130
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?