github-starred/gitea
2
0
Fork 0
mirror of https://github.com/go-gitea/gitea.git synced 2026-03-09 04:25:18 -05:00
Code Issues 2.6k Packages Projects Releases 100 Wiki Activity

High cpu comsuming cause gitea very slow #7063

New Issue
Closed
opened 2025-11-02 07:14:46 -06:00 by GiteaMirror · 25 comments
GiteaMirror commented 2025-11-02 07:14:46 -06:00
Owner
Copy Link

Originally created by @0000005 on GitHub (Mar 24, 2021).

  • Gitea version (or commit ref): 1.13.2
  • Git version: 1.8.3.1
  • Operating system:
  • Database (use [x]):
    • PostgreSQL
    • MySQL
    • MSSQL
    • SQLite
  • Can you reproduce the bug at https://try.gitea.io:
    • Yes (provide example URL)
    • No
  • Log gist:

Description

...
I don't know what action will be causing this problem yet. But it happened frequently.

Screenshots

image
image
The cpu high usage will last for tens of minutes.

Originally created by @0000005 on GitHub (Mar 24, 2021). <!-- NOTE: If your issue is a security concern, please send an email to security@gitea.io instead of opening a public issue --> <!-- 1. Please speak English, this is the language all maintainers can speak and write. 2. Please ask questions or configuration/deploy problems on our Discord server (https://discord.gg/gitea) or forum (https://discourse.gitea.io). 3. Please take a moment to check that your issue doesn't already exist. 4. Make sure it's not mentioned in the FAQ (https://docs.gitea.io/en-us/faq) 5. Please give all relevant information below for bug reports, because incomplete details will be handled as an invalid report. --> - Gitea version (or commit ref): 1.13.2 - Git version: 1.8.3.1 - Operating system: <!-- Please include information on whether you built gitea yourself, used one of our downloads or are using some other package --> <!-- Please also tell us how you are running gitea, e.g. if it is being run from docker, a command-line, systemd etc. ---> <!-- If you are using a package or systemd tell us what distribution you are using --> - Database (use `[x]`): - [ ] PostgreSQL - [x] MySQL - [ ] MSSQL - [ ] SQLite - Can you reproduce the bug at https://try.gitea.io: - [ ] Yes (provide example URL) - [x] No - Log gist: <!-- It really is important to provide pertinent logs --> <!-- Please read https://docs.gitea.io/en-us/logging-configuration/#debugging-problems --> <!-- In addition, if your problem relates to git commands set `RUN_MODE=dev` at the top of app.ini --> ## Description <!-- If using a proxy or a CDN (e.g. CloudFlare) in front of gitea, please disable the proxy/CDN fully and connect to gitea directly to confirm the issue still persists without those services. --> ... I don't know what action will be causing this problem yet. But it happened frequently. ## Screenshots ![image](https://user-images.githubusercontent.com/9690089/112288956-d2a04a00-8cc8-11eb-82df-bcaf8cf6ec9e.png) ![image](https://user-images.githubusercontent.com/9690089/112289154-024f5200-8cc9-11eb-984e-1f4c63ad9fa3.png) The cpu high usage will last for tens of minutes. <!-- **If this issue involves the Web Interface, please include a screenshot** -->
GiteaMirror added the performance/bigrepo label 2025-11-02 07:14:46 -06:00
GiteaMirror commented 2025-11-02 07:14:47 -06:00
Author
Owner
Copy Link

@0000005 commented on GitHub (Mar 24, 2021):

I found some related issue. https://github.com/go-gitea/gitea/issues/9642
But there is no more feedback.

@0000005 commented on GitHub (Mar 24, 2021): I found some related issue. https://github.com/go-gitea/gitea/issues/9642 But there is no more feedback.
GiteaMirror commented 2025-11-02 07:14:47 -06:00
Author
Owner
Copy Link

@zeripath commented on GitHub (Mar 24, 2021):

Please update - this is likely fixed by #14649 which is in 1.13.3

@zeripath commented on GitHub (Mar 24, 2021): Please update - this is likely fixed by #14649 which is in 1.13.3
GiteaMirror commented 2025-11-02 07:14:47 -06:00
Author
Owner
Copy Link

@zeripath commented on GitHub (Mar 24, 2021):

The latest point release for 1.13 is 1.13.6 - you should always try the latest point release.

@zeripath commented on GitHub (Mar 24, 2021): The latest point release for 1.13 is 1.13.6 - you should always try the latest point release.
GiteaMirror commented 2025-11-02 07:14:48 -06:00
Author
Owner
Copy Link

@0000005 commented on GitHub (Mar 24, 2021):

Thank you.
I will try the latest stable version to see if the problem remains.

@0000005 commented on GitHub (Mar 24, 2021): Thank you. I will try the latest stable version to see if the problem remains.
GiteaMirror commented 2025-11-02 07:14:48 -06:00
Author
Owner
Copy Link

@techknowlogick commented on GitHub (Mar 24, 2021):

Closing, please re-open if it remains.

@techknowlogick commented on GitHub (Mar 24, 2021): Closing, please re-open if it remains.
GiteaMirror commented 2025-11-02 07:14:48 -06:00
Author
Owner
Copy Link

@0000005 commented on GitHub (Mar 25, 2021):

Please re-open this issue, because after upgrade to 1.13.6 the problem remains.

@0000005 commented on GitHub (Mar 25, 2021): Please re-open this issue, because after upgrade to 1.13.6 the problem remains.
GiteaMirror commented 2025-11-02 07:14:49 -06:00
Author
Owner
Copy Link

@0000005 commented on GitHub (Mar 25, 2021):

image

@0000005 commented on GitHub (Mar 25, 2021): ![image](https://user-images.githubusercontent.com/9690089/112410842-d5e21700-8d56-11eb-85e6-2148e29f8dfe.png)
GiteaMirror commented 2025-11-02 07:14:49 -06:00
Author
Owner
Copy Link

@0000005 commented on GitHub (Mar 25, 2021):

tree model
image

@0000005 commented on GitHub (Mar 25, 2021): tree model ![image](https://user-images.githubusercontent.com/9690089/112411027-2ce7ec00-8d57-11eb-8d2a-219dca703558.png)
GiteaMirror commented 2025-11-02 07:14:49 -06:00
Author
Owner
Copy Link

@0000005 commented on GitHub (Mar 25, 2021):

@techknowlogick

@0000005 commented on GitHub (Mar 25, 2021): @techknowlogick
GiteaMirror commented 2025-11-02 07:14:49 -06:00
Author
Owner
Copy Link

@0000005 commented on GitHub (Mar 25, 2021):

what should i do?

@0000005 commented on GitHub (Mar 25, 2021): what should i do?
GiteaMirror commented 2025-11-02 07:14:49 -06:00
Author
Owner
Copy Link

@0000005 commented on GitHub (Mar 25, 2021):

I am pretty sure that merge pr will cause this problem, although this pr is very small.
image

@0000005 commented on GitHub (Mar 25, 2021): I am pretty sure that merge pr will cause this problem, although this pr is very small. ![image](https://user-images.githubusercontent.com/9690089/112435356-954ac380-8d7f-11eb-8f61-bd08414fd3e0.png)
GiteaMirror commented 2025-11-02 07:14:50 -06:00
Author
Owner
Copy Link

@noerw commented on GitHub (Mar 26, 2021):

The PR may be small, but does that repo have a long history (i.e. > 10.000 commits)?

@noerw commented on GitHub (Mar 26, 2021): The PR may be small, but does that repo have a long history (i.e. > 10.000 commits)?
GiteaMirror commented 2025-11-02 07:14:50 -06:00
Author
Owner
Copy Link

@0000005 commented on GitHub (Mar 26, 2021):

The PR may be small, but does that repo have a long history (i.e. > 10.000 commits)?

image

@0000005 commented on GitHub (Mar 26, 2021): > The PR may be small, but does that repo have a long history (i.e. > 10.000 commits)? ![image](https://user-images.githubusercontent.com/9690089/112606255-1e7efa80-8e53-11eb-9e99-911d4c695075.png)
GiteaMirror commented 2025-11-02 07:14:50 -06:00
Author
Owner
Copy Link

@0000005 commented on GitHub (Mar 26, 2021):

It is very fast if I merge the branch in the local.

@0000005 commented on GitHub (Mar 26, 2021): It is very fast if I merge the branch in the local.
GiteaMirror commented 2025-11-02 07:14:50 -06:00
Author
Owner
Copy Link

@zeripath commented on GitHub (Mar 27, 2021):

I think pprof would be useful here:

[server]
ENABLE_PPROF = true

Just before the slow merge please start:

wget -O cpuprofile.out http://localhost:6060/debug/pprof/profile?seconds=60

Then do the merge and either attach the profile here or talk to me on discord.

I will need to know what version of gitea you are running exactly.

We also need logs. Read the issue template. We do not ask for htop as essentially it is useless. Give us some logs and the results of the pprof I've asked for above.

@zeripath commented on GitHub (Mar 27, 2021): I think pprof would be useful here: ```ini [server] ENABLE_PPROF = true ``` Just before the slow merge please start: `wget -O cpuprofile.out http://localhost:6060/debug/pprof/profile?seconds=60` Then do the merge and either attach the profile here or talk to me on discord. I will need to know what version of gitea you are running exactly. --- We also need logs. Read the issue template. We do not ask for htop as essentially it is useless. Give us some logs and the results of the pprof I've asked for above.
GiteaMirror commented 2025-11-02 07:14:51 -06:00
Author
Owner
Copy Link

@0000005 commented on GitHub (Mar 30, 2021):

cpuprofile.zip

@0000005 commented on GitHub (Mar 30, 2021): [cpuprofile.zip](https://github.com/go-gitea/gitea/files/6226981/cpuprofile.zip)
GiteaMirror commented 2025-11-02 07:14:51 -06:00
Author
Owner
Copy Link

@0000005 commented on GitHub (Mar 30, 2021):

gitea.0330.log
merge action happend at 2021-03-30 14:35:38

@0000005 commented on GitHub (Mar 30, 2021): [gitea.0330.log](https://github.com/go-gitea/gitea/files/6227043/gitea.0330.log) merge action happend at 2021-03-30 14:35:38
GiteaMirror commented 2025-11-02 07:14:51 -06:00
Author
Owner
Copy Link

@0000005 commented on GitHub (Mar 30, 2021):

@zeripath Can you analyze this problem for me? thank you very much!

@0000005 commented on GitHub (Mar 30, 2021): @zeripath Can you analyze this problem for me? thank you very much!
GiteaMirror commented 2025-11-02 07:14:51 -06:00
Author
Owner
Copy Link

@zeripath commented on GitHub (Mar 30, 2021):

OK you've only sent me the xorm logs which are the least useful logs. Please read: https://docs.gitea.io/en-us/logging-configuration/#debugging-problems.

And the profile doesn't actually contain all of the merge - however, it probably does show where the inefficiency is:

43% of the time in that profile is spent in 6dfa92bb1c/modules/git/repo_language_stats.go (L23-L91)

In particular 27.3% (3.46s) is spent on line 47 above in go-enry.IsVendor:

// IsVendor returns whether or not path is a vendor path.
func IsVendor(path string) bool {
	return matchRegexSlice(data.VendorMatchers, path)
}

Which is remarkably inefficiently written because it simply iterates over a list of regexps checking each one in turn without combining the regexps in to a single regexp.

There are a number of other inefficiencies that this flame graph identifies but I think this is a particularly glaring issue.

@zeripath commented on GitHub (Mar 30, 2021): OK you've only sent me the xorm logs which are the least useful logs. Please read: https://docs.gitea.io/en-us/logging-configuration/#debugging-problems. And the profile doesn't actually contain all of the merge - however, it probably does show where the inefficiency is: 43% of the time in that profile is spent in https://github.com/go-gitea/gitea/blob/6dfa92bb1ca97d4e93e07dd5c289a8035761d89c/modules/git/repo_language_stats.go#L23-L91 In particular 27.3% (3.46s) is spent on line 47 above in go-enry.IsVendor: ```go // IsVendor returns whether or not path is a vendor path. func IsVendor(path string) bool { return matchRegexSlice(data.VendorMatchers, path) } ``` Which is remarkably inefficiently written because it simply iterates over a list of regexps checking each one in turn without combining the regexps in to a single regexp. There are a number of other inefficiencies that this flame graph identifies but I think this is a particularly glaring issue.
GiteaMirror commented 2025-11-02 07:14:52 -06:00
Author
Owner
Copy Link

@zeripath commented on GitHub (Mar 30, 2021):

@81519434 would it be possible to for you give us a profile file for this push on 1.14-rc2 and master?

(I think at least a few things should have been improved even without the isvendor pr I've proposed above but it would be good to know for sure.)

@zeripath commented on GitHub (Mar 30, 2021): @81519434 would it be possible to for you give us a profile file for this push on 1.14-rc2 and master? (I think at least a few things should have been improved even without the isvendor pr I've proposed above but it would be good to know for sure.)
GiteaMirror commented 2025-11-02 07:14:52 -06:00
Author
Owner
Copy Link

@0000005 commented on GitHub (Apr 1, 2021):

The high CPU will keep 10-20 minutes after the merge action complete.
Do you mean that I should capture the profile for this long?
@zeripath

@0000005 commented on GitHub (Apr 1, 2021): The high CPU will keep 10-20 minutes after the merge action complete. Do you mean that I should capture the profile for this long? @zeripath
GiteaMirror commented 2025-11-02 07:14:52 -06:00
Author
Owner
Copy Link

@0000005 commented on GitHub (Apr 1, 2021):

We can see git rev-list process is the reason why CPU keeps high consumption from the htop.
Is that mean gitea incorrectly triggered the git command line?
If it is like this, I think that pprof will not find the cause. Because git command line is not managed by go.
Of course, I am not familiar with go. This is all my guess.

@0000005 commented on GitHub (Apr 1, 2021): We can see `git rev-list` process is the reason why CPU keeps high consumption from the `htop`. Is that mean `gitea` incorrectly triggered the git command line? If it is like this, I think that `pprof` will not find the cause. Because git command line is not managed by `go`. Of course, I am not familiar with `go`. This is all my guess.
GiteaMirror commented 2025-11-02 07:14:53 -06:00
Author
Owner
Copy Link

@johanvdw commented on GitHub (Apr 3, 2021):

What happens if you create a new repo on the same server, push the original data and open the same pull request? Your git directory (on the server) might be damaged if git itself is very slow.

More recent versions of git also contain speedups to the mentioned functions, but it should be a fast operation anyway.

@johanvdw commented on GitHub (Apr 3, 2021): What happens if you create a new repo on the same server, push the original data and open the same pull request? Your git directory (on the server) might be damaged if git itself is very slow. More recent versions of git also contain speedups to the mentioned functions, but it should be a fast operation anyway.
GiteaMirror commented 2025-11-02 07:14:53 -06:00
Author
Owner
Copy Link

@0000005 commented on GitHub (Apr 6, 2021):

I will try to upgrade the latest git client verstion, and see if the problem remains.

@0000005 commented on GitHub (Apr 6, 2021): I will try to upgrade the latest git client verstion, and see if the problem remains.
GiteaMirror commented 2025-11-02 07:14:53 -06:00
Author
Owner
Copy Link

@wxiaoguang commented on GitHub (Dec 3, 2021):

@0000005 Has the problem been resolved? Since Gitea 1.16 is coming and a lot of code was changed(improved), I think we can close this one. If there is any performance problem, we can continue to improve on 1.16

@wxiaoguang commented on GitHub (Dec 3, 2021): @0000005 Has the problem been resolved? Since Gitea 1.16 is coming and a lot of code was changed(improved), I think we can close this one. If there is any performance problem, we can continue to improve on 1.16
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
$20
$250
$50
$500
backport/done
💎 Bounty
docs-update-needed
good first issue
hacktoberfest
issue/bounty
issue/confirmed
issue/critical
issue/duplicate
issue/needs-feedback
issue/not-a-bug
issue/regression
issue/stale
issue/workaround
lgtm/need 2
modifies/api
modifies/translation
outdated/backport/v1.18
outdated/theme/markdown
outdated/theme/timetracker
performance/bigrepo
performance/cpu
performance/memory
performance/speed
pr/breaking
proposal/accepted
proposal/rejected
pr/wip
pull-request
Mirrored from GitHub Pull Request
 reviewed/wontfix
💰 Rewarded
skip-changelog
status/blocked
topic/accessibility
topic/api
topic/authentication
topic/build
topic/code-linting
topic/commit-signing
topic/content-rendering
topic/deployment
topic/distribution
topic/federation
topic/gitea-actions
topic/issues
topic/lfs
topic/mobile
topic/moderation
topic/packages
topic/pr
topic/projects
topic/repo
topic/repo-migration
topic/security
topic/theme
topic/ui
topic/ui-interaction
topic/ux
topic/webhooks
topic/wiki
type/bug
type/deprecation
type/docs
type/enhancement
type/feature
type/miscellaneous
type/proposal
type/question
type/refactoring
type/summary
type/testing
type/upstream
No Label performance/bigrepo
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/gitea#7063
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?