Gitea depletes Ram + CPU #5823

@stale[bot] commented on GitHub (Oct 12, 2020):

This issue has been automatically marked as stale because it has not had recent activity. I am here to help clear issues left open even if solved or waiting for more insight. This issue will be closed if no further activity occurs during the next 2 weeks. If the issue is still valid just add a comment to keep it alive. Thank you for your contributions.

@stale[bot] commented on GitHub (Oct 12, 2020): This issue has been automatically marked as stale because it has not had recent activity. I am here to help clear issues left open even if solved or waiting for more insight. This issue will be closed if no further activity occurs during the next 2 weeks. If the issue is still valid just add a comment to keep it alive. Thank you for your contributions.

GiteaMirror commented

@6543 commented on GitHub (Oct 12, 2020):

@JoeyWrk how many user and how many repos do this instance have?

@6543 commented on GitHub (Oct 12, 2020): @JoeyWrk how many user and how many repos do this instance have?

GiteaMirror commented

@maxsnts commented on GitHub (Dec 9, 2020):

Hi.
Going from 1.12.2 to 1.13.0 i faced the same problem.
Server unusable with all RAM (and swap) used.
Downgraded, problem went away.
Its not a huge instance, been working fine with 4GB, increased to 8GB but did not help.

Thanks

@maxsnts commented on GitHub (Dec 9, 2020): Hi. Going from 1.12.2 to 1.13.0 i faced the same problem. Server unusable with all RAM (and swap) used. Downgraded, problem went away. Its not a huge instance, been working fine with 4GB, increased to 8GB but did not help. Thanks

GiteaMirror commented

@zeripath commented on GitHub (Dec 10, 2020):

Hi.
Going from 1.12.2 to 1.13.0 i faced the same problem.
Server unusable with all RAM (and swap) used.
Downgraded, problem went away.
Its not a huge instance, been working fine with 4GB, increased to 8GB but did not help.

Thanks

At the risk of repeating myself...

When did this happen?

Had gitea finished it's migration to 1.13 or was gitea up and running?

What caused the memory to spike? Was it a push? Are you pushing LFS or just normal files? How big?

Etc.

You're not giving us any information to help you.

@zeripath commented on GitHub (Dec 10, 2020): > Hi. > Going from 1.12.2 to 1.13.0 i faced the same problem. > Server unusable with all RAM (and swap) used. > Downgraded, problem went away. > Its not a huge instance, been working fine with 4GB, increased to 8GB but did not help. > > Thanks At the risk of repeating myself... When did this happen? Had gitea finished it's migration to 1.13 or was gitea up and running? What caused the memory to spike? Was it a push? Are you pushing LFS or just normal files? How big? Etc. You're not giving us any information to help you.

GiteaMirror commented

@6543 commented on GitHub (Dec 10, 2020):

Its not a huge instance

1 repo, 10 repos 1000 ?

@6543 commented on GitHub (Dec 10, 2020): > Its not a huge instance 1 repo, 10 repos 1000 ?

GiteaMirror commented

@maxsnts commented on GitHub (Dec 11, 2020):

47 Repos. No one was doing any operation.
I'm running gitea as a daemon, maybe that is why +im not seeing the upgrade process?
Next time i will try running it the first time in the shell so that i can see the output.

@maxsnts commented on GitHub (Dec 11, 2020): 47 Repos. No one was doing any operation. I'm running gitea as a daemon, maybe that is why +im not seeing the upgrade process? Next time i will try running it the first time in the shell so that i can see the output.

GiteaMirror commented

@maxsnts commented on GitHub (Dec 11, 2020):

Its something to do with many pulls at the same time.

We have a repo that gets checked periodically as part of a continuous delivery system.
Its about 150 VMs pulling a repo to see if there are new stuff.
Some times many of them happen to be at the same time.
In 1.12.5, these pulls go by with no problem at all.
In 1.13.0, gitea immediately exhausts all resources (RAM+CPU+SWAP) never recovering by it self.

Tested multiple times, the concurrent pulls is what kills gitea.

@maxsnts commented on GitHub (Dec 11, 2020): Its something to do with many pulls at the same time. We have a repo that gets checked periodically as part of a continuous delivery system. Its about 150 VMs pulling a repo to see if there are new stuff. Some times many of them happen to be at the same time. In 1.12.5, these pulls go by with no problem at all. In 1.13.0, gitea immediately exhausts all resources (RAM+CPU+SWAP) never recovering by it self. Tested multiple times, the concurrent pulls is what kills gitea.

GiteaMirror commented

@zeripath commented on GitHub (Dec 12, 2020):

OK so this is very likely related to the go-git reading objects in to memory - you might want to try the no-go-git PR. (#13673)

@zeripath commented on GitHub (Dec 12, 2020): OK so this is very likely related to the go-git reading objects in to memory - you might want to try the no-go-git PR. (#13673)

GiteaMirror commented

@maxsnts commented on GitHub (Dec 12, 2020):

I'm sorry i'm a bit lost.
Thank you for the help, but when you can, can you point me to what it is "no-go-git PR"?
It may be obvious, but i'm not getting there.

@maxsnts commented on GitHub (Dec 12, 2020): I'm sorry i'm a bit lost. Thank you for the help, but when you can, can you point me to what it is "no-go-git PR"? It may be obvious, but i'm not getting there.

GiteaMirror commented

@zeripath commented on GitHub (Dec 12, 2020):

#13673

@zeripath commented on GitHub (Dec 12, 2020): #13673

GiteaMirror commented

@maxsnts commented on GitHub (Dec 12, 2020):

Got it.
I will try to get a build going.
Thanks.

@maxsnts commented on GitHub (Dec 12, 2020): Got it. I will try to get a build going. Thanks.

GiteaMirror commented

@zeripath commented on GitHub (Dec 12, 2020):

One other thing is that it might be helpful to set in your app.ini:

[server]
...
ENABLE_PPROF = true

Then use:

go tool pprof http://localhost:6060/debug/pprof/heap

and either type: top to get a top memory use and/or web to generate an svg of memory use.

Then at least we'll be able to trace where the memory is being used.

@zeripath commented on GitHub (Dec 12, 2020): One other thing is that it might be helpful to set in your app.ini: ``` [server] ... ENABLE_PPROF = true ``` Then use: ``` go tool pprof http://localhost:6060/debug/pprof/heap ``` and either type: `top` to get a top memory use and/or `web` to generate an svg of memory use. Then at least we'll be able to trace where the memory is being used.

GiteaMirror commented

@dragospe commented on GitHub (Dec 16, 2020):

Just wanted to chime in and say that I was having a similar issue that was fixed by building from #13673.
Thanks zeripath!

@dragospe commented on GitHub (Dec 16, 2020): Just wanted to chime in and say that I was having a similar issue that was fixed by building from #13673. Thanks zeripath!

GiteaMirror commented