Possible Memory Leak in LFS push #6216

New Issue

GiteaMirror · 2025-11-02T06:48:41-06:00

GiteaMirror commented

2025-11-02 06:48:41 -06:00

Originally created by @v-byte-cpu on GitHub (Oct 26, 2020).

Gitea version (or commit ref): 1.12.1
Git version:
Operating system:
Database (use [x]):
- PostgreSQL
- MySQL
- MSSQL
- SQLite
Can you reproduce the bug at https://try.gitea.io:
- Yes (provide example URL)
- No
Log gist:

Description

I fetched all lfs files from one repo (not in gitea) with git lfs fetch --all and pushed them to private gitea instance with git lfs push --all. First I allocated 500mb to gitea in container orchestrator, it was always enough for gitea that consumed only 128mb. I noticed that gitea was killed with OOM right after lfs push. Then I increased ram to 1gb, lfs push succeeded but memory consumption significantly increased to 780mb.

Screenshots

stats from orchestrator:

Originally created by @v-byte-cpu on GitHub (Oct 26, 2020).  - Gitea version (or commit ref): 1.12.1 - Git version: - Operating system:    - Database (use `[x]`): - [x] PostgreSQL - [ ] MySQL - [ ] MSSQL - [ ] SQLite - Can you reproduce the bug at https://try.gitea.io: - [ ] Yes (provide example URL) - [ ] No - Log gist:    ## Description I fetched all lfs files from one repo (not in gitea) with `git lfs fetch --all` and pushed them to private gitea instance with `git lfs push --all`. First I allocated 500mb to gitea in container orchestrator, it was always enough for gitea that consumed only 128mb. I noticed that gitea was killed with OOM right after lfs push. Then I increased ram to 1gb, `lfs push` succeeded but memory consumption significantly increased to 780mb. ## Screenshots  ![image](https://user-images.githubusercontent.com/65545655/97220530-aa292800-17dc-11eb-8d63-3ab31a06bf73.png) stats from orchestrator: ![image](https://user-images.githubusercontent.com/65545655/97220733-fbd1b280-17dc-11eb-8cfe-8808ba92c7c0.png)

GiteaMirror added the performance/memory label 2025-11-02 06:48:41 -06:00

GiteaMirror closed this issue

2025-11-02 06:48:41 -06:00

GiteaMirror commented

2025-11-02 06:48:42 -06:00

@lunny commented on GitHub (Oct 27, 2020):

Golang holds 780MB, but it in fact use 116MB.

@lunny commented on GitHub (Oct 27, 2020): <img width="410" alt="图片" src="https://user-images.githubusercontent.com/81045/97247633-028f1280-183b-11eb-9d9d-c5916ba62631.png"> Golang holds 780MB, but it in fact use 116MB.

GiteaMirror commented

2025-11-02 06:48:42 -06:00

@v-byte-cpu commented on GitHub (Oct 27, 2020):

@lunny Yes, but in fact app memory consumption is still high. Can it be released back to OS ?

@v-byte-cpu commented on GitHub (Oct 27, 2020): @lunny Yes, but in fact app memory consumption is still high. Can it be released back to OS ?

GiteaMirror commented

2025-11-02 06:48:43 -06:00

@zeripath commented on GitHub (Oct 27, 2020):

As @lunny says I don't think your graphs show a memory leak at all.

However the spike in memory consumption is concerning to me that the LFS stream is somehow being read entirely in to memory (temporarily) before being released.

That's weird though because we definitely try not to do that - so the question is where is that occurring.

Now I see that you're using 1.12.1 - it would be helpful is you could update to at least 1.12.5 as there was a problem with repo stats reading entire files in to memory in that version and then try again.

@zeripath commented on GitHub (Oct 27, 2020): As @lunny says I don't think your graphs show a memory leak at all. However the spike in memory consumption is concerning to me that the LFS stream is somehow being read entirely in to memory (temporarily) before being released. That's weird though because we definitely try not to do that - so the question is where is that occurring. Now I see that you're using 1.12.1 - it would be helpful is you could update to at least 1.12.5 as there was a problem with repo stats reading entire files in to memory in that version and then try again.

GiteaMirror commented

2025-11-02 06:48:44 -06:00

@v-byte-cpu commented on GitHub (Oct 27, 2020):

@zeripath ok, I will update gitea version and try again

@v-byte-cpu commented on GitHub (Oct 27, 2020): @zeripath ok, I will update gitea version and try again

GiteaMirror commented

2025-11-02 06:48:44 -06:00

@v-byte-cpu commented on GitHub (Oct 28, 2020):

@zeripath I updated gitea to 1.12.5

Before lfs push:

After lfs push files with total summary size 40mb (like in original case):

The situation is much better now. However, Heap Memory Obtained is still increasing (1.5x). Is it possible to release Heap Memory Idle ? And yes, it is still pretty high spike in memory consumption in relation to upload file size (40mb).

@v-byte-cpu commented on GitHub (Oct 28, 2020): @zeripath I updated gitea to 1.12.5 Before lfs push: ![image](https://user-images.githubusercontent.com/65545655/97437650-d952ad00-1934-11eb-93d9-81acdfcd896c.png) After lfs push files with total summary size 40mb (like in original case): ![image](https://user-images.githubusercontent.com/65545655/97437725-ee2f4080-1934-11eb-8cbe-c1b195705b3e.png) ![image](https://user-images.githubusercontent.com/65545655/97437778-0606c480-1935-11eb-90ea-082474c7aae2.png) The situation is much better now. However, **Heap Memory Obtained** is still increasing (1.5x). Is it possible to release **Heap Memory Idle** ? And yes, it is still pretty high spike in memory consumption in relation to upload file size (40mb).

GiteaMirror commented

2025-11-02 06:48:44 -06:00

@zeripath commented on GitHub (Oct 28, 2020):

In regards to relating heap memory that's something that you'd have to look at the golang tuning parameters. Garbage collected languages always have a delay between releasing memory and releasing heap. Go won't be using all that memory it's just there in case it needs it again.

Now the real question is the cause of that spike - because that worries me that we're loading a lot of stuff directly in to memory - which even if temporary is still bad.

The question is how to figure out what's doing that, where and how to stop it.

We do have a pprof option the problem is I genuinely don't know how to use it or where to look to learn about how to use it. But if you do - it's there for this reason.

@zeripath commented on GitHub (Oct 28, 2020): In regards to relating heap memory that's something that you'd have to look at the golang tuning parameters. Garbage collected languages always have a delay between releasing memory and releasing heap. Go won't be using all that memory it's just there in case it needs it again. Now the real question is the cause of that spike - because that worries me that we're loading a lot of stuff directly in to memory - which even if temporary is still bad. The question is how to figure out what's doing that, where and how to stop it. We do have a pprof option the problem is I genuinely don't know how to use it or where to look to learn about how to use it. But if you do - it's there for this reason.

GiteaMirror commented

2025-11-02 06:48:44 -06:00

@zeripath commented on GitHub (Jan 14, 2021):

I think this is likely related to the go-git memory leak problems and should now be fixed after the no-go-git PR

@v-byte-cpu would it be possible to recheck on master?

@zeripath commented on GitHub (Jan 14, 2021): I think this is likely related to the go-git memory leak problems and should now be fixed after the no-go-git PR @v-byte-cpu would it be possible to recheck on master?

GiteaMirror commented

2025-11-02 06:48:45 -06:00

@v-byte-cpu commented on GitHub (Jan 15, 2021):

yes, sure. Do you have pre-built docker images for master branch ?

@v-byte-cpu commented on GitHub (Jan 15, 2021): yes, sure. Do you have pre-built docker images for master branch ?

GiteaMirror commented

2025-11-02 06:48:45 -06:00

@JamesYFC commented on GitHub (Feb 11, 2021):

Not sure if this is a separate issue or not but I'm seeing massive memory spikes and a consistent server crash on pushing a fairly large repo (2gb), seemingly at the end of the lfs upload.

This is the latest version of gitea (1.13.2) in a docker container on ubuntu 18.04.

Our gitea lfs is set to an external bucket (backblaze b2).

Our instance has 8gb memory and 8gb storage, but running df -h shows we use less than half the storage.
Upping the storage is no issue if that's the problem, but the crash is worrying considering I'd like to upload even bigger repos (even 20gb ones, but pretty much everything big goes to lfs)

@JamesYFC commented on GitHub (Feb 11, 2021): Not sure if this is a separate issue or not but I'm seeing massive memory spikes and a consistent server crash on pushing a fairly large repo (2gb), seemingly at the end of the lfs upload. This is the latest version of gitea (1.13.2) in a docker container on ubuntu 18.04. Our gitea lfs is set to an external bucket (backblaze b2). Our instance has 8gb memory and 8gb storage, but running `df -h` shows we use less than half the storage. Upping the storage is no issue if that's the problem, but the crash is worrying considering I'd like to upload even bigger repos (even 20gb ones, but pretty much everything big goes to lfs)

GiteaMirror commented

2025-11-02 06:48:45 -06:00

@zeripath commented on GitHub (Feb 11, 2021):

What version?

Have you tried changing you password hashing algorithm from argon2 to a different algorithm?

@zeripath commented on GitHub (Feb 11, 2021): What version? Have you tried changing you password hashing algorithm from argon2 to a different algorithm?

GiteaMirror commented

2025-11-02 06:48:45 -06:00

@JamesYFC commented on GitHub (Feb 11, 2021):

I mentioned it at the end, 1.13.2.
I'll edit the comment to clarify.

I'll try that and get back to you, see if it makes a difference.

@JamesYFC commented on GitHub (Feb 11, 2021): I mentioned it at the end, 1.13.2. I'll edit the comment to clarify. I'll try that and get back to you, see if it makes a difference.

GiteaMirror commented

2025-11-02 06:48:46 -06:00

@lunny commented on GitHub (Feb 13, 2021):

I think #14673 and #14675 will fix the problem.

@lunny commented on GitHub (Feb 13, 2021): I think #14673 and #14675 will fix the problem.

GiteaMirror commented

2025-11-02 06:48:46 -06:00

@zeripath commented on GitHub (Feb 15, 2021):

OK I think we can probably close this - as @lunny says the likely issue here is the memory use of argon2 on password hashing if you are using http(s) pushing. I do expect that the no-go-git PR will also have helped.

Therefore I'm going to close this.

(Of note we will need to think carefully about argon2 and other hashing limits in future - perhaps with some maximal hashing pool or the like.)

@zeripath commented on GitHub (Feb 15, 2021): OK I think we can probably close this - as @lunny says the likely issue here is the memory use of argon2 on password hashing if you are using http(s) pushing. I do expect that the no-go-git PR will also have helped. Therefore I'm going to close this. (Of note we will need to think carefully about argon2 and other hashing limits in future - perhaps with some maximal hashing pool or the like.)

GiteaMirror commented

2025-11-02 06:48:46 -06:00

@zeripath commented on GitHub (Feb 15, 2021):

Resolved by #14673 and #14675

@zeripath commented on GitHub (Feb 15, 2021): Resolved by #14673 and #14675

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/gitea#6216