During git clone, gitea fails with accept4: too many open files; retrying in 1s #15023

@wxiaoguang commented on GitHub (Oct 14, 2025):

It seems that there are some resource leaking.

@wxiaoguang commented on GitHub (Oct 14, 2025): It seems that there are some resource leaking.

GiteaMirror commented

@wxiaoguang commented on GitHub (Oct 14, 2025):

Propose a PR to try to fix: LFS SSH: Fix missing Close when error occurs and add more tests #35658

If you can collect more information (for example: when error occurs, what are the leaked resources? file description or tcp socket connection), it would be very helpful.

@wxiaoguang commented on GitHub (Oct 14, 2025): Propose a PR to try to fix: LFS SSH: Fix missing Close when error occurs and add more tests #35658 If you can collect more information (for example: when error occurs, what are the leaked resources? file description or tcp socket connection), it would be very helpful.

GiteaMirror commented

@galets commented on GitHub (Oct 14, 2025):

I have environment where the issue is reproducible. Can you write me up the steps I need to follow to collect the data you want?

Also, would you like me to deploy the fix you proposed and see if it resolves the issue?

@galets commented on GitHub (Oct 14, 2025): I have environment where the issue is reproducible. Can you write me up the steps I need to follow to collect the data you want? Also, would you like me to deploy the fix you proposed and see if it resolves the issue?

GiteaMirror commented

2025-11-02 11:28:16 -06:00

@wxiaoguang commented on GitHub (Oct 14, 2025):

Some tools like ss or lsof can help to list the open FD or TCP sockets in a process.

For example: ss -anp and lsof -anp $(pidof gitea)

@wxiaoguang commented on GitHub (Oct 14, 2025): Some tools like `ss` or `lsof` can help to list the open FD or TCP sockets in a process. For example: `ss -anp` and `lsof -anp $(pidof gitea)`

GiteaMirror commented

@galets commented on GitHub (Oct 14, 2025):

I dumped the whole lsof, as there weren't many other processes in the container

(Update: deleted the log attachment)

Please let me know if you need anything else. I will restart gitea in a few hours

@galets commented on GitHub (Oct 14, 2025): I dumped the whole lsof, as there weren't many other processes in the container (Update: deleted the log attachment) Please let me know if you need anything else. I will restart gitea in a few hours

GiteaMirror commented

2025-11-02 11:28:16 -06:00

@galets commented on GitHub (Oct 14, 2025):

Also, something occured to me worth mentioning: a lot of exhausted connections belong to web ui process, not workers:

# ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 Oct13 ?        00:00:01 /usr/lib/systemd/systemd
root          16       1  0 Oct13 ?        00:00:42 /usr/lib/systemd/systemd-journald
systemd+      42       1  0 Oct13 ?        00:00:01 /usr/lib/systemd/systemd-networkd
root          65       1  0 Oct13 ?        00:00:00 /usr/sbin/cron -f
message+      66       1  0 Oct13 ?        00:00:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root          67       1  0 Oct13 ?        00:00:00 /usr/lib/systemd/systemd-logind
gitea         69       1 21 Oct13 ?        07:05:42 /usr/local/lib/gitea/gitea web --config /usr/local/var/gitea/custom/conf/app.ini
root          87       1  0 Oct13 pts/0    00:00:00 /sbin/agetty -o -- \u --noreset --noclear --keep-baud 115200,57600,38400,9600 - vt220
root          90       1  0 Oct13 ?        00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
root         228       1  0 Oct13 ?        00:00:00 /usr/lib/systemd/systemd --user
root         230     228  0 Oct13 ?        00:00:00 (sd-pam)
gitea       2128       1  0 23:22 ?        00:00:00 /usr/lib/systemd/systemd --user
gitea       2130    2128  0 23:22 ?        00:00:00 (sd-pam)
gitea       2203       1  1 23:22 ?        00:00:31 /usr/local/lib/gitea/gitea --config=/usr/local/var/gitea/custom/conf/app.ini serv key-4
gitea       2230       1  0 23:22 ?        00:00:18 /usr/local/lib/gitea/gitea --config=/usr/local/var/gitea/custom/conf/app.ini serv key-4
gitea       2257       1  0 23:22 ?        00:00:14 /usr/local/lib/gitea/gitea --config=/usr/local/var/gitea/custom/conf/app.ini serv key-4
gitea       2286       1  0 23:22 ?        00:00:19 /usr/local/lib/gitea/gitea --config=/usr/local/var/gitea/custom/conf/app.ini serv key-4
gitea       2311       1  0 23:22 ?        00:00:15 /usr/local/lib/gitea/gitea --config=/usr/local/var/gitea/custom/conf/app.ini serv key-4
gitea       2333       1  1 23:22 ?        00:00:22 /usr/local/lib/gitea/gitea --config=/usr/local/var/gitea/custom/conf/app.ini serv key-4
gitea       2359       1  0 23:22 ?        00:00:19 /usr/local/lib/gitea/gitea --config=/usr/local/var/gitea/custom/conf/app.ini serv key-4
gitea       2385       1  1 23:22 ?        00:00:43 /usr/local/lib/gitea/gitea --config=/usr/local/var/gitea/custom/conf/app.ini serv key-4
root        2442       1  0 23:38 pts/2    00:00:00 /bin/bash -l
root        2444    2442  0 23:38 pts/2    00:00:00 (sd-
root        2573    2442  0 23:59 pts/2    00:00:00 ps -ef

@galets commented on GitHub (Oct 14, 2025): Also, something occured to me worth mentioning: a lot of exhausted connections belong to web ui process, not workers: ``` # ps -ef UID PID PPID C STIME TTY TIME CMD root 1 0 0 Oct13 ? 00:00:01 /usr/lib/systemd/systemd root 16 1 0 Oct13 ? 00:00:42 /usr/lib/systemd/systemd-journald systemd+ 42 1 0 Oct13 ? 00:00:01 /usr/lib/systemd/systemd-networkd root 65 1 0 Oct13 ? 00:00:00 /usr/sbin/cron -f message+ 66 1 0 Oct13 ? 00:00:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only root 67 1 0 Oct13 ? 00:00:00 /usr/lib/systemd/systemd-logind gitea 69 1 21 Oct13 ? 07:05:42 /usr/local/lib/gitea/gitea web --config /usr/local/var/gitea/custom/conf/app.ini root 87 1 0 Oct13 pts/0 00:00:00 /sbin/agetty -o -- \u --noreset --noclear --keep-baud 115200,57600,38400,9600 - vt220 root 90 1 0 Oct13 ? 00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups root 228 1 0 Oct13 ? 00:00:00 /usr/lib/systemd/systemd --user root 230 228 0 Oct13 ? 00:00:00 (sd-pam) gitea 2128 1 0 23:22 ? 00:00:00 /usr/lib/systemd/systemd --user gitea 2130 2128 0 23:22 ? 00:00:00 (sd-pam) gitea 2203 1 1 23:22 ? 00:00:31 /usr/local/lib/gitea/gitea --config=/usr/local/var/gitea/custom/conf/app.ini serv key-4 gitea 2230 1 0 23:22 ? 00:00:18 /usr/local/lib/gitea/gitea --config=/usr/local/var/gitea/custom/conf/app.ini serv key-4 gitea 2257 1 0 23:22 ? 00:00:14 /usr/local/lib/gitea/gitea --config=/usr/local/var/gitea/custom/conf/app.ini serv key-4 gitea 2286 1 0 23:22 ? 00:00:19 /usr/local/lib/gitea/gitea --config=/usr/local/var/gitea/custom/conf/app.ini serv key-4 gitea 2311 1 0 23:22 ? 00:00:15 /usr/local/lib/gitea/gitea --config=/usr/local/var/gitea/custom/conf/app.ini serv key-4 gitea 2333 1 1 23:22 ? 00:00:22 /usr/local/lib/gitea/gitea --config=/usr/local/var/gitea/custom/conf/app.ini serv key-4 gitea 2359 1 0 23:22 ? 00:00:19 /usr/local/lib/gitea/gitea --config=/usr/local/var/gitea/custom/conf/app.ini serv key-4 gitea 2385 1 1 23:22 ? 00:00:43 /usr/local/lib/gitea/gitea --config=/usr/local/var/gitea/custom/conf/app.ini serv key-4 root 2442 1 0 23:38 pts/2 00:00:00 /bin/bash -l root 2444 2442 0 23:38 pts/2 00:00:00 (sd- root 2573 2442 0 23:59 pts/2 00:00:00 ps -ef ```

GiteaMirror commented

2025-11-02 11:28:16 -06:00

@wxiaoguang commented on GitHub (Oct 15, 2025):

Hmm, just realized that it's difficult to "catch" a snapshot when the resource leaking occurs.

The reason is that the leaking occurs in the "git hook gitea subcommand" (but not gitea server), when the leaking occurs, git client is likely to exit soon, then the "git hook gitea subcommand" also exits, then everything recovers. It needs a quite good timing to catch the resource leaking in "git hook gitea subcommand" ....

@wxiaoguang commented on GitHub (Oct 15, 2025): Hmm, just realized that it's difficult to "catch" a snapshot when the resource leaking occurs. The reason is that the leaking occurs in the "git hook gitea subcommand" (but not gitea server), when the leaking occurs, git client is likely to exit soon, then the "git hook gitea subcommand" also exits, then everything recovers. It needs a quite good timing to catch the resource leaking in "git hook gitea subcommand" ....

GiteaMirror commented

@wxiaoguang commented on GitHub (Oct 15, 2025):

I dumped the whole lsof, as there weren't many other processes in the container

gitea-lsof.txt.gz

Please let me know if you need anything else. I will restart gitea in a few hours

OK, I think we can know the problem is caused by the gitea subcommand to gitea server connection.

There are plenty of these logs:

gitea       69   81 gitea             gitea 3459u     IPv6           31151935       0t0        TCP localhost:3000->localhost:36762 (ESTABLISHED)

I guess "Fix missing Close when error occurs and add more tests #35658" might be able to fix. You can restart the server :)

@wxiaoguang commented on GitHub (Oct 15, 2025): > I dumped the whole lsof, as there weren't many other processes in the container > > [gitea-lsof.txt.gz](https://github.com/user-attachments/files/22915213/gitea-lsof.txt.gz) > > Please let me know if you need anything else. I will restart gitea in a few hours OK, I think we can know the problem is caused by the gitea subcommand to gitea server connection. There are plenty of these logs: ``` gitea 69 81 gitea gitea 3459u IPv6 31151935 0t0 TCP localhost:3000->localhost:36762 (ESTABLISHED) ``` I guess "Fix missing Close when error occurs and add more tests #35658" might be able to fix. You can restart the server :)

GiteaMirror commented

@galets commented on GitHub (Oct 15, 2025):

Would you like me to deploy the commit that fixes is to test?

@galets commented on GitHub (Oct 15, 2025): Would you like me to deploy the commit that fixes is to test?

GiteaMirror commented

@wxiaoguang commented on GitHub (Oct 15, 2025):

Would you like me to deploy the commit that fixes is to test?

If you are able to build your own docker image and deploy, it would be quite helpful. Thank you very much.

(And I am still working on it to try to debug more details)

@wxiaoguang commented on GitHub (Oct 15, 2025): > Would you like me to deploy the commit that fixes is to test? If you are able to build your own docker image and deploy, it would be quite helpful. Thank you very much. (And I am still working on it to try to debug more details)

GiteaMirror commented

@galets commented on GitHub (Oct 15, 2025):

Can you supply exactly the commit hash?

@galets commented on GitHub (Oct 15, 2025): Can you supply exactly the commit hash?

GiteaMirror commented

@wxiaoguang commented on GitHub (Oct 15, 2025):

You can checkout my branch: https://github.com/wxiaoguang/gitea/tree/fix-lfs-ssh

At the moment there is only one commit db70766908. I might add more commits later if I find something new.

@wxiaoguang commented on GitHub (Oct 15, 2025): You can checkout my branch: https://github.com/wxiaoguang/gitea/tree/fix-lfs-ssh At the moment there is only one commit https://github.com/go-gitea/gitea/pull/35658/commits/db70766908e7bc2aaa3b3e6303e8446aded89b0d. I might add more commits later if I find something new.

GiteaMirror commented

@wxiaoguang commented on GitHub (Oct 15, 2025):

If you'd like to apply it to 1.24, the only related change is modules/lfstransfer/backend/backend.go, others are not directly related.

(ps: I think it's also safe to run my fix-lfs-ssh branch, although it is based on main branch, it should be stable enough and can be switch to 1.25 which is going to be released soon)

@wxiaoguang commented on GitHub (Oct 15, 2025): If you'd like to apply it to 1.24, the only related change is `modules/lfstransfer/backend/backend.go`, others are not directly related. (ps: I think it's also safe to run my `fix-lfs-ssh` branch, although it is based on main branch, it should be stable enough and can be switch to 1.25 which is going to be released soon)

GiteaMirror commented

@galets commented on GitHub (Oct 15, 2025):

I applied following and re-testing now:

$ git diff
diff --git a/modules/lfstransfer/backend/backend.go b/modules/lfstransfer/backend/backend.go
index dd4108ea56..f4e6157091 100644
--- a/modules/lfstransfer/backend/backend.go
+++ b/modules/lfstransfer/backend/backend.go
@@ -157,7 +157,7 @@ func (g *GiteaBackend) Batch(_ string, pointers []transfer.BatchItem, args trans
 }
 
 // Download implements transfer.Backend. The returned reader must be closed by the caller.
-func (g *GiteaBackend) Download(oid string, args transfer.Args) (io.ReadCloser, int64, error) {
+func (g *GiteaBackend) Download(oid string, args transfer.Args) (_ io.ReadCloser, _ int64, retErr error) {
        idMapStr, exists := args[argID]
        if !exists {
                return nil, 0, ErrMissingID
@@ -188,7 +188,15 @@ func (g *GiteaBackend) Download(oid string, args transfer.Args) (io.ReadCloser,
        if err != nil {
                return nil, 0, fmt.Errorf("failed to get response: %w", err)
        }
-       // no need to close the body here by "defer resp.Body.Close()", see below
+       // We must return the ReaderCloser but not "ReadAll", to avoid OOM.
+       // "transfer.Backend" will check io.Closer interface and close the Body reader.
+       // So only close the Body when error occurs
+       defer func() {
+               if retErr != nil {
+                       _ = resp.Body.Close()
+               }
+       }()
+
        if resp.StatusCode != http.StatusOK {
                return nil, 0, statusCodeToErr(resp.StatusCode)
        }
@@ -197,7 +205,6 @@ func (g *GiteaBackend) Download(oid string, args transfer.Args) (io.ReadCloser,
        if err != nil {
                return nil, 0, fmt.Errorf("failed to parse content length: %w", err)
        }
-       // transfer.Backend will check io.Closer interface and close this Body reader
        return resp.Body, respSize, nil
 }

@galets commented on GitHub (Oct 15, 2025): I applied following and re-testing now: ``` $ git diff diff --git a/modules/lfstransfer/backend/backend.go b/modules/lfstransfer/backend/backend.go index dd4108ea56..f4e6157091 100644 --- a/modules/lfstransfer/backend/backend.go +++ b/modules/lfstransfer/backend/backend.go @@ -157,7 +157,7 @@ func (g *GiteaBackend) Batch(_ string, pointers []transfer.BatchItem, args trans } // Download implements transfer.Backend. The returned reader must be closed by the caller. -func (g *GiteaBackend) Download(oid string, args transfer.Args) (io.ReadCloser, int64, error) { +func (g *GiteaBackend) Download(oid string, args transfer.Args) (_ io.ReadCloser, _ int64, retErr error) { idMapStr, exists := args[argID] if !exists { return nil, 0, ErrMissingID @@ -188,7 +188,15 @@ func (g *GiteaBackend) Download(oid string, args transfer.Args) (io.ReadCloser, if err != nil { return nil, 0, fmt.Errorf("failed to get response: %w", err) } - // no need to close the body here by "defer resp.Body.Close()", see below + // We must return the ReaderCloser but not "ReadAll", to avoid OOM. + // "transfer.Backend" will check io.Closer interface and close the Body reader. + // So only close the Body when error occurs + defer func() { + if retErr != nil { + _ = resp.Body.Close() + } + }() + if resp.StatusCode != http.StatusOK { return nil, 0, statusCodeToErr(resp.StatusCode) } @@ -197,7 +205,6 @@ func (g *GiteaBackend) Download(oid string, args transfer.Args) (io.ReadCloser, if err != nil { return nil, 0, fmt.Errorf("failed to parse content length: %w", err) } - // transfer.Backend will check io.Closer interface and close this Body reader return resp.Body, respSize, nil } ```

GiteaMirror commented

@wxiaoguang commented on GitHub (Oct 15, 2025):

Good news, I found a new resource leaking point.

The "http client" is abused, then it creates a lot of "connection pool", every pool holds an open connection.

Will propose a fix soon.

@wxiaoguang commented on GitHub (Oct 15, 2025): Good news, I found a new resource leaking point. The "http client" is abused, then it creates a lot of "connection pool", every pool holds an open connection. Will propose a fix soon.

GiteaMirror commented

@galets commented on GitHub (Oct 15, 2025):

I confirm that backend.go fix did not resolve the issue, still getting same error

@galets commented on GitHub (Oct 15, 2025): I confirm that backend.go fix did not resolve the issue, still getting same error

GiteaMirror commented

@wxiaoguang commented on GitHub (Oct 15, 2025):

I think (highly likely) this patch will fix, just add two DisableKeepAlives: true:

diff --git a/modules/httplib/request.go b/modules/httplib/request.go
index 49ea6f4b73..3c65dfe820 100644
--- a/modules/httplib/request.go
+++ b/modules/httplib/request.go
@@ -167,6 +167,8 @@ func (r *Request) getResponse() (*http.Response, error) {
                        TLSClientConfig: r.setting.TLSClientConfig,
                        Proxy:           http.ProxyFromEnvironment,
                        DialContext:     TimeoutDialer(r.setting.ConnectTimeout),
+
+                       DisableKeepAlives: true,
                }
        } else if t, ok := trans.(*http.Transport); ok {
                if t.TLSClientConfig == nil {
@@ -175,6 +177,7 @@ func (r *Request) getResponse() (*http.Response, error) {
                if t.DialContext == nil {
                        t.DialContext = TimeoutDialer(r.setting.ConnectTimeout)
                }
+               t.DisableKeepAlives = true
        }

@wxiaoguang commented on GitHub (Oct 15, 2025): I think (highly likely) this patch will fix, just add two `DisableKeepAlives: true`: ```diff diff --git a/modules/httplib/request.go b/modules/httplib/request.go index 49ea6f4b73..3c65dfe820 100644 --- a/modules/httplib/request.go +++ b/modules/httplib/request.go @@ -167,6 +167,8 @@ func (r *Request) getResponse() (*http.Response, error) { TLSClientConfig: r.setting.TLSClientConfig, Proxy: http.ProxyFromEnvironment, DialContext: TimeoutDialer(r.setting.ConnectTimeout), + + DisableKeepAlives: true, } } else if t, ok := trans.(*http.Transport); ok { if t.TLSClientConfig == nil { @@ -175,6 +177,7 @@ func (r *Request) getResponse() (*http.Response, error) { if t.DialContext == nil { t.DialContext = TimeoutDialer(r.setting.ConnectTimeout) } + t.DisableKeepAlives = true } ```

GiteaMirror commented

@galets commented on GitHub (Oct 15, 2025):

Trying following (previous change rolled back):

$ git diff
diff --git a/modules/httplib/request.go b/modules/httplib/request.go
index 49ea6f4b73..32e42c7f8e 100644
--- a/modules/httplib/request.go
+++ b/modules/httplib/request.go
@@ -167,6 +167,7 @@ func (r *Request) getResponse() (*http.Response, error) {
                        TLSClientConfig: r.setting.TLSClientConfig,
                        Proxy:           http.ProxyFromEnvironment,
                        DialContext:     TimeoutDialer(r.setting.ConnectTimeout),
+                        DisableKeepAlives: true,
                }
        } else if t, ok := trans.(*http.Transport); ok {
                if t.TLSClientConfig == nil {
@@ -175,6 +176,7 @@ func (r *Request) getResponse() (*http.Response, error) {
                if t.DialContext == nil {
                        t.DialContext = TimeoutDialer(r.setting.ConnectTimeout)
                }
+                t.DisableKeepAlives = true
        }
 
        client := &http.Client{

@galets commented on GitHub (Oct 15, 2025): Trying following (previous change rolled back): ``` $ git diff diff --git a/modules/httplib/request.go b/modules/httplib/request.go index 49ea6f4b73..32e42c7f8e 100644 --- a/modules/httplib/request.go +++ b/modules/httplib/request.go @@ -167,6 +167,7 @@ func (r *Request) getResponse() (*http.Response, error) { TLSClientConfig: r.setting.TLSClientConfig, Proxy: http.ProxyFromEnvironment, DialContext: TimeoutDialer(r.setting.ConnectTimeout), + DisableKeepAlives: true, } } else if t, ok := trans.(*http.Transport); ok { if t.TLSClientConfig == nil { @@ -175,6 +176,7 @@ func (r *Request) getResponse() (*http.Response, error) { if t.DialContext == nil { t.DialContext = TimeoutDialer(r.setting.ConnectTimeout) } + t.DisableKeepAlives = true } client := &http.Client{ ```

GiteaMirror commented

@galets commented on GitHub (Oct 15, 2025):

As checkout is at 96%:

root@gitea:/usr/local/lib/gitea# lsof | wc -l
4006

@galets commented on GitHub (Oct 15, 2025): As checkout is at 96%: ``` root@gitea:/usr/local/lib/gitea# lsof | wc -l 4006 ```

GiteaMirror commented

@galets commented on GitHub (Oct 15, 2025):

git clone is complete, the fix is working

@galets commented on GitHub (Oct 15, 2025): `git clone` is complete, the fix is working

GiteaMirror commented