OOM then uploading file fails with: Error during upload to repo: ... open /var/lib/gitea/data/tmp/uploads/a/a/aa...(uuid): no such file of directory #12371

Closed
opened 2025-11-02 10:07:36 -06:00 by GiteaMirror · 27 comments
Owner

Originally created by @viknsagit on GitHub (Jan 22, 2024).

Description

Set up git lfs and changed it so I could upload large files, up to
4500mb.

Gitea Version

1.21.3

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

2.34.1

Operating System

ubuntu 22.04

How are you running Gitea?

Launched by gitea via systemd. And the nginx proxy installed.

Database

PostgreSQL

Originally created by @viknsagit on GitHub (Jan 22, 2024). ### Description Set up git lfs and changed it so I could upload large files, up to 4500mb. ### Gitea Version 1.21.3 ### Can you reproduce the bug on the Gitea demo site? No ### Log Gist _No response_ ### Screenshots _No response_ ### Git Version 2.34.1 ### Operating System ubuntu 22.04 ### How are you running Gitea? Launched by gitea via systemd. And the nginx proxy installed. ### Database PostgreSQL
GiteaMirror added the type/bug label 2025-11-02 10:07:36 -06:00
Author
Owner

@viknsagit commented on GitHub (Jan 22, 2024):

my gitea : https://git.vikncrew.ru/

@viknsagit commented on GitHub (Jan 22, 2024): my gitea : https://git.vikncrew.ru/
Author
Owner

@pmandalka commented on GitHub (Jan 22, 2024):

What's the limit of your nginx upload file size? see: https://stackoverflow.com/questions/26717013/how-to-edit-nginx-conf-to-increase-file-size-upload

@pmandalka commented on GitHub (Jan 22, 2024): What's the limit of your nginx upload file size? see: https://stackoverflow.com/questions/26717013/how-to-edit-nginx-conf-to-increase-file-size-upload
Author
Owner

@wxiaoguang commented on GitHub (Jan 22, 2024):

nginx? have you checked the manual: https://docs.gitea.com/administration/reverse-proxies#resolving-error-413-request-entity-too-large

nginx config: client_max_body_size

If it still doesn't work, you need to figure out where the failure occurs (on your client, on your network, on the nginx, or on the gitea).

@wxiaoguang commented on GitHub (Jan 22, 2024): nginx? have you checked the manual: https://docs.gitea.com/administration/reverse-proxies#resolving-error-413-request-entity-too-large nginx config: client_max_body_size If it still doesn't work, you need to figure out where the failure occurs (on your client, on your network, on the nginx, or on the gitea).
Author
Owner

@viknsagit commented on GitHub (Jan 22, 2024):

current nginx config

server{

server_name git.vikncrew.ru;

location /
{
        proxy_pass http://localhost:3000;


        client_max_body_size 3000m;
        client_body_timeout 120s;
        proxy_read_timeout 300s;
        proxy_connect_timeout 300s;
        proxy_buffer_size          128k;
        proxy_buffers              4 256k;
        proxy_busy_buffers_size    256k;
}


    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/git.vikncrew.ru/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/git.vikncrew.ru/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot




}
server{
    if ($host = git.vikncrew.ru) {
        return 301 https://$host$request_uri;
    } # managed by Certbot



server_name git.vikncrew.ru;
    listen 80;
    return 404; # managed by Certbot


}
@viknsagit commented on GitHub (Jan 22, 2024): current nginx config ``` server{ server_name git.vikncrew.ru; location / { proxy_pass http://localhost:3000; client_max_body_size 3000m; client_body_timeout 120s; proxy_read_timeout 300s; proxy_connect_timeout 300s; proxy_buffer_size 128k; proxy_buffers 4 256k; proxy_busy_buffers_size 256k; } listen 443 ssl; # managed by Certbot ssl_certificate /etc/letsencrypt/live/git.vikncrew.ru/fullchain.pem; # managed by Certbot ssl_certificate_key /etc/letsencrypt/live/git.vikncrew.ru/privkey.pem; # managed by Certbot include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot } server{ if ($host = git.vikncrew.ru) { return 301 https://$host$request_uri; } # managed by Certbot server_name git.vikncrew.ru; listen 80; return 404; # managed by Certbot } ```
Author
Owner

@viknsagit commented on GitHub (Jan 22, 2024):

I'm trying to upload files up to 400MB

@viknsagit commented on GitHub (Jan 22, 2024): I'm trying to upload files up to 400MB
Author
Owner

@wxiaoguang commented on GitHub (Jan 22, 2024):

Could you figure out where the failure occurs (on your client, on your network, on the nginx, or on the gitea).

At least, when an error occurs, there usually are related logs. And you could time the request to see whether it gets timed out.

@wxiaoguang commented on GitHub (Jan 22, 2024): Could you figure out where the failure occurs (on your client, on your network, on the nginx, or on the gitea). At least, when an error occurs, there usually are related logs. And you could time the request to see whether it gets timed out.
Author
Owner

@viknsagit commented on GitHub (Jan 22, 2024):

I'm going to try it now

@viknsagit commented on GitHub (Jan 22, 2024): I'm going to try it now
Author
Owner

@viknsagit commented on GitHub (Jan 22, 2024):

in systemctl error
Error during upload to repo: <Repository 20:viknsagit/nsfw_classification> to filepath: on main from viknsagit-patch-1: open /var/lib/gitea/data/tmp/uploads/a/a/aa502c1d-df44-4aa1-a8ce-0176a419b3c9: no such file of directory

@viknsagit commented on GitHub (Jan 22, 2024): in systemctl error Error during upload to repo: <Repository 20:viknsagit/nsfw_classification> to filepath: on main from viknsagit-patch-1: open /var/lib/gitea/data/tmp/uploads/a/a/aa502c1d-df44-4aa1-a8ce-0176a419b3c9: no such file of directory
Author
Owner

@wxiaoguang commented on GitHub (Jan 22, 2024):

Thank you very much. Some more questions: Is it the only error log? Does nginx have any error log? Is the failure related to file size (eg: small files succeed but large files fail)?

(At the beginning I thought it might be related to nginx config, but if it would be caused by Gitea code, it might need some people who have enough time to investigate if they could reproduce it ....)

@wxiaoguang commented on GitHub (Jan 22, 2024): Thank you very much. Some more questions: Is it the only error log? Does nginx have any error log? Is the failure related to file size (eg: small files succeed but large files fail)? (At the beginning I thought it might be related to nginx config, but if it would be caused by Gitea code, it might need some people who have enough time to investigate if they could reproduce it ....)
Author
Owner

@viknsagit commented on GitHub (Jan 22, 2024):

when i start upload a file in logs show this error
[E] Error during upload to repo: <Repository 20:viknsagit/nsfw_classification> to filepath: on main from viknsagit-patch-1: open /var/lib/gitea/data/tmp/uploads/a/a/aa502c1d-df44-4aa1-a8ce-0176a419b3c9:no such file or directory.

@viknsagit commented on GitHub (Jan 22, 2024): when i start upload a file in logs show this error [E] Error during upload to repo: <Repository 20:viknsagit/nsfw_classification> to filepath: on main from viknsagit-patch-1: open /var/lib/gitea/data/tmp/uploads/a/a/aa502c1d-df44-4aa1-a8ce-0176a419b3c9:no such file or directory.
Author
Owner

@viknsagit commented on GitHub (Jan 22, 2024):

And clicking on the save commit button takes me to a page with a 502 Bad gateway error

@viknsagit commented on GitHub (Jan 22, 2024): And clicking on the save commit button takes me to a page with a 502 Bad gateway error
Author
Owner

@viknsagit commented on GitHub (Jan 22, 2024):

gitea.log
Log with error

@viknsagit commented on GitHub (Jan 22, 2024): [gitea.log](https://github.com/go-gitea/gitea/files/14012399/gitea.log) Log with error
Author
Owner

@wxiaoguang commented on GitHub (Jan 22, 2024):

Hmm, that's strange.

Have you tried other files? For example, a small file? Or large file with different content? It looks like the uploaded file is deleted from the temp directory after your first uploading.

Uploading a file in Gitea needs 2 steps:

  1. Upload single file, to temp directory, then server returns a UUID
  2. User submit the form with the UUIDs, then server will read the files in temp dir by UUID and commit them into git repo.

It looks like you finished the first step, then the file disappears?


(too late in my timezone, I will go to bed .....)

@wxiaoguang commented on GitHub (Jan 22, 2024): Hmm, that's strange. Have you tried other files? For example, a small file? Or large file with different content? It looks like the uploaded file is deleted from the temp directory after your first uploading. Uploading a file in Gitea needs 2 steps: 1. Upload single file, to temp directory, then server returns a UUID 2. User submit the form with the UUIDs, then server will read the files in temp dir by UUID and commit them into git repo. It looks like you finished the first step, then the file disappears? ---- (too late in my timezone, I will go to bed .....)
Author
Owner

@viknsagit commented on GitHub (Jan 24, 2024):

Small files load fine

@viknsagit commented on GitHub (Jan 24, 2024): Small files load fine
Author
Owner

@wxiaoguang commented on GitHub (Jan 25, 2024):

Small files load fine

TBH, I have no more idea about this problem at the moment. I could only try to provide some information for reference.

  1. The question above: "Or large file with different content?" It is still unclear that when & why & how it fails.
    • If small files succeed, large files fail, what's the threshold? (or timeout?) Is it caused by reverse proxy or Gitea itself, you could try to bypass the reverse proxy and use Gitea's http server directly to test.
    • If some large files succeed, some large files fail, is it related to file content? I can see you were playing with "nsfw" contents.
  2. The logs seem strange
    • I didn't find "uploading" requests, there are only "saving" requests.
    • The log shows that the Gitea instance restarted during your operation, is it caused by OOM, or you restarted it manually?

There are too many unclear details .... maybe you need to investigate and try to figure out more clues.

@wxiaoguang commented on GitHub (Jan 25, 2024): > Small files load fine TBH, I have no more idea about this problem at the moment. I could only try to provide some information for reference. 1. The question above: "Or large file with different content?" It is still unclear that when & why & how it fails. * If small files succeed, large files fail, what's the threshold? (or timeout?) Is it caused by reverse proxy or Gitea itself, you could try to bypass the reverse proxy and use Gitea's http server directly to test. * If some large files succeed, some large files fail, is it related to file content? I can see you were playing with "nsfw" contents. 2. The logs seem strange * I didn't find "uploading" requests, there are only "saving" requests. * The log shows that the Gitea instance restarted during your operation, is it caused by OOM, or you restarted it manually? There are too many unclear details .... maybe you need to investigate and try to figure out more clues.
Author
Owner

@viknsagit commented on GitHub (Jan 27, 2024):

I tried to upload a 135mb file, and after I clicked the commit changes button, an error came out:
Не удалось загрузить файлы в «» из-за ошибки: Unable to push back to repo from temporary repo: viknsagit/test (/var/lib/gitea/data/tmp/local-repo/upload.git587484273) Error: push failed: exit status 1 - To /var/lib/gitea/data/gitea-repositories/viknsagit/test.git ! [rejected] e1bace633aa6c6c4bfd07b292e5df0a570d3017c -> main (fetch first) error: failed to push some refs to '/var/lib/gitea/data/gitea-repositories/viknsagit/test.git' hint: Updates were rejected because the remote contains work that you do hint: not have locally. This is usually caused by another repository pushing hint: to the same ref. You may want to first integrate the remote changes hint: (e.g., 'git pull ...') before pushing again. hint: See the 'Note about fast-forwards' in 'git push --help' for details. - To /var/lib/gitea/data/gitea-repositories/viknsagit/test.git ! [rejected] e1bace633aa6c6c4bfd07b292e5df0a570d3017c -> main (fetch first) error: failed to push some refs to '/var/lib/gitea/data/gitea-repositories/viknsagit/test.git' hint: Updates were rejected because the remote contains work that you do hint: not have locally. This is usually caused by another repository pushing hint: to the same ref. You may want to first integrate the remote changes hint: (e.g., 'git pull ...') before pushing again. hint: See the 'Note about fast-forwards' in 'git push --help' for details. - To /var/lib/gitea/data/gitea-repositories/viknsagit/test.git ! [rejected] e1bace633aa6c6c4bfd07b292e5df0a570d3017c -> main (fetch first) error: failed to push some refs to '/var/lib/gitea/data/gitea-repositories/viknsagit/test.git' hint: Updates were rejected because the remote contains work that you do hint: not have locally. This is usually caused by another repository pushing hint: to the same ref. You may want to first integrate the remote changes hint: (e.g., 'git pull ...') before pushing again. hint: See the 'Note about fast-forwards' in 'git push --help' for details.

@viknsagit commented on GitHub (Jan 27, 2024): I tried to upload a 135mb file, and after I clicked the commit changes button, an error came out: Не удалось загрузить файлы в «» из-за ошибки: Unable to push back to repo from temporary repo: viknsagit/test (/var/lib/gitea/data/tmp/local-repo/upload.git587484273) Error: push failed: exit status 1 - To /var/lib/gitea/data/gitea-repositories/viknsagit/test.git ! [rejected] e1bace633aa6c6c4bfd07b292e5df0a570d3017c -> main (fetch first) error: failed to push some refs to '/var/lib/gitea/data/gitea-repositories/viknsagit/test.git' hint: Updates were rejected because the remote contains work that you do hint: not have locally. This is usually caused by another repository pushing hint: to the same ref. You may want to first integrate the remote changes hint: (e.g., 'git pull ...') before pushing again. hint: See the 'Note about fast-forwards' in 'git push --help' for details. - To /var/lib/gitea/data/gitea-repositories/viknsagit/test.git ! [rejected] e1bace633aa6c6c4bfd07b292e5df0a570d3017c -> main (fetch first) error: failed to push some refs to '/var/lib/gitea/data/gitea-repositories/viknsagit/test.git' hint: Updates were rejected because the remote contains work that you do hint: not have locally. This is usually caused by another repository pushing hint: to the same ref. You may want to first integrate the remote changes hint: (e.g., 'git pull ...') before pushing again. hint: See the 'Note about fast-forwards' in 'git push --help' for details. - To /var/lib/gitea/data/gitea-repositories/viknsagit/test.git ! [rejected] e1bace633aa6c6c4bfd07b292e5df0a570d3017c -> main (fetch first) error: failed to push some refs to '/var/lib/gitea/data/gitea-repositories/viknsagit/test.git' hint: Updates were rejected because the remote contains work that you do hint: not have locally. This is usually caused by another repository pushing hint: to the same ref. You may want to first integrate the remote changes hint: (e.g., 'git pull ...') before pushing again. hint: See the 'Note about fast-forwards' in 'git push --help' for details.
Author
Owner

@viknsagit commented on GitHub (Jan 27, 2024):

Tried to download again and it worked. Now I'm going to try heavier files.

@viknsagit commented on GitHub (Jan 27, 2024): Tried to download again and it worked. Now I'm going to try heavier files.
Author
Owner

@viknsagit commented on GitHub (Jan 27, 2024):

A 254mb file has been uploaded.

@viknsagit commented on GitHub (Jan 27, 2024): A 254mb file has been uploaded.
Author
Owner

@viknsagit commented on GitHub (Jan 27, 2024):

In another repository where I'm trying to upload files from the neural network to classify nsfw, the same error came out as last time. Now I'm going to try other files.

@viknsagit commented on GitHub (Jan 27, 2024): In another repository where I'm trying to upload files from the neural network to classify nsfw, the same error came out as last time. Now I'm going to try other files.
Author
Owner

@viknsagit commented on GitHub (Jan 27, 2024):

Tried to download a 0.7 GB file
image
I have a suspicion of nginx

@viknsagit commented on GitHub (Jan 27, 2024): Tried to download a 0.7 GB file ![image](https://github.com/go-gitea/gitea/assets/80544536/7a1d9b16-f9e0-4abe-92a1-1f7c65a88530) I have a suspicion of nginx
Author
Owner

@viknsagit commented on GitHub (Jan 27, 2024):

When uploading large files, an error occurs and gitea restarts

@viknsagit commented on GitHub (Jan 27, 2024): When uploading large files, an error occurs and gitea restarts
Author
Owner

@viknsagit commented on GitHub (Jan 27, 2024):

image
image
Accessed the site via HTTP

@viknsagit commented on GitHub (Jan 27, 2024): ![image](https://github.com/go-gitea/gitea/assets/80544536/31acd710-3dd7-4fb1-8f82-e96d784bf088) ![image](https://github.com/go-gitea/gitea/assets/80544536/83fee130-b386-4d5f-a900-d844d0ca6fe8) Accessed the site via HTTP
Author
Owner

@wxiaoguang commented on GitHub (Jan 27, 2024):

When uploading large files, an error occurs and gitea restarts

It seems related. What does dmesg -T say? Maybe Gitea itself is killed by OOM.

@wxiaoguang commented on GitHub (Jan 27, 2024): > When uploading large files, an error occurs and gitea restarts It seems related. What does `dmesg -T` say? Maybe Gitea itself is killed by OOM.
Author
Owner

@viknsagit commented on GitHub (Jan 27, 2024):

image
I don't have enough memory)

@viknsagit commented on GitHub (Jan 27, 2024): ![image](https://github.com/go-gitea/gitea/assets/80544536/6978262c-8862-47df-8cdc-a93d49ad11b2) I don't have enough memory)
Author
Owner

@viknsagit commented on GitHub (Jan 27, 2024):

It turns out that the server was created with 2 gb of ram

@viknsagit commented on GitHub (Jan 27, 2024): It turns out that the server was created with 2 gb of ram
Author
Owner

@viknsagit commented on GitHub (Jan 27, 2024):

I added more RAM to the server and everything worked.

@viknsagit commented on GitHub (Jan 27, 2024): I added more RAM to the server and everything worked.
Author
Owner

@github-actions[bot] commented on GitHub (Feb 29, 2024):

Automatically locked because of our CONTRIBUTING guidelines

@github-actions[bot] commented on GitHub (Feb 29, 2024): Automatically locked because of our [CONTRIBUTING guidelines](https://github.com/go-gitea/gitea/blob/main/CONTRIBUTING.md#issue-locking)
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/gitea#12371