LFS: file is undownloadable with EOF error, git lfs push --all origin master from source repo (where the full file is still present) doesn't fix it #868

New Issue

GiteaMirror · 2025-11-02T03:39:41-06:00

GiteaMirror commented

2025-11-02 03:39:41 -06:00

Originally created by @ghost on GitHub (Jul 4, 2017).

Gitea version (or commit ref): 9e627af built with: bindata, sqlite
Git version: 2.11.2
Operating system: whatever the default docker image uses
Database (use [x]):
- PostgreSQL
- MySQL
- MSSQL
- SQLite
Can you reproduce the bug at https://try.gitea.io:
- Yes (provide example URL)
- No - don't know how to reproduce!
- Not relevant
Log gist:

Description

I have an LFS file which cannot be downloaded: every pull that involves it will give an EOF error as shown below. git lfs push --all origin master from source repo (where the full file is still present) doesn't fix it, the lfs push doesn't return any error but the problem simply doesn't go away.

Originally created by @ghost on GitHub (Jul 4, 2017). - Gitea version (or commit ref): `9e627af built with: bindata, sqlite` - Git version: `2.11.2` - Operating system: whatever the default docker image uses - Database (use `[x]`): - [X] PostgreSQL - [ ] MySQL - [ ] MSSQL - [ ] SQLite - Can you reproduce the bug at https://try.gitea.io: - [ ] Yes (provide example URL) - [X] No - don't know how to reproduce! - [ ] Not relevant - Log gist: ## Description I have an LFS file which cannot be downloaded: every pull that involves it will give an EOF error as shown below. `git lfs push --all origin master` from source repo (where the full file is still present) doesn't fix it, the lfs push doesn't return any error but the problem simply doesn't go away. ``` ```

GiteaMirror added the type/bug label 2025-11-02 03:39:41 -06:00

GiteaMirror closed this issue

2025-11-02 03:39:42 -06:00

GiteaMirror commented

2025-11-02 03:39:42 -06:00

@ghost commented on GitHub (Jul 5, 2017):

Also please note that the git-lfs client is built to absolutely rely on the server to have successfully stored the file when it gets a positive HTTP response at the end of the upload. Therefore if gitea's LFS server doesn't do this yet, it would probably be a good idea to have some sort of hook verify that the whole file arrived before sending a positive HTTP response code. (There is also the "verify" protocol feature I believe where you can require the client to check on some URL itself if the whole file is available)

@ghost commented on GitHub (Jul 5, 2017): Also please note that the git-lfs client is built to absolutely rely on the server to have successfully stored the file when it gets a positive HTTP response at the end of the upload. Therefore if gitea's LFS server doesn't do this yet, it would probably be a good idea to have some sort of hook verify that the whole file arrived before sending a positive HTTP response code. (There is also the "verify" protocol feature I believe where you can require the client to check on some URL itself if the whole file is available)

GiteaMirror commented

2025-11-02 03:39:42 -06:00

@threadproc commented on GitHub (Nov 6, 2017):

+1 on this being required, using Git LFS for any number of large files frequently results in them not being able to be pulled anymore.

@JonasT have you found a way to re-upload the files manually in this circumstance?

@threadproc commented on GitHub (Nov 6, 2017): +1 on this being required, using Git LFS for any number of large files frequently results in them not being able to be pulled anymore. @JonasT have you found a way to re-upload the files manually in this circumstance?

GiteaMirror commented

2025-11-02 03:39:43 -06:00

@technoweenie commented on GitHub (Nov 6, 2017):

Hey 👋, I'm one of the Git LFS authors. I don't know how Gitea works, but the Git LFS client will attempt to verify uploads if the server desires.

https://github.com/git-lfs/git-lfs/blob/master/docs/api/basic-transfers.md#verification

This is useful in cases where an LFS server uses external cloud storage (like S3) without any proxying. By adding a verify action to the Batch API response, Git LFS can make another call to the LFS server, giving it a chance to verify the file has been successfully uploaded.

If you can delete the file from the server, you can then run git lfs push --all on a clone with a valid .git/lfs/objects/03/4c/034c77f1f05ec89421e4a63f0e3a4ca1ecf852cc6d2bf611f126f275728e017d file to re-upload to the server.

@technoweenie commented on GitHub (Nov 6, 2017): Hey :wave:, I'm one of the Git LFS authors. I don't know how Gitea works, but the Git LFS client will attempt to verify uploads if the server desires. https://github.com/git-lfs/git-lfs/blob/master/docs/api/basic-transfers.md#verification This is useful in cases where an LFS server uses external cloud storage (like S3) without any proxying. By adding a `verify` action to the Batch API response, Git LFS can make another call to the LFS server, giving it a chance to verify the file has been successfully uploaded. If you can delete the file from the server, you can then run `git lfs push --all` on a clone with a valid `.git/lfs/objects/03/4c/034c77f1f05ec89421e4a63f0e3a4ca1ecf852cc6d2bf611f126f275728e017d` file to re-upload to the server.

GiteaMirror commented

2025-11-02 03:39:43 -06:00

@threadproc commented on GitHub (Nov 6, 2017):

@technoweenie Interestingly enough, I've now deleted and repushed the particular file that I'm running into issues with to the server and I'm still getting the same error. I actually reuploaded all of the LFS objects, and it's still the same file. I've checked the MD5 sum of all of the files (the original file, the file in .git/lfs/objects/{OID_PATH} on the original repo, the LFS object on the server, and the incomplete file on the broken clone .git/lfs/objects/incomplete/{OID}.tmp) and they all actually match. I'm going to poke around some more and see why it might be getting an EOF error on this.

I will also look into adding the verify action, thanks for that hint!

Edit: just did a fresh clone, md5sums still match and now it's giving me several files that aren't quite right. I'll update with anything new I find out while I poke at this.

@threadproc commented on GitHub (Nov 6, 2017): @technoweenie Interestingly enough, I've now deleted and repushed the particular file that I'm running into issues with to the server and I'm still getting the same error. I actually reuploaded all of the LFS objects, and it's still the same file. I've checked the MD5 sum of all of the files (the original file, the file in `.git/lfs/objects/{OID_PATH}` on the original repo, the LFS object on the server, and the incomplete file on the broken clone `.git/lfs/objects/incomplete/{OID}.tmp`) and they all actually match. I'm going to poke around some more and see why it might be getting an EOF error on this. I will also look into adding the `verify` action, thanks for that hint! Edit: just did a fresh clone, md5sums still match and now it's giving me several files that aren't quite right. I'll update with anything new I find out while I poke at this.

GiteaMirror commented

2025-11-02 03:39:43 -06:00

@lafriks commented on GitHub (Nov 6, 2017):

@technoweenie is it common behaviour for LFS servers to return verify in response even if LFS files are stored localy on server?
Just for info gitea LFS implementation is mostly based on LFS test server code

@lafriks commented on GitHub (Nov 6, 2017): @technoweenie is it common behaviour for LFS servers to return verify in response even if LFS files are stored localy on server? Just for info gitea LFS implementation is mostly based on LFS test server code

GiteaMirror commented

2025-11-02 03:39:43 -06:00

@threadproc commented on GitHub (Nov 6, 2017):

Submitted a PR that fixes this issue for me. The problem was that it was returning a 206 response (Partial Content) but providing the full file size as the Content-Length instead of the truncated size.

@threadproc commented on GitHub (Nov 6, 2017): Submitted a PR that fixes this issue for me. The problem was that it was returning a 206 response (Partial Content) but providing the full file size as the Content-Length instead of the truncated size.

GiteaMirror commented

2025-11-02 03:39:43 -06:00

@technoweenie commented on GitHub (Nov 6, 2017):

@lafriks I don't know. GitHub.com currently uses verify, while GitHub Enterprise does not (due to a custom upload server). So, we test both.

If you're comfortable that your LFS server can verify that the exact contents have been saved to the storage layer, adding the verify action will slow things down. But, I suppose double checking doesn't hurt much :) This only happens the first time a unique LFS OID is uploaded, so it's not a big perf hit.

@technoweenie commented on GitHub (Nov 6, 2017): @lafriks I don't know. GitHub.com currently uses `verify`, while GitHub Enterprise does not (due to a custom upload server). So, we test both. If you're comfortable that your LFS server can verify that the _exact_ contents have been saved to the storage layer, adding the `verify` action will slow things down. But, I suppose double checking doesn't hurt much :) This only happens the first time a unique LFS OID is uploaded, so it's not a big perf hit.

GiteaMirror commented

2025-11-02 03:39:44 -06:00

@lafriks commented on GitHub (Nov 6, 2017):

@techknowlogick thanks for your answer. So probably we need to add that verify until we implement proper server-side verification

@lafriks commented on GitHub (Nov 6, 2017): @techknowlogick thanks for your answer. So probably we need to add that `verify` until we implement proper server-side verification

GiteaMirror commented

2025-11-02 03:39:44 -06:00

@techknowlogick commented on GitHub (Nov 6, 2017):

lol. wrong tech

ping @technoweenie

@techknowlogick commented on GitHub (Nov 6, 2017): lol. wrong tech ping @technoweenie

GiteaMirror commented

2025-11-02 03:39:44 -06:00

@lafriks commented on GitHub (Nov 6, 2017):

@techknowlogick sorry :)

@lafriks commented on GitHub (Nov 6, 2017): @techknowlogick sorry :)

GiteaMirror commented

2025-11-02 03:39:44 -06:00

@lafriks commented on GitHub (Nov 6, 2017):

@technoweenie what status should /verify handler return if verification fails and should it remove lfs object from server store if for example size differs?

@lafriks commented on GitHub (Nov 6, 2017): @technoweenie what status should /verify handler return if verification fails and should it remove lfs object from server store if for example size differs?

GiteaMirror commented

2025-11-02 03:39:44 -06:00

@technoweenie commented on GitHub (Nov 7, 2017):

I'd say a 422. LFS will follow any 3xx redirects, and treats 4xx and 5xx statuses as errors. A 4xx error is more appropriate in this case, as a 5xx would imply some kind of server issue (DB down, disk full, etc).

@technoweenie commented on GitHub (Nov 7, 2017): I'd say a 422. LFS will follow any 3xx redirects, and treats 4xx and 5xx statuses as errors. A 4xx error is more appropriate in this case, as a 5xx would imply some kind of server issue (DB down, disk full, etc).

GiteaMirror referenced this issue

2025-11-02 11:51:39 -06:00

[PR #926] [MERGED] Batch updates for issues #15656

Sign in to join this conversation.