Error when cloning: Smudge error #3567

Originally created by @bkodre on GitHub (Jul 10, 2019).

Gitea version (or commit ref):1.8.3
Git version:2.22.0.windows.1
Operating system:Windows server 2012 R2
Database (use [x]):
- PostgreSQL
- MySQL
- MSSQL
- SQLite
Can you reproduce the bug at https://try.gitea.io:
- Yes (provide example URL)
- No
- Not relevant
Log gist:

Description

When cloning repository i get error on downloading file tracked in lfs. Clone does not fail every time, but when it does it has always different file as a reason.

Error downloading object: Flash/lib/greensock.swc (1b0c435): Smudge error: Error downloading Flash/lib/greensock.swc (1b0c43569cad83be13b26dc11253aa696c42a8797d57a95983b3fde97bdbcf5d): batch response: Repository or object not found: http://hostname:3000/company/product.git/info/lfs/objects/batch
Check that it exists and that you have proper access to it

Originally created by @bkodre on GitHub (Jul 10, 2019). - Gitea version (or commit ref):1.8.3 - Git version:2.22.0.windows.1 - Operating system:Windows server 2012 R2 - Database (use `[x]`): - [ ] PostgreSQL - [x] MySQL - [ ] MSSQL - [ ] SQLite - Can you reproduce the bug at https://try.gitea.io: - [ ] Yes (provide example URL) - [x] No - [ ] Not relevant - Log gist: ## Description When cloning repository i get error on downloading file tracked in lfs. Clone does not fail every time, but when it does it has always different file as a reason. Error downloading object: Flash/lib/greensock.swc (1b0c435): Smudge error: Error downloading Flash/lib/greensock.swc (1b0c43569cad83be13b26dc11253aa696c42a8797d57a95983b3fde97bdbcf5d): batch response: Repository or object not found: http://hostname:3000/company/product.git/info/lfs/objects/batch Check that it exists and that you have proper access to it

GiteaMirror closed this issue

GiteaMirror commented

@lunny commented on GitHub (Jul 10, 2019):

It seems v1.9 fixed that.

@lunny commented on GitHub (Jul 10, 2019): It seems v1.9 fixed that.

GiteaMirror commented

@bkodre commented on GitHub (Jul 10, 2019):

When version 1.9 will be released.
I get this error verfy frequently

@bkodre commented on GitHub (Jul 10, 2019): When version 1.9 will be released. I get this error verfy frequently

GiteaMirror commented

@zeripath commented on GitHub (Jul 10, 2019):

Basically the error is saying that LFS was unable to download the file.

I think there are a number of possible reasons for this. However from your response it's not easy to determine which one of these is definitely responsible.

The most scary for you should be the situation that these LFS files are not in your repository. I think it's worth ensuring that they are and if not get someone who does have them to do a git lfs push --all (see here
https://github.com/git-lfs/git-lfs/issues/1113#issuecomment-203484024 )

I have a pr open that will allow you to look at your repo and try to find all of the possible LFS pointers your repo has and compare them with the LFS files it knows about and those you can access allowing you to match these up and fix errors.

It's not necessarily Gitea's fault that this has happened if it has but converse of #732 (i.e. if you could merge LFS adding prs from forks) might have caused it. On 1.9 that should no longer happen but I seriously apologize if you were affected by that - all I can say is that getting LFS right is not as easy as it first appears.

@zeripath commented on GitHub (Jul 10, 2019): Basically the error is saying that LFS was unable to download the file. I think there are a number of possible reasons for this. However from your response it's not easy to determine which one of these is definitely responsible. The most scary for you should be the situation that these LFS files are not in your repository. I think it's worth ensuring that they are and if not get someone who does have them to do a git lfs push --all (see here https://github.com/git-lfs/git-lfs/issues/1113#issuecomment-203484024 ) I have a pr open that will allow you to look at your repo and try to find all of the possible LFS pointers your repo has and compare them with the LFS files it knows about and those you can access allowing you to match these up and fix errors. It's not necessarily Gitea's fault that this has happened if it has but converse of #732 (i.e. if you could merge LFS adding prs from forks) might have caused it. On 1.9 that should no longer happen but I seriously apologize if you were affected by that - all I can say is that getting LFS right is not as easy as it first appears.

GiteaMirror commented

@zeripath commented on GitHub (Jul 10, 2019):

A good test might be to try git lfs fetch --all to see if there are LFS objects missing - check the Gitea logs as well on errors to see if

@zeripath commented on GitHub (Jul 10, 2019): A good test might be to try `git lfs fetch --all` to see if there are LFS objects missing - check the Gitea logs as well on errors to see if

GiteaMirror commented

@bkodre commented on GitHub (Jul 11, 2019):

i executed git lfs fetch --all command and it completed successfully ( Downloading LFS objects: 100% (1277/1277)).

The strange thing is that on my pc most of the times clone is done successfully, but my colleague gets smudge error almost every time.

@bkodre commented on GitHub (Jul 11, 2019): i executed `git lfs fetch --all` command and it completed successfully ( Downloading LFS objects: 100% (1277/1277)). The strange thing is that on my pc most of the times clone is done successfully, but my colleague gets smudge error almost every time.

GiteaMirror commented

@zeripath commented on GitHub (Jul 11, 2019):

It might be worth them trying the fetch all technique - it could be that the repository still doesn't have all the objects but you have the missing ones.

Are you cloning over SSH or HTTP? It would be helpful to see the logs when these failures happen.

@zeripath commented on GitHub (Jul 11, 2019): It might be worth them trying the fetch all technique - it could be that the repository still doesn't have all the objects but you have the missing ones. Are you cloning over SSH or HTTP? It would be helpful to see the logs when these failures happen.

GiteaMirror commented

20190711T102700.309593.log

@bkodre commented on GitHub (Jul 11, 2019):

HTTP was used for cloning.

He tried fetch all and got this error in lsf\logs(file attached):

batch response: Repository or object not found: http://hostname:3000/company/product.git/info/lfs/objects/batch
Check that it exists and that you have proper access to it
batch response: Fatal error: Server error: http://hostname:3000/company/product.git/info/lfs/objects/batch

@bkodre commented on GitHub (Jul 11, 2019): HTTP was used for cloning. He tried fetch all and got this error in lsf\logs(file attached): ``` batch response: Repository or object not found: http://hostname:3000/company/product.git/info/lfs/objects/batch Check that it exists and that you have proper access to it batch response: Fatal error: Server error: http://hostname:3000/company/product.git/info/lfs/objects/batch ``` [20190711T102700.309593.log](https://github.com/go-gitea/gitea/files/3381511/20190711T102700.309593.log)

GiteaMirror commented

@zeripath commented on GitHub (Jul 11, 2019):

Ok that implies you have some LFS files missing from your repo - the Gitea logs would be more helpful however.

I don't really know how this has happened. I suspect the underlying issue #732 is to blame.

You need to from your computer, that has all the objects, do a git lfs push --all and then see if git lfs fetch --all on their machine works.

@zeripath commented on GitHub (Jul 11, 2019): Ok that implies you have some LFS files missing from your repo - the Gitea logs would be more helpful however. I don't really know how this has happened. I suspect the underlying issue #732 is to blame. You need to from your computer, that has all the objects, do a `git lfs push --all` and then see if `git lfs fetch --all` on their machine works.

GiteaMirror commented

@bkodre commented on GitHub (Jul 11, 2019):

I'm not sure that files are missing, because on my pc fresh clone most of times works ok.

I attached gitea logs:
gitea.log

I tried git lfs push --all and it failed:

Uploading LFS objects:  38% (3200/8409), 121 MB \| 0 B/s, done                    
batch response: Repository or object not found: http://hostname:3000/company/product.git/info/lfs/objects/batch
Check that it exists and that you have proper access to it

@bkodre commented on GitHub (Jul 11, 2019): I'm not sure that files are missing, because on my pc fresh clone most of times works ok. I attached gitea logs: [gitea.log](https://github.com/go-gitea/gitea/files/3381722/gitea.log) I tried `git lfs push --all` and it failed: ``` Uploading LFS objects: 38% (3200/8409), 121 MB \| 0 B/s, done batch response: Repository or object not found: http://hostname:3000/company/product.git/info/lfs/objects/batch Check that it exists and that you have proper access to it ```

GiteaMirror commented

@zeripath commented on GitHub (Jul 11, 2019):

Thanks for the Gitea log it looks like this could be a mysql connector issue! Someone else recently reported something similar.

[...s/context/context.go:238 func1()] [E] GetAccessTokenBySha: dial tcp 127.0.0.1:3307: connectex: Only one usage of each socket address (protocol/network address/port) is normally permitted.
2019/07/11 08:10:48 [...s/context/context.go:238 func1()] [E] UserSignIn: dial tcp 127.0.0.1:3307: connectex: Only one usage of each socket address (protocol/network address/port) is normally permitted.

@zeripath commented on GitHub (Jul 11, 2019): Thanks for the Gitea log it looks like this could be a mysql connector issue! Someone else recently reported something similar. ``` [...s/context/context.go:238 func1()] [E] GetAccessTokenBySha: dial tcp 127.0.0.1:3307: connectex: Only one usage of each socket address (protocol/network address/port) is normally permitted. 2019/07/11 08:10:48 [...s/context/context.go:238 func1()] [E] UserSignIn: dial tcp 127.0.0.1:3307: connectex: Only one usage of each socket address (protocol/network address/port) is normally permitted. ```

GiteaMirror commented

@bkodre commented on GitHub (Jul 11, 2019):

Any idea what can be done to resolve this issue with mysql connector?

@bkodre commented on GitHub (Jul 11, 2019): Any idea what can be done to resolve this issue with mysql connector?

GiteaMirror commented

@zeripath commented on GitHub (Jul 11, 2019):

Ah it was you who reported it...

@zeripath commented on GitHub (Jul 11, 2019): Ah it was you who reported it...

GiteaMirror commented

@bkodre commented on GitHub (Jul 11, 2019):

Yes it was me, but i didn't found solution to it and i was not sure if these errors are connected.

@bkodre commented on GitHub (Jul 11, 2019): Yes it was me, but i didn't found solution to it and i was not sure if these errors are connected.

GiteaMirror commented

@zeripath commented on GitHub (Jul 11, 2019):

If you look at your log you get some authentication errors in the middle of these errors - I wonder if the db connection failure is causing authentication to fail.

@zeripath commented on GitHub (Jul 11, 2019): If you look at your log you get some authentication errors in the middle of these errors - I wonder if the db connection failure is causing authentication to fail.

GiteaMirror commented

@bkodre commented on GitHub (Jul 11, 2019):

My credentials are saved in git client, so probably db connection is to blame for authentication fail.

@bkodre commented on GitHub (Jul 11, 2019): My credentials are saved in git client, so probably db connection is to blame for authentication fail.

GiteaMirror commented

https://blogs.msdn.microsoft.com/dgorti/2005/09/18/only-one-usage-of-each-socket-address-protocolnetwork-addressport-is-normally-permitted/

@zeripath commented on GitHub (Jul 11, 2019):

@zeripath commented on GitHub (Jul 11, 2019): https://blogs.msdn.microsoft.com/dgorti/2005/09/18/only-one-usage-of-each-socket-address-protocolnetwork-addressport-is-normally-permitted/

GiteaMirror commented

@zeripath commented on GitHub (Jul 11, 2019):

But we're not the first people to come across this issue and I wonder if following the techniques here might fix this: https://github.com/restic/restic/issues/791

@zeripath commented on GitHub (Jul 11, 2019): But we're not the first people to come across this issue and I wonder if following the techniques here might fix this: https://github.com/restic/restic/issues/791

GiteaMirror commented

@bkodre commented on GitHub (Jul 11, 2019):

I have set MaxUserPort to 65535 and TcpTimedWaitDelay to 30. Now it's working ok.

@bkodre commented on GitHub (Jul 11, 2019): I have set MaxUserPort to 65535 and TcpTimedWaitDelay to 30. Now it's working ok.

GiteaMirror commented

@zeripath commented on GitHub (Jul 11, 2019):

Awesome.

@zeripath commented on GitHub (Jul 11, 2019): Awesome.

GiteaMirror commented