checking out repo with LFS fails - keeps on trying to use http authentication #533

@fabian-z commented on GitHub (Mar 17, 2017):

It shows

AccessDownload=none
AccessUpload=none

for me when checking a LFS repository I only used over SSH.

If you are not already using the built-in SSH server, you could try that to rule out issues with your SSH setup. Just add START_SSH_SERVER = true to your configuration.

@fabian-z commented on GitHub (Mar 17, 2017): It shows ``` AccessDownload=none AccessUpload=none ``` for me when checking a LFS repository I only used over SSH. If you are not already using the built-in SSH server, you could try that to rule out issues with your SSH setup. Just add ```START_SSH_SERVER = true``` to your [configuration](https://docs.gitea.io/en-us/config-cheat-sheet/).

GiteaMirror commented

@fabian-z commented on GitHub (Mar 17, 2017):

Please also include Gitea debug logs for a clone attempt.

@fabian-z commented on GitHub (Mar 17, 2017): Please also include Gitea debug logs for a clone attempt.

GiteaMirror commented

@kubatyszko commented on GitHub (Mar 17, 2017):

I am using builtin SSH server.
I am cloning repo over SSH (and it correctly gets the JWT token over SSH).

I'll be posting logs later investigating another issue related to private repo with LFS cloning

@kubatyszko commented on GitHub (Mar 17, 2017): I am using builtin SSH server. I am cloning repo over SSH (and it correctly gets the JWT token over SSH). I'll be posting logs later investigating another issue related to private repo with LFS cloning

GiteaMirror commented

@kubatyszko commented on GitHub (Mar 17, 2017):

I'm going to post it here, since the issue may be related, basically I'm trying to clone a repo (that's marked as private).
I added a ton of debugging information, and I found that at some point it keeps on trying to authenticate me to a repoID==2 while token is granted (claims) for repo==4.
Repo 2 belongs to completely different user, Repo 4 is correct.

The debug info that I added is here:


diff --git a/modules/lfs/server.go b/modules/lfs/server.go
index 44ab082..c18e966 100644
--- a/modules/lfs/server.go
+++ b/modules/lfs/server.go
@@ -84,6 +84,7 @@ func ObjectOidHandler(ctx *context.Context) {
 		return
 	}

+	log.Debug("LFS content OID handler")
 	if ctx.Req.Method == "GET" || ctx.Req.Method == "HEAD" {
 		if MetaMatcher(ctx.Req) {
 			GetMetaHandler(ctx)
@@ -118,6 +119,7 @@ func GetContentHandler(ctx *context.Context) {
 		return
 	}

+	log.Debug("LFS content handler, repo %d", repository.ID)
 	if !authenticate(ctx, repository, rv.Authorization, false) {
 		requireAuth(ctx)
 		return
@@ -178,6 +180,7 @@ func GetMetaHandler(ctx *context.Context) {
 		return
 	}

+	log.Debug("LFS content meta handler, repo %d", repository.ID)
 	if !authenticate(ctx, repository, rv.Authorization, false) {
 		requireAuth(ctx)
 		return
@@ -217,6 +220,7 @@ func PostHandler(ctx *context.Context) {
 		return
 	}

+	log.Debug("LFS content post handler, repo %d", repository.ID)
 	if !authenticate(ctx, repository, rv.Authorization, true) {
 		requireAuth(ctx)
 	}
@@ -276,6 +280,7 @@ func BatchHandler(ctx *context.Context) {
 			requireWrite = true
 		}

+	log.Debug("LFS content batch handler, repo %d", repository.ID)
 		if !authenticate(ctx, repository, object.Authorization, requireWrite) {
 			requireAuth(ctx)
 			return
@@ -324,6 +329,7 @@ func PutHandler(ctx *context.Context) {
 		return
 	}

+	log.Debug("LFS content put handler, repo %d", repository.ID)
 	if !authenticate(ctx, repository, rv.Authorization, true) {
 		requireAuth(ctx)
 		return
@@ -453,6 +459,9 @@ func logRequest(r macaron.Request, status int) {
 // or not to proceed. This server assumes an HTTP Basic auth format.
 func authenticate(ctx *context.Context, repository *models.Repository, authorization string, requireWrite bool) bool {

+
+	log.Debug("LFS KUBA, authenticate, requirewrite: %b, auth string: %s , for repository %d", requireWrite, authorization, repository.ID)
+
 	accessMode := models.AccessModeRead
 	if requireWrite {
 		accessMode = models.AccessModeWrite
@@ -461,23 +470,29 @@ func authenticate(ctx *context.Context, repository *models.Repository, authoriza
 	if !repository.IsPrivate && !requireWrite {
 		return true
 	}
+	log.Debug("LFS KUBA, repo is not public or requires write access")

 	if ctx.IsSigned {
 		accessCheck, _ := models.HasAccess(ctx.User.ID, repository, accessMode)
 		return accessCheck
 	}
+	log.Debug("LFS KUBA, request is signed" )

 	if authorization == "" {
 		return false
 	}
+	log.Debug("LFS KUBA, auth is not empty")

 	if authenticateToken(repository, authorization, requireWrite) {
+	 log.Debug("LFS KUBA, token did return true ")
 		return true
 	}
+	log.Debug("LFS KUBA, token did not return true ")

 	if !strings.HasPrefix(authorization, "Basic ") {
 		return false
 	}
+	log.Debug("LFS KUBA, authorization does not start with basic")

 	c, err := base64.StdEncoding.DecodeString(strings.TrimPrefix(authorization, "Basic "))
 	if err != nil {
@@ -507,6 +522,7 @@ func authenticateToken(repository *models.Repository, authorization string, requ
 	if !strings.HasPrefix(authorization, "Bearer ") {
 		return false
 	}
+	log.Debug("LFS KUBA, token auth, bearer is there")

 	token, err := jwt.Parse(authorization[7:], func(t *jwt.Token) (interface{}, error) {
 		if _, ok := t.Method.(*jwt.SigningMethodHMAC); !ok {
@@ -521,22 +537,29 @@ func authenticateToken(repository *models.Repository, authorization string, requ
 	if !token.Valid || !claimsOk {
 		return false
 	}
+	log.Debug("LFS KUBA, token auth,  claims")

 	opStr, ok := claims["op"].(string)
 	if !ok {
 		return false
 	}
+	log.Debug("LFS KUBA, before require write and upload bug")
+
+	if requireWrite && strings.Split(opStr," ")[0] != "upload" {

-	if requireWrite && opStr != "upload" {
 		return false
 	}
+	log.Debug("LFS KUBA, after require write and upload bug")

 	repoID, ok := claims["repo"].(float64)
 	if !ok {
 		return false
 	}
+	log.Debug("LFS KUBA, claims ")
+	log.Debug("LFS KUBA, repository %d %d %s %d",repository.ID, int64(repoID) , claims["op"], claims["repo"])

 	if repository.ID != int64(repoID) {
+	   log.Debug("LFS KUBA, matching repository.ID to repoID failed")
 		return false
 	}

debug log from gitea is here:


2017/03/17 23:28:23 [T] SSH: Arguments: [serv key-4 --config=/usr/gitea/custom/conf/app.ini]
2017/03/17 23:28:23 [T] SSH: Handshaking for 10.61.2.133:41200
2017/03/17 23:28:23 [T] SSH: Connection from 10.61.2.133:41200 (SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1)
2017/03/17 23:28:23 [T] SSH: Payload: git-lfs-authenticate avd/test.git download
2017/03/17 23:28:23 [T] SSH: Arguments: [serv key-4 --config=/usr/gitea/custom/conf/app.ini]
2017/03/17 23:28:24 [D] Session ID: 3a81e679656eb3ce
2017/03/17 23:28:24 [D] CSRF Token: kT9rQsbxdiGzV-aJpDCT11sDSZI6MTQ4OTc5MzMwNDAyMDE3NzU1OA==
2017/03/17 23:28:24 [D] LFS content batch handler, repo 4
2017/03/17 23:28:24 [D] LFS KUBA, authenticate, requirewrite: %!b(bool=false), auth string: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE0ODk3OTM2MDQsIm5iZiI6MTQ4OTc5MzMwNCwib3AiOiJkb3dubG9hZCIsInJlcG8iOjR9.WziOwS88tT_0Xy6CwXmkGLEWPoscyLdV4mJlNPmb3OQ , for repository 4
2017/03/17 23:28:24 [D] LFS KUBA, repo is not public or requires write access
2017/03/17 23:28:24 [D] LFS KUBA, request is signed
2017/03/17 23:28:24 [D] LFS KUBA, auth is not empty
2017/03/17 23:28:24 [D] LFS KUBA, token auth, bearer is there
2017/03/17 23:28:24 [D] LFS KUBA, token auth,  claims
2017/03/17 23:28:24 [D] LFS KUBA, before require write and upload bug
2017/03/17 23:28:24 [D] LFS KUBA, after require write and upload bug
2017/03/17 23:28:24 [D] LFS KUBA, claims
2017/03/17 23:28:24 [D] LFS KUBA, repository 4 4 download %!d(float64=4)   *************
2017/03/17 23:28:24 [D] LFS KUBA, token did return true
2017/03/17 23:28:24 [D] LFS request - Method: POST, URL: /avd/test.git/info/lfs/objects/batch, Status 200    ************
2017/03/17 23:28:24 [D] Session ID: e187e772a703a8de
2017/03/17 23:28:24 [D] CSRF Token: HPQSLdGSsuU0tVqxfay4FOyJeuk6MTQ4OTc5MzMwNDA0ODI1NjM5MQ==
2017/03/17 23:28:24 [D] LFS content OID handler
2017/03/17 23:28:24 [D] LFS content handler, repo 2       ******************
2017/03/17 23:28:24 [D] LFS KUBA, authenticate, requirewrite: %!b(bool=false), auth string: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE0ODk3OTM2MDQsIm5iZiI6MTQ4OTc5MzMwNCwib3AiOiJkb3dubG9hZCIsInJlcG8iOjR9.WziOwS88tT_0Xy6CwXmkGLEWPoscyLdV4mJlNPmb3OQ , for repository 2
2017/03/17 23:28:24 [D] LFS KUBA, repo is not public or requires write access
2017/03/17 23:28:24 [D] LFS KUBA, request is signed
2017/03/17 23:28:24 [D] LFS KUBA, auth is not empty
2017/03/17 23:28:24 [D] LFS KUBA, token auth, bearer is there
2017/03/17 23:28:24 [D] LFS KUBA, token auth,  claims
2017/03/17 23:28:24 [D] LFS KUBA, before require write and upload bug
2017/03/17 23:28:24 [D] LFS KUBA, after require write and upload bug
2017/03/17 23:28:24 [D] LFS KUBA, claims
2017/03/17 23:28:24 [D] LFS KUBA, repository 2 4 download %!d(float64=4)
2017/03/17 23:28:24 [D] LFS KUBA, matching repository.ID to repoID failed
2017/03/17 23:28:24 [D] LFS KUBA, token did not return true
2017/03/17 23:28:24 [D] LFS request - Method: GET, URL: /avd/test/info/lfs/objects/17e682f060b5f8e47ea04c5c4855908b0a5ad612022260fe50e11ecb0cc0ab76, Status 401
2017/03/17 23:28:25 [D] Session ID: b691b70c9939f4fe
2017/03/17 23:28:25 [D] CSRF Token: lo_8ELrlvpQP-EVBJF44zfYLxuY6MTQ4OTc5MzMwNTU4MjkyNDk3Nw==
2017/03/17 23:28:25 [D] LFS content OID handler
2017/03/17 23:28:25 [D] LFS content handler, repo 2
2017/03/17 23:28:25 [D] LFS KUBA, authenticate, requirewrite: %!b(bool=false), auth string:  , for repository 2
2017/03/17 23:28:25 [D] LFS KUBA, repo is not public or requires write access
2017/03/17 23:28:25 [D] LFS KUBA, request is signed
2017/03/17 23:28:25 [D] LFS request - Method: GET, URL: /avd/test/info/lfs/objects/17e682f060b5f8e47ea04c5c4855908b0a5ad612022260fe50e11ecb0cc0ab76, Status 401
2017/03/17 23:28:27 [T] SSH: Handshaking for 10.61.2.133:41204
2017/03/17 23:28:27 [T] SSH: Connection from 10.61.2.133:41204 (SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1)
2017/03/17 23:28:27 [T] SSH: Payload: git-lfs-authenticate avd/test.git download
2017/03/17 23:28:27 [T] SSH: Arguments: [serv key-4 --config=/usr/gitea/custom/conf/app.ini]

I marked the relevant bit with asterisks, you can see that the first time it goes to batch handler, it passes correct repo (id==4) and the http status on the POST method is 200.
Then, it goes to OID handler and content handler, but then it ends up with repo id==2. - this is why it fails because in the end, the auth token function compares repository.ID with int64(repoID) and simply 2!=4

I haven't investigated why the OID -> content handler thinks it should be repo id==2 yet...

@kubatyszko commented on GitHub (Mar 17, 2017): I'm going to post it here, since the issue may be related, basically I'm trying to clone a repo (that's marked as private). I added a ton of debugging information, and I found that at some point it keeps on trying to authenticate me to a repoID==2 while token is granted (claims) for repo==4. Repo 2 belongs to completely different user, Repo 4 is correct. The debug info that I added is here: ``` diff --git a/modules/lfs/server.go b/modules/lfs/server.go index 44ab082..c18e966 100644 --- a/modules/lfs/server.go +++ b/modules/lfs/server.go @@ -84,6 +84,7 @@ func ObjectOidHandler(ctx *context.Context) { return } + log.Debug("LFS content OID handler") if ctx.Req.Method == "GET" || ctx.Req.Method == "HEAD" { if MetaMatcher(ctx.Req) { GetMetaHandler(ctx) @@ -118,6 +119,7 @@ func GetContentHandler(ctx *context.Context) { return } + log.Debug("LFS content handler, repo %d", repository.ID) if !authenticate(ctx, repository, rv.Authorization, false) { requireAuth(ctx) return @@ -178,6 +180,7 @@ func GetMetaHandler(ctx *context.Context) { return } + log.Debug("LFS content meta handler, repo %d", repository.ID) if !authenticate(ctx, repository, rv.Authorization, false) { requireAuth(ctx) return @@ -217,6 +220,7 @@ func PostHandler(ctx *context.Context) { return } + log.Debug("LFS content post handler, repo %d", repository.ID) if !authenticate(ctx, repository, rv.Authorization, true) { requireAuth(ctx) } @@ -276,6 +280,7 @@ func BatchHandler(ctx *context.Context) { requireWrite = true } + log.Debug("LFS content batch handler, repo %d", repository.ID) if !authenticate(ctx, repository, object.Authorization, requireWrite) { requireAuth(ctx) return @@ -324,6 +329,7 @@ func PutHandler(ctx *context.Context) { return } + log.Debug("LFS content put handler, repo %d", repository.ID) if !authenticate(ctx, repository, rv.Authorization, true) { requireAuth(ctx) return @@ -453,6 +459,9 @@ func logRequest(r macaron.Request, status int) { // or not to proceed. This server assumes an HTTP Basic auth format. func authenticate(ctx *context.Context, repository *models.Repository, authorization string, requireWrite bool) bool { + + log.Debug("LFS KUBA, authenticate, requirewrite: %b, auth string: %s , for repository %d", requireWrite, authorization, repository.ID) + accessMode := models.AccessModeRead if requireWrite { accessMode = models.AccessModeWrite @@ -461,23 +470,29 @@ func authenticate(ctx *context.Context, repository *models.Repository, authoriza if !repository.IsPrivate && !requireWrite { return true } + log.Debug("LFS KUBA, repo is not public or requires write access") if ctx.IsSigned { accessCheck, _ := models.HasAccess(ctx.User.ID, repository, accessMode) return accessCheck } + log.Debug("LFS KUBA, request is signed" ) if authorization == "" { return false } + log.Debug("LFS KUBA, auth is not empty") if authenticateToken(repository, authorization, requireWrite) { + log.Debug("LFS KUBA, token did return true ") return true } + log.Debug("LFS KUBA, token did not return true ") if !strings.HasPrefix(authorization, "Basic ") { return false } + log.Debug("LFS KUBA, authorization does not start with basic") c, err := base64.StdEncoding.DecodeString(strings.TrimPrefix(authorization, "Basic ")) if err != nil { @@ -507,6 +522,7 @@ func authenticateToken(repository *models.Repository, authorization string, requ if !strings.HasPrefix(authorization, "Bearer ") { return false } + log.Debug("LFS KUBA, token auth, bearer is there") token, err := jwt.Parse(authorization[7:], func(t *jwt.Token) (interface{}, error) { if _, ok := t.Method.(*jwt.SigningMethodHMAC); !ok { @@ -521,22 +537,29 @@ func authenticateToken(repository *models.Repository, authorization string, requ if !token.Valid || !claimsOk { return false } + log.Debug("LFS KUBA, token auth, claims") opStr, ok := claims["op"].(string) if !ok { return false } + log.Debug("LFS KUBA, before require write and upload bug") + + if requireWrite && strings.Split(opStr," ")[0] != "upload" { - if requireWrite && opStr != "upload" { return false } + log.Debug("LFS KUBA, after require write and upload bug") repoID, ok := claims["repo"].(float64) if !ok { return false } + log.Debug("LFS KUBA, claims ") + log.Debug("LFS KUBA, repository %d %d %s %d",repository.ID, int64(repoID) , claims["op"], claims["repo"]) if repository.ID != int64(repoID) { + log.Debug("LFS KUBA, matching repository.ID to repoID failed") return false } ``` debug log from gitea is here: ``` 2017/03/17 23:28:23 [T] SSH: Arguments: [serv key-4 --config=/usr/gitea/custom/conf/app.ini] 2017/03/17 23:28:23 [T] SSH: Handshaking for 10.61.2.133:41200 2017/03/17 23:28:23 [T] SSH: Connection from 10.61.2.133:41200 (SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1) 2017/03/17 23:28:23 [T] SSH: Payload: git-lfs-authenticate avd/test.git download 2017/03/17 23:28:23 [T] SSH: Arguments: [serv key-4 --config=/usr/gitea/custom/conf/app.ini] 2017/03/17 23:28:24 [D] Session ID: 3a81e679656eb3ce 2017/03/17 23:28:24 [D] CSRF Token: kT9rQsbxdiGzV-aJpDCT11sDSZI6MTQ4OTc5MzMwNDAyMDE3NzU1OA== 2017/03/17 23:28:24 [D] LFS content batch handler, repo 4 2017/03/17 23:28:24 [D] LFS KUBA, authenticate, requirewrite: %!b(bool=false), auth string: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE0ODk3OTM2MDQsIm5iZiI6MTQ4OTc5MzMwNCwib3AiOiJkb3dubG9hZCIsInJlcG8iOjR9.WziOwS88tT_0Xy6CwXmkGLEWPoscyLdV4mJlNPmb3OQ , for repository 4 2017/03/17 23:28:24 [D] LFS KUBA, repo is not public or requires write access 2017/03/17 23:28:24 [D] LFS KUBA, request is signed 2017/03/17 23:28:24 [D] LFS KUBA, auth is not empty 2017/03/17 23:28:24 [D] LFS KUBA, token auth, bearer is there 2017/03/17 23:28:24 [D] LFS KUBA, token auth, claims 2017/03/17 23:28:24 [D] LFS KUBA, before require write and upload bug 2017/03/17 23:28:24 [D] LFS KUBA, after require write and upload bug 2017/03/17 23:28:24 [D] LFS KUBA, claims 2017/03/17 23:28:24 [D] LFS KUBA, repository 4 4 download %!d(float64=4) ************* 2017/03/17 23:28:24 [D] LFS KUBA, token did return true 2017/03/17 23:28:24 [D] LFS request - Method: POST, URL: /avd/test.git/info/lfs/objects/batch, Status 200 ************ 2017/03/17 23:28:24 [D] Session ID: e187e772a703a8de 2017/03/17 23:28:24 [D] CSRF Token: HPQSLdGSsuU0tVqxfay4FOyJeuk6MTQ4OTc5MzMwNDA0ODI1NjM5MQ== 2017/03/17 23:28:24 [D] LFS content OID handler 2017/03/17 23:28:24 [D] LFS content handler, repo 2 ****************** 2017/03/17 23:28:24 [D] LFS KUBA, authenticate, requirewrite: %!b(bool=false), auth string: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE0ODk3OTM2MDQsIm5iZiI6MTQ4OTc5MzMwNCwib3AiOiJkb3dubG9hZCIsInJlcG8iOjR9.WziOwS88tT_0Xy6CwXmkGLEWPoscyLdV4mJlNPmb3OQ , for repository 2 2017/03/17 23:28:24 [D] LFS KUBA, repo is not public or requires write access 2017/03/17 23:28:24 [D] LFS KUBA, request is signed 2017/03/17 23:28:24 [D] LFS KUBA, auth is not empty 2017/03/17 23:28:24 [D] LFS KUBA, token auth, bearer is there 2017/03/17 23:28:24 [D] LFS KUBA, token auth, claims 2017/03/17 23:28:24 [D] LFS KUBA, before require write and upload bug 2017/03/17 23:28:24 [D] LFS KUBA, after require write and upload bug 2017/03/17 23:28:24 [D] LFS KUBA, claims 2017/03/17 23:28:24 [D] LFS KUBA, repository 2 4 download %!d(float64=4) 2017/03/17 23:28:24 [D] LFS KUBA, matching repository.ID to repoID failed 2017/03/17 23:28:24 [D] LFS KUBA, token did not return true 2017/03/17 23:28:24 [D] LFS request - Method: GET, URL: /avd/test/info/lfs/objects/17e682f060b5f8e47ea04c5c4855908b0a5ad612022260fe50e11ecb0cc0ab76, Status 401 2017/03/17 23:28:25 [D] Session ID: b691b70c9939f4fe 2017/03/17 23:28:25 [D] CSRF Token: lo_8ELrlvpQP-EVBJF44zfYLxuY6MTQ4OTc5MzMwNTU4MjkyNDk3Nw== 2017/03/17 23:28:25 [D] LFS content OID handler 2017/03/17 23:28:25 [D] LFS content handler, repo 2 2017/03/17 23:28:25 [D] LFS KUBA, authenticate, requirewrite: %!b(bool=false), auth string: , for repository 2 2017/03/17 23:28:25 [D] LFS KUBA, repo is not public or requires write access 2017/03/17 23:28:25 [D] LFS KUBA, request is signed 2017/03/17 23:28:25 [D] LFS request - Method: GET, URL: /avd/test/info/lfs/objects/17e682f060b5f8e47ea04c5c4855908b0a5ad612022260fe50e11ecb0cc0ab76, Status 401 2017/03/17 23:28:27 [T] SSH: Handshaking for 10.61.2.133:41204 2017/03/17 23:28:27 [T] SSH: Connection from 10.61.2.133:41204 (SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1) 2017/03/17 23:28:27 [T] SSH: Payload: git-lfs-authenticate avd/test.git download 2017/03/17 23:28:27 [T] SSH: Arguments: [serv key-4 --config=/usr/gitea/custom/conf/app.ini] ``` I marked the relevant bit with asterisks, you can see that the first time it goes to batch handler, it passes correct repo (id==4) and the http status on the POST method is 200. Then, it goes to OID handler and content handler, but then it ends up with repo id==2. - this is why it fails because in the end, the auth token function compares repository.ID with int64(repoID) and simply 2!=4 I haven't investigated why the OID -> content handler thinks it should be repo id==2 yet...

GiteaMirror commented

@kubatyszko commented on GitHub (Mar 18, 2017):

Minor update, so far I traced to the GetLFSMetaObjectByOid, when passed OID (a correct object - I verified in the physical lfs storage), somehow it returns a structure that thinks it belongs to repo id==2 not 4.

This may have something to do with the fact I have 4 repos (across different users), and the reason why it's hard to reproduce is because you might have only 1 repo for testing - this is likely why I can't reproduce it my my own mac either.

My repos might be somehow broken - but I'll keep tracing, since I somehow ended up with this situation there may be a bug somewhere...

@kubatyszko commented on GitHub (Mar 18, 2017): Minor update, so far I traced to the GetLFSMetaObjectByOid, when passed OID (a correct object - I verified in the physical lfs storage), somehow it returns a structure that thinks it belongs to repo id==2 not 4. This *may* have something to do with the fact I have 4 repos (across different users), and the reason why it's hard to reproduce is because you might have only 1 repo for testing - this is likely why I can't reproduce it my my own mac either. My repos might be somehow broken - but I'll keep tracing, since I somehow ended up with this situation there may be a bug somewhere...

GiteaMirror commented

@kubatyszko commented on GitHub (Mar 18, 2017):

More investigating:

gitea=# select * from lfs_meta_object
gitea-# ;
 id |                               oid                                | size | repository_id | created_unix
----+------------------------------------------------------------------+------+---------------+--------------
  1 | 11a77c3d96c06974b53d7f40a577e6813739eb5c811b2a86f59038ea90add772 |    5 |             2 |   1489708249
  2 | 17e682f060b5f8e47ea04c5c4855908b0a5ad612022260fe50e11ecb0cc0ab76 |    4 |             2 |   1489761216
  3 | 87428fc522803d31065e7bce3cf03fe475096631e5e07bbd7a0fde60c4cf25c7 |    2 |             2 |   1489773133
  4 | 11d1d5bba135cd5b16ccd12fd5a1db43065d6692418223b28a17484c36adbac4 |   10 |             2 |   1489773202
  5 | 4b59d598410964bccc7cab68af50e1a35a329b6c50bf90e6f2d27800ebf859af |    5 |             4 |   1489789248
(5 rows)

The object in question (17e682) in database clearly belongs to repo id==2 but is should belong to repo id==4...

I'm not wondering whether it's the batch handler that gave the wrong object OID or whether the OID was incorrectly saved in database somehow...

@kubatyszko commented on GitHub (Mar 18, 2017): More investigating: ``` gitea=# select * from lfs_meta_object gitea-# ; id | oid | size | repository_id | created_unix ----+------------------------------------------------------------------+------+---------------+-------------- 1 | 11a77c3d96c06974b53d7f40a577e6813739eb5c811b2a86f59038ea90add772 | 5 | 2 | 1489708249 2 | 17e682f060b5f8e47ea04c5c4855908b0a5ad612022260fe50e11ecb0cc0ab76 | 4 | 2 | 1489761216 3 | 87428fc522803d31065e7bce3cf03fe475096631e5e07bbd7a0fde60c4cf25c7 | 2 | 2 | 1489773133 4 | 11d1d5bba135cd5b16ccd12fd5a1db43065d6692418223b28a17484c36adbac4 | 10 | 2 | 1489773202 5 | 4b59d598410964bccc7cab68af50e1a35a329b6c50bf90e6f2d27800ebf859af | 5 | 4 | 1489789248 (5 rows) ``` The object in question (17e682) in database clearly belongs to repo id==2 but is should belong to repo id==4... I'm not wondering whether it's the batch handler that gave the wrong object OID or whether the OID was incorrectly saved in database somehow...

GiteaMirror commented

@kubatyszko commented on GitHub (Mar 18, 2017):

I know!!!!

The reason for the OID clash between repos 2 and 4 is that both have a file with the same content - resulting in the same SHA and therefore the same OID.

Database keeps track of what repo the file belongs to (first time added) but when the second repo ended up with the same file it already was in LFS (kind of deduplication).

The solution is likely to have a list of repos in that table instead of a single value.

@kubatyszko commented on GitHub (Mar 18, 2017): I know!!!! The reason for the OID clash between repos 2 and 4 is that both have a file with the same content - resulting in the same SHA and therefore the same OID. Database keeps track of what repo the file belongs to (first time added) but when the second repo ended up with the same file it already was in LFS (kind of deduplication). The solution is likely to have a list of repos in that table instead of a single value.

GiteaMirror commented

@fabian-z commented on GitHub (Mar 18, 2017):

Great debugging, thank you!
Funnily enough, I was just about to post the same conclusion since I could reproduce after your hint with multiple repositories.

Since we now know that the OID collision triggers the failure condition, I will look into reworking the OID <-> Repository relation later today.

@fabian-z commented on GitHub (Mar 18, 2017): Great debugging, thank you! Funnily enough, I was just about to post the same conclusion since I could reproduce after your hint with multiple repositories. Since we now know that the OID collision triggers the failure condition, I will look into reworking the OID <-> Repository relation later today.

GiteaMirror commented

@kubatyszko commented on GitHub (Mar 18, 2017):

Or alternatively have different index on that table so that the same sha can be stored multiple times with each repo it belongs to.
This will actually work because it will keep track of all the repos.

The only catch will be in the content handler - it has to match one of object's repo Id's (a list perhaps?) with repo is the token is for.

@kubatyszko commented on GitHub (Mar 18, 2017): Or alternatively have different index on that table so that the same sha can be stored multiple times with each repo it belongs to. This will actually work because it will keep track of all the repos. The only catch will be in the content handler - it has to match one of object's repo Id's (a list perhaps?) with repo is the token is for.

GiteaMirror commented

@kubatyszko commented on GitHub (Mar 18, 2017):

Great stuff. I'm glad we figured it out.
Looking forward to seeing your solution to this, happy to test whenever.

@kubatyszko commented on GitHub (Mar 18, 2017): Great stuff. I'm glad we figured it out. Looking forward to seeing your solution to this, happy to test whenever.

GiteaMirror commented

@fabian-z commented on GitHub (Mar 19, 2017):

Could you test out my duplicateoids branch?

It did resolve the issue for me in my first tests. However, this still needs validation (see TODOs) in order to prevent a malicious client (with push access to the server) from obtaining object contents by OID without actually proving ownership of or access right for the file.

@fabian-z commented on GitHub (Mar 19, 2017): Could you test out my [duplicateoids branch](https://github.com/fabian-z/gitea/tree/duplicateoids)? It did resolve the issue for me in my first tests. However, this still needs validation (see TODOs) in order to prevent a malicious client (with push access to the server) from obtaining object contents by OID without actually proving ownership of or access right for the file.

GiteaMirror commented

@kubatyszko commented on GitHub (Mar 20, 2017):

Will check together with the other PR. Looking at the code, I see that you didn't change database schema - how are you planning on keeping track of all the repositories that have given file and ensure that it stays there until the last repository deletes it?

I see that you're trying to access the object this way:

-	m := &LFSMetaObject{Oid: oid}
 +	m := &LFSMetaObject{Oid: oid, RepositoryID: repository.ID}

but without changes to database - will it even allow multiple rows with the same OID ?
Looking at the schema, both OID and repository are unique - I don't think database will let you insert multiple rows like that, it would need joint index on both fields together...

@kubatyszko commented on GitHub (Mar 20, 2017): Will check together with the other PR. Looking at the code, I see that you didn't change database schema - how are you planning on keeping track of all the repositories that have given file and ensure that it stays there until the last repository deletes it? I see that you're trying to access the object this way: ``` - m := &LFSMetaObject{Oid: oid} + m := &LFSMetaObject{Oid: oid, RepositoryID: repository.ID} ``` but without changes to database - will it even allow multiple rows with the same OID ? Looking at the schema, both OID and repository are unique - I don't think database will let you insert multiple rows like that, it would need joint index on both fields together...

GiteaMirror commented

@lunny commented on GitHub (Mar 20, 2017):

Oid and RepositoryID is a composite unique

@lunny commented on GitHub (Mar 20, 2017): `Oid` and `RepositoryID` is a composite unique

GiteaMirror commented

@kubatyszko commented on GitHub (Mar 20, 2017):

Ok, I'll verify that as well tomorrow, I'm pretty good with databases, but looking at the xorm spec it's not clear to me that the index (and therefore uniqueness) is composite.

@kubatyszko commented on GitHub (Mar 20, 2017): Ok, I'll verify that as well tomorrow, I'm pretty good with databases, but looking at the xorm spec it's not clear to me that the index (and therefore uniqueness) is composite.

GiteaMirror commented

@fabian-z commented on GitHub (Mar 20, 2017):

how are you planning on keeping track of all the repositories that have given file and ensure that it stays there until the last repository deletes it?

The data structure is already there since each object is saved together with its repository id. It already works with forks and deletions of repositories.

looking at the xorm spec it's not clear to me that the index (and therefore uniqueness) is composite.

Have a look at the column definiton manual

@fabian-z commented on GitHub (Mar 20, 2017): > how are you planning on keeping track of all the repositories that have given file and ensure that it stays there until the last repository deletes it? The data structure is already there since each object is saved together with its repository id. It already works with forks and deletions of repositories. > looking at the xorm spec it's not clear to me that the index (and therefore uniqueness) is composite. Have a look at the [column definiton manual](https://github.com/go-xorm/manual-en-US/blob/master/chapter-02/4.columns.md)

GiteaMirror commented

@kubatyszko commented on GitHub (Mar 20, 2017):

Ok, I'm clear now on the indexing, verified that my database shows correct index:

gitea=# \d lfs_meta_object;
                                    Table "public.lfs_meta_object"
    Column     |          Type          |                          Modifiers
---------------+------------------------+--------------------------------------------------------------
 id            | bigint                 | not null default nextval('lfs_meta_object_id_seq'::regclass)
 oid           | character varying(255) | not null
 size          | bigint                 | not null
 repository_id | bigint                 | not null
 created_unix  | bigint                 |
Indexes:
    "lfs_meta_object_pkey" PRIMARY KEY, btree (id)
    "UQE_lfs_meta_object_S" **UNIQUE, btree (oid, repository_id)**
    "IDX_lfs_meta_object_oid" btree (oid)
    "IDX_lfs_meta_object_repository_id" btree (repository_id)

gitea=#

I verified both of your patches - the sshfix and duplicateoids,

I tried interacting with multiple repos and ensure that the files with the same oid are saved in the database, you can see at least 87428 and a6a00 repeated twice (3,13 ; 9,12 ).

gitea=# select * from lfs_meta_object;
 id |                               oid                                | size  | repository_id | created_unix
----+------------------------------------------------------------------+-------+---------------+--------------
  1 | 11a77c3d96c06974b53d7f40a577e6813739eb5c811b2a86f59038ea90add772 |     5 |             2 |   1489708249
  2 | 17e682f060b5f8e47ea04c5c4855908b0a5ad612022260fe50e11ecb0cc0ab76 |     4 |             2 |   1489761216
  3 | 87428fc522803d31065e7bce3cf03fe475096631e5e07bbd7a0fde60c4cf25c7 |     2 |             2 |   1489773133
  4 | 11d1d5bba135cd5b16ccd12fd5a1db43065d6692418223b28a17484c36adbac4 |    10 |             2 |   1489773202
  5 | 4b59d598410964bccc7cab68af50e1a35a329b6c50bf90e6f2d27800ebf859af |     5 |             4 |   1489789248
  6 | 01b031d1ac043038117415718d17cebd5b1fe7a7625e80fa13e9bace51f8a7e0 | 56832 |             5 |   1489809799
  7 | 186819b8b188e3708c522d8f24de87b085d3e7d483f7c0327c453fd55a578031 | 56832 |             5 |   1489810672
  8 | 2f866aa2e1a8717d94bebacf25e86e94f03b7a3de1d8d5fb728840257d4b6b4e |    11 |             2 |   1490029453
  9 | a6a00c11d892f403bba36281d9cbfed574ea562c666aeca47098fddef07d0b15 |    14 |             2 |   1490029631
 10 | 06060c3fbdf2873ce55fdb95fa16f901143e7ffc473183000c29fb3858462d78 |    26 |             2 |   1490029749
 11 | 7f8b1dfc466b6249f06cbe55c9174df2578e7754da793fded244ef5cba2a38f1 |     8 |             2 |   1490029784
 12 | a6a00c11d892f403bba36281d9cbfed574ea562c666aeca47098fddef07d0b15 |    14 |             5 |   1490030070
 13 | 87428fc522803d31065e7bce3cf03fe475096631e5e07bbd7a0fde60c4cf25c7 |     2 |             5 |   1490030070
(13 rows)

gitea=#

All seems to be working well, I'm not getting any prompts for http user/password and no issues with lfs so far.

Thank you.

Consider this my LGTM for both patches.

@kubatyszko commented on GitHub (Mar 20, 2017): Ok, I'm clear now on the indexing, verified that my database shows correct index: ``` gitea=# \d lfs_meta_object; Table "public.lfs_meta_object" Column | Type | Modifiers ---------------+------------------------+-------------------------------------------------------------- id | bigint | not null default nextval('lfs_meta_object_id_seq'::regclass) oid | character varying(255) | not null size | bigint | not null repository_id | bigint | not null created_unix | bigint | Indexes: "lfs_meta_object_pkey" PRIMARY KEY, btree (id) "UQE_lfs_meta_object_S" **UNIQUE, btree (oid, repository_id)** "IDX_lfs_meta_object_oid" btree (oid) "IDX_lfs_meta_object_repository_id" btree (repository_id) gitea=# ``` I verified both of your patches - the sshfix and duplicateoids, I tried interacting with multiple repos and ensure that the files with the same oid are saved in the database, you can see at least 87428 and a6a00 repeated twice (3,13 ; 9,12 ). ``` gitea=# select * from lfs_meta_object; id | oid | size | repository_id | created_unix ----+------------------------------------------------------------------+-------+---------------+-------------- 1 | 11a77c3d96c06974b53d7f40a577e6813739eb5c811b2a86f59038ea90add772 | 5 | 2 | 1489708249 2 | 17e682f060b5f8e47ea04c5c4855908b0a5ad612022260fe50e11ecb0cc0ab76 | 4 | 2 | 1489761216 3 | 87428fc522803d31065e7bce3cf03fe475096631e5e07bbd7a0fde60c4cf25c7 | 2 | 2 | 1489773133 4 | 11d1d5bba135cd5b16ccd12fd5a1db43065d6692418223b28a17484c36adbac4 | 10 | 2 | 1489773202 5 | 4b59d598410964bccc7cab68af50e1a35a329b6c50bf90e6f2d27800ebf859af | 5 | 4 | 1489789248 6 | 01b031d1ac043038117415718d17cebd5b1fe7a7625e80fa13e9bace51f8a7e0 | 56832 | 5 | 1489809799 7 | 186819b8b188e3708c522d8f24de87b085d3e7d483f7c0327c453fd55a578031 | 56832 | 5 | 1489810672 8 | 2f866aa2e1a8717d94bebacf25e86e94f03b7a3de1d8d5fb728840257d4b6b4e | 11 | 2 | 1490029453 9 | a6a00c11d892f403bba36281d9cbfed574ea562c666aeca47098fddef07d0b15 | 14 | 2 | 1490029631 10 | 06060c3fbdf2873ce55fdb95fa16f901143e7ffc473183000c29fb3858462d78 | 26 | 2 | 1490029749 11 | 7f8b1dfc466b6249f06cbe55c9174df2578e7754da793fded244ef5cba2a38f1 | 8 | 2 | 1490029784 12 | a6a00c11d892f403bba36281d9cbfed574ea562c666aeca47098fddef07d0b15 | 14 | 5 | 1490030070 13 | 87428fc522803d31065e7bce3cf03fe475096631e5e07bbd7a0fde60c4cf25c7 | 2 | 5 | 1490030070 (13 rows) gitea=# ``` All seems to be working well, I'm not getting any prompts for http user/password and no issues with lfs so far. Thank you. Consider this my LGTM for both patches.

GiteaMirror commented