On Prem Azure Devops Migration fails #8980

@bjornbouetsmith commented on GitHub (May 21, 2022):

Also tested on debian 11 with git version 2.34.1 and its the same issue.

So git on Linux in my very simple test does simply not handle escaping of characters correctly,

@bjornbouetsmith commented on GitHub (May 21, 2022): Also tested on debian 11 with `git version 2.34.1` and its the same issue. So git on Linux in my very simple test does simply not handle escaping of characters correctly,

GiteaMirror commented

@bjornbouetsmith commented on GitHub (May 21, 2022):

Tested with a different username and a plain password without any special characters in it - and its the same issue. I think git does not support proper http authentication as such, i.e. challenge and response.

When I use the same password with wget
It writes out:

Resolving devops.root.dom (devops.root.dom)... 192.168.1.230
Connecting to devops.root.dom (devops.root.dom)|192.168.1.230|:80... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Authentication selected: NTLM
Connecting to devops.root.dom (devops.root.dom)|192.168.1.230|:80... connected.
HTTP request sent, awaiting response... 401 Unauthorized

Then wget properly encodes a response with the username/password the and request suceeds.

So possibly its a git on linux vs azure devops on prem incompatibility - and what might need to be done instead is use ssh for git cloning, since that would fix the issue entirely.
Then whoever wants to migrate a azure devops repository would need to enable ssh on the server, create an authentication token and then it should just work if gitea pipes the git clone with a ssh url a ssh "url".

@bjornbouetsmith commented on GitHub (May 21, 2022): Tested with a different username and a plain password without any special characters in it - and its the same issue. I think git does not support proper http authentication as such, i.e. challenge and response. When I use the same password with `wget` It writes out: ~~~ Resolving devops.root.dom (devops.root.dom)... 192.168.1.230 Connecting to devops.root.dom (devops.root.dom)|192.168.1.230|:80... connected. HTTP request sent, awaiting response... 401 Unauthorized Authentication selected: NTLM Connecting to devops.root.dom (devops.root.dom)|192.168.1.230|:80... connected. HTTP request sent, awaiting response... 401 Unauthorized ~~~ Then wget properly encodes a response with the username/password the and request suceeds. So possibly its a git on linux vs azure devops on prem incompatibility - and what might need to be done instead is use ssh for git cloning, since that would fix the issue entirely. Then whoever wants to migrate a azure devops repository would need to enable ssh on the server, create an authentication token and then it should just work if gitea pipes the git clone with a ssh url a ssh "url".

GiteaMirror commented

@bjornbouetsmith commented on GitHub (May 21, 2022):

I can verify that adding a ssh public key to azure devops allows me to git clone.
So I think what needs to be done for "Azure Devops" in gitea is that it gets its own "migration" type and then tell people to use ssh.

Basically I would guess that what needs to be done is

User running gitea needs ssh-keys generated
public key needs to be configured in azure devops
migrate/clone url should only accept ssh

Success.

If I knew how to program in GO and my UI skills weren't shit - I would create a pull request for this.

@bjornbouetsmith commented on GitHub (May 21, 2022): I can verify that adding a ssh public key to azure devops allows me to git clone. So I think what needs to be done for "Azure Devops" in gitea is that it gets its own "migration" type and then tell people to use ssh. Basically I would guess that what needs to be done is * User running gitea needs ssh-keys generated * public key needs to be configured in azure devops * migrate/clone url should only accept ssh Success. If I knew how to program in GO and my UI skills weren't shit - I would create a pull request for this.

GiteaMirror commented

@6543 commented on GitHub (May 21, 2022):

related: #1635

@6543 commented on GitHub (May 21, 2022): related: #1635

GiteaMirror commented

@KN4CK3R commented on GitHub (Oct 28, 2022):

Is this still an issue? I'v tested it with a pull and push mirror and both worked.

@KN4CK3R commented on GitHub (Oct 28, 2022): Is this still an issue? I'v tested it with a pull and push mirror and both worked.

GiteaMirror commented

@bjornbouetsmith commented on GitHub (Oct 28, 2022):

Is this still an issue? I'v tested it with a pull and push mirror and both worked.

Yes, I just tried to fire up my gitea installation - updated to 17.3 and same exact issue.

Authentication error as I described above.

What type of migration did you try to do?

And please provide exact steps, was it http/https or other url you used?

@bjornbouetsmith commented on GitHub (Oct 28, 2022): > Is this still an issue? I'v tested it with a pull and push mirror and both worked. Yes, I just tried to fire up my gitea installation - updated to 17.3 and same exact issue. Authentication error as I described above. What type of migration did you try to do? And please provide exact steps, was it http/https or other url you used? ![image](https://user-images.githubusercontent.com/1437162/198580827-bb0a10e7-9cfa-46d0-acfe-9ecb060662d8.png)

GiteaMirror commented

@KN4CK3R commented on GitHub (Oct 29, 2022):

@kdumontnu asked me about this issue, so I registered at Devops and created a repo. The Devops UI told me to use admin as username and zpv5hppqgmk5p76b...h4bwrrpaq as password. Then I setup a push mirror from Gitea to Devops with the provided url and credentials. Gitea created a mirror information like this in the git config file:

[remote "remote_mirror_CDjCL4odYe"]
    url = https://admin:zpv5hppqgmk5p76b...h4bwrrpaq@dev.azure.com/kn4ck3r/test/_git/test
    mirror = true
    push = +refs/heads/*:refs/heads/*
    push = +refs/tags/*:refs/tags/*

That resulted in

After that I created a (mirror) migration for that repo and that worked too.

@KN4CK3R commented on GitHub (Oct 29, 2022): @kdumontnu asked me about this issue, so I registered at Devops and created a repo. The Devops UI told me to use `admin` as username and `zpv5hppqgmk5p76b...h4bwrrpaq` as password. Then I setup a push mirror from Gitea to Devops with the provided url and credentials. Gitea created a mirror information like this in the git config file: ``` [remote "remote_mirror_CDjCL4odYe"] url = https://admin:zpv5hppqgmk5p76b...h4bwrrpaq@dev.azure.com/kn4ck3r/test/_git/test mirror = true push = +refs/heads/*:refs/heads/* push = +refs/tags/*:refs/tags/* ``` That resulted in ![grafik](https://user-images.githubusercontent.com/1666336/198582163-5a4faa44-0e2b-4989-9894-a344a797ebb8.png) After that I created a (mirror) migration for that repo and that worked too.

GiteaMirror commented

@bjornbouetsmith commented on GitHub (Oct 29, 2022):

[remote "remote_mirror_CDjCL4odYe"]
url = https://admin:zpv5hppqgmk5p76b...h4bwrrpaq@dev.azure.com/kn4ck3r/test/_git/test
mirror = true
push = +refs/heads/:refs/heads/
push = +refs/tags/:refs/tags/

You are using azure devops "cloud version" - this issue is about on prem dev ops.

Also - I am not trying to set up a mirror - simply migrate a git repository from on prem azure devops to gitea.

@bjornbouetsmith commented on GitHub (Oct 29, 2022): > [remote "remote_mirror_CDjCL4odYe"] > url = https://admin:zpv5hppqgmk5p76b...h4bwrrpaq@dev.azure.com/kn4ck3r/test/_git/test > mirror = true > push = +refs/heads/*:refs/heads/* > push = +refs/tags/*:refs/tags/* > ``` You are using azure devops "cloud version" - this issue is about **on prem** dev ops. Also - I am not trying to set up a mirror - simply migrate a git repository from on prem azure devops to gitea.

GiteaMirror commented

@KN4CK3R commented on GitHub (Oct 29, 2022):

Oh, sorry about that. I will try with an on prem dev ops.

@KN4CK3R commented on GitHub (Oct 29, 2022): Oh, sorry about that. I will try with an on prem dev ops.

GiteaMirror commented

@bjornbouetsmith commented on GitHub (Oct 29, 2022):

Oh, sorry about that. I will try with an on prem dev ops.

No worries - but make sure you try a migration - and not try to set up any mirrors.

@bjornbouetsmith commented on GitHub (Oct 29, 2022): > Oh, sorry about that. I will try with an on prem dev ops. No worries - but make sure you try a migration - and not try to set up any mirrors.

GiteaMirror commented

@KN4CK3R commented on GitHub (Oct 30, 2022):

Got it working. Basically the on prem Devops needs the Authorization header for cloning. https://learn.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops&tabs=Linux#use-a-pat

I will not create a PR for that because this should be implemented as a generic "add custom headers to git requests" instead of just the Authorization header. A possible implementation needs to support the migration/clone, pull/push mirror, LFS client pull/push, ... and everything I forgot. It may not be worth the effort just to enable migrations from on prem Devops as no other service needs this.

These are the code changes I made:

diff --git a/modules/git/repo.go b/modules/git/repo.go
index 8ba3ae4fd..f601b7300 100644
--- a/modules/git/repo.go
+++ b/modules/git/repo.go
@@ -98,16 +98,17 @@ func (repo *Repository) IsEmpty() (bool, error) {
 
 // CloneRepoOptions options when clone a repository
 type CloneRepoOptions struct {
-       Timeout       time.Duration
-       Mirror        bool
-       Bare          bool
-       Quiet         bool
-       Branch        string
-       Shared        bool
-       NoCheckout    bool
-       Depth         int
-       Filter        string
-       SkipTLSVerify bool
+       Timeout             time.Duration
+       Mirror              bool
+       Bare                bool
+       Quiet               bool
+       Branch              string
+       Shared              bool
+       NoCheckout          bool
+       Depth               int
+       Filter              string
+       SkipTLSVerify       bool
+       AuthorizationHeader string
 }
 
 // Clone clones original repository to target path.
@@ -126,6 +127,9 @@ func CloneWithArgs(ctx context.Context, args []CmdArg, from, to string, opts Clo
        if opts.SkipTLSVerify {
                cmd.AddArguments("-c", "http.sslVerify=false")
        }
+       if opts.AuthorizationHeader != "" {
+               cmd.AddArguments("-c").AddDynamicArguments("http.extraHeader=" + opts.AuthorizationHeader)
+       }
        if opts.Mirror {
                cmd.AddArguments("--mirror")
        }
diff --git a/modules/migration/options.go b/modules/migration/options.go
index 1e92a1b0b..944d83bc1 100644
--- a/modules/migration/options.go
+++ b/modules/migration/options.go
@@ -11,13 +11,15 @@ import "code.gitea.io/gitea/modules/structs"
 // this is for internal usage by migrations module and func who interact with it
 type MigrateOptions struct {
        // required: true
-       CloneAddr             string `json:"clone_addr" binding:"Required"`
-       CloneAddrEncrypted    string `json:"clone_addr_encrypted,omitempty"`
-       AuthUsername          string `json:"auth_username"`
-       AuthPassword          string `json:"-"`
-       AuthPasswordEncrypted string `json:"auth_password_encrypted,omitempty"`
-       AuthToken             string `json:"-"`
-       AuthTokenEncrypted    string `json:"auth_token_encrypted,omitempty"`
+       CloneAddr              string `json:"clone_addr" binding:"Required"`
+       CloneAddrEncrypted     string `json:"clone_addr_encrypted,omitempty"`
+       AuthUsername           string `json:"auth_username"`
+       AuthPassword           string `json:"-"`
+       AuthPasswordEncrypted  string `json:"auth_password_encrypted,omitempty"`
+       AuthToken              string `json:"-"`
+       AuthTokenEncrypted     string `json:"auth_token_encrypted,omitempty"`
+       UseAuthorizationHeader bool   `json:"use_authorization_header"`
+       AuthorizationHeader    string `json:"-"`
        // required: true
        UID int `json:"uid" binding:"Required"`
        // required: true
diff --git a/modules/repository/repo.go b/modules/repository/repo.go
index de6de3bda..1e69acb67 100644
--- a/modules/repository/repo.go
+++ b/modules/repository/repo.go
@@ -74,10 +74,11 @@ func MigrateRepositoryGitData(ctx context.Context, u *user_model.User,
        }
 
        if err = git.Clone(ctx, opts.CloneAddr, repoPath, git.CloneRepoOptions{
-               Mirror:        true,
-               Quiet:         true,
-               Timeout:       migrateTimeout,
-               SkipTLSVerify: setting.Migrations.SkipTLSVerify,
+               Mirror:              true,
+               Quiet:               true,
+               Timeout:             migrateTimeout,
+               SkipTLSVerify:       setting.Migrations.SkipTLSVerify,
+               AuthorizationHeader: opts.AuthorizationHeader,
        }); err != nil {
                return repo, fmt.Errorf("Clone: %w", err)
        }
@@ -95,11 +96,12 @@ func MigrateRepositoryGitData(ctx context.Context, u *user_model.User,
                        }
 
                        if err := git.Clone(ctx, wikiRemotePath, wikiPath, git.CloneRepoOptions{
-                               Mirror:        true,
-                               Quiet:         true,
-                               Timeout:       migrateTimeout,
-                               Branch:        "master",
-                               SkipTLSVerify: setting.Migrations.SkipTLSVerify,
+                               Mirror:              true,
+                               Quiet:               true,
+                               Timeout:             migrateTimeout,
+                               Branch:              "master",
+                               SkipTLSVerify:       setting.Migrations.SkipTLSVerify,
+                               AuthorizationHeader: opts.AuthorizationHeader,
                        }); err != nil {
                                log.Warn("Clone wiki: %v", err)
                                if err := util.RemoveAll(wikiPath); err != nil {
diff --git a/services/migrations/gitea_uploader.go b/services/migrations/gitea_uploader.go
index 8a7533b3d..5de25dfb6 100644
--- a/services/migrations/gitea_uploader.go
+++ b/services/migrations/gitea_uploader.go
@@ -7,6 +7,7 @@ package migrations
 
 import (
        "context"
+       "encoding/base64"
        "fmt"
        "io"
        "os"
@@ -118,19 +119,24 @@ func (g *GiteaLocalUploader) CreateRepo(repo *base.Repository, opts base.Migrate
        r.DefaultBranch = repo.DefaultBranch
        r.Description = repo.Description
 
+       if opts.UseAuthorizationHeader {
+               opts.AuthorizationHeader = fmt.Sprintf("Authorization: Basic %s", base64.StdEncoding.EncodeToString([]byte(fmt.Sprintf("%s:%s", opts.AuthUsername, opts.AuthPassword))))
+       }
+
        r, err = repo_module.MigrateRepositoryGitData(g.ctx, owner, r, base.MigrateOptions{
-               RepoName:       g.repoName,
-               Description:    repo.Description,
-               OriginalURL:    repo.OriginalURL,
-               GitServiceType: opts.GitServiceType,
-               Mirror:         repo.IsMirror,
-               LFS:            opts.LFS,
-               LFSEndpoint:    opts.LFSEndpoint,
-               CloneAddr:      repo.CloneURL, // SECURITY: we will assume that this has already been checked
-               Private:        repo.IsPrivate,
-               Wiki:           opts.Wiki,
-               Releases:       opts.Releases, // if didn't get releases, then sync them from tags
-               MirrorInterval: opts.MirrorInterval,
+               RepoName:            g.repoName,
+               Description:         repo.Description,
+               OriginalURL:         repo.OriginalURL,
+               GitServiceType:      opts.GitServiceType,
+               Mirror:              repo.IsMirror,
+               LFS:                 opts.LFS,
+               LFSEndpoint:         opts.LFSEndpoint,
+               CloneAddr:           repo.CloneURL, // SECURITY: we will assume that this has already been checked
+               Private:             repo.IsPrivate,
+               Wiki:                opts.Wiki,
+               Releases:            opts.Releases, // if didn't get releases, then sync them from tags
+               MirrorInterval:      opts.MirrorInterval,
+               AuthorizationHeader: opts.AuthorizationHeader,
        }, NewMigrationHTTPTransport())
 
        g.sameApp = strings.HasPrefix(repo.OriginalURL, setting.AppURL)

@KN4CK3R commented on GitHub (Oct 30, 2022): Got it working. Basically the on prem Devops needs the `Authorization` header for cloning. https://learn.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops&tabs=Linux#use-a-pat I will not create a PR for that because this should be implemented as a generic "add custom headers to git requests" instead of just the `Authorization` header. A possible implementation needs to support the migration/clone, pull/push mirror, LFS client pull/push, ... and everything I forgot. It may not be worth the effort just to enable migrations from on prem Devops as no other service needs this. These are the code changes I made: ```diff diff --git a/modules/git/repo.go b/modules/git/repo.go index 8ba3ae4fd..f601b7300 100644 --- a/modules/git/repo.go +++ b/modules/git/repo.go @@ -98,16 +98,17 @@ func (repo *Repository) IsEmpty() (bool, error) { // CloneRepoOptions options when clone a repository type CloneRepoOptions struct { - Timeout time.Duration - Mirror bool - Bare bool - Quiet bool - Branch string - Shared bool - NoCheckout bool - Depth int - Filter string - SkipTLSVerify bool + Timeout time.Duration + Mirror bool + Bare bool + Quiet bool + Branch string + Shared bool + NoCheckout bool + Depth int + Filter string + SkipTLSVerify bool + AuthorizationHeader string } // Clone clones original repository to target path. @@ -126,6 +127,9 @@ func CloneWithArgs(ctx context.Context, args []CmdArg, from, to string, opts Clo if opts.SkipTLSVerify { cmd.AddArguments("-c", "http.sslVerify=false") } + if opts.AuthorizationHeader != "" { + cmd.AddArguments("-c").AddDynamicArguments("http.extraHeader=" + opts.AuthorizationHeader) + } if opts.Mirror { cmd.AddArguments("--mirror") } diff --git a/modules/migration/options.go b/modules/migration/options.go index 1e92a1b0b..944d83bc1 100644 --- a/modules/migration/options.go +++ b/modules/migration/options.go @@ -11,13 +11,15 @@ import "code.gitea.io/gitea/modules/structs" // this is for internal usage by migrations module and func who interact with it type MigrateOptions struct { // required: true - CloneAddr string `json:"clone_addr" binding:"Required"` - CloneAddrEncrypted string `json:"clone_addr_encrypted,omitempty"` - AuthUsername string `json:"auth_username"` - AuthPassword string `json:"-"` - AuthPasswordEncrypted string `json:"auth_password_encrypted,omitempty"` - AuthToken string `json:"-"` - AuthTokenEncrypted string `json:"auth_token_encrypted,omitempty"` + CloneAddr string `json:"clone_addr" binding:"Required"` + CloneAddrEncrypted string `json:"clone_addr_encrypted,omitempty"` + AuthUsername string `json:"auth_username"` + AuthPassword string `json:"-"` + AuthPasswordEncrypted string `json:"auth_password_encrypted,omitempty"` + AuthToken string `json:"-"` + AuthTokenEncrypted string `json:"auth_token_encrypted,omitempty"` + UseAuthorizationHeader bool `json:"use_authorization_header"` + AuthorizationHeader string `json:"-"` // required: true UID int `json:"uid" binding:"Required"` // required: true diff --git a/modules/repository/repo.go b/modules/repository/repo.go index de6de3bda..1e69acb67 100644 --- a/modules/repository/repo.go +++ b/modules/repository/repo.go @@ -74,10 +74,11 @@ func MigrateRepositoryGitData(ctx context.Context, u *user_model.User, } if err = git.Clone(ctx, opts.CloneAddr, repoPath, git.CloneRepoOptions{ - Mirror: true, - Quiet: true, - Timeout: migrateTimeout, - SkipTLSVerify: setting.Migrations.SkipTLSVerify, + Mirror: true, + Quiet: true, + Timeout: migrateTimeout, + SkipTLSVerify: setting.Migrations.SkipTLSVerify, + AuthorizationHeader: opts.AuthorizationHeader, }); err != nil { return repo, fmt.Errorf("Clone: %w", err) } @@ -95,11 +96,12 @@ func MigrateRepositoryGitData(ctx context.Context, u *user_model.User, } if err := git.Clone(ctx, wikiRemotePath, wikiPath, git.CloneRepoOptions{ - Mirror: true, - Quiet: true, - Timeout: migrateTimeout, - Branch: "master", - SkipTLSVerify: setting.Migrations.SkipTLSVerify, + Mirror: true, + Quiet: true, + Timeout: migrateTimeout, + Branch: "master", + SkipTLSVerify: setting.Migrations.SkipTLSVerify, + AuthorizationHeader: opts.AuthorizationHeader, }); err != nil { log.Warn("Clone wiki: %v", err) if err := util.RemoveAll(wikiPath); err != nil { diff --git a/services/migrations/gitea_uploader.go b/services/migrations/gitea_uploader.go index 8a7533b3d..5de25dfb6 100644 --- a/services/migrations/gitea_uploader.go +++ b/services/migrations/gitea_uploader.go @@ -7,6 +7,7 @@ package migrations import ( "context" + "encoding/base64" "fmt" "io" "os" @@ -118,19 +119,24 @@ func (g *GiteaLocalUploader) CreateRepo(repo *base.Repository, opts base.Migrate r.DefaultBranch = repo.DefaultBranch r.Description = repo.Description + if opts.UseAuthorizationHeader { + opts.AuthorizationHeader = fmt.Sprintf("Authorization: Basic %s", base64.StdEncoding.EncodeToString([]byte(fmt.Sprintf("%s:%s", opts.AuthUsername, opts.AuthPassword)))) + } + r, err = repo_module.MigrateRepositoryGitData(g.ctx, owner, r, base.MigrateOptions{ - RepoName: g.repoName, - Description: repo.Description, - OriginalURL: repo.OriginalURL, - GitServiceType: opts.GitServiceType, - Mirror: repo.IsMirror, - LFS: opts.LFS, - LFSEndpoint: opts.LFSEndpoint, - CloneAddr: repo.CloneURL, // SECURITY: we will assume that this has already been checked - Private: repo.IsPrivate, - Wiki: opts.Wiki, - Releases: opts.Releases, // if didn't get releases, then sync them from tags - MirrorInterval: opts.MirrorInterval, + RepoName: g.repoName, + Description: repo.Description, + OriginalURL: repo.OriginalURL, + GitServiceType: opts.GitServiceType, + Mirror: repo.IsMirror, + LFS: opts.LFS, + LFSEndpoint: opts.LFSEndpoint, + CloneAddr: repo.CloneURL, // SECURITY: we will assume that this has already been checked + Private: repo.IsPrivate, + Wiki: opts.Wiki, + Releases: opts.Releases, // if didn't get releases, then sync them from tags + MirrorInterval: opts.MirrorInterval, + AuthorizationHeader: opts.AuthorizationHeader, }, NewMigrationHTTPTransport()) g.sameApp = strings.HasPrefix(repo.OriginalURL, setting.AppURL) ```

GiteaMirror commented

@bjornbouetsmith commented on GitHub (Oct 31, 2022):

Nice find @KN4CK3R - but what I don't understand is why other implementations do not require this - surely they all use username/password authentication.

Other repositories probably just relies on the git binary doing the authentication for them.

All basic authentication works like that.

Server sends a challenge, client sends a "Authorization" header.

And I am sorry you feel that its not worth it to fix this just because its "only" azure devops on prem that needs this - I can see other settings the the "MigrateOptions" - that are specific other other types, i.e. Wiki.

Also I think it can also be a bug in the git client possibly, it should be able to handle the authentication, but possibly the azure devops server requests "NTLM" authenticaion and then git just says "nope" - and on windows perhaps there is a special handling of that that translates that into "BASIC" instead.

It would be nice to see the actual headers the azure devops server sends to see if its possible to get a hint to why git cannot just provide basic authentication header automatically, as I am sure that is what it does to all other http/https git urls.

@bjornbouetsmith commented on GitHub (Oct 31, 2022): Nice find @KN4CK3R - but what I don't understand is why other implementations do not require this - surely they all use username/password authentication. Other repositories probably just relies on the git binary doing the authentication for them. All basic authentication works like that. Server sends a challenge, client sends a "Authorization" header. And I am sorry you feel that its not worth it to fix this just because its "only" azure devops on prem that needs this - I can see other settings the the "MigrateOptions" - that are specific other other types, i.e. Wiki. Also I think it can also be a bug in the git client possibly, it should be able to handle the authentication, but possibly the azure devops server requests "NTLM" authenticaion and then git just says "nope" - and on windows perhaps there is a special handling of that that translates that into "BASIC" instead. It would be nice to see the actual headers the azure devops server sends to see if its possible to get a hint to why git cannot just provide basic authentication header automatically, as I am sure that is what it does to all other http/https git urls.

GiteaMirror commented