Page not reacting while long-running migration/mirror initialization #1685

@tankerkiller125 commented on GitHub (Dec 4, 2018):

This really needs to be resolved, on large repos the page will timeout and break the repo causing a 500 error.

@tankerkiller125 commented on GitHub (Dec 4, 2018): This really needs to be resolved, on large repos the page will timeout and break the repo causing a 500 error.

GiteaMirror commented

@stale[bot] commented on GitHub (Feb 2, 2019):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

@stale[bot] commented on GitHub (Feb 2, 2019): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

GiteaMirror commented

@techknowlogick commented on GitHub (Feb 2, 2019):

Please review https://docs.gitea.io/en-us/config-cheat-sheet/ for git.timeout settings and increase the timeouts.

@techknowlogick commented on GitHub (Feb 2, 2019): Please review https://docs.gitea.io/en-us/config-cheat-sheet/ for `git.timeout` settings and increase the timeouts.

GiteaMirror commented

@kb-1000 commented on GitHub (Feb 3, 2019):

This is not the cause of the issue... the issue is that the server waits until completion before sending the response page.

@kb-1000 commented on GitHub (Feb 3, 2019): This is not the cause of the issue... the issue is that the server waits until completion before sending the response page.

GiteaMirror commented

@kb-1000 commented on GitHub (Feb 3, 2019):

Please reopen this issue!

@kb-1000 commented on GitHub (Feb 3, 2019): Please reopen this issue!

GiteaMirror commented

https://github.com/go-gitea/gitea/blob/master/routers/repo/repo.go#L211

@zeripath commented on GitHub (Feb 3, 2019):

Thank you @kaeptmblaubaer1000 I think I understand the issue.

func MigratePost(ctx *context.Context, form auth.MigrateRepoForm) {
	...
	repo, err := models.MigrateRepository(ctx.User, ctxUser, models.MigrateRepoOptions{
		Name:        form.RepoName,
		Description: form.Description,
		IsPrivate:   form.Private || setting.Repository.ForcePrivate,
		IsMirror:    form.Mirror,
		RemoteAddr:  remoteAddr,
	})
	...
}

https://github.com/go-gitea/gitea/blob/master/models/repo.go#L899

func MigrateRepository(doer, u *User, opts MigrateRepoOptions) (*Repository, error) {
	...
	if err = git.Clone(opts.RemoteAddr, repoPath, git.CloneRepoOptions{
		Mirror:  true,
		Quiet:   true,
		Timeout: migrateTimeout,
	}); err != nil {
		return repo, fmt.Errorf("Clone: %v", err)
	}
	...
}

In the same goroutine as the POST! For big repositories this clone could take a very long time indeed. Meanwhile the user is sat on the other end of that POST waiting.

@zeripath commented on GitHub (Feb 3, 2019): Thank you @kaeptmblaubaer1000 I think I understand the issue. https://github.com/go-gitea/gitea/blob/master/routers/repo/repo.go#L211 ```go func MigratePost(ctx *context.Context, form auth.MigrateRepoForm) { ... repo, err := models.MigrateRepository(ctx.User, ctxUser, models.MigrateRepoOptions{ Name: form.RepoName, Description: form.Description, IsPrivate: form.Private || setting.Repository.ForcePrivate, IsMirror: form.Mirror, RemoteAddr: remoteAddr, }) ... } ``` https://github.com/go-gitea/gitea/blob/master/models/repo.go#L899 ```go func MigrateRepository(doer, u *User, opts MigrateRepoOptions) (*Repository, error) { ... if err = git.Clone(opts.RemoteAddr, repoPath, git.CloneRepoOptions{ Mirror: true, Quiet: true, Timeout: migrateTimeout, }); err != nil { return repo, fmt.Errorf("Clone: %v", err) } ... } ``` In the same goroutine as the POST! For big repositories this clone could take a very long time indeed. Meanwhile the user is sat on the other end of that POST waiting.

GiteaMirror commented

@zeripath commented on GitHub (Feb 3, 2019):

The problem is how to fix this. Clearly we should put this clone in a separate goroutine and return. But we probably need to monitor the migration somehow.

I guess a quick fix is to see if just sticking the clone and it's following parts in a goroutine is enough.

@zeripath commented on GitHub (Feb 3, 2019): The problem is how to fix this. Clearly we should put this clone in a separate goroutine and return. But we probably need to monitor the migration somehow. I guess a quick fix is to see if just sticking the clone and it's following parts in a goroutine is enough.

GiteaMirror commented

@stale[bot] commented on GitHub (Feb 17, 2019):

This issue has been automatically closed because of inactivity. You can re-open it if needed.

@stale[bot] commented on GitHub (Feb 17, 2019): This issue has been automatically closed because of inactivity. You can re-open it if needed.

GiteaMirror commented

@zeripath commented on GitHub (Feb 17, 2019):

Not stale - sorry that was my fault for not removing the stale tag. It shouldn't happen anymore.

@zeripath commented on GitHub (Feb 17, 2019): Not stale - sorry that was my fault for not removing the stale tag. It shouldn't happen anymore.

GiteaMirror commented

@zeripath commented on GitHub (Feb 17, 2019):

@kaeptmblaubaer1000 is there any chance you could test my pr to see if it solves your issue?

@zeripath commented on GitHub (Feb 17, 2019): @kaeptmblaubaer1000 is there any chance you could test my pr to see if it solves your issue?

GiteaMirror commented

@kb-1000 commented on GitHub (Feb 18, 2019):

If it doesn't wait for completion of mirror / import clone before returning the response page, then it solved the issue... but unfortunately I can't test it, as I don't have Go on my computer (only on my phone) and I don't currently use Gitea anyway, because there is currently no need for it for me.

@kb-1000 commented on GitHub (Feb 18, 2019): If it doesn't wait for completion of mirror / import clone before returning the response page, then it solved the issue... but unfortunately I can't test it, as I don't have Go on my computer (only on my phone) and I don't currently use Gitea anyway, because there is currently no need for it for me.

GiteaMirror commented