Migration stuck for some repos #4235

GiteaMirror commented

@ashimokawa commented on GitHub (Nov 4, 2019):

@lunny
This is clearly a kind/bug.

The migration fails silently and the animation is running forever.
Not even the first issue (which is a PR with deleted branches) is migrated properly.

@ashimokawa commented on GitHub (Nov 4, 2019): @lunny This is clearly a kind/bug. The migration fails silently and the animation is running forever. Not even the first issue (which is a PR with deleted branches) is migrated properly.

GiteaMirror commented

@lunny commented on GitHub (Nov 4, 2019):

I think we can add a cancel button here.

@lunny commented on GitHub (Nov 4, 2019): I think we can add a `cancel` button here.

GiteaMirror commented

@ashimokawa commented on GitHub (Nov 4, 2019):

@lunny
Sure, but that does not fix the underlying problem. The migrations fails somehow, always, reproducible with the repo mentioned above https://github.com/kriztan/Pix-Art-Messenger

@ashimokawa commented on GitHub (Nov 4, 2019): @lunny Sure, but that does not fix the underlying problem. The migrations fails somehow, always, reproducible with the repo mentioned above https://github.com/kriztan/Pix-Art-Messenger

GiteaMirror commented

@6543 commented on GitHub (Nov 4, 2019):

it works on gitea.com ( 1.11.0+dev-110-gc4bc5abda)
and 1.9.5 (with a timeout message but it works)

and https://try.gitea.io/abcKing/s

@6543 commented on GitHub (Nov 4, 2019): it works on gitea.com ( 1.11.0+dev-110-gc4bc5abda) and 1.9.5 (with a timeout message but it works) and https://try.gitea.io/abcKing/s

GiteaMirror commented

@6543 commented on GitHub (Nov 4, 2019):

maby some space limitations based on user? or something like that?

@6543 commented on GitHub (Nov 4, 2019): maby some space limitations based on user? or something like that?

GiteaMirror commented

@6543 commented on GitHub (Nov 4, 2019):

@lunny what happen wen a user run out of granted space when he has a migration job?

@6543 commented on GitHub (Nov 4, 2019): @lunny what happen wen a user run out of granted space when he has a migration job?

GiteaMirror commented

@lunny commented on GitHub (Nov 5, 2019):

@6543 The repository should be deleted if any error found on the migrating. And a notice will be displayed on admin panel notices.

@lunny commented on GitHub (Nov 5, 2019): @6543 The repository should be deleted if any error found on the migrating. And a notice will be displayed on admin panel notices.

GiteaMirror commented

@lunny commented on GitHub (Nov 5, 2019):

As an enhancements for the migration process, I will send another PRs to split the process as serval parts.

@lunny commented on GitHub (Nov 5, 2019): As an enhancements for the migration process, I will send another PRs to split the process as serval parts.

GiteaMirror commented

@ashimokawa commented on GitHub (Nov 5, 2019):

@6543
No it does not work when migration with issues and pr, please read the original issue description:

with issues, wiki, pull requests and releases to Gitea, the migration is stuck, and the repo inaccessible.

EDIT: This also has nothing to do with space restrictions, happens on all gitea instances, always. I think above repository is perfect for testing and fixing the problem.

Maybe it was not wise to create a 2-in-1 issue, actually these should be two:

FULL migration of https://github.com/kriztan/Pix-Art-Messenger ALWAYS fails
can't cancel failed migration

@ashimokawa commented on GitHub (Nov 5, 2019): @6543 No it does *not* work when migration with issues and pr, please read the original issue description: > with issues, wiki, pull requests and releases to Gitea, the migration is stuck, and the repo inaccessible. EDIT: This also has nothing to do with space restrictions, happens on all gitea instances, always. I think above repository is perfect for testing and fixing the problem. Maybe it was not wise to create a 2-in-1 issue, actually these should be two: - FULL migration of https://github.com/kriztan/Pix-Art-Messenger ALWAYS fails - can't cancel failed migration

GiteaMirror commented

@lafriks commented on GitHub (Nov 5, 2019):

Imho problem is that if Gitea is restarted while migration is running it will be left unfinished

@lafriks commented on GitHub (Nov 5, 2019): Imho problem is that if Gitea is restarted while migration is running it will be left unfinished

GiteaMirror commented

@ashimokawa commented on GitHub (Nov 5, 2019):

@lafriks
@lunny

No, it happens always on any instance, not only when restarting.

We did not restart try.gitea.io (we can't) and it happens on our test instance also. You can reproduce it easily by migrating https://github.com/kriztan/Pix-Art-Messenger

@ashimokawa commented on GitHub (Nov 5, 2019): @lafriks @lunny No, it happens always on any instance, not only when restarting. We did not restart try.gitea.io (we can't) and it happens on our test instance also. You can reproduce it easily by migrating https://github.com/kriztan/Pix-Art-Messenger

GiteaMirror commented

@lunny commented on GitHub (Nov 5, 2019):

@ashimokawa I will try that repository.

@lunny commented on GitHub (Nov 5, 2019): @ashimokawa I will try that repository.

GiteaMirror commented

45f219239e/vendor/xorm.io/xorm/session_delete.go (L211)

@mrsdizzie commented on GitHub (Nov 5, 2019):

I started to debug this a bit and ran out of time but if it is helpful for you @lunny what is happening is that at some point inserting a batch of releases from this repo is throwing the error:

UNIQUE constraint failed: release.repo_id, release.tag_name

And then it goes into rollback but never comes out. It seems to get here:

And then the it hits this code and seems to never really do anything after that:

45f219239e/vendor/xorm.io/xorm/session_raw.go (L177-L179)

That runs some code in the specific driver for the database, but I'm able to reproduce on sqlite3 and assume try.gitea.io and codeberg are using something else so I wonder if it just isn't returning an error where it should at some point. Not familiar with xorm at all though :-/

I tried with lafriks patch too but seems unrelated in this case (though helpful in general)

@mrsdizzie commented on GitHub (Nov 5, 2019): I started to debug this a bit and ran out of time but if it is helpful for you @lunny what is happening is that at some point inserting a batch of releases from this repo is throwing the error: ``` UNIQUE constraint failed: release.repo_id, release.tag_name ``` And then it goes into rollback but never comes out. It seems to get here: https://github.com/go-gitea/gitea/blob/45f219239e420f707a2452b99b27a68f45d5d4ca/vendor/xorm.io/xorm/session_delete.go#L211 And then the it hits this code and seems to never really do anything after that: https://github.com/go-gitea/gitea/blob/45f219239e420f707a2452b99b27a68f45d5d4ca/vendor/xorm.io/xorm/session_raw.go#L177-L179 That runs some code in the specific driver for the database, but I'm able to reproduce on sqlite3 and assume try.gitea.io and codeberg are using something else so I wonder if it just isn't returning an error where it should at some point. Not familiar with xorm at all though :-/ I tried with lafriks patch too but seems unrelated in this case (though helpful in general)

GiteaMirror commented

@ashimokawa commented on GitHub (Nov 5, 2019):

@mrsdizzie
Yep, codeberg.org runs mariadb and not sqlite.

I assume you tried the repo I mentioned with full migration?

The only thing a saw in logs is that a rollback failed (timout? stuck?), I really do not remember because I was also in a hurry.

@ashimokawa commented on GitHub (Nov 5, 2019): @mrsdizzie Yep, codeberg.org runs mariadb and not sqlite. I assume you tried the repo I mentioned with full migration? The only thing a saw in logs is that a rollback failed (timout? stuck?), I really do not remember because I was also in a hurry.

GiteaMirror commented

@mrsdizzie commented on GitHub (Nov 5, 2019):

I assume you tried the repo I mentioned with full migration?

Yes, it is specifically the releases that are causing the initial migration of this particular repo to fail and then there is a separate (and much worse) problem of the rollback not finishing properly after the migration fails.

@mrsdizzie commented on GitHub (Nov 5, 2019): > I assume you tried the repo I mentioned with full migration? Yes, it is specifically the releases that are causing the initial migration of this particular repo to fail and then there is a separate (and much worse) problem of the rollback not finishing properly after the migration fails.

GiteaMirror commented

@lunny commented on GitHub (Nov 6, 2019):

@mrsdizzie Thanks for your work!
So

UNIQUE constraint failed: release.repo_id, release.tag_name

means there are duplicated releases on that repository??? I will take a look at that code.

@lunny commented on GitHub (Nov 6, 2019): @mrsdizzie Thanks for your work! So ``` UNIQUE constraint failed: release.repo_id, release.tag_name ``` means there are duplicated releases on that repository??? I will take a look at that code.

GiteaMirror commented

@ashimokawa commented on GitHub (Nov 6, 2019):

I can confirm that migrating the repository without releases works.

@ashimokawa commented on GitHub (Nov 6, 2019): I can confirm that migrating the repository without releases works.

GiteaMirror commented

@techknowlogick commented on GitHub (Dec 2, 2019):

It seems unusual that there would be a duplicate release, as I'm not sure that git allows same tag to be used twice. Regardless of that assumption, I am able to replicate same error as above, but able to have successful migration when release is unchecked (I tested on sqlite as that's what I am using on my dev instance, but will try with postgres later, as that's whats on my prod instance).

@techknowlogick commented on GitHub (Dec 2, 2019): It seems unusual that there would be a duplicate release, as I'm not sure that git allows same tag to be used twice. Regardless of that assumption, I am able to replicate same error as above, but able to have successful migration when release is unchecked (I tested on sqlite as that's what I am using on my dev instance, but will try with postgres later, as that's whats on my prod instance).

GiteaMirror commented

@mrsdizzie commented on GitHub (Dec 2, 2019):

It could be a bug in the migrating that perhaps tries to insert the same release twice as well.

The 100% certain bug is that the rollback isn't finishing but I didn't know enough about xorm etc to really figure out why that is happening here.

@mrsdizzie commented on GitHub (Dec 2, 2019): It could be a bug in the migrating that perhaps tries to insert the same release twice as well. The 100% certain bug is that the rollback isn't finishing but I didn't know enough about xorm etc to really figure out why that is happening here.

GiteaMirror commented

@lunny commented on GitHub (Dec 11, 2019):

#9319 shoud fix another bug but not this one. But @ashimokawa , you can still try it.

@lunny commented on GitHub (Dec 11, 2019): #9319 shoud fix another bug but not this one. But @ashimokawa , you can still try it.

GiteaMirror commented

https://try.gitea.io/mrsdizzie/Pix-Art-Messenger

@mrsdizzie commented on GitHub (Dec 11, 2019):

The specific example from this issue should be fixd, see:

But the original issue exposed a lower level bug in rollback that is still present (though not triggered anymore through this issue) and should probably stay open until that is fixed.

Plus there are the other aspects like not being able to cancel a migration that is stuck for any reason which should be possible

@mrsdizzie commented on GitHub (Dec 11, 2019): The specific example from this issue should be fixd, see: https://try.gitea.io/mrsdizzie/Pix-Art-Messenger But the original issue exposed a lower level bug in rollback that is still present (though not triggered anymore through this issue) and should probably stay open until that is fixed. Plus there are the other aspects like not being able to cancel a migration that is stuck for any reason which should be possible

GiteaMirror commented

@lunny commented on GitHub (Dec 11, 2019):

@mrsdizzie it could be deleted from admin panel currently. :)

@lunny commented on GitHub (Dec 11, 2019): @mrsdizzie it could be deleted from admin panel currently. :)

GiteaMirror commented

@ashimokawa commented on GitHub (Dec 12, 2019):

@lunny
I can confirm that the specific case I opened this issue for is fixed. So normally I would close this one.

@ashimokawa commented on GitHub (Dec 12, 2019): @lunny I can confirm that the specific case I opened this issue for is fixed. So normally I would close this one.

GiteaMirror commented

@ashimokawa commented on GitHub (Dec 12, 2019):

@lunny @mrsdizzie

Stuck repos can be deleted by the user by just manually navigating to his settings, even in that stuck state.

https://try.gitea.io/mrsdizzie/Pix-Art-Messenger/settings (example, I tried this with a real stuck repo)

Don't know if that is good or not.

@ashimokawa commented on GitHub (Dec 12, 2019): @lunny @mrsdizzie Stuck repos can be deleted by the user by just manually navigating to his settings, even in that stuck state. https://try.gitea.io/mrsdizzie/Pix-Art-Messenger/settings (example, I tried this with a real stuck repo) Don't know if that is good or not.

GiteaMirror commented

@lunny commented on GitHub (Dec 12, 2019):

@ashimokawa :)

@lunny commented on GitHub (Dec 12, 2019): @ashimokawa :)

GiteaMirror commented

@mrsdizzie commented on GitHub (Dec 12, 2019):

There is no link to that settings page if it is stuck migrating (though visiting the URL directly works). So it is good but could be better.

I think the general bug still exists that it shouldn't be possible for it to get stuck at all (it should work or fail), but now this example will not trigger it so perhaps harder to reproduce. @lunny maybe if you have time to look at the xorm code from above, if not then I'm not sure : ) Glad that this example helped fix the bug causing it anyway so thanks!

@mrsdizzie commented on GitHub (Dec 12, 2019): There is no link to that settings page if it is stuck migrating (though visiting the URL directly works). So it is good but could be better. I think the general bug still exists that it shouldn't be possible for it to get stuck at all (it should work or fail), but now this example will not trigger it so perhaps harder to reproduce. @lunny maybe if you have time to look at the xorm code from above, if not then I'm not sure : ) Glad that this example helped fix the bug causing it anyway so thanks!

GiteaMirror commented

@saurasambit commented on GitHub (Feb 9, 2020):

I faced the issue while migrating https://github.com/jitsi/jitsi-meet.
See https://try.gitea.io/dashadsh/jitsi-meet for the reproduced problem.

@saurasambit commented on GitHub (Feb 9, 2020): I faced the issue while migrating https://github.com/jitsi/jitsi-meet. See https://try.gitea.io/dashadsh/jitsi-meet for the reproduced problem.

GiteaMirror commented

@dkebler commented on GitHub (Mar 16, 2020):

Just had this happen to me after migrating many repos to my backup site. How does one get rid of this aborted migration? Not via the webui as there is no access to "settings" since the migration was not completed. I restarted my instance and web server but it must be in the db itself. How can on repair the git.db?

Of Note. I see that I had 7000+ pages of system notices all one date all about not being able to access the main git from which the migrations come. This backup install has been acting funny too. Slow page loads, popup notices above loosing data if moving to another page which I wasn't. I'm guessing this is a corrupted data base maybe from loading up on all those system notifications. 7000 pages is excessive no :-). Luckily I have daily backups and a backup from when I upgraded in Jan. Will try those if I can solve this otherwise.

Of real note is that my main gitea db is only 2mb
but the backup gitea (of the same repos as migrations) is 47mb! Maybe 7000 pages i.e. 45mb of system notifications? Anyway something has gone horribly wrong with the db of my backup gitea install. It's look more and more like the logging system gone amok. Today's gitea.log is 179mb! with unending entries like so
2020/03/16 00:00:11 ...eue/queue_wrapped.go:69:setInternal() [W] [Attempt: 706] Failed to create queue: level for task cfg: {/mnt/data/git/data/queues/task 1000 20 1 6 1s 5m0s 5 task-level} error: leveldb: manifest corrupted (field 'comparer'): missing [file=MANIFEST-000020]

@dkebler commented on GitHub (Mar 16, 2020): Just had this happen to me after migrating many repos to my backup site. How does one get rid of this aborted migration? Not via the webui as there is no access to "settings" since the migration was not completed. I restarted my instance and web server but it must be in the db itself. How can on repair the `git.db`? Of Note. I see that I had 7000+ pages of system notices all one date all about not being able to access the main git from which the migrations come. This backup install has been acting funny too. Slow page loads, popup notices above loosing data if moving to another page which I wasn't. I'm guessing this is a corrupted data base maybe from loading up on all those system notifications. 7000 pages is excessive no :-). Luckily I have daily backups and a backup from when I upgraded in Jan. Will try those if I can solve this otherwise. Of real note is that my main gitea db is only 2mb but the backup gitea (of the same repos as migrations) is 47mb! Maybe 7000 pages i.e. 45mb of system notifications? Anyway something has gone horribly wrong with the db of my backup gitea install. It's look more and more like the logging system gone amok. Today's gitea.log is 179mb! with unending entries like so ` 2020/03/16 00:00:11 ...eue/queue_wrapped.go:69:setInternal() [W] [Attempt: 706] Failed to create queue: level for task cfg: {/mnt/data/git/data/queues/task 1000 20 1 6 1s 5m0s 5 task-level} error: leveldb: manifest corrupted (field 'comparer'): missing [file=MANIFEST-000020]`

GiteaMirror commented

@dkebler commented on GitHub (Mar 16, 2020):

I was able to delete all the notifications via the UI and under site administration /admin/repos I was able to delete the offending migration.
I then attempted to make anew the migration and it was the same issue :-(

I have been using 1.1 now going to try upgrading to 1.96.

@dkebler commented on GitHub (Mar 16, 2020): I was able to delete all the notifications via the UI and under site administration /admin/repos I was able to delete the offending migration. I then attempted to make anew the migration and it was the same issue :-( I have been using 1.1 now going to try upgrading to 1.96.

GiteaMirror commented

@dkebler commented on GitHub (Mar 17, 2020):

I upgraded both installs to 1.9.6. I deleted and recreated the offending repo on the main gitea install and pushed it again from local. Then tried again to make a migration on my backup gitea from main gitea install and YEA that worked.

So only 2 issues for me now.

that my git.db on my backup install is still 47mb. Apparently sqlite has no way to compact after deleting all those pages of notifications? https://www.recoveryandmanagement.com/repair-sqlite-database-manually/
Why did gitea create 7000 pages worth of notifications/logs. Was that 'bug' fixed going from 1.1 to 1.9.6?

@dkebler commented on GitHub (Mar 17, 2020): I upgraded both installs to 1.9.6. I deleted and recreated the offending repo on the main gitea install and pushed it again from local. Then tried again to make a migration on my backup gitea from main gitea install and YEA that worked. So only 2 issues for me now. 1. that my git.db on my backup install is still 47mb. Apparently sqlite has no way to compact after deleting all those pages of notifications? https://www.recoveryandmanagement.com/repair-sqlite-database-manually/ 2. Why did gitea create 7000 pages worth of notifications/logs. Was that 'bug' fixed going from 1.1 to 1.9.6?

GiteaMirror commented

@lunny commented on GitHub (Mar 17, 2020):

That admin notices could be deleted after your problem resolved.

@lunny commented on GitHub (Mar 17, 2020): That admin notices could be deleted after your problem resolved.

GiteaMirror commented

@guillep2k commented on GitHub (Mar 17, 2020):

@dkebler Sincerely, 1.1 was so long ago... so many things have changed since then: refactors, rewriting, updating libraries... I'd even suggest you upgrade to 1.11.3 while you're at it; a lot of good features and bugfixes were added since 1.9.6 already.

@guillep2k commented on GitHub (Mar 17, 2020): @dkebler Sincerely, 1.1 was so long ago... so many things have changed since then: refactors, rewriting, updating libraries... I'd even suggest you upgrade to 1.11.3 while you're at it; a lot of good features and bugfixes were added since 1.9.6 already.

GiteaMirror commented

@qdzhaov commented on GitHub (Apr 14, 2020):

I found the bug like this.

Gitea version (or commit ref):1.11.4
Git version:2.17.1
Operating system:linux
Database : SQLite
When I Migration a Repositories from gogs(,need a user name a password ,but I don't give,so
the task is in dead loop.(This bug is also in gogs.)

@qdzhaov commented on GitHub (Apr 14, 2020): I found the bug like this. - Gitea version (or commit ref):1.11.4 - Git version:2.17.1 - Operating system:linux - Database : SQLite When I Migration a Repositories from gogs(,need a user name a password ,but I don't give,so the task is in dead loop.(This bug is also in gogs.)

GiteaMirror commented

@guillep2k commented on GitHub (Apr 15, 2020):

@qdzhaov What version of Gogs are you migrating from? (I take it's not an upgrade but a site-to-site migration)

@guillep2k commented on GitHub (Apr 15, 2020): @qdzhaov What version of Gogs are you migrating from? (I take it's not an upgrade but a site-to-site migration)

GiteaMirror commented

@stale[bot] commented on GitHub (Jun 14, 2020):

This issue has been automatically marked as stale because it has not had recent activity. I am here to help clear issues left open even if solved or waiting for more insight. This issue will be closed if no further activity occurs during the next 2 weeks. If the issue is still valid just add a comment to keep it alive. Thank you for your contributions.

@stale[bot] commented on GitHub (Jun 14, 2020): This issue has been automatically marked as stale because it has not had recent activity. I am here to help clear issues left open even if solved or waiting for more insight. This issue will be closed if no further activity occurs during the next 2 weeks. If the issue is still valid just add a comment to keep it alive. Thank you for your contributions.

GiteaMirror commented

@theAkito commented on GitHub (Jul 5, 2020):

I've upgraded from 6af13dbac to 15d3aa7b95, because I had this issue on the former version. Now the issue is gone and mirroring works.

I encountered that issue with another repository. The mirroring worked after restarting Gitea.

@theAkito commented on GitHub (Jul 5, 2020): I've upgraded from 6af13dbac to 15d3aa7b95e37cc2e9a06208089169cbe04383b5, because I had this issue on the former version. Now the issue is gone and mirroring works. ____ I encountered that issue with another repository. The mirroring worked after restarting Gitea.

GiteaMirror commented

@1byte2bytes commented on GitHub (Sep 14, 2020):

I encounter the issue of migration getting stuck while trying to mirror https://github.com/apache/spamassassin to my Gitea instance, using Gitea 1.14.8 on Debian 10. Restarting the Gitea server did not fix the importing for me.

@1byte2bytes commented on GitHub (Sep 14, 2020): I encounter the issue of migration getting stuck while trying to mirror https://github.com/apache/spamassassin to my Gitea instance, using Gitea 1.14.8 on Debian 10. Restarting the Gitea server did not fix the importing for me.

GiteaMirror commented

@jolheiser commented on GitHub (Sep 14, 2020):

using Gitea 1.14.8 on Debian 10

Currently Gitea is version 1.12.4 heading towards 1.13.

@jolheiser commented on GitHub (Sep 14, 2020): > using Gitea 1.14.8 on Debian 10 Currently Gitea is version `1.12.4` heading towards `1.13`.

GiteaMirror commented