ORM issue upgrading from 1.13.7 to 1.14.3 on error 500 repo #7513

Closed
opened 2025-11-02 07:28:10 -06:00 by GiteaMirror · 22 comments
Owner

Originally created by @Jack64 on GitHub (Jun 27, 2021).

  • Gitea version (or commit ref): 1.14.0
  • Git version: 2.30.2
  • Operating system: linux (docker)
  • Database (use [x]):
    • PostgreSQL
    • MySQL
    • MSSQL
    • SQLite
  • Can you reproduce the bug at https://try.gitea.io:
    • Yes (provide example URL)
    • No
  • Log gist: https://pastebin.com/ZPdByCge

Description

When trying to upgrade from 1.13.7 to 1.14.0, I get an ORM initialization error as shown in the logs above. It is unclear which repository is causing this, please help.
EDIT: added log line below to help others facing this problem find this issue

ORM engine initialization attempt #3/10 failed. Error: migrate: do migrate: GetTagCommit: exit status 128 - fatal: not a git repository (or any parent up to mount point /)
Originally created by @Jack64 on GitHub (Jun 27, 2021). <!-- NOTE: If your issue is a security concern, please send an email to security@gitea.io instead of opening a public issue --> <!-- 1. Please speak English, this is the language all maintainers can speak and write. 2. Please ask questions or configuration/deploy problems on our Discord server (https://discord.gg/gitea) or forum (https://discourse.gitea.io). 3. Please take a moment to check that your issue doesn't already exist. 4. Make sure it's not mentioned in the FAQ (https://docs.gitea.io/en-us/faq) 5. Please give all relevant information below for bug reports, because incomplete details will be handled as an invalid report. --> - Gitea version (or commit ref): 1.14.0 - Git version: 2.30.2 - Operating system: linux (docker) <!-- Please include information on whether you built gitea yourself, used one of our downloads or are using some other package --> <!-- Please also tell us how you are running gitea, e.g. if it is being run from docker, a command-line, systemd etc. ---> <!-- If you are using a package or systemd tell us what distribution you are using --> - Database (use `[x]`): - [x] PostgreSQL - [ ] MySQL - [ ] MSSQL - [ ] SQLite - Can you reproduce the bug at https://try.gitea.io: - [ ] Yes (provide example URL) - [x] No - Log gist: https://pastebin.com/ZPdByCge <!-- It really is important to provide pertinent logs --> <!-- Please read https://docs.gitea.io/en-us/logging-configuration/#debugging-problems --> <!-- In addition, if your problem relates to git commands set `RUN_MODE=dev` at the top of app.ini --> ## Description <!-- If using a proxy or a CDN (e.g. CloudFlare) in front of gitea, please disable the proxy/CDN fully and connect to gitea directly to confirm the issue still persists without those services. --> When trying to upgrade from 1.13.7 to 1.14.0, I get an ORM initialization error as shown in the logs above. It is unclear which repository is causing this, please help. EDIT: added log line below to help others facing this problem find this issue ```bash ORM engine initialization attempt #3/10 failed. Error: migrate: do migrate: GetTagCommit: exit status 128 - fatal: not a git repository (or any parent up to mount point /) ```
GiteaMirror added the type/questionissue/duplicate labels 2025-11-02 07:28:10 -06:00
Author
Owner

@zeripath commented on GitHub (Jun 27, 2021):

Please try 1.14.3 - it will at least give you more information even if it continues to fail.

In general once a point release is release just use the latest point release - never use the lower point releases unless absolutely necessary.

@zeripath commented on GitHub (Jun 27, 2021): Please try 1.14.3 - it will at least give you more information even if it continues to fail. In general once a point release is release just use the latest point release - never use the lower point releases unless absolutely necessary.
Author
Owner

@zeripath commented on GitHub (Jun 27, 2021):

I just don't understand why you would ever not just use the latest point release? How can we improve our documentation so people don't do this?

I'm serious here - we fix a large number of bugs in between point releases - sometimes with migrations - sometimes bugs that would cause data loss. If you use the lowest point release instead of jumping to the latest point release you're exposing yourself to already fixed bugs and wasting our time responding to bug reports that are duplicates.

@zeripath commented on GitHub (Jun 27, 2021): I just don't understand why you would ever not just use the latest point release? How can we improve our documentation so people don't do this? I'm serious here - we fix a large number of bugs in between point releases - sometimes with migrations - sometimes bugs that would cause data loss. If you use the lowest point release instead of jumping to the latest point release you're exposing yourself to already fixed bugs and wasting our time responding to bug reports that are duplicates.
Author
Owner

@Jack64 commented on GitHub (Jun 27, 2021):

@zeripath I wasn't actually planning on staying at 1.14.0. I moved from 1.13.1 to 1.13.7 prior to upgrading to 1.14.0, assuming this would improve my chances of a successful minor release upgrade. If I understand correctly, you are recommending that if someone is at 1.X."latest-patch" they should upgrade directly to 1.Y."latest-patch" ?

In any case, this produces the same result, curiously on a mirror of gitea itself:

2021/06/27 13:15:33 ...s/migrations/v156.go:89:fixPublisherIDforTagReleases() [I] [SQL] SELECT "id", "owner_id", "owner_name", "name" FROM "repository" WHERE "id"=$1 LIMIT 1 [151] - 1.221616ms
2021/06/27 13:15:33 ...s/migrations/v156.go:125:fixPublisherIDforTagReleases() [E] Error whilst getting commit for Tag: v0.11rc in [151]mirrors/gitea. Error: exit status 128 - fatal: not a git repository (or any parent up to mount point /)
        Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).

2021/06/27 13:15:33 ...ations/migrations.go:397:Migrate() [I] [SQL] ROLLBACK [] - 605.608µs
2021/06/27 13:15:33 routers/init.go:81:initDBEngine() [E] ORM engine initialization attempt #8/10 failed. Error: migrate: do migrate: GetTagCommit: exit status 128 - fatal: not a git repository (or any parent up to mount point /)
        Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
@Jack64 commented on GitHub (Jun 27, 2021): @zeripath I wasn't actually planning on staying at 1.14.0. I moved from 1.13.1 to 1.13.7 prior to upgrading to 1.14.0, assuming this would improve my chances of a successful minor release upgrade. If I understand correctly, you are recommending that if someone is at 1.X."latest-patch" they should upgrade directly to 1.Y."latest-patch" ? In any case, this produces the same result, curiously on a mirror of gitea itself: ``` 2021/06/27 13:15:33 ...s/migrations/v156.go:89:fixPublisherIDforTagReleases() [I] [SQL] SELECT "id", "owner_id", "owner_name", "name" FROM "repository" WHERE "id"=$1 LIMIT 1 [151] - 1.221616ms 2021/06/27 13:15:33 ...s/migrations/v156.go:125:fixPublisherIDforTagReleases() [E] Error whilst getting commit for Tag: v0.11rc in [151]mirrors/gitea. Error: exit status 128 - fatal: not a git repository (or any parent up to mount point /) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). 2021/06/27 13:15:33 ...ations/migrations.go:397:Migrate() [I] [SQL] ROLLBACK [] - 605.608µs 2021/06/27 13:15:33 routers/init.go:81:initDBEngine() [E] ORM engine initialization attempt #8/10 failed. Error: migrate: do migrate: GetTagCommit: exit status 128 - fatal: not a git repository (or any parent up to mount point /) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). ```
Author
Owner

@Jack64 commented on GitHub (Jun 27, 2021):

Do you have any suggestions for a next step? I can always remove the mirror while on 1.13.7 and try again, but what if this happens on a repo I can't remove?

@Jack64 commented on GitHub (Jun 27, 2021): Do you have any suggestions for a next step? I can always remove the mirror while on 1.13.7 and try again, but what if this happens on a repo I can't remove?
Author
Owner

@zeripath commented on GitHub (Jun 27, 2021):

Yes always go from latest point release to latest point release.

Anyway this is a duplicate issue. The error message on 1.14.3 is different if you read the logs:

...s/migrations/v156.go:125:fixPublisherIDforTagReleases() [E] Error whilst getting commit for Tag: v0.11rc in [151]mirrors/gitea. Error: exit status 128 - fatal: not a git repository (or any parent up to mount point /)
        Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).

The migration can't find the repository in the place it is looking for. That is likely because your [repository] ROOT is incorrect - likely you left it unset in your app.ini but unfortunately we had to change this default location because of snap.

The breaking notes for 1.14 state:

 The default location for repository data ([repository] ROOT) will now be in a subdirectory of the APP_DATA_PATH path. (#13991)

Likely all you have to do is set [repository] ROOT correctly or move them to the new location.

See: #16217 and #16210

@zeripath commented on GitHub (Jun 27, 2021): Yes always go from latest point release to latest point release. Anyway this is a duplicate issue. The error message on 1.14.3 is different if you read the logs: ``` ...s/migrations/v156.go:125:fixPublisherIDforTagReleases() [E] Error whilst getting commit for Tag: v0.11rc in [151]mirrors/gitea. Error: exit status 128 - fatal: not a git repository (or any parent up to mount point /) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). ``` The migration can't find the repository in the place it is looking for. That is likely because your `[repository]` `ROOT` is incorrect - likely you left it unset in your app.ini but unfortunately we had to change this default location because of snap. The breaking notes for 1.14 state: ``` The default location for repository data ([repository] ROOT) will now be in a subdirectory of the APP_DATA_PATH path. (#13991) ``` Likely all you have to do is set `[repository]` `ROOT` correctly or move them to the new location. See: #16217 and #16210
Author
Owner

@Jack64 commented on GitHub (Jun 27, 2021):

OK in the future I'll remember to upgrade minor versions from latest point release to latest point release.

I have the [repository] ROOT set in the app.ini but it fails regardless. I have no other keys set under [repository] and I don't have [server] APP_DATA_PATH defined.

I'm using "/data/git/gitea-repositories" as [repository]ROOT, from what I gathered from the other issues I need to move it to something like "/data/gitea/git/gitea-repositories" and also set [server]APP_DATA_PATH = /data/gitea so that [repository]ROOT becomes a subdirectory of APP_DATA_PATH. Is this correct?

@Jack64 commented on GitHub (Jun 27, 2021): OK in the future I'll remember to upgrade minor versions from latest point release to latest point release. I have the [repository] ROOT set in the app.ini but it fails regardless. I have no other keys set under [repository] and I don't have [server] APP_DATA_PATH defined. I'm using "/data/git/gitea-repositories" as [repository]ROOT, from what I gathered from the other issues I need to move it to something like "/data/gitea/git/gitea-repositories" and also set [server]APP_DATA_PATH = /data/gitea so that [repository]ROOT becomes a subdirectory of APP_DATA_PATH. Is this correct?
Author
Owner

@zeripath commented on GitHub (Jun 27, 2021):

What command are you running? Are you running that command in the same environment as you would normally run Gitea and with the same arguments?

@zeripath commented on GitHub (Jun 27, 2021): What command are you running? Are you running that command in the same environment as you would normally run Gitea and with the same arguments?
Author
Owner

@Jack64 commented on GitHub (Jun 27, 2021):

Yes, nothing has changed in that regard. I'm simply running the Docker container pointing to the app.ini I'm mounting in there.

The thing is, I have a terraform module that I use to run gitea which allows automatic periodic backups and restores from a minio / S3 bucket so since I was moving it from one Kubernetes cluster to another, it ran in migration mode and had to pull all the data from the S3 to put it in the correct folder (i.e. [repository]ROOT). This has worked flawlessly since (IIRC) 1.11 . If [repository]ROOT now must be a subdir of APP_DATA_PATH (which I have not set), I have to change the backup/restore scripts to adjust to this change.

@Jack64 commented on GitHub (Jun 27, 2021): Yes, nothing has changed in that regard. I'm simply running the Docker container pointing to the app.ini I'm mounting in there. The thing is, I have a terraform module that I use to run gitea which allows automatic periodic backups and restores from a minio / S3 bucket so since I was moving it from one Kubernetes cluster to another, it ran in migration mode and had to pull all the data from the S3 to put it in the correct folder (i.e. [repository]ROOT). This has worked flawlessly since (IIRC) 1.11 . If [repository]ROOT now must be a subdir of APP_DATA_PATH (which I have not set), I have to change the backup/restore scripts to adjust to this change.
Author
Owner

@zeripath commented on GitHub (Jun 27, 2021):

it doesn't have to be - you can set it anywhere.

If you're absolutely certain that you have [repository] ROOT set and that you're reading in the config correctly - then the problem is going to be that the repository this migration is looking for simply doesn't exist.

So ... are you sure that $ROOT/mirrors/gitea.git is a bare git repository ?

@zeripath commented on GitHub (Jun 27, 2021): it doesn't have to be - you can set it anywhere. If you're absolutely certain that you have `[repository]` `ROOT` set and that you're reading in the config correctly - then the problem is going to be that the repository this migration is looking for simply doesn't exist. So ... are you sure that `$ROOT/mirrors/gitea.git` is a bare git repository ?
Author
Owner

@zeripath commented on GitHub (Jun 27, 2021):

I have the [repository] ROOT set in the app.ini but it fails regardless. I have no other keys set under [repository] and I don't have [server] APP_DATA_PATH defined.

I'm using "/data/git/gitea-repositories" as [repository] ROOT, from what I gathered from the other issues I need to move it to something like "/data/gitea/git/gitea-repositories" and also set [server] APP_DATA_PATH = /data/gitea so that [repository] ROOT becomes a subdirectory of APP_DATA_PATH. Is this correct?

No. So long as you have set the ROOT to an absolute path the APP_DATA_PATH is irrelevant regarding this.

@zeripath commented on GitHub (Jun 27, 2021): > I have the `[repository]` `ROOT` set in the `app.ini` but it fails regardless. I have no other keys set under `[repository]` and I don't have `[server]` `APP_DATA_PATH` defined. > > I'm using `"/data/git/gitea-repositories"` as `[repository]` `ROOT`, from what I gathered from the other issues I need to move it to something like `"/data/gitea/git/gitea-repositories"` and also set `[server]` `APP_DATA_PATH = /data/gitea` so that `[repository]` `ROOT` becomes a subdirectory of `APP_DATA_PATH`. Is this correct? No. So long as you have set the ROOT to an absolute path the APP_DATA_PATH is irrelevant regarding this.
Author
Owner

@Jack64 commented on GitHub (Jun 27, 2021):

Good hint. It shows in the repo search but then it throws a 500 error when I try to access it (on 1.13.7).
image

I don't see any error logs anywhere to explain this 500 error.

@Jack64 commented on GitHub (Jun 27, 2021): Good hint. It shows in the repo search but then it throws a 500 error when I try to access it (on 1.13.7). ![image](https://user-images.githubusercontent.com/11641285/123547291-15eed680-d758-11eb-8397-f1c086147845.png) I don't see any error logs anywhere to explain this 500 error.
Author
Owner

@zeripath commented on GitHub (Jun 27, 2021):

Likely it's not there and you just need to remove it from the db. Probably there was a migration failure at some point and the repository wasn't properly removed - 1.14.x won't do that.

you might be able to use an url like http://gitea/mirors/gitea/settings to get to the settings page directly and delete the repository that way if you don't know how to remove it directly from the db.

@zeripath commented on GitHub (Jun 27, 2021): Likely it's not there and you just need to remove it from the db. Probably there was a migration failure at some point and the repository wasn't properly removed - 1.14.x won't do that. you might be able to use an url like `http://gitea/mirors/gitea/settings` to get to the settings page directly and delete the repository that way if you don't know how to remove it directly from the db.
Author
Owner

@Jack64 commented on GitHub (Jun 27, 2021):

Probably there was a migration failure at some point and the repository wasn't properly removed - 1.14.x won't do that.

Yes that's likely.

Unfortunately /settings throws an error as well. I guess all that's left is removing it from the DB directly, do you by any chance have the SQL command at hand ? I see that I have other repos in this situation so I'll have to do it more than once ..

@Jack64 commented on GitHub (Jun 27, 2021): > Probably there was a migration failure at some point and the repository wasn't properly removed - 1.14.x won't do that. Yes that's likely. Unfortunately /settings throws an error as well. I guess all that's left is removing it from the DB directly, do you by any chance have the SQL command at hand ? I see that I have other repos in this situation so I'll have to do it more than once ..
Author
Owner

@zeripath commented on GitHub (Jun 27, 2021):

DELETE FROM repository WHERE owner_name = 'mirrors' AND name = 'gitea'

@zeripath commented on GitHub (Jun 27, 2021): `DELETE FROM repository WHERE owner_name = 'mirrors' AND name = 'gitea'`
Author
Owner

@zeripath commented on GitHub (Jun 27, 2021):

normally we'd just have the migrations delete these sort of things but as the ROOT location can be wrong it's not really the right thing to do in this case.

@zeripath commented on GitHub (Jun 27, 2021): normally we'd just have the migrations delete these sort of things but as the ROOT location can be wrong it's not really the right thing to do in this case.
Author
Owner

@zeripath commented on GitHub (Jun 27, 2021):

An alternate option is to just do a git init --bare owner/repo.git in your $ROOT as that should allow stop the 500.

@zeripath commented on GitHub (Jun 27, 2021): An alternate option is to just do a `git init --bare owner/repo.git` in your `$ROOT` as that should allow stop the 500.
Author
Owner

@Jack64 commented on GitHub (Jun 27, 2021):

normally we'd just have the migrations delete these sort of things but as the ROOT location can be wrong it's not really the right thing to do in this case.

Yes I realized that might be why it's not "auto-fixed". Would be nice to have somewhere in the UI (administration section maybe?) where we could do this without having to jump into the SQL, though. But perhaps it's not that common an issue to make this a priority.

The DELETE command worked flawlessly, thank you very much. I'll re-run the migration now and delete any other repos in this situation.

@Jack64 commented on GitHub (Jun 27, 2021): > normally we'd just have the migrations delete these sort of things but as the ROOT location can be wrong it's not really the right thing to do in this case. Yes I realized that might be why it's not "auto-fixed". Would be nice to have somewhere in the UI (administration section maybe?) where we could do this without having to jump into the SQL, though. But perhaps it's not that common an issue to make this a priority. The DELETE command worked flawlessly, thank you very much. I'll re-run the migration now and delete any other repos in this situation.
Author
Owner

@Jack64 commented on GitHub (Jun 27, 2021):

Boom - 1.14.3 running 👍 thanks again for your invaluable help @zeripath , this sort of incredibly fast and helpful support puts some commercial solutions to shame!! 🙏 🙏

@Jack64 commented on GitHub (Jun 27, 2021): Boom - 1.14.3 running :+1: thanks again for your invaluable help @zeripath , this sort of incredibly fast and helpful support puts some commercial solutions to shame!! :pray: :pray:
Author
Owner

@zeripath commented on GitHub (Jun 27, 2021):

Yes I realized that might be why it's not "auto-fixed". Would be nice to have somewhere in the UI (administration section maybe?) where we could do this without having to jump into the SQL, though. But perhaps it's not that common an issue to make this a priority.

Trouble is to get to any fixed UI would require you to pass the migration step first...

The other issue is that the loss of a repo should cause a 500 - it is an internal server error. I'm not sure I really want these to be tolerated. Something has gone drastically wrong if the git repo has disappeared - but yeah it's not great that then you're stuck in a broken db situation even if the rest of the situation is resolvable.

@zeripath commented on GitHub (Jun 27, 2021): > Yes I realized that might be why it's not "auto-fixed". Would be nice to have somewhere in the UI (administration section maybe?) where we could do this without having to jump into the SQL, though. But perhaps it's not that common an issue to make this a priority. Trouble is to get to any fixed UI would require you to pass the migration step first... The other issue is that the loss of a repo should cause a 500 - it is an internal server error. I'm not sure I really want these to be tolerated. Something has gone drastically wrong if the git repo has disappeared - but yeah it's not great that then you're stuck in a broken db situation even if the rest of the situation is resolvable.
Author
Owner

@zeripath commented on GitHub (Jun 27, 2021):

I'll close this and think a bit more about the 500 issue.

@zeripath commented on GitHub (Jun 27, 2021): I'll close this and think a bit more about the 500 issue.
Author
Owner

@Jack64 commented on GitHub (Jun 27, 2021):

Trouble is to get to any fixed UI would require you to pass the migration step first...

The common solution for this is simply rolling-back to the last working version (in my case 1.13.7). After that, I would be able to go to the Administration panel and see that there is an "errored repo". If I had a list of such repos there, I could manually decide which ones could be safely deleted (pressing a button that ran this DELETE SQL command on the back-end) or leave it alone in order to restore the data, depending on which case it were.

At least that's how I imagined this would work. By putting this on the Administration tab and not on the repo settings it'd bypass the 500 error.

The other issue is that the loss of a repo should cause a 500 - it is an internal server error. I'm not sure I really want these to be tolerated. Something has gone drastically wrong if the git repo has disappeared - but yeah it's not great that then you're stuck in a broken db situation even if the rest of the situation is resolvable.

Fully agree here that there shouldn't be an "auto-fix", because it may use the "wrong fix" by deleting or keeping it. The point of that UI would be to point out to the admin that there are repos in this situation (fortunately for me it was only this mirror + my own fork of it) and allow them to fix it. I'm not sure if this is common enough to warrant the new UI, but if this happened by a rogue rm -rf or ransomware or something, it'd be invisible if it's not a regularly used repo (which was the case).

EDIT: Another thing to consider is the improved verbosity of the logs on 1.14.3 now allow a relatively quick diagnosis, so having to do this SQL command once in 3 years of running Gitea and using it every day didn't "kill me" and shouldn't be that big a deal for anyone else (aside from the effort of actually knowing the right command to run).

@Jack64 commented on GitHub (Jun 27, 2021): > Trouble is to get to any fixed UI would require you to pass the migration step first... The common solution for this is simply rolling-back to the last working version (in my case 1.13.7). After that, I would be able to go to the Administration panel and see that there is an "errored repo". If I had a list of such repos there, I could manually decide which ones could be safely deleted (pressing a button that ran this DELETE SQL command on the back-end) or leave it alone in order to restore the data, depending on which case it were. At least that's how I imagined this would work. By putting this on the Administration tab and not on the repo settings it'd bypass the 500 error. > The other issue is that the loss of a repo should cause a 500 - it is an internal server error. I'm not sure I really want these to be tolerated. Something has gone drastically wrong if the git repo has disappeared - but yeah it's not great that then you're stuck in a broken db situation even if the rest of the situation is resolvable. Fully agree here that there shouldn't be an "auto-fix", because it may use the "wrong fix" by deleting or keeping it. The point of that UI would be to point out to the admin that there are repos in this situation (fortunately for me it was only this mirror + my own fork of it) and allow them to fix it. I'm not sure if this is common enough to warrant the new UI, but if this happened by a rogue rm -rf or ransomware or something, it'd be invisible if it's not a regularly used repo (which was the case). EDIT: Another thing to consider is the improved verbosity of the logs on 1.14.3 now allow a relatively quick diagnosis, so having to do this SQL command once in 3 years of running Gitea and using it every day didn't "kill me" and shouldn't be that big a deal for anyone else (aside from the effort of actually knowing the right command to run).
Author
Owner

@Jack64 commented on GitHub (Jun 27, 2021):

updated the title, might help anyone with this exact problem find this issue.

@Jack64 commented on GitHub (Jun 27, 2021): updated the title, might help anyone with this exact problem find this issue.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/gitea#7513