Failed migration v67 remove stale watches causes Gitea to not start #2815

New Issue

2025-11-02T04:49:23-06:00

GiteaMirror commented

2025-11-02 04:49:23 -06:00

Originally created by @KubqoA on GitHub (Jan 25, 2019).

Gitea version (or commit ref): v8a9f5b3
Git version: don't know, but most probably not related to the issue
Operating system: Alpine Linux v3.8 running the Gitea's Docker container
Database (use [x]):
- PostgreSQL
- MySQL
- MSSQL
- SQLite
Can you reproduce the bug at https://try.gitea.io:
- Yes (provide example URL)
- No
- Not relevant
Log gist: https://privatebin.net/?73672c209412b2f4#DlI7DnhdZH8DkSoFl7qPxXzYXOhAa8J+B9hp/voyrug=
docker-compose.yml: https://privatebin.net/?58383a8ad08c6bd2#kMhn10MYdNrIR8vZtEIe1DXPdIal/ZTA53vrch3+vyU=
app.ini: https://privatebin.net/?d9584bd98f475c6f#eGvSbTLT7VjAQdZxA7ccjZgCbGkiXfhE6NTtVLSH/AQ=

Description

I am running mailcow with gitea so I can authenticate users using their mail account. Yesterday I updated my mailcow stack and also Gitea with it. Since then Gitea doesn't launch. I tried to debug it and it looks like it gets stuck when it tries to run migration remove stale watches. If I specify another database file in my app.ini, so that Gitea creates a new one, everything seems to work and Gitea loads without any problems. However this means that I'd lose all my data, which is not a very good solution.

Originally created by @KubqoA on GitHub (Jan 25, 2019). - Gitea version (or commit ref): v8a9f5b3 - Git version: don't know, but most probably not related to the issue - Operating system: Alpine Linux v3.8 running the Gitea's Docker container - Database (use `[x]`): - [ ] PostgreSQL - [ ] MySQL - [ ] MSSQL - [x] SQLite - Can you reproduce the bug at https://try.gitea.io: - [ ] Yes (provide example URL) - [ ] No - [x] Not relevant - Log gist: [https://privatebin.net/?73672c209412b2f4#DlI7DnhdZH8DkSoFl7qPxXzYXOhAa8J+B9hp/voyrug=](https://privatebin.net/?73672c209412b2f4#DlI7DnhdZH8DkSoFl7qPxXzYXOhAa8J+B9hp/voyrug=) - docker-compose.yml: [https://privatebin.net/?58383a8ad08c6bd2#kMhn10MYdNrIR8vZtEIe1DXPdIal/ZTA53vrch3+vyU=](https://privatebin.net/?58383a8ad08c6bd2#kMhn10MYdNrIR8vZtEIe1DXPdIal/ZTA53vrch3+vyU=) - app.ini: [https://privatebin.net/?d9584bd98f475c6f#eGvSbTLT7VjAQdZxA7ccjZgCbGkiXfhE6NTtVLSH/AQ=](https://privatebin.net/?d9584bd98f475c6f#eGvSbTLT7VjAQdZxA7ccjZgCbGkiXfhE6NTtVLSH/AQ=) ## Description I am running [mailcow with gitea](https://mailcow.github.io/mailcow-dockerized-docs/third_party-gitea/) so I can authenticate users using their mail account. Yesterday I updated my mailcow stack and also Gitea with it. Since then Gitea doesn't launch. I tried to debug it and it looks like it gets stuck when it tries to run migration `remove stale watches`. If I specify another database file in my `app.ini`, so that Gitea creates a new one, everything seems to work and Gitea loads without any problems. However this means that I'd lose all my data, which is not a very good solution.

GiteaMirror added the type/bug label 2025-11-02 04:49:23 -06:00

GiteaMirror closed this issue

2025-11-02 04:49:23 -06:00

GiteaMirror commented

2025-11-02 04:49:24 -06:00

@lafriks commented on GitHub (Jan 25, 2019):

Can you check contents of gitea.log file, it should contain error where it fails

@lafriks commented on GitHub (Jan 25, 2019): Can you check contents of gitea.log file, it should contain error where it fails

GiteaMirror commented

2025-11-02 04:49:25 -06:00

@KubqoA commented on GitHub (Jan 25, 2019):

I checked the /var/lib/gitea/gitea/logs/gitea.log, xorm.log and serv.log, but there are no other logs. The log gist I gave you has all the log messages I am able to get my hands on.

@KubqoA commented on GitHub (Jan 25, 2019): I checked the `/var/lib/gitea/gitea/logs/gitea.log`, `xorm.log` and `serv.log`, but there are no other logs. The log gist I gave you has all the log messages I am able to get my hands on.

GiteaMirror commented

2025-11-02 04:49:25 -06:00

@zeripath commented on GitHub (Jan 25, 2019):

Hmm is this an sqlite database? I wonder if you're hitting a deadlock somehow

@zeripath commented on GitHub (Jan 25, 2019): Hmm is this an sqlite database? I wonder if you're hitting a deadlock somehow

GiteaMirror commented

2025-11-02 04:49:25 -06:00

@KubqoA commented on GitHub (Jan 25, 2019):

Well, if migrating to different type of database is a possible solution I am down for it. However, I don't know would I do such a thing.

@KubqoA commented on GitHub (Jan 25, 2019): Well, if migrating to different type of database is a possible solution I am down for it. However, I don't know would I do such a thing.

GiteaMirror commented

2025-11-02 04:49:25 -06:00

@typeless commented on GitHub (Jan 25, 2019):

@KubqoA What was the version before the migration?

@typeless commented on GitHub (Jan 25, 2019): @KubqoA What was the version before the migration?

GiteaMirror commented

2025-11-02 04:49:25 -06:00

@zeripath commented on GitHub (Jan 25, 2019):

The logs you've printed refer to migration v67.go. the final log appears to be due to a select on the repository table and is probably this line: https://github.com/go-gitea/gitea/blob/d9b51a781cbdaeea53de9888831b33c680e43d46/models/migrations/v67.go#L92

Which probably shouldn't be x.Get but rather be sess.Get

@zeripath commented on GitHub (Jan 25, 2019): The logs you've printed refer to migration v67.go. the final log appears to be due to a select on the repository table and is probably this line: https://github.com/go-gitea/gitea/blob/d9b51a781cbdaeea53de9888831b33c680e43d46/models/migrations/v67.go#L92 Which probably shouldn't be `x.Get` but rather be `sess.Get`

GiteaMirror commented

2025-11-02 04:49:26 -06:00

@KubqoA commented on GitHub (Jan 25, 2019):

So the issue is in that migration file?

@KubqoA commented on GitHub (Jan 25, 2019): So the issue is in that migration file?

GiteaMirror commented

2025-11-02 04:49:26 -06:00

@typeless commented on GitHub (Jan 25, 2019):

It is the exact line @zeripath demonstrated.

@typeless commented on GitHub (Jan 25, 2019): It is the exact line @zeripath demonstrated.

GiteaMirror commented

2025-11-02 04:49:26 -06:00

@KubqoA commented on GitHub (Jan 25, 2019):

So should I create a pull request so this can get fixed?

@KubqoA commented on GitHub (Jan 25, 2019): So should I create a pull request so this can get fixed?

GiteaMirror commented

2025-11-02 04:49:26 -06:00

@zeripath commented on GitHub (Jan 25, 2019):

Yup. You're hitting an SQL deadlock in the v67 migration.

I recently did a check up of the rest of the codebase looking for deadlocks but skipped the migrations coz I assumed incorrectly that these would have been caught already. Clearly I need to check them aswell.

In your case this should be a very easy fix - although I won't be able to make a PR for several hours possibly not till Saturday so if someone else could do it I can lgtm it. The simplest way to prevent deadlocks is to ensure that all dependent db queries are done within the same transaction.

(You can be a bit cleverer but it will almost always bite you later.)

@zeripath commented on GitHub (Jan 25, 2019): Yup. You're hitting an SQL deadlock in the v67 migration. I recently did a check up of the rest of the codebase looking for deadlocks but skipped the migrations coz I assumed incorrectly that these would have been caught already. Clearly I need to check them aswell. In your case this should be a very easy fix - although I won't be able to make a PR for several hours possibly not till Saturday so if someone else could do it I can lgtm it. The simplest way to prevent deadlocks is to ensure that all dependent db queries are done within the same transaction. (You can be a bit cleverer but it will almost always bite you later.)

GiteaMirror commented

2025-11-02 04:49:27 -06:00