Internal error 500 on pull requests or other forms of submission #3106

New Issue

GiteaMirror · 2025-11-02T05:00:55-06:00

GiteaMirror commented

2025-11-02 05:00:55 -06:00

Originally created by @nicovince on GitHub (Mar 26, 2019).

Gitea version (or commit ref): 1.7.3
Git version: 2.18.1
Operating system: alpine 3.8
Database:
- PostgreSQL
- MySQL
- MSSQL
- SQLite 3.26.0
Can you reproduce the bug at https://try.gitea.io:
- Yes (provide example URL)
- No
- Not relevant
Log gist:

Description

Sometimes, when I submit a form from a pull request creation or update or from a ticket creation or in order to close it, I get an internal error 500.

==> /var/log/gitea/gitea.log <==
2019/03/26 14:56:58 [...user/notification.go:33 GetNotificationCount()] [E] GetNotificationCount: database table is locked: notification

Originally created by @nicovince on GitHub (Mar 26, 2019). - Gitea version (or commit ref): 1.7.3 - Git version: 2.18.1 - Operating system: alpine 3.8 - Database: - [ ] PostgreSQL - [ ] MySQL - [ ] MSSQL - [x] SQLite 3.26.0 - Can you reproduce the bug at https://try.gitea.io: - [ ] Yes (provide example URL) - [ ] No - [x] Not relevant - Log gist: ## Description Sometimes, when I submit a form from a pull request creation or update or from a ticket creation or in order to close it, I get an internal error 500. ``` ==> /var/log/gitea/gitea.log <== 2019/03/26 14:56:58 [...user/notification.go:33 GetNotificationCount()] [E] GetNotificationCount: database table is locked: notification ```

GiteaMirror closed this issue

2025-11-02 05:00:55 -06:00

GiteaMirror commented

2025-11-02 05:00:56 -06:00

@nicovince commented on GitHub (Mar 26, 2019):

I updated the OS to alpine 3.9 and I still randomly encounter the issue.

@nicovince commented on GitHub (Mar 26, 2019): I updated the OS to alpine 3.9 and I still randomly encounter the issue.

GiteaMirror commented

2025-11-02 05:00:56 -06:00

@nicovince commented on GitHub (Mar 26, 2019):

By the way the version of sqlite installed on the server have no incidence because sqlite is embedded in the go application. (sqlite has been removed from the server and gitea keeps on working).

@nicovince commented on GitHub (Mar 26, 2019): By the way the version of sqlite installed on the server have no incidence because sqlite is embedded in the go application. (sqlite has been removed from the server and gitea keeps on working).

GiteaMirror commented

2025-11-02 05:00:57 -06:00

@jolheiser commented on GitHub (Mar 26, 2019):

I believe the issue here is likely because our notification services are running in goroutines and thus async.

I'm not an expert with channels, but I think we would need to make sure the channel is drained before continuing (which sort of defeats the purpose of async...) in order to avoid an instance where a notifier is still looping/creating notifications when Gitea attempts to get a count using the same table, which it does when it fills the context.

Notifier consumes and starts making DB transactions async:
82e08a3364/modules/notification/ui/ui.go (L37-L46)

Gitea tries to serve next page, context attempts to load notifications:
801843b011/routers/user/notification.go (L21-L38)

Attempts to count from notification table, which may be locked by the notifications?
b34996a629/routers/routes/routes.go (L193)

Or am I missing somewhere that handles this?

@jolheiser commented on GitHub (Mar 26, 2019): I believe the issue here is likely because our notification services are running in goroutines and thus async. I'm not an expert with channels, but I think we would need to make sure the channel is drained before continuing (which sort of defeats the purpose of async...) in order to avoid an instance where a notifier is still looping/creating notifications when Gitea attempts to get a count using the same table, which it does when it fills the context. Notifier consumes and starts making DB transactions async: https://github.com/go-gitea/gitea/blob/82e08a3364195b515a005180c2bdc08e78aac208/modules/notification/ui/ui.go#L37-L46 Gitea tries to serve next page, context attempts to load notifications: https://github.com/go-gitea/gitea/blob/801843b0115e29ba2304fa6a5bea1ae169a58e02/routers/user/notification.go#L21-L38 Attempts to count from notification table, which may be locked by the notifications? https://github.com/go-gitea/gitea/blob/b34996a62937b23121d19912b37ed2b1023f1479/routers/routes/routes.go#L193 Or am I missing somewhere that handles this?

GiteaMirror commented

2025-11-02 05:00:57 -06:00

@zeripath commented on GitHub (Mar 26, 2019):

Deadlocks like this are the definition of random events. They are the original Heisenbug.

Ok, so I have a few questions:

Have you changed the logging? It's this the default configuration?
It's probably hard to do, but I'd like to see the XORM logs around this point if possible. Also the general logs from gitea near here too.
How high is your load?

Now, unlike most of the deadlocks we've experienced, I suspect this might be a different type. This might actually be the case where two processes are trying to write to the database. It might be that we have to make the Gitea serv completely a client of the main process.

This is probably an example of where sqlite is not appropriate for installations with more than a few users.

@zeripath commented on GitHub (Mar 26, 2019): Deadlocks like this are the definition of random events. They are the original Heisenbug. Ok, so I have a few questions: 1. Have you changed the logging? It's this the default configuration? 2. It's probably hard to do, but I'd like to see the XORM logs around this point if possible. Also the general logs from gitea near here too. 3. How high is your load? Now, unlike most of the deadlocks we've experienced, I suspect this might be a different type. This might actually be the case where two processes are trying to write to the database. It might be that we have to make the Gitea serv completely a client of the main process. This is probably an example of where sqlite is not appropriate for installations with more than a few users.

GiteaMirror commented

2025-11-02 05:00:57 -06:00

@zeripath commented on GitHub (Mar 26, 2019):

Hmm. I think I have completely misread the issue. For some reason I was thinking this might have something to do with when actually pushing!

Instead this might just be another deadlock bug. I suspect that @jolheiser is on the right track and that the problem lies in models.CreateOrUpdateIssueNotifications. I wonder if that is somehow causing a deadlock.

This is where the XORM logs would be helpful.

@zeripath commented on GitHub (Mar 26, 2019): Hmm. I think I have completely misread the issue. For some reason I was thinking this might have something to do with when actually pushing! Instead this might just be another deadlock bug. I suspect that @jolheiser is on the right track and that the problem lies in models.CreateOrUpdateIssueNotifications. I wonder if that is somehow causing a deadlock. This is where the XORM logs would be helpful.

GiteaMirror commented

2025-11-02 05:00:57 -06:00

@lunny commented on GitHub (Mar 27, 2019):

This is not related with pull request, it's a random event. I think @typeless had sent a PR to fix the sqlite lock issue. @nicovince how did you get gitea? I think the gitea version should be with all the build tags.

@lunny commented on GitHub (Mar 27, 2019): This is not related with pull request, it's a random event. I think @typeless had sent a PR to fix the sqlite lock issue. @nicovince how did you get gitea? I think the gitea version should be with all the build tags.

GiteaMirror commented

2025-11-02 05:00:58 -06:00

@typeless commented on GitHub (Mar 27, 2019):

@nicovince Did you built Gitea from source? Please check if Gitea is built with sqlite_unlock_notify.

@typeless commented on GitHub (Mar 27, 2019): @nicovince Did you built Gitea from source? Please check if Gitea is built with `sqlite_unlock_notify`.

GiteaMirror commented

2025-11-02 05:00:58 -06:00

@nicovince commented on GitHub (Mar 27, 2019):

gitea-all-logs-500-anon.txt
Here is some more logs (http.log, gitea.log, xorm.log) which happens around a 500.

To give you an idea of my setup :

11 users, some of which constantly are logged on the gitea web interface
2 administators constantly logged on the gitea web interface
~80 repositories (including forks)

During installation sqlite seemed a good candidate and it has the advantage of being easy to make a backup of the VM running gitea (no need to dump the database beforehand).
In the future we may switch to another database (postgre probably) but it would require the resolution of #6090

Let me know if you need more informations.

@nicovince commented on GitHub (Mar 27, 2019): [gitea-all-logs-500-anon.txt](https://github.com/go-gitea/gitea/files/3012740/gitea-all-logs-500-anon.txt) Here is some more logs (http.log, gitea.log, xorm.log) which happens around a 500. To give you an idea of my setup : - 11 users, some of which constantly are logged on the gitea web interface - 2 administators constantly logged on the gitea web interface - ~80 repositories (including forks) During installation sqlite seemed a good candidate and it has the advantage of being easy to make a backup of the VM running gitea (no need to dump the database beforehand). In the future we may switch to another database (postgre probably) but it would require the resolution of #6090 Let me know if you need more informations.

GiteaMirror commented

2025-11-02 05:00:58 -06:00

@nicovince commented on GitHub (Mar 27, 2019):

@lunny @typeless Regarding how i got gitea, it is a package from alpine : https://git.alpinelinux.org/aports/tree/community/gitea/APKBUILD?id=ef3cd926643f971359265dd121a05ccb96733968 and it looks like the sqlite_unlock_notify flag might be missing.

From the site admin/config page i have the following :
Gitea Version : 1.7.3 built with go1.11.5 : tidb, sqlite

@nicovince commented on GitHub (Mar 27, 2019): @lunny @typeless Regarding how i got gitea, it is a package from alpine : https://git.alpinelinux.org/aports/tree/community/gitea/APKBUILD?id=ef3cd926643f971359265dd121a05ccb96733968 and it looks like the sqlite_unlock_notify flag might be missing. From the site admin/config page i have the following : Gitea Version : 1.7.3 built with go1.11.5 : tidb, sqlite

GiteaMirror commented

2025-11-02 05:00:58 -06:00

@lunny commented on GitHub (Mar 27, 2019):

@nicovince missing sqlite_unlock_notify flag will make sqlite lock table sometimes. For your situation(11 users), we recommend you to migrate to other databases.

@lunny commented on GitHub (Mar 27, 2019): @nicovince missing sqlite_unlock_notify flag will make sqlite lock table sometimes. For your situation(11 users), we recommend you to migrate to other databases.

GiteaMirror commented

2025-11-02 05:00:59 -06:00

@nicovince commented on GitHub (Mar 28, 2019):

Thanks for the update, I'll look into recompiling gitea alpine's package with proper flag. Hopefully that will resolve my issue.

In order to migrate to another database I'm blocked by #6090 and #6034 (I'll try to update that one so you can reproduce it).

@nicovince commented on GitHub (Mar 28, 2019): Thanks for the update, I'll look into recompiling gitea alpine's package with proper flag. Hopefully that will resolve my issue. In order to migrate to another database I'm blocked by #6090 and #6034 (I'll try to update that one so you can reproduce it).

GiteaMirror commented

2025-11-02 05:00:59 -06:00

@nicovince commented on GitHub (Apr 10, 2019):

Problem fixed with new alpine package since : https://github.com/alpinelinux/aports/pull/6859

Thank you all for your support :)

@nicovince commented on GitHub (Apr 10, 2019): Problem fixed with new alpine package since : https://github.com/alpinelinux/aports/pull/6859 Thank you all for your support :)

GiteaMirror referenced this issue

2025-11-02 12:18:41 -06:00

[PR #3106] [MERGED] Fix legacy URL redirects (#3100) #16769

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/gitea#3106