Check storage readiness and show meaningful error message when DB is busy #11288

Closed
opened 2025-11-02 09:33:26 -06:00 by GiteaMirror · 6 comments
Owner

Originally created by @aku on GitHub (Jul 20, 2023).

Feature Description

I've tried to install Gitea with PostgreSQL storage via Helm chart and faced the following problem.
When you try to access Gitea right after it has been installed, some actions will fail with 500 errors because PostgreSQL is doing some replication/recovery.

It would be nice to add some sort of DB readiness check and show a meaningful error explaining that we need to wait for some time until the DB is ready

Example of error in a log file:

2023/07/20 08:09:19 ...ers/web/auth/auth.go:543:createUserInContext() [E] CreateUser: pq: cannot set transaction read-write mode during recovery
2023/07/20 08:09:19 ...eb/routing/logger.go:102:func1() [I] router: completed POST /user/sign_up for 10.244.0.1:1504, 500 Internal Server Error in 0.8ms @ auth/auth.go:405(auth.SignUpPost)

Screenshots

No response

Originally created by @aku on GitHub (Jul 20, 2023). ### Feature Description I've tried to install Gitea with PostgreSQL storage via Helm chart and faced the following problem. When you try to access Gitea right after it has been installed, some actions will fail with 500 errors because PostgreSQL is doing some replication/recovery. It would be nice to add some sort of DB readiness check and show a meaningful error explaining that we need to wait for some time until the DB is ready Example of error in a log file: ``` 2023/07/20 08:09:19 ...ers/web/auth/auth.go:543:createUserInContext() [E] CreateUser: pq: cannot set transaction read-write mode during recovery 2023/07/20 08:09:19 ...eb/routing/logger.go:102:func1() [I] router: completed POST /user/sign_up for 10.244.0.1:1504, 500 Internal Server Error in 0.8ms @ auth/auth.go:405(auth.SignUpPost) ``` ### Screenshots _No response_
GiteaMirror added the issue/needs-feedback label 2025-11-02 09:33:26 -06:00
Author
Owner

@weironz commented on GitHub (Aug 25, 2023):

same here, v1.20.3 with helm install:

helm install gitea gitea-charts/gitea \
-n gitea --create-namespace \
--set gitea.admin.password=gitea_admin \
--set ingress.enabled=true \
--set ingress.className=nginx

i must wait some time and pods will running

root@ubuntu:~# kubectl -n gitea get pods
NAME                                          READY   STATUS             RESTARTS        AGE
gitea-9cf775dfb-xmd4s                         0/1     CrashLoopBackOff   4 (77s ago)     5m
gitea-postgresql-ha-pgpool-55ddb7cbf9-8v4qq   1/1     Running            0               5m
gitea-postgresql-ha-postgresql-0              1/1     Running            0               5m
gitea-postgresql-ha-postgresql-1              1/1     Running            0               5m
gitea-postgresql-ha-postgresql-2              1/1     Running            0               5m
gitea-redis-cluster-0                         1/1     Running            2 (3m51s ago)   5m
gitea-redis-cluster-1                         1/1     Running            3 (4m4s ago)    5m
gitea-redis-cluster-2                         1/1     Running            2 (4m4s ago)    5m
gitea-redis-cluster-3                         1/1     Running            2 (3m58s ago)   5m
gitea-redis-cluster-4                         1/1     Running            2 (3m18s ago)   5m
gitea-redis-cluster-5                         1/1     Running            1 (3m17s ago)   5m

root@ubuntu:~# kubectl -n gitea logs -f gitea-9cf775dfb-xmd4s
Defaulted container "gitea" out of: gitea, init-directories (init), init-app-ini (init), configure-gitea (init)
2023/08/25 06:40:00 cmd/web.go:223:runWeb() [I] Starting Gitea on PID: 7
2023/08/25 06:40:00 cmd/web.go:148:serveInstalled() [I] Gitea version: 1.20.3 built with GNU Make 4.4.1, go1.20.7 : bindata, timetzdata, sqlite, sqlite_unlock_notify
2023/08/25 06:40:00 cmd/web.go:149:serveInstalled() [I] App path: /usr/local/bin/gitea
2023/08/25 06:40:00 cmd/web.go:150:serveInstalled() [I] Work path: /data
2023/08/25 06:40:00 cmd/web.go:151:serveInstalled() [I] Custom path: /data/gitea
2023/08/25 06:40:00 cmd/web.go:152:serveInstalled() [I] Config file: /data/gitea/conf/app.ini
2023/08/25 06:40:00 cmd/web.go:153:serveInstalled() [I] Run mode: prod
2023/08/25 06:40:00 cmd/web.go:154:serveInstalled() [I] Prepare to run web server
2023/08/25 06:40:00 routers/init.go:112:InitWebInstalled() [I] Git version: 2.40.1, Wire Protocol Version 2 Enabled (home: /data/home)
2023/08/25 06:40:01 ...les/setting/cache.go:75:loadCacheFrom() [I] Cache Service Enabled
2023/08/25 06:40:01 ...les/setting/cache.go:90:loadCacheFrom() [I] Last Commit Cache Service Enabled
2023/08/25 06:40:01 ...s/setting/session.go:74:loadSessionFrom() [I] Session Service Enabled
2023/08/25 06:40:01 ...s/storage/storage.go:176:initAttachments() [I] Initialising Attachment storage with type: local
2023/08/25 06:40:01 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /data/attachments
2023/08/25 06:40:01 ...s/storage/storage.go:166:initAvatars() [I] Initialising Avatar storage with type: local
2023/08/25 06:40:01 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /data/avatars
2023/08/25 06:40:01 ...s/storage/storage.go:192:initRepoAvatars() [I] Initialising Repository Avatar storage with type: local
2023/08/25 06:40:01 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /data/repo-avatars
2023/08/25 06:40:01 ...s/storage/storage.go:198:initRepoArchives() [I] Initialising Repository Archive storage with type: local
2023/08/25 06:40:01 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /data/repo-archive
2023/08/25 06:40:01 ...s/storage/storage.go:208:initPackages() [I] Initialising Packages storage with type: local
2023/08/25 06:40:01 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /data/packages
2023/08/25 06:40:01 routers/init.go:130:InitWebInstalled() [I] SQLite3 support is enabled
2023/08/25 06:40:01 routers/common/db.go:21:InitDBEngine() [I] Beginning ORM engine initialization.
2023/08/25 06:40:01 routers/common/db.go:28:InitDBEngine() [I] ORM engine initialization attempt #1/10...
2023/08/25 06:40:01 cmd/web.go:175:serveInstalled() [I] PING DATABASE postgres
2023/08/25 06:40:01 cmd/web.go:175:serveInstalled() [W] Table system_setting Column version db default is , struct default is 1
2023/08/25 06:40:01 cmd/web.go:175:serveInstalled() [W] Table oauth2_application Column confidential_client db default is true, struct default is TRUE
2023/08/25 06:40:01 cmd/web.go:175:serveInstalled() [W] Table user Column max_repo_creation db default is '-1', struct default is -1
2023/08/25 06:40:01 routers/init.go:136:InitWebInstalled() [I] ORM engine initialization successful!
2023/08/25 06:40:01 ...er/issues/indexer.go:156:func2() [I] PID 7: Initializing Issue Indexer: db
2023/08/25 06:40:01 ...er/issues/indexer.go:246:func3() [I] Issue Indexer Initialization took 48.389µs
2023/08/25 06:40:01 ...xer/stats/indexer.go:38:populateRepoIndexer() [I] Populating the repo stats indexer with existing repositories
2023/08/25 06:40:01 routers/init.go:81:syncAppConfForGit() [I] AppPath changed from '' to '/usr/local/bin/gitea'
2023/08/25 06:40:01 routers/init.go:86:syncAppConfForGit() [I] CustomConf changed from '' to '/data/gitea/conf/app.ini'
2023/08/25 06:40:01 routers/init.go:92:syncAppConfForGit() [I] re-sync repository hooks ...
2023/08/25 06:40:01 routers/init.go:95:syncAppConfForGit() [I] re-write ssh public keys ...
2023/08/25 06:40:01 routers/init.go:69:mustInitCtx() [F] code.gitea.io/gitea/routers.syncAppConfForGit(ctx) failed: pq: cannot set transaction read-write mode during recovery
root@ubuntu:~# 

but UI still have 500 error
image

@weironz commented on GitHub (Aug 25, 2023): same here, v1.20.3 with helm install: ``` helm install gitea gitea-charts/gitea \ -n gitea --create-namespace \ --set gitea.admin.password=gitea_admin \ --set ingress.enabled=true \ --set ingress.className=nginx ``` i must wait some time and pods will running ``` root@ubuntu:~# kubectl -n gitea get pods NAME READY STATUS RESTARTS AGE gitea-9cf775dfb-xmd4s 0/1 CrashLoopBackOff 4 (77s ago) 5m gitea-postgresql-ha-pgpool-55ddb7cbf9-8v4qq 1/1 Running 0 5m gitea-postgresql-ha-postgresql-0 1/1 Running 0 5m gitea-postgresql-ha-postgresql-1 1/1 Running 0 5m gitea-postgresql-ha-postgresql-2 1/1 Running 0 5m gitea-redis-cluster-0 1/1 Running 2 (3m51s ago) 5m gitea-redis-cluster-1 1/1 Running 3 (4m4s ago) 5m gitea-redis-cluster-2 1/1 Running 2 (4m4s ago) 5m gitea-redis-cluster-3 1/1 Running 2 (3m58s ago) 5m gitea-redis-cluster-4 1/1 Running 2 (3m18s ago) 5m gitea-redis-cluster-5 1/1 Running 1 (3m17s ago) 5m root@ubuntu:~# kubectl -n gitea logs -f gitea-9cf775dfb-xmd4s Defaulted container "gitea" out of: gitea, init-directories (init), init-app-ini (init), configure-gitea (init) 2023/08/25 06:40:00 cmd/web.go:223:runWeb() [I] Starting Gitea on PID: 7 2023/08/25 06:40:00 cmd/web.go:148:serveInstalled() [I] Gitea version: 1.20.3 built with GNU Make 4.4.1, go1.20.7 : bindata, timetzdata, sqlite, sqlite_unlock_notify 2023/08/25 06:40:00 cmd/web.go:149:serveInstalled() [I] App path: /usr/local/bin/gitea 2023/08/25 06:40:00 cmd/web.go:150:serveInstalled() [I] Work path: /data 2023/08/25 06:40:00 cmd/web.go:151:serveInstalled() [I] Custom path: /data/gitea 2023/08/25 06:40:00 cmd/web.go:152:serveInstalled() [I] Config file: /data/gitea/conf/app.ini 2023/08/25 06:40:00 cmd/web.go:153:serveInstalled() [I] Run mode: prod 2023/08/25 06:40:00 cmd/web.go:154:serveInstalled() [I] Prepare to run web server 2023/08/25 06:40:00 routers/init.go:112:InitWebInstalled() [I] Git version: 2.40.1, Wire Protocol Version 2 Enabled (home: /data/home) 2023/08/25 06:40:01 ...les/setting/cache.go:75:loadCacheFrom() [I] Cache Service Enabled 2023/08/25 06:40:01 ...les/setting/cache.go:90:loadCacheFrom() [I] Last Commit Cache Service Enabled 2023/08/25 06:40:01 ...s/setting/session.go:74:loadSessionFrom() [I] Session Service Enabled 2023/08/25 06:40:01 ...s/storage/storage.go:176:initAttachments() [I] Initialising Attachment storage with type: local 2023/08/25 06:40:01 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /data/attachments 2023/08/25 06:40:01 ...s/storage/storage.go:166:initAvatars() [I] Initialising Avatar storage with type: local 2023/08/25 06:40:01 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /data/avatars 2023/08/25 06:40:01 ...s/storage/storage.go:192:initRepoAvatars() [I] Initialising Repository Avatar storage with type: local 2023/08/25 06:40:01 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /data/repo-avatars 2023/08/25 06:40:01 ...s/storage/storage.go:198:initRepoArchives() [I] Initialising Repository Archive storage with type: local 2023/08/25 06:40:01 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /data/repo-archive 2023/08/25 06:40:01 ...s/storage/storage.go:208:initPackages() [I] Initialising Packages storage with type: local 2023/08/25 06:40:01 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /data/packages 2023/08/25 06:40:01 routers/init.go:130:InitWebInstalled() [I] SQLite3 support is enabled 2023/08/25 06:40:01 routers/common/db.go:21:InitDBEngine() [I] Beginning ORM engine initialization. 2023/08/25 06:40:01 routers/common/db.go:28:InitDBEngine() [I] ORM engine initialization attempt #1/10... 2023/08/25 06:40:01 cmd/web.go:175:serveInstalled() [I] PING DATABASE postgres 2023/08/25 06:40:01 cmd/web.go:175:serveInstalled() [W] Table system_setting Column version db default is , struct default is 1 2023/08/25 06:40:01 cmd/web.go:175:serveInstalled() [W] Table oauth2_application Column confidential_client db default is true, struct default is TRUE 2023/08/25 06:40:01 cmd/web.go:175:serveInstalled() [W] Table user Column max_repo_creation db default is '-1', struct default is -1 2023/08/25 06:40:01 routers/init.go:136:InitWebInstalled() [I] ORM engine initialization successful! 2023/08/25 06:40:01 ...er/issues/indexer.go:156:func2() [I] PID 7: Initializing Issue Indexer: db 2023/08/25 06:40:01 ...er/issues/indexer.go:246:func3() [I] Issue Indexer Initialization took 48.389µs 2023/08/25 06:40:01 ...xer/stats/indexer.go:38:populateRepoIndexer() [I] Populating the repo stats indexer with existing repositories 2023/08/25 06:40:01 routers/init.go:81:syncAppConfForGit() [I] AppPath changed from '' to '/usr/local/bin/gitea' 2023/08/25 06:40:01 routers/init.go:86:syncAppConfForGit() [I] CustomConf changed from '' to '/data/gitea/conf/app.ini' 2023/08/25 06:40:01 routers/init.go:92:syncAppConfForGit() [I] re-sync repository hooks ... 2023/08/25 06:40:01 routers/init.go:95:syncAppConfForGit() [I] re-write ssh public keys ... 2023/08/25 06:40:01 routers/init.go:69:mustInitCtx() [F] code.gitea.io/gitea/routers.syncAppConfForGit(ctx) failed: pq: cannot set transaction read-write mode during recovery root@ubuntu:~# ``` but UI still have 500 error ![image](https://github.com/go-gitea/gitea/assets/33590311/8d53685e-afa1-4ce2-9b88-e50fe96e8f54)
Author
Owner

@wxiaoguang commented on GitHub (Aug 25, 2023):

Are your Gitea connecting to a read-only replica instance?

https://stackoverflow.com/questions/24384407/cannot-set-transaction-read-write-mode-during-recovery-error

The answer is: Connect to the master server instead.

@wxiaoguang commented on GitHub (Aug 25, 2023): Are your Gitea connecting to a read-only replica instance? https://stackoverflow.com/questions/24384407/cannot-set-transaction-read-write-mode-during-recovery-error The answer is: `Connect to the master server instead.`
Author
Owner

@phardelain-kouassi commented on GitHub (Aug 25, 2023):

Check the logs of one of the posgres-db and restart the pod where you see the error message in the log. this help to solve the issue.
Did it and it works
https://github.com/zalando/postgres-operator/issues/2354#issuecomment-1646446060

@phardelain-kouassi commented on GitHub (Aug 25, 2023): Check the logs of one of the posgres-db and restart the pod where you see the error message in the log. this help to solve the issue. Did it and it works https://github.com/zalando/postgres-operator/issues/2354#issuecomment-1646446060
Author
Owner

@weironz commented on GitHub (Aug 25, 2023):

Thanks, it work , i delete two standby pg to restart them, and login again, no 500 error

kubectl -n gitea delete pods gitea-postgresql-ha-postgresql-1  gitea-postgresql-ha-postgresql-2
@weironz commented on GitHub (Aug 25, 2023): Thanks, it work , i delete two standby pg to restart them, and login again, no 500 error ``` kubectl -n gitea delete pods gitea-postgresql-ha-postgresql-1 gitea-postgresql-ha-postgresql-2 ```
Author
Owner

@SchisandraX commented on GitHub (Sep 1, 2023):

I also encountered the same problem, and my expectation was to deploy automatically through helm without human intervention

@SchisandraX commented on GitHub (Sep 1, 2023): I also encountered the same problem, and my expectation was to deploy automatically through helm without human intervention
Author
Owner

@GiteaBot commented on GitHub (Oct 1, 2023):

We close issues that need feedback from the author if there were no new comments for a month. 🍵

@GiteaBot commented on GitHub (Oct 1, 2023): We close issues that need feedback from the author if there were no new comments for a month. :tea:
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/gitea#11288