Initial Git hook syncing causes downtime for large instances #9792

Originally created by @fnetX on GitHub (Nov 7, 2022).

Description

https://github.com/go-gitea/gitea/pull/21619, #21625 & #21626 seem to do a Git hook sync on first run, because previous app config path is empty.

In our logs, we see

2022/11/07 02:57:24 routers/init.go:90:syncAppConfForGit() [I] CustomConf changed from '' to '/etc/gitea/conf/app.ini'

and for several tens of thousands of repositories, this takes quite a while, and caused unexpected downtime.

We reverted the patch for now in 26b92e3f80

Gitea Version

1.17.3 7a2daae7c3

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

No response

Operating System

No response

How are you running Gitea?

Custom build

Database

MySQL

Originally created by @fnetX on GitHub (Nov 7, 2022). ### Description https://github.com/go-gitea/gitea/pull/21619, #21625 & #21626 seem to do a Git hook sync on first run, because previous app config path is empty. In our logs, we see ~~~ 2022/11/07 02:57:24 routers/init.go:90:syncAppConfForGit() [I] CustomConf changed from '' to '/etc/gitea/conf/app.ini' ~~~ and for several tens of thousands of repositories, this takes quite a while, and caused unexpected downtime. We reverted the patch for now in https://codeberg.org/Codeberg/gitea/commit/26b92e3f80961de11a48d3d307d50065aca8922a ### Gitea Version 1.17.3 7a2daae7c3a203a3716d653bfcdcdbb2fe48cb2d ### Can you reproduce the bug on the Gitea demo site? No ### Log Gist _No response_ ### Screenshots _No response_ ### Git Version _No response_ ### Operating System _No response_ ### How are you running Gitea? Custom build ### Database MySQL

GiteaMirror added the type/bug label 2025-11-02 08:49:47 -06:00

GiteaMirror closed this issue

GiteaMirror commented

@fnetX commented on GitHub (Nov 7, 2022):

If at all, this should be done async, after the software is back online IMHO. Or the value for the config should not be empty on first run.

Another solution I proposed a while ago: https://github.com/go-gitea/gitea/issues/18995

@fnetX commented on GitHub (Nov 7, 2022): If at all, this should be done async, after the software is back online IMHO. Or the value for the config should not be empty on first run. Another solution I proposed a while ago: https://github.com/go-gitea/gitea/issues/18995

GiteaMirror commented

@lunny commented on GitHub (Nov 7, 2022):

If at all, this should be done async, after the software is back online IMHO. Or the value for the config should not be empty on first run.

Another solution I proposed a while ago: #18995

You can also run the sync task on admin panel. And for a large instance, maybe more than 1 instance is better so that you could upgrade one by one.

@lunny commented on GitHub (Nov 7, 2022): > If at all, this should be done async, after the software is back online IMHO. Or the value for the config should not be empty on first run. > > Another solution I proposed a while ago: #18995 You can also run the sync task on admin panel. And for a large instance, maybe more than 1 instance is better so that you could upgrade one by one.

GiteaMirror commented

@fnetX commented on GitHub (Nov 7, 2022):

My point is: Our app location didn't change, it is where it always was, so we don't even need to update 47k project's hooks. This sync is only triggered because our alleged app location is empty.

@fnetX commented on GitHub (Nov 7, 2022): My point is: Our app location didn't change, it is where it always was, so we don't even need to update 47k project's hooks. This sync is only triggered because our alleged app location is empty.

GiteaMirror commented

@lunny commented on GitHub (Nov 7, 2022):

It should have been checked if the configuration file path changed. ref https://github.com/go-gitea/gitea/pull/21626/files#diff-3d9812b7e268b55c71e312a7dd154900ab1036c47493f4cb91f608b32a7541fcR91

@lunny commented on GitHub (Nov 7, 2022): It should have been checked if the configuration file path changed. ref https://github.com/go-gitea/gitea/pull/21626/files#diff-3d9812b7e268b55c71e312a7dd154900ab1036c47493f4cb91f608b32a7541fcR91

GiteaMirror commented

@fnetX commented on GitHub (Nov 7, 2022):

Yes, and as you surely read in my initial report, it is considered "empty" for some reason. But the config file wasn't moved.

2022/11/07 02:57:24 routers/init.go:90:syncAppConfForGit() [I] CustomConf changed from '' to '/etc/gitea/conf/app.ini'

@fnetX commented on GitHub (Nov 7, 2022): Yes, and as you surely read in my initial report, it is considered "empty" for some reason. But the config file wasn't moved. ~~~ 2022/11/07 02:57:24 routers/init.go:90:syncAppConfForGit() [I] CustomConf changed from '' to '/etc/gitea/conf/app.ini' ~~~

GiteaMirror commented

@lunny commented on GitHub (Nov 7, 2022):

Yes, and as you surely read in my initial report, it is considered "empty" for some reason. But the config file wasn't moved.
2022/11/07 02:57:24 routers/init.go:90:syncAppConfForGit() [I] CustomConf changed from '' to '/etc/gitea/conf/app.ini'

Ah, yes. It's a problem for the first running. The new system state item was introduced from that PR, so before it's empty, but next time it should be right value.

@lunny commented on GitHub (Nov 7, 2022): > Yes, and as you surely read in my initial report, it is considered "empty" for some reason. But the config file wasn't moved. > > ``` > 2022/11/07 02:57:24 routers/init.go:90:syncAppConfForGit() [I] CustomConf changed from '' to '/etc/gitea/conf/app.ini' > ``` Ah, yes. It's a problem for the first running. The new system state item was introduced from that PR, so before it's empty, but next time it should be right value.

GiteaMirror commented

@fnetX commented on GitHub (Nov 7, 2022):

Yes, but in order to reach next time, we need to face a prolonged downtime in order to sync the hooks of many repositories, this is what the bug report is about.

@fnetX commented on GitHub (Nov 7, 2022): Yes, but in order to reach next time, we need to face a prolonged downtime in order to sync the hooks of many repositories, this is what the bug report is about.

GiteaMirror commented

@wxiaoguang commented on GitHub (Dec 6, 2022):

It helps poor users like https://github.com/go-gitea/gitea/issues/22049

@wxiaoguang commented on GitHub (Dec 6, 2022): It helps poor users like https://github.com/go-gitea/gitea/issues/22049

GiteaMirror commented

@fnetX commented on GitHub (Dec 13, 2022):

The implemented feature helps users, yes. The bug that the initial value is empty does not help anyone, but is a bug.

@fnetX commented on GitHub (Dec 13, 2022): The implemented feature helps users, yes. The bug that the initial value is empty does not help anyone, but is a bug.

GiteaMirror commented

@wxiaoguang commented on GitHub (Dec 14, 2022):

But what's the proper initial value? Gitea knows nothing about the correct config path at the beginning.

Users are usually affected by such bug when then do upgrade:

User is running an old Gitea with /etc/gitea/app-1.17.ini
Then the user decides to upgrade Gitea and use a config /etc/gitea/app-1.18.ini
When new Gitea runs, Gitea only sees /etc/gitea/app-1.18.ini, if it's stored as the default value, the git hooks are out-of-sync then.

ps: it could be a rare case, but it could happen, it only happens for users who first upgrade to Gitea with config path sync support and they change the config path.

@wxiaoguang commented on GitHub (Dec 14, 2022): But what's the proper initial value? Gitea knows nothing about the correct config path at the beginning. Users are usually affected by such bug when then do upgrade: * User is running an old Gitea with `/etc/gitea/app-1.17.ini` * Then the user decides to upgrade Gitea and use a config `/etc/gitea/app-1.18.ini` * When new Gitea runs, Gitea only sees `/etc/gitea/app-1.18.ini`, if it's stored as the default value, the git hooks are out-of-sync then. ps: it could be a rare case, but it could happen, it only happens for users who first upgrade to Gitea with config path sync support and they change the config path.

GiteaMirror commented

@lunny commented on GitHub (Jul 27, 2023):

The newly installed big instances will not be affected by this if they doesn't change the configuration file path.

@lunny commented on GitHub (Jul 27, 2023): The newly installed big instances will not be affected by this if they doesn't change the configuration file path.

GiteaMirror commented