systemd: gitea.service not started after database upgrade #7337

New Issue

GiteaMirror · 2025-11-02T07:23:18-06:00

GiteaMirror commented

2025-11-02 07:23:18 -06:00

Originally created by @silverwind on GitHub (May 14, 2021).

In the systemd service we recommend a Requires= dependency, but it seems this causes a issue with the service not being started back up when apt upgrades the database dependency. I observed the following sequence with such a configuration during a unattended APT upgrade of mysql.service on Ubuntu 20.04.2 using systemd 245:

May 13 06:34:13 systemd[1]: Stopping Gitea (Git with a cup of tea)...
May 13 06:34:13 systemd[1]: Stopping MySQL Community Server...
May 13 06:34:14 systemd[1]: Stopped MySQL Community Server.
May 13 06:34:14 systemd[1]: Stopped Gitea (Git with a cup of tea).
May 13 06:34:39 systemd[1]: Starting MySQL Community Server...
May 13 06:34:40 systemd[1]: Started MySQL Community Server.

Essentially gitea.service is not started up again after the database upgrade for unknown reason.

I don't fully understand why systemd would not bring up gitea.service in such a case, but I think we may want to consider using Wants= instead of Requires= to indicate a "weak" dependency which may help in such a case, or find out what systemd exactly does here.

https://unix.stackexchange.com/questions/375091 sounds like the same issue but I don't think the solution is acceptable.

Originally created by @silverwind on GitHub (May 14, 2021). In the [systemd service](https://github.com/go-gitea/gitea/blob/05fb1f61d56c29f5637dc7fba218e92a24145f92/contrib/systemd/gitea.service#L9-L13) we recommend a [`Requires=`](https://www.freedesktop.org/software/systemd/man/systemd.unit.html) dependency, but it seems this causes a issue with the service not being started back up when `apt` upgrades the database dependency. I observed the following sequence with such a configuration during a unattended APT upgrade of `mysql.service` on Ubuntu 20.04.2 using systemd 245: ``` May 13 06:34:13 systemd[1]: Stopping Gitea (Git with a cup of tea)... May 13 06:34:13 systemd[1]: Stopping MySQL Community Server... May 13 06:34:14 systemd[1]: Stopped MySQL Community Server. May 13 06:34:14 systemd[1]: Stopped Gitea (Git with a cup of tea). May 13 06:34:39 systemd[1]: Starting MySQL Community Server... May 13 06:34:40 systemd[1]: Started MySQL Community Server. ``` Essentially `gitea.service` is not started up again after the database upgrade for unknown reason. I don't fully understand why systemd would not bring up `gitea.service` in such a case, but I think we may want to consider using `Wants=` instead of `Requires=` to indicate a "weak" dependency which may help in such a case, or find out what systemd exactly does here. https://unix.stackexchange.com/questions/375091 sounds like the same issue but I don't think the solution is acceptable.

GiteaMirror added the topic/deployment label 2025-11-02 07:23:18 -06:00

GiteaMirror closed this issue

2025-11-02 07:23:18 -06:00

GiteaMirror commented

2025-11-02 07:23:19 -06:00

@zeripath commented on GitHub (May 14, 2021):

After= might be better?

@zeripath commented on GitHub (May 14, 2021): `After=` might be better?

GiteaMirror commented

2025-11-02 07:23:19 -06:00

@silverwind commented on GitHub (May 14, 2021):

We had After= before but is was changed to Requires= in dbb4aeead9. I do think a dependency like Requires= or Wants= is appropriate but there is something particular that apt/systemd do that prevents the service from starting after the dependency was restarted.

@silverwind commented on GitHub (May 14, 2021): We had `After=` before but is was changed to `Requires=` in https://github.com/go-gitea/gitea/commit/dbb4aeead993a419f109a23c610d5e6d2b45697f. I do think a dependency like `Requires=` or `Wants=` is appropriate but there is something particular that apt/systemd do that prevents the service from starting after the dependency was restarted.

GiteaMirror commented

2025-11-02 07:23:19 -06:00

@zeripath commented on GitHub (May 14, 2021):

I think there's nothing stopping us from having a Requires and After for the same value.

I suspect the problem is that without the After the mysqld.service doesn't know that it needs to restart the service. All systemd knew is that it needed to stop the service because a Required dependency no longer existed.

I guess we need to know the semantics of Requires, Wants and After.

My suspicion is that there are subtle implications of these and that Requires won't start a required service and will shutdown if a Requires service is shutdown. Wants will start the wanted service and will shutdown if the wanted service shutsdown. Whereas After will tell the service that it comes after that it should be started after that service is started.

One test of that theory would be to restart the network.target and see if Gitea is restarted after that.

@zeripath commented on GitHub (May 14, 2021): I think there's nothing stopping us from having a Requires and After for the same value. I suspect the problem is that without the After the mysqld.service doesn't know that it needs to restart the service. All systemd knew is that it needed to stop the service because a Required dependency no longer existed. I guess we need to know the semantics of Requires, Wants and After. My suspicion is that there are subtle implications of these and that Requires won't start a required service and will shutdown if a Requires service is shutdown. Wants will start the wanted service and will shutdown if the wanted service shutsdown. Whereas After will tell the service that it comes after that it should be started after that service is started. One test of that theory would be to restart the network.target and see if Gitea is restarted after that.

GiteaMirror commented

2025-11-02 07:23:19 -06:00

@silverwind commented on GitHub (May 14, 2021):

One particular interesting comment I found on above stackexchange question is:

Amusingly enough, when you do systemctl restart postgresql, systemd remembers to restart its dependent services again. It seems like apt, for some reason, does combination of stop and start instead of restart

This should be testable by creating 2 units with a Require= relation and then stop/start the one that is being required. Systemd docs don't really mention what happens in such a situation in detail.

@silverwind commented on GitHub (May 14, 2021): One particular interesting comment I found on above stackexchange question is: > Amusingly enough, when you do systemctl restart postgresql, systemd remembers to restart its dependent services again. It seems like apt, for some reason, does combination of stop and start instead of restart This should be testable by creating 2 units with a `Require=` relation and then stop/start the one that is being required. Systemd docs don't really mention what happens in such a situation in detail.

GiteaMirror commented

2025-11-02 07:23:19 -06:00

@noerw commented on GitHub (May 14, 2021):

@silverwind Oh wow thanks for the report, you saved me hours of debugging on my instance, where the same happened the last two days..

@noerw commented on GitHub (May 14, 2021): @silverwind Oh wow thanks for the report, you saved me hours of debugging on my instance, where the same happened the last two days..

GiteaMirror commented

2025-11-02 07:23:20 -06:00

@hubertbanas commented on GitHub (May 15, 2021):

I ran into the same issue some time age. This is what resolved it for me

After=syslog.target network.target nginx.service mariadb.service

@hubertbanas commented on GitHub (May 15, 2021): I ran into the same issue some time age. This is what resolved it for me ``` After=syslog.target network.target nginx.service mariadb.service ```

GiteaMirror commented

2025-11-02 07:23:20 -06:00

@zeripath commented on GitHub (May 15, 2021):

OK so it sounds like my suspicion above (https://github.com/go-gitea/gitea/issues/15866#issuecomment-841104659) is correct and if we change our suggested definition to include After= and Require= we're gonna be fine.

In fact it's notable that the lines below regarding socket activation actually suggest this:

###
# If using socket activation for main http/s
###
#
#After=gitea.main.socket
#Requires=gitea.main.socket
#

@zeripath commented on GitHub (May 15, 2021): OK so it sounds like my suspicion above (https://github.com/go-gitea/gitea/issues/15866#issuecomment-841104659) is correct and if we change our suggested definition to include `After=` and `Require=` we're gonna be fine. --- In fact it's notable that the lines below regarding socket activation actually suggest this: ``` ### # If using socket activation for main http/s ### # #After=gitea.main.socket #Requires=gitea.main.socket # ```

GiteaMirror commented

2025-11-02 07:23:20 -06:00

@lunny commented on GitHub (May 16, 2021):

Reopen per https://github.com/go-gitea/gitea/pull/15881#issuecomment-841651089

@lunny commented on GitHub (May 16, 2021): Reopen per https://github.com/go-gitea/gitea/pull/15881#issuecomment-841651089

GiteaMirror commented

2025-11-02 07:23:20 -06:00

@silverwind commented on GitHub (May 16, 2021):

Doesn't work for me either, simple test:

/etc/systemd/system/unit1.service

[Unit]
Description=unit1
Requires=unit2.service
After=unit2.service

[Service]
ExecStart=ping 127.0.0.1
ExecStop=kill -- $MAINPID
StandardOutput=file:/dev/null

[Install]
WantedBy=multi-user.target

/etc/systemd/system/unit2.service

[Unit]
Description=unit2

[Service]
ExecStart=ping 127.0.0.1
ExecStop=kill -- $MAINPID
StandardOutput=file:/dev/null

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl start unit1 unit2
systemctl stop unit2
systemctl start unit2
systemctl status unit1
# Active: inactive (dead)

Putting Wants= instead of Requires= leads to unit1 never being stopped when unit2 is stopped, better but not ideal.

@silverwind commented on GitHub (May 16, 2021): Doesn't work for me either, simple test: /etc/systemd/system/unit1.service ````ini [Unit] Description=unit1 Requires=unit2.service After=unit2.service [Service] ExecStart=ping 127.0.0.1 ExecStop=kill -- $MAINPID StandardOutput=file:/dev/null [Install] WantedBy=multi-user.target ```` /etc/systemd/system/unit2.service ````ini [Unit] Description=unit2 [Service] ExecStart=ping 127.0.0.1 ExecStop=kill -- $MAINPID StandardOutput=file:/dev/null [Install] WantedBy=multi-user.target ```` ````bash systemctl daemon-reload systemctl start unit1 unit2 systemctl stop unit2 systemctl start unit2 systemctl status unit1 # Active: inactive (dead) ```` Putting `Wants=` instead of `Requires=` leads to unit1 never being stopped when unit2 is stopped, better but not ideal.

GiteaMirror commented

2025-11-02 07:23:20 -06:00

@silverwind commented on GitHub (May 16, 2021):

I think systemd can not provide a solution and we're better off using Wants= which means gitea will not be stopped when the database stops. Assuming gitea can reconnect to the database, this will work fine.

@silverwind commented on GitHub (May 16, 2021): I think systemd can not provide a solution and we're better off using `Wants=` which means gitea will not be stopped when the database stops. Assuming gitea can reconnect to the database, this will work fine.

GiteaMirror commented

2025-11-02 07:23:21 -06:00

@hubertbanas commented on GitHub (May 16, 2021):

Here is my working gitea.service for your references. I'm running Debian 10 Buster

[Unit]
Description=Gitea (Git with a cup of tea)
After=syslog.target network.target nginx.service mariadb.service

[Service]
Restart=always
RestartSec=2s
Type=simple
User=git
Group=git
WorkingDirectory=/var/lib/gitea/
ExecStart=/usr/local/bin/gitea web -c /etc/gitea/app.ini
Environment=USER=git HOME=/home/git GITEA_WORK_DIR=/var/lib/gitea

[Install]
WantedBy=multi-user.target

@hubertbanas commented on GitHub (May 16, 2021): Here is my working ```gitea.service``` for your references. I'm running ```Debian 10 Buster``` ``` [Unit] Description=Gitea (Git with a cup of tea) After=syslog.target network.target nginx.service mariadb.service [Service] Restart=always RestartSec=2s Type=simple User=git Group=git WorkingDirectory=/var/lib/gitea/ ExecStart=/usr/local/bin/gitea web -c /etc/gitea/app.ini Environment=USER=git HOME=/home/git GITEA_WORK_DIR=/var/lib/gitea [Install] WantedBy=multi-user.target ```

GiteaMirror commented

2025-11-02 07:23:21 -06:00

@silverwind commented on GitHub (May 16, 2021):

Yes, it works without Requires= because Requires= is the option that make the service stop but not start back up. I think either After= alone or After= and Wants= are both fine, but we simply can not have Requires= (or BindsTo=) if we want to avoid the service being stopped when the database stops. Quote from Requires= docs:

this unit will be stopped if one of the other units is explicitly stopped

@silverwind commented on GitHub (May 16, 2021): Yes, it works without `Requires=` because `Requires=` is the option that make the service stop but not start back up. I think either `After=` alone or `After=` and `Wants=` are both fine, but we simply can not have `Requires=` (or `BindsTo=`) if we want to avoid the service being stopped when the database stops. Quote from `Requires=` docs: > this unit will be stopped if one of the other units is explicitly stopped

GiteaMirror commented

2025-11-02 07:23:21 -06:00

@silverwind commented on GitHub (May 16, 2021):

Related systemd issue: https://github.com/systemd/systemd/issues/19626

@silverwind commented on GitHub (May 16, 2021): Related systemd issue: https://github.com/systemd/systemd/issues/19626

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/gitea#7337