A Push ends in: ! [remote rejected] lab -> lab (pre-receive hook declined) #1383

Closed
opened 2025-11-02 03:58:46 -06:00 by GiteaMirror · 19 comments
Owner

Originally created by @minoru7 on GitHub (Dec 18, 2017).

  • Gitea version (or commit ref): 1.3.2
  • Git version: 2.11.0
  • Operating system: Debian Stretch (9.3)
  • Database (use [x]):
    • PostgreSQL
    • MySQL (MariaDB, actually)
    • MSSQL
    • SQLite
  • Can you reproduce the bug at https://try.gitea.io:
    • Yes (provide example URL)
    • No
    • Not relevant
  • Log gist: [...io/gitea/cmd/hook.go:115 runHookPreReceive()] [F] retrieve protected branches information failed

Description

I have seen similar issues, but the solutions to those are not fixing the current issue. Example: https://github.com/go-gitea/gitea/issues/1685.

What's strange is that it will fail and the logs will show the above. However, if I restart the gitea service, a push happens correctly, as expected. This will be fine for a while, but somewhere after about 10 minutes or so, I would have to restart the service again to be able to push changes to my repos. This is affecting all of my repos. As I find more information, I'll be glad to post details here. Do you have any suggestions what may be causing this?

I've attached my config file for review (private info redacted). This config worked well with versions in the 1.2 branches.

app.ini.txt

Originally created by @minoru7 on GitHub (Dec 18, 2017). - Gitea version (or commit ref): 1.3.2 - Git version: 2.11.0 - Operating system: Debian Stretch (9.3) - Database (use `[x]`): - [ ] PostgreSQL - [x] MySQL (MariaDB, actually) - [ ] MSSQL - [ ] SQLite - Can you reproduce the bug at https://try.gitea.io: - [ ] Yes (provide example URL) - [ ] No - [x] Not relevant - Log gist: [...io/gitea/cmd/hook.go:115 runHookPreReceive()] [F] retrieve protected branches information failed ## Description I have seen similar issues, but the solutions to those are not fixing the current issue. Example: https://github.com/go-gitea/gitea/issues/1685. What's strange is that it will fail and the logs will show the above. However, if I restart the gitea service, a push happens correctly, as expected. This will be fine for a while, but somewhere after about 10 minutes or so, I would have to restart the service again to be able to push changes to my repos. This is affecting all of my repos. As I find more information, I'll be glad to post details here. Do you have any suggestions what may be causing this? I've attached my config file for review (private info redacted). This config worked well with versions in the 1.2 branches. [app.ini.txt](https://github.com/go-gitea/gitea/files/1568540/app.ini.txt)
GiteaMirror added the type/question label 2025-11-02 03:58:46 -06:00
Author
Owner

@minoru7 commented on GitHub (Dec 18, 2017):

I have also found the following in the service logs, which may be relevant:
Completed /<my-path-to-repo.git>/info/refs?service=git-receive-pack 401 Unauthorized in 4.739506ms
Completed /<my-path-to-repo.git>/info/refs?service=git-upload-pack 401 Unauthorized in 2.956383ms

It does, however, show a "200 ok" later for each of these items. Secondly, if we were having a rights issue, why would a restart of the service fix it?

Still looking for more details to provide. Thank you for any help that could be given.

@minoru7 commented on GitHub (Dec 18, 2017): I have also found the following in the service logs, which may be relevant: Completed /<my-path-to-repo.git>/info/refs?service=git-receive-pack 401 Unauthorized in 4.739506ms Completed /<my-path-to-repo.git>/info/refs?service=git-upload-pack 401 Unauthorized in 2.956383ms It does, however, show a "200 ok" later for each of these items. Secondly, if we were having a rights issue, why would a restart of the service fix it? Still looking for more details to provide. Thank you for any help that could be given.
Author
Owner

@minoru7 commented on GitHub (Dec 18, 2017):

And I've found this:

Dec 18 00:09:53 git01.mydomain.net gitea[21940]: [Macaron] 2017-12-18 00:09:53: Started GET / for <myIP_redacted>
Dec 18 00:09:53 git01.mydomain.net gitea[21940]: [Macaron] 2017-12-18 00:09:53: Completed / 302 Found in 681.322µs
Dec 18 00:12:58 git01.mydomain.net gitea[21940]: [mysql] 2017/12/18 00:12:58 packets.go:33: unexpected EOF

@minoru7 commented on GitHub (Dec 18, 2017): And I've found this: Dec 18 00:09:53 git01.mydomain.net gitea[21940]: [Macaron] 2017-12-18 00:09:53: Started GET / for <myIP_redacted> Dec 18 00:09:53 git01.mydomain.net gitea[21940]: [Macaron] 2017-12-18 00:09:53: Completed / 302 Found in 681.322µs Dec 18 00:12:58 git01.mydomain.net gitea[21940]: [mysql] 2017/12/18 00:12:58 packets.go:33: unexpected EOF
Author
Owner

@lunny commented on GitHub (Dec 19, 2017):

@minoru7 that means your config LOCAL_ROOT_URL is not corrected.

; Local (DMZ) URL for Gitea workers (such as SSH update) accessing web service.
; In most cases you do not need to change the default value.
; Alter it only if your SSH server node is not the same as HTTP node.
LOCAL_ROOT_URL = %(PROTOCOL)s://%(HTTP_ADDR)s:%(HTTP_PORT)s/
@lunny commented on GitHub (Dec 19, 2017): @minoru7 that means your config `LOCAL_ROOT_URL` is not corrected. ```ini ; Local (DMZ) URL for Gitea workers (such as SSH update) accessing web service. ; In most cases you do not need to change the default value. ; Alter it only if your SSH server node is not the same as HTTP node. LOCAL_ROOT_URL = %(PROTOCOL)s://%(HTTP_ADDR)s:%(HTTP_PORT)s/ ```
Author
Owner

@minoru7 commented on GitHub (Dec 19, 2017):

@lunny I'm confused. Why does it work for a while after I restart the service? Also, we're doing a proxy per what that comment says:

    ProxyPreserveHost On
    ProxyRequests off
    ProxyPass / http://localhost:3000/
    ProxyPassReverse / http://localhost:3000/

This worked with prior versions. What has changed? Thank you for your assistance.

@minoru7 commented on GitHub (Dec 19, 2017): @lunny I'm confused. Why does it work for a while after I restart the service? Also, we're doing a proxy per what that comment says: ProxyPreserveHost On ProxyRequests off ProxyPass / http://localhost:3000/ ProxyPassReverse / http://localhost:3000/ This worked with prior versions. What has changed? Thank you for your assistance.
Author
Owner

@lafriks commented on GitHub (Dec 19, 2017):

@minoru7 because it was changed to use http internal API for operations that change DB from ssh to have less problems with sqlite database locking

@lafriks commented on GitHub (Dec 19, 2017): @minoru7 because it was changed to use http internal API for operations that change DB from ssh to have less problems with sqlite database locking
Author
Owner

@minoru7 commented on GitHub (Dec 19, 2017):

Ah, ok. Was that done after version 1.2.3?

So we're unable to use the same proxying we were doing before?

@minoru7 commented on GitHub (Dec 19, 2017): Ah, ok. Was that done after version 1.2.3? So we're unable to use the same proxying we were doing before?
Author
Owner

@lafriks commented on GitHub (Dec 19, 2017):

You can do, you just need to set LOCAL_ROOT_URL so that ssh server can connect to gitea address, usually that is http://localhost....

@lafriks commented on GitHub (Dec 19, 2017): You can do, you just need to set `LOCAL_ROOT_URL` so that ssh server can connect to gitea address, usually that is http://localhost....
Author
Owner

@minoru7 commented on GitHub (Dec 19, 2017):

I guess I was under the impression that the following were variables pulling from other values:
LOCAL_ROOT_URL = %(PROTOCOL)s://%(HTTP_ADDR)s:%(HTTP_PORT)s/

I thought I could just place PROTOCOL=http, HTTP_ADDR=localhost, and HTTP_PORT=3000 and then that string (above) would take those values and fill in accordingly. Kind of like what happens with apache configs. That's not the case?

@minoru7 commented on GitHub (Dec 19, 2017): I guess I was under the impression that the following were variables pulling from other values: LOCAL_ROOT_URL = %(PROTOCOL)s://%(HTTP_ADDR)s:%(HTTP_PORT)s/ I thought I could just place PROTOCOL=http, HTTP_ADDR=localhost, and HTTP_PORT=3000 and then that string (above) would take those values and fill in accordingly. Kind of like what happens with apache configs. That's not the case?
Author
Owner

@lafriks commented on GitHub (Dec 19, 2017):

In such case yes that should be so

@lafriks commented on GitHub (Dec 19, 2017): In such case yes that should be so
Author
Owner

@minoru7 commented on GitHub (Dec 19, 2017):

Ok, in that case, if you look at my config file that I attached, the values are there and should turn up as LOCAL_ROOT_URL = http://localhost:3000. Does it need to be the external address listed in ROOT_URL, or is having it "localhost" sufficient? All functions for the Gitea server are local to this server (ssh, http(s), etc.).

@minoru7 commented on GitHub (Dec 19, 2017): Ok, in that case, if you look at my config file that I attached, the values are there and should turn up as LOCAL_ROOT_URL = http://localhost:3000. Does it need to be the external address listed in ROOT_URL, or is having it "localhost" sufficient? All functions for the Gitea server are local to this server (ssh, http(s), etc.).
Author
Owner

@minoru7 commented on GitHub (Dec 19, 2017):

I think I may have figured this out. We use Puppet to lay out our configurations. I noticed that every time the Puppet agent ran, it was breaking it. So then I also noticed that after I restarted the gitea service, a change was made to the /opt/gitea/custom/conf/app.ini that was removing INTERNAL_TOKEN = <some_number> under the [security] heading.

It looks like Gitea places that into the config when it starts up? And then Puppet is removing it after the fact. What is this item? And was this added after version 1.2.3?

@minoru7 commented on GitHub (Dec 19, 2017): I think I may have figured this out. We use Puppet to lay out our configurations. I noticed that every time the Puppet agent ran, it was breaking it. So then I also noticed that after I restarted the gitea service, a change was made to the /opt/gitea/custom/conf/app.ini that was removing INTERNAL_TOKEN = <some_number> under the [security] heading. It looks like Gitea places that into the config when it starts up? And then Puppet is removing it after the fact. What is this item? And was this added after version 1.2.3?
Author
Owner

@lunny commented on GitHub (Dec 20, 2017):

@minoru7 I think it should be. The INTERNAL_TOKEN config was added in v1.2.x. Gitea web and Gitea serve/hook will share the token so that that should not be changed after Gitea web running.

@lunny commented on GitHub (Dec 20, 2017): @minoru7 I think it should be. The `INTERNAL_TOKEN` config was added in v1.2.x. `Gitea web` and `Gitea serve/hook` will share the token so that that should not be changed after `Gitea web` running.
Author
Owner

@minoru7 commented on GitHub (Dec 20, 2017):

I never saw this in version 1.2.3, which was our previous version. We're now running 1.3.2. Is the removal of the INTERNAL_TOKEN by Puppet causing the error I listed above, you think?

@minoru7 commented on GitHub (Dec 20, 2017): I never saw this in version 1.2.3, which was our previous version. We're now running 1.3.2. Is the removal of the INTERNAL_TOKEN by Puppet causing the error I listed above, you think?
Author
Owner

@lunny commented on GitHub (Dec 20, 2017):

@minoru7 I think yes.

@lunny commented on GitHub (Dec 20, 2017): @minoru7 I think yes.
Author
Owner

@minoru7 commented on GitHub (Dec 20, 2017):

Well, that's a bit unfortunate. If the token is being used by the running of the system, it'd be better to have that controlled elsewhere rather than in the config file that is modified with my personal changes. I'm going to have to figure out another way of laying down the config using Puppet.

Thank you for your help in figuring this issue out. You guys do a great job!

@minoru7 commented on GitHub (Dec 20, 2017): Well, that's a bit unfortunate. If the token is being used by the running of the system, it'd be better to have that controlled elsewhere rather than in the config file that is modified with my personal changes. I'm going to have to figure out another way of laying down the config using Puppet. Thank you for your help in figuring this issue out. You guys do a great job!
Author
Owner

@minoru7 commented on GitHub (Dec 20, 2017):

I have the older server still up where I can compare the configs. I was wrong, I see the INTERNAL_TOKEN being removed on the old server too. So that's not what is causing the issue then. Back to the drawing board...

However, the main point that I see is that it breaks right after a Puppet run. So I'm unsure what it may be doing to Gitea. It removes that item and then tries to initiate a reload of the service, I think. Maybe the issue lies somewhere within that "reload".

@minoru7 commented on GitHub (Dec 20, 2017): I have the older server still up where I can compare the configs. I was wrong, I see the INTERNAL_TOKEN being removed on the old server too. So that's not what is causing the issue then. Back to the drawing board... However, the main point that I see is that it breaks right after a Puppet run. So I'm unsure what it may be doing to Gitea. It removes that item and then tries to initiate a reload of the service, I think. Maybe the issue lies somewhere within that "reload".
Author
Owner

@minoru7 commented on GitHub (Dec 20, 2017):

@lunny or @lafriks,

FYI. I have made my file resource in Puppet only apply if the app.ini doesn't exist. It's unfortunate to have to do this since now any changes to app.ini in Puppet will have to be replaced manually on the git servers. However, since doing this, whatever was happening with Gitea has been fixed. I don't understand why the removal of INTERNAL_TOKEN (or the reload of the service) caused issues with 1.3.2 but not 1.2.3, but I at least have a fix in place for now.

If you think it would be possible to move the INTERNAL_TOKEN feature outside of the app.ini, I will be glad to put in a feature request. Honestly, a security item such as this, that is internal to the running of the application should probably be placed elsewhere (a lockfile, pid, or a separate config file for the system to edit, etc.). Just my two cents though. ;-)

@minoru7 commented on GitHub (Dec 20, 2017): @lunny or @lafriks, FYI. I have made my file resource in Puppet only apply if the app.ini doesn't exist. It's unfortunate to have to do this since now any changes to app.ini in Puppet will have to be replaced manually on the git servers. However, since doing this, whatever was happening with Gitea has been fixed. I don't understand why the removal of INTERNAL_TOKEN (or the reload of the service) caused issues with 1.3.2 but not 1.2.3, but I at least have a fix in place for now. If you think it would be possible to move the INTERNAL_TOKEN feature outside of the app.ini, I will be glad to put in a feature request. Honestly, a security item such as this, that is internal to the running of the application should probably be placed elsewhere (a lockfile, pid, or a separate config file for the system to edit, etc.). Just my two cents though. ;-)
Author
Owner

@lunny commented on GitHub (Dec 20, 2017):

@minoru7 You are welcome to send a PR to do that. The INTERNAL_TOKEN will be shared on gitea web process and gitea serv process when SSH is called. The variable will be read or created when gitea web start and saved on memory. When a git pull or push via SSH, gitea serv/ gitea hook will be called and will send a HTTP request with the token to HTTP service started on gitea web.

@lunny commented on GitHub (Dec 20, 2017): @minoru7 You are welcome to send a PR to do that. The INTERNAL_TOKEN will be shared on `gitea web` process and `gitea serv` process when SSH is called. The variable will be read or created when `gitea web` start and saved on memory. When a git pull or push via SSH, `gitea serv`/ `gitea hook` will be called and will send a HTTP request with the token to HTTP service started on `gitea web`.
Author
Owner

@minoru7 commented on GitHub (Dec 20, 2017):

@lunny, thank you for your help. It's greatly appreciated.

@minoru7 commented on GitHub (Dec 20, 2017): @lunny, thank you for your help. It's greatly appreciated.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/gitea#1383