Push fails with "retrieve protected branches information failed: invalid character '<' looking for beginning of value" #6318

New Issue

GiteaMirror · 2025-11-02T06:52:26-06:00

GiteaMirror commented

2025-11-02 06:52:26 -06:00

Originally created by @joepie91 on GitHub (Nov 15, 2020).

Gitea version (or commit ref): 1.12.5
Git version: 2.25.4
Operating system: NixOS 20.09 - installed from distro package
Database (use [x]):
- PostgreSQL
- MySQL
- MSSQL
- SQLite
Can you reproduce the bug at https://try.gitea.io:
- Yes (provide example URL)
- No
Log gist: https://gist.github.com/joepie91/54a7a140c6a7e0ea60732f3a81c12657

Description

When pushing to one of my existing repositories, I get the following output:

$ git push
Enumerating objects: 59, done.
Counting objects: 100% (59/59), done.
Delta compression using up to 24 threads
Compressing objects: 100% (43/43), done.
Writing objects: 100% (44/44), 17.25 KiB | 8.62 MiB/s, done.
Total 44 (delta 25), reused 0 (delta 0)
remote: Gitea: Internal error
remote: retrieve protected branches information failed: invalid character '<' looking for beginning of value
To git.cryto.net:joepie91/morph-rc.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'git@git.cryto.net:joepie91/morph-rc.git'

Pushing to a newly-created test repository, however, works fine.

Based on trawling through other issues for several hours, I have done/verified the following:

Regenerated authorized_keys, verified that the binary path in there matches that of the running Gitea instance
Verified that app.ini contains a static INTERNAL_TOKEN
Set LOCAL_ROOT_URL to http://localhost:3000/
Restarted the entire server

This instance of Gitea started as a Gogs instance, and was recently upgraded to Gitea 1.0.2 -> 1.6.4 -> 1.11.8 -> 1.12.5. The failing repository dates from before the move to Gitea.

I have not tried doctor recreate-tables as this command does not yet exist in 1.12.x, and I don't know how safe it is to run an RC (and presumably there's no way to downgrade back to 1.12.x if the RC breaks, due to database migrations?).

Weirdly, the entire error message in the git push output does not seem to exist in the Gitea source code since 1.9.x, but I have double-, triple- and quadruple-checked that I really am running 1.12.x. I'm not really sure what's going on there...

Screenshots

N/A

Originally created by @joepie91 on GitHub (Nov 15, 2020).   - Gitea version (or commit ref): 1.12.5 - Git version: 2.25.4 - Operating system: NixOS 20.09 - installed from distro package    - Database (use `[x]`): - [X] PostgreSQL - [ ] MySQL - [ ] MSSQL - [ ] SQLite - Can you reproduce the bug at https://try.gitea.io: - [ ] Yes (provide example URL) - [x] No - Log gist: https://gist.github.com/joepie91/54a7a140c6a7e0ea60732f3a81c12657    ## Description When pushing to one of my existing repositories, I get the following output: ``` $ git push Enumerating objects: 59, done. Counting objects: 100% (59/59), done. Delta compression using up to 24 threads Compressing objects: 100% (43/43), done. Writing objects: 100% (44/44), 17.25 KiB | 8.62 MiB/s, done. Total 44 (delta 25), reused 0 (delta 0) remote: Gitea: Internal error remote: retrieve protected branches information failed: invalid character '<' looking for beginning of value To git.cryto.net:joepie91/morph-rc.git ! [remote rejected] master -> master (pre-receive hook declined) error: failed to push some refs to 'git@git.cryto.net:joepie91/morph-rc.git' ``` Pushing to a newly-created test repository, however, works fine. Based on trawling through other issues for several hours, I have done/verified the following: - Regenerated `authorized_keys`, verified that the binary path in there matches that of the running Gitea instance - Verified that app.ini contains a static `INTERNAL_TOKEN` - Set `LOCAL_ROOT_URL` to `http://localhost:3000/` - Restarted the entire server This instance of Gitea started as a Gogs instance, and was recently upgraded to Gitea 1.0.2 -> 1.6.4 -> 1.11.8 -> 1.12.5. The failing repository dates from before the move to Gitea. I have not tried `doctor recreate-tables` as this command does not yet exist in `1.12.x`, and I don't know how safe it is to run an RC (and presumably there's no way to downgrade back to `1.12.x` if the RC breaks, due to database migrations?). Weirdly, the entire error message in the `git push` output does not seem to exist in the Gitea source code since `1.9.x`, but I have double-, triple- and quadruple-checked that I really *am* running `1.12.x`. I'm not really sure what's going on there... ## Screenshots N/A

GiteaMirror closed this issue

2025-11-02 06:52:26 -06:00

GiteaMirror commented

2025-11-02 06:52:27 -06:00

@zeripath commented on GitHub (Nov 15, 2020):

In adminstration pages run resynchronize hooks

@zeripath commented on GitHub (Nov 15, 2020): In adminstration pages run resynchronize hooks

GiteaMirror commented

2025-11-02 06:52:27 -06:00

@zeripath commented on GitHub (Nov 15, 2020):

Nov 15 23:51:13 machine-konjassiem-02 gitea[25273]: [Macaron] 2020-11-15 23:51:13: Completed GET /api/internal/branch/332/master 404 Not Found in 5.366685ms
Nov 15 23:51:13 machine-konjassiem-02 gitea[25273]: [Macaron] 2020-11-15 23:51:13: Started POST /api/internal/ssh/1/update/332 for 127.0.0.1

These are old endpoints that have not been present since 1.9 and imply that the hooks in Gitea's git repositories are out of date

@zeripath commented on GitHub (Nov 15, 2020): ``` Nov 15 23:51:13 machine-konjassiem-02 gitea[25273]: [Macaron] 2020-11-15 23:51:13: Completed GET /api/internal/branch/332/master 404 Not Found in 5.366685ms Nov 15 23:51:13 machine-konjassiem-02 gitea[25273]: [Macaron] 2020-11-15 23:51:13: Started POST /api/internal/ssh/1/update/332 for 127.0.0.1 ``` These are old endpoints that have not been present since 1.9 and imply that the hooks in Gitea's git repositories are out of date

GiteaMirror commented

2025-11-02 06:52:27 -06:00

@joepie91 commented on GitHub (Nov 15, 2020):

Ah, that indeed fixed it, thanks!

Shouldn't this have been done automatically as a part of the upgrade migration process, though?

@joepie91 commented on GitHub (Nov 15, 2020): Ah, that indeed fixed it, thanks! Shouldn't this have been done automatically as a part of the upgrade migration process, though?

GiteaMirror commented

2025-11-02 06:52:27 -06:00

@zeripath commented on GitHub (Nov 15, 2020):

Yes iirc there is a migration that should have done this.

Did you ever step forward to master and then step back?

@zeripath commented on GitHub (Nov 15, 2020): Yes iirc there is a migration that should have done this. Did you ever step forward to master and then step back?

GiteaMirror commented

2025-11-02 06:52:27 -06:00

@joepie91 commented on GitHub (Nov 15, 2020):

I'm not sure I understand what you mean. If the question is whether I've ever run Gitea itself from master, then no - I only upgraded through stable releases (1.0.2 -> 1.6.4 -> 1.11.8 as static binaries, then 1.11.8 from distro package, then 1.12.5 from newer distro package).

Or are you talking about master in the repository that was failing to push?

@joepie91 commented on GitHub (Nov 15, 2020): I'm not sure I understand what you mean. If the question is whether I've ever run Gitea itself from `master`, then no - I only upgraded through stable releases (1.0.2 -> 1.6.4 -> 1.11.8 as static binaries, then 1.11.8 from distro package, then 1.12.5 from newer distro package). Or are you talking about `master` in the repository that was failing to push?

GiteaMirror commented

2025-11-02 06:52:27 -06:00

@zeripath commented on GitHub (Nov 15, 2020):

Hmm can't find the migration actually. I'm not certain what originally happened when I made that particular change but the hooks were automatically regenerated - perhaps that relied on something that has since been refactored away in an efficiency drive.

I guess it's just one of those things, it's very easy to fix and the error should be distinctive enough to search for so I'm not certain that we should actually do anything in code to change it. If you fancy you can add a docs pr to add it as advice to the faq or some where else appropriate.

@zeripath commented on GitHub (Nov 15, 2020): Hmm can't find the migration actually. I'm not certain what originally happened when I made that particular change but the hooks were automatically regenerated - perhaps that relied on something that has since been refactored away in an efficiency drive. I guess it's just one of those things, it's very easy to fix and the error should be distinctive enough to search for so I'm not certain that we should actually do anything in code to change it. If you fancy you can add a docs pr to add it as advice to the faq or some where else appropriate.

GiteaMirror commented

2025-11-02 06:52:28 -06:00

@joepie91 commented on GitHub (Nov 15, 2020):

Ah, I think I know why I ran into this whereas others don't - I think there was never a hook-rewrite migration to begin with, but for most people it works because they replace the binary in-place (at /usr/bin/gitea or something). But in my case, this is what the pre-receive hook looked like:

#!/usr/bin/env bash
"/var/lib/gitea/gitea-1.6.4-linux-amd64" hook --config='/var/lib/gitea/custom/conf/app.ini' pre-receive

... which is a versioned binary (put there for the multi-step Gogs -> Gitea upgrade process), and therefore the old version kept being called instead of the current version. This also explains the error message that shouldn't have existed anymore :)

So yeah, this would probably reoccur in any case where someone doesn't upgrade the binary in-place, with varying errors and failure modes. I just checked, and the NixOS module for Gitea actually has some code to deal with this exact issue (as it does atomic upgrades, and so the binary path changes with each upgrade).

Maybe it would make sense to just re-generate all the hooks on each start of Gitea? That would cover all of these cases in one go.

@joepie91 commented on GitHub (Nov 15, 2020): Ah, I think I know why I ran into this whereas others don't - I think there was never a hook-rewrite migration to begin with, but for most people it works because they replace the binary in-place (at `/usr/bin/gitea` or something). But in my case, this is what the `pre-receive` hook looked like: ``` #!/usr/bin/env bash "/var/lib/gitea/gitea-1.6.4-linux-amd64" hook --config='/var/lib/gitea/custom/conf/app.ini' pre-receive ``` ... which is a versioned binary (put there for the multi-step Gogs -> Gitea upgrade process), and therefore the old version kept being called instead of the current version. This also explains the error message that shouldn't have existed anymore :) So yeah, this would probably reoccur in any case where someone doesn't upgrade the binary in-place, with varying errors and failure modes. I just checked, and the NixOS module for Gitea actually [has some code](https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/misc/gitea.nix#L491-L499) to deal with this exact issue (as it does atomic upgrades, and so the binary path changes with each upgrade). Maybe it would make sense to just re-generate all the hooks on each start of Gitea? That would cover all of these cases in one go.

GiteaMirror commented

2025-11-02 06:52:28 -06:00

@lunny commented on GitHub (Nov 16, 2020):

We have a ./gitea doctor to help to find these problems.

@lunny commented on GitHub (Nov 16, 2020): We have a `./gitea doctor` to help to find these problems.

GiteaMirror commented

2025-11-02 06:52:28 -06:00

@joepie91 commented on GitHub (Nov 16, 2020):

I actually tried gitea doctor first (but forgot to mention that in my original issue), and it did not detect this issue.

Aside from that, it would be more practical to just have Gitea do this on every start, as this would break in different ways depending on the exact upgrade path - and so it won't be obvious what's wrong when it does. Automatically dealing with this situation as it occurs would be a much better UX than having to run doctor all the time after an upgrade, whether it was an automatic or a manual upgrade.

The code for this regeneration is already there, it seems to be a fast process, and as I understand it you already shouldn't be manually modifying the auto-generated hooks; so I don't see a downside to just making it a startup operation.

@joepie91 commented on GitHub (Nov 16, 2020): I actually tried `gitea doctor` first (but forgot to mention that in my original issue), and it did not detect this issue. Aside from that, it would be more practical to just have Gitea do this on every start, as this would break in different ways depending on the exact upgrade path - and so it won't be obvious what's wrong when it does. Automatically dealing with this situation as it occurs would be a much better UX than having to run `doctor` all the time after an upgrade, whether it was an automatic or a manual upgrade. The code for this regeneration is already there, it seems to be a fast process, and as I understand it you *already* shouldn't be manually modifying the auto-generated hooks; so I don't see a downside to just making it a startup operation.

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/gitea#6318