SSH Keys Not Accepted After Migration #12555

Closed
opened 2025-11-02 10:13:55 -06:00 by GiteaMirror · 14 comments
Owner

Originally created by @inferenceus on GitHub (Feb 28, 2024).

Description

Firstly, I'd like to state that I've attempted fixes for hours for SSH at this point since I thought it was related to that, and it may be; however, I have changed nothing in relation to clients, and have checked multiple times across everything to ensure client SSH configurations and keys etc are correct. The clients' keys are also added to ssh-agent as they were before the migration. I have also deleted and added the same keys back to the clients via the web UI, and verified the keys.

Whenever I attempt to either push to or clone repositories from the new server after migrating the Gitea configurations, database, and /var/lib/gitea/ directory, I receive the following error:

Received disconnect from 10.0.0.20 port 22:2: Too many authentication failures
Disconnected from 10.0.0.20 port 22
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

This occurs to multiple clients. I have checked the /var/lib/gitea/.ssh/authorized_keys file and everything seems correct in there, too. I'm unsure what to do at this point as everything seems fine, yet it's still not accepted. To clarify, the repositories do exist and I can log into the Gitea web UI as the clients which are failing via SSH to view them there; all issues are also intact.

Gitea Version

1.21.7

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

2.43.0

Operating System

Gentoo Linux

How are you running Gitea?

Compiled from source via Gentoo Linux's package manager.

Database

PostgreSQL

Originally created by @inferenceus on GitHub (Feb 28, 2024). ### Description Firstly, I'd like to state that I've attempted fixes for hours for SSH at this point since I thought it was related to that, and it may be; however, I have changed nothing in relation to clients, and have checked multiple times across everything to ensure client SSH configurations and keys etc are correct. The clients' keys are also added to ssh-agent as they were before the migration. I have also deleted and added the same keys back to the clients via the web UI, and verified the keys. Whenever I attempt to either push to or clone repositories from the new server after migrating the Gitea configurations, database, and `/var/lib/gitea/` directory, I receive the following error: ``` Received disconnect from 10.0.0.20 port 22:2: Too many authentication failures Disconnected from 10.0.0.20 port 22 fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. ``` This occurs to multiple clients. I have checked the `/var/lib/gitea/.ssh/authorized_keys` file and everything seems correct in there, too. I'm unsure what to do at this point as everything seems fine, yet it's still not accepted. To clarify, the repositories do exist and I can log into the Gitea web UI as the clients which are failing via SSH to view them there; all issues are also intact. ### Gitea Version 1.21.7 ### Can you reproduce the bug on the Gitea demo site? No ### Log Gist _No response_ ### Screenshots _No response_ ### Git Version 2.43.0 ### Operating System Gentoo Linux ### How are you running Gitea? Compiled from source via Gentoo Linux's package manager. ### Database PostgreSQL
GiteaMirror added the type/bug label 2025-11-02 10:13:56 -06:00
Author
Owner

@inferenceus commented on GitHub (Feb 28, 2024):

As an update to this issue, I found a small issue with the /var/lib/gitea/.ssh/ directory; the permissions were set to 750 rather than 700. After setting the permissions correctly, it still fails, but I get a different error:

git@src.inferencium.net: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

The /var/lib/gitea/.ssh/authorized_keys file is set to 600.

@inferenceus commented on GitHub (Feb 28, 2024): As an update to this issue, I found a small issue with the `/var/lib/gitea/.ssh/` directory; the permissions were set to `750` rather than `700`. After setting the permissions correctly, it still fails, but I get a different error: ``` git@src.inferencium.net: Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. ``` The `/var/lib/gitea/.ssh/authorized_keys` file is set to `600`.
Author
Owner

@jolheiser commented on GitHub (Feb 28, 2024):

Do you have logs or a config you can redact and upload?
As well, can you attempt to SSH into git@src.inferencium.net directly? You should get a message

Hi there, [user]! You've successfully authenticated with the key named [key], but Gitea does not provide shell access.
If this is unexpected, please log in with password and setup Gitea under another user.

@jolheiser commented on GitHub (Feb 28, 2024): Do you have logs or a config you can redact and upload? As well, can you attempt to SSH into `git@src.inferencium.net` directly? You should get a message > Hi there, [user]! You've successfully authenticated with the key named [key], but Gitea does not provide shell access. If this is unexpected, please log in with password and setup Gitea under another user.
Author
Owner

@inferenceus commented on GitHub (Feb 28, 2024):

SSH debug information via GIT_SSH_COMMAND="ssh -vvv" git push:

debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey
debug3: authmethod_lookup publickey
debug3: remaining preferred:
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug3: ssh_get_authentication_socket_path: path '/tmp/ssh-XXXXXXgjpkJm/agent.13130'
debug1: get_agent_identities: bound agent to hostkey
debug1: get_agent_identities: agent returned 1 keys
debug1: Will attempt key: /home/a00-sys/.ssh/ssh-inf-a00-sys-ed25519 ED25519 SHA256:FtEVfx1CmTKMy40VwZvF4k+3TC+QhCWy+EmPRg50Nnc explicit agent
debug2: pubkey_prepare: done
debug1: Offering public key: /home/a00-sys/.ssh/ssh-inf-a00-sys-ed25519 ED25519 SHA256:FtEVfx1CmTKMy40VwZvF4k+3TC+QhCWy+EmPRg50Nnc explicit agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
git@src.inferencium.net: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
@inferenceus commented on GitHub (Feb 28, 2024): SSH debug information via `GIT_SSH_COMMAND="ssh -vvv" git push`: ``` debug1: Authentications that can continue: publickey debug3: start over, passed a different list publickey debug3: preferred publickey debug3: authmethod_lookup publickey debug3: remaining preferred: debug3: authmethod_is_enabled publickey debug1: Next authentication method: publickey debug3: ssh_get_authentication_socket_path: path '/tmp/ssh-XXXXXXgjpkJm/agent.13130' debug1: get_agent_identities: bound agent to hostkey debug1: get_agent_identities: agent returned 1 keys debug1: Will attempt key: /home/a00-sys/.ssh/ssh-inf-a00-sys-ed25519 ED25519 SHA256:FtEVfx1CmTKMy40VwZvF4k+3TC+QhCWy+EmPRg50Nnc explicit agent debug2: pubkey_prepare: done debug1: Offering public key: /home/a00-sys/.ssh/ssh-inf-a00-sys-ed25519 ED25519 SHA256:FtEVfx1CmTKMy40VwZvF4k+3TC+QhCWy+EmPRg50Nnc explicit agent debug3: send packet: type 50 debug2: we sent a publickey packet, wait for reply debug3: receive packet: type 51 debug1: Authentications that can continue: publickey debug2: we did not send a packet, disable method debug1: No more authentication methods to try. git@src.inferencium.net: Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. ```
Author
Owner

@inferenceus commented on GitHub (Feb 28, 2024):

@jolheiser

As well, can you attempt to SSH into git@src.inferencium.net directly? You should get a message

user@hostname: % ssh git@src.inferencium.net
git@src.inferencium.net: Permission denied (publickey).
@inferenceus commented on GitHub (Feb 28, 2024): @jolheiser > As well, can you attempt to SSH into git@src.inferencium.net directly? You should get a message ``` user@hostname: % ssh git@src.inferencium.net git@src.inferencium.net: Permission denied (publickey). ```
Author
Owner

@jolheiser commented on GitHub (Feb 28, 2024):

That tells me you aren't quite getting to Gitea, it must be somewhere in between the openssh side of things.

Some common issues/fixes are documented at https://docs.gitea.com/next/help/faq#ssh-issues if you hadn't come across that page yet.

@jolheiser commented on GitHub (Feb 28, 2024): That tells me you aren't quite getting to Gitea, it must be somewhere in between the openssh side of things. Some common issues/fixes are documented at https://docs.gitea.com/next/help/faq#ssh-issues if you hadn't come across that page yet.
Author
Owner

@inferenceus commented on GitHub (Feb 28, 2024):

@jolheiser

That tells me you aren't quite getting to Gitea, it must be somewhere in between the openssh side of things.

There's actually been 2 migrations I've performed today (public server and internal server) and both systems are behaving the same way; they're even on different subnets (internal is 10.0.0.0, public is 10.1.0.0). What could potentially have changed between the migrations? The clients and servers have identical SSH configurations as before the migrations occurred. Connecting via SSH as root works, so it seems to be something related to the git user, no?

@inferenceus commented on GitHub (Feb 28, 2024): @jolheiser > That tells me you aren't quite getting to Gitea, it must be somewhere in between the openssh side of things. There's actually been 2 migrations I've performed today (public server and internal server) and both systems are behaving the same way; they're even on different subnets (internal is `10.0.0.0`, public is `10.1.0.0`). What could potentially have changed between the migrations? The clients and servers have identical SSH configurations as before the migrations occurred. Connecting via SSH as `root` works, so it seems to be something related to the `git` user, no?
Author
Owner

@jolheiser commented on GitHub (Feb 28, 2024):

I'd be curious to know if it's related to your issue earlier today.
Is the environment set up such that it's finding the correct .ssh directory?

It's interesting that attempting to access a repo gives an error indicative of Gitea, but direct ssh isn't giving the correct error about lack of shell. Did the docs page give any other ideas to try or logs with more info?

@jolheiser commented on GitHub (Feb 28, 2024): I'd be curious to know if it's related to your issue earlier today. Is the environment set up such that it's finding the correct .ssh directory? It's interesting that attempting to access a repo gives an error indicative of Gitea, but direct ssh isn't giving the correct error about lack of shell. Did the docs page give any other ideas to try or logs with more info?
Author
Owner

@inferenceus commented on GitHub (Feb 28, 2024):

@jolheiser

I'd be curious to know if it's related to your issue earlier today.

Potentially. While that issue was resolved, perhaps there's a configuration file line I need to change along with the previous one? I haven't touched the SSH section until earlier today since everything just used to work. I will also say that I didn't have this issue when I migrated from Gitea to Forgejo last year. I'm not saying this is a Gitea issue, because I don't think it is; they even share the same core codebase as of their version 1.21.5-0. This is a migration from Forgejo back to Gitea. I'm trying to think of what didn't happen last migration which did happen this time, or what did happen last time which hasn't happened this time.

Is the environment set up such that it's finding the correct .ssh directory?

If you refer to the previous issue I had regarding SSH, which was resolved, I set that to be /var/lib/gitea/.ssh/ and I got Gitea running other than this current issue.

It's interesting that attempting to access a repo gives an error indicative of Gitea, but direct ssh isn't giving the correct error about lack of shell.

And I can SSH as root, which makes me think it's related to Gitea and/or the git user.

Did the docs page give any other ideas to try or logs with more info?

Nothing on the docs page gave me any indication of what could be wrong. I'm happy to provide logs; what would you need? Router log, process log, error log (which was empty last time I checked)?

@inferenceus commented on GitHub (Feb 28, 2024): @jolheiser > I'd be curious to know if it's related to your issue earlier today. Potentially. While that issue was resolved, perhaps there's a configuration file line I need to change along with the previous one? I haven't touched the SSH section until earlier today since everything just used to work. I will also say that I didn't have this issue when I migrated from Gitea to Forgejo last year. I'm not saying this is a Gitea issue, because I don't think it is; they even share the same core codebase as of their version 1.21.5-0. This is a migration from Forgejo back to Gitea. I'm trying to think of what didn't happen last migration which did happen this time, or what did happen last time which hasn't happened this time. > Is the environment set up such that it's finding the correct .ssh directory? If you refer to the previous issue I had regarding SSH, which was resolved, I set that to be `/var/lib/gitea/.ssh/` and I got Gitea running other than this current issue. > It's interesting that attempting to access a repo gives an error indicative of Gitea, but direct ssh isn't giving the correct error about lack of shell. And I can SSH as `root`, which makes me think it's related to Gitea and/or the `git` user. > Did the docs page give any other ideas to try or logs with more info? Nothing on the docs page gave me any indication of what could be wrong. I'm happy to provide logs; what would you need? Router log, process log, error log (which was empty last time I checked)?
Author
Owner

@inferenceus commented on GitHub (Feb 28, 2024):

@jolheiser

I've just looked through /etc/passwd as it's the only place I thought of which I haven't looked today. git has a shell (/bin/sh, as usual), but the home directory was set to /var/lib/forgejo/. Pushing now works. This isn't a bug, it was a misconfiguration due to that single file I missed during the migration.

Thanks for the very responsive assistance as always.

@inferenceus commented on GitHub (Feb 28, 2024): @jolheiser I've just looked through `/etc/passwd` as it's the only place I thought of which I haven't looked today. `git` has a shell (`/bin/sh`, as usual), but the home directory was set to `/var/lib/forgejo/`. Pushing now works. This isn't a bug, it was a misconfiguration due to that single file I missed during the migration. Thanks for the very responsive assistance as always.
Author
Owner

@jolheiser commented on GitHub (Feb 28, 2024):

Aha, good to know for reference. Happy to help!

@jolheiser commented on GitHub (Feb 28, 2024): Aha, good to know for reference. Happy to help!
Author
Owner

@inferenceus commented on GitHub (Feb 28, 2024):

@jolheiser

Aha, good to know for reference. Happy to help!

Despite being a security researcher, you'd be surprised at how many times the 700 permissions get me with .ssh/ directories. This time, it was something else. These things are so small, I wouldn't blame anyone during migrations for missing a file like this, because there's so much to do, but yes, it was my bad, but now we both know that /etc/passwd could always be a suspect for this issue in future.

@inferenceus commented on GitHub (Feb 28, 2024): @jolheiser > Aha, good to know for reference. Happy to help! Despite being a security researcher, you'd be surprised at how many times the `700` permissions get me with `.ssh/` directories. This time, it was something else. These things are so small, I wouldn't blame anyone during migrations for missing a file like this, because there's so much to do, but yes, it was my bad, but now we both know that `/etc/passwd` could always be a suspect for this issue in future.
Author
Owner

@lunny commented on GitHub (Feb 28, 2024):

Maybe we can have some checks on gitea doctor for the file permissions.

@lunny commented on GitHub (Feb 28, 2024): Maybe we can have some checks on `gitea doctor` for the file permissions.
Author
Owner

@inferenceus commented on GitHub (Feb 28, 2024):

@lunny

Maybe we can have some checks on gitea doctor for the file permissions.

This would be useful. Sanity checks make a huge difference for debugging.

@inferenceus commented on GitHub (Feb 28, 2024): @lunny > Maybe we can have some checks on gitea doctor for the file permissions. This would be useful. Sanity checks make a huge difference for debugging.
Author
Owner

@inferenceus commented on GitHub (Feb 28, 2024):

Perhaps also add checks for a working shell and home directory? I don't know how difficult it would be to implement this stuff, but it likely would have made this issue self-explanatory had they existed.

@inferenceus commented on GitHub (Feb 28, 2024): Perhaps also add checks for a working shell and home directory? I don't know how difficult it would be to implement this stuff, but it likely would have made this issue self-explanatory had they existed.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/gitea#12555