Clone new project works, pushing commits fails with internal error #6440

@LordOfDragons commented on GitHub (Dec 9, 2020):

I've tried now to clone the repository directly on the server using direct access to Gitea without going through Apache reverse proxy. The result is the same:

01:21:44.954733 git.c:439               trace: built-in: git push
01:21:44.954908 run-command.c:663       trace: run_command: GIT_DIR=.git git-remote-http origin http://localhost:3000/dragondreams/lfstest.git
Username for 'http://localhost:3000': XXX
Password for 'http://XXX@localhost:3000': XXX
01:21:51.279304 run-command.c:663       trace: run_command: git send-pack --stateless-rpc --helper-status --thin --progress http://localhost:3000/dragondreams/lfstest.git/ --stdin
01:21:51.280148 git.c:439               trace: built-in: git send-pack --stateless-rpc --helper-status --thin --progress http://localhost:3000/dragondreams/lfstest.git/ --stdin
01:21:51.280328 run-command.c:663       trace: run_command: git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress
01:21:51.280958 git.c:439               trace: built-in: git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress
Objekte aufzählen: 3, Fertig.
Zähle Objekte: 100% (3/3), Fertig.
Schreibe Objekte: 100% (3/3), 206 Bytes | 206.00 KiB/s, Fertig.
Gesamt 3 (Delta 0), Wiederverwendet 0 (Delta 0), Pack wiederverwendet 0
remote: fatal: Kein Git-Repository: '.'
error: Entpacken auf der Gegenseite fehlgeschlagen: unpack-objects abnormal exit
To http://localhost:3000/dragondreams/lfstest.git
 ! [remote rejected] master -> master (unpacker error)
error: Fehler beim Versenden einiger Referenzen nach 'http://localhost:3000/dragondreams/lfstest.git'

The gitea.log file contains nothing. Something inside Gite is going horribly wrong. Especially without any logs I get nowhere.

@LordOfDragons commented on GitHub (Dec 9, 2020): I've tried now to clone the repository directly on the server using direct access to Gitea without going through Apache reverse proxy. The result is the same: ``` 01:21:44.954733 git.c:439 trace: built-in: git push 01:21:44.954908 run-command.c:663 trace: run_command: GIT_DIR=.git git-remote-http origin http://localhost:3000/dragondreams/lfstest.git Username for 'http://localhost:3000': XXX Password for 'http://XXX@localhost:3000': XXX 01:21:51.279304 run-command.c:663 trace: run_command: git send-pack --stateless-rpc --helper-status --thin --progress http://localhost:3000/dragondreams/lfstest.git/ --stdin 01:21:51.280148 git.c:439 trace: built-in: git send-pack --stateless-rpc --helper-status --thin --progress http://localhost:3000/dragondreams/lfstest.git/ --stdin 01:21:51.280328 run-command.c:663 trace: run_command: git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress 01:21:51.280958 git.c:439 trace: built-in: git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress Objekte aufzählen: 3, Fertig. Zähle Objekte: 100% (3/3), Fertig. Schreibe Objekte: 100% (3/3), 206 Bytes | 206.00 KiB/s, Fertig. Gesamt 3 (Delta 0), Wiederverwendet 0 (Delta 0), Pack wiederverwendet 0 remote: fatal: Kein Git-Repository: '.' error: Entpacken auf der Gegenseite fehlgeschlagen: unpack-objects abnormal exit To http://localhost:3000/dragondreams/lfstest.git ! [remote rejected] master -> master (unpacker error) error: Fehler beim Versenden einiger Referenzen nach 'http://localhost:3000/dragondreams/lfstest.git' ``` The gitea.log file contains nothing. Something inside Gite is going horribly wrong. Especially without any logs I get nowhere.

GiteaMirror commented

@techknowlogick commented on GitHub (Dec 9, 2020):

I see that the repo name is lfstest are you pushing any LFS files as well?
Are you able to upgrade to latest stable version of Gitea (1.13.0)?
Did you initialize the repo via Gitea web interface, or is the repo initialized via your local computer?
Can you create another test repo to see if you can replicate there (as it could be that there is an issue with git hooks with your first repo)?

@techknowlogick commented on GitHub (Dec 9, 2020): I see that the repo name is `lfstest` are you pushing any LFS files as well? Are you able to upgrade to latest stable version of Gitea (1.13.0)? Did you initialize the repo via Gitea web interface, or is the repo initialized via your local computer? Can you create another test repo to see if you can replicate there (as it could be that there is an issue with git hooks with your first repo)?

GiteaMirror commented

@LordOfDragons commented on GitHub (Dec 9, 2020):

I plan to test pushing LFS files but the initial test has been about getting GIT repository working using a regular file commit
About upgrading I don't know: https://packages.gentoo.org/packages/www-apps/gitea . Version is 1.12.6 . 9999 gives GIT master.
Init using the web interface is not working. See below
3b) I tried now cloning the repo, calling "git init", commit a file and pushing. Still failing with the same error.
Yes, it happens with any repository I tried creating.

Web Interface Git Init Problem

Results in the following error:

An error has occurred:

initRepository: prepareRepoCommit: git clone: exit status 128 - Cloning into '/tmp/gitea-testrepo981834641'...
fatal: unable to access '/tmp/gitea-testrepo981834641/.git/config': Permission denied

Gitea Version: 1.12.6

A directory "gitea-testrepo981834641" does not exist in /tmp . /tmp is owned root:root with permission drwxrwxrwt. Maybe Gitea does not use a correct way to access tmp-files? Can this path be specified to avoid problems?

EDIT: gitea.log contains something for the first time

2020/12/09 10:05:02 ...s/repository/init.go:43:prepareRepoCommit() [E] Failed to clone from &{5 1 dragonlord 0xc000374280 testrepo testrepo   0   0 0 0 0 0 0 0 0 0 0 0 0 true false false false <nil> 0 map[] map[] [] <nil> false 0 <nil> false 0 <nil> 0 <nil> <nil> true false []  0 0} into /tmp/gitea-testrepo981834641: stdout: 
        Error: exit status 128 - Cloning into '/tmp/gitea-testrepo981834641'...
        fatal: unable to access '/tmp/gitea-testrepo981834641/.git/config': Permission denied
        
2020/12/09 10:05:02 ...sitory/repository.go:23:CreateRepository() [E] Rollback deleteRepository: repository does not exist [id: 5, uid: 1, owner_name: , name: ]
2020/12/09 10:05:02 routers/repo/repo.go:176:handleCreateError() [E] CreatePost: initRepository: prepareRepoCommit: git clone: exit status 128 - Cloning into '/tmp/gitea-testrepo981834641'...
        fatal: unable to access '/tmp/gitea-testrepo981834641/.git/config': Permission denied

EDIT EDIT: I don't know if this has an influence on /tmp but I'm running a hardened kernel on the server.

EDIT EDIT EDIT: I cloned the git repo as regular user into /tmp which is working. Gitea has to do something wrong to get permission denied where I as regular user do not get it.

EDIT EDIT EDIT EDIT: I sudo-ed into the "git" user, which is what Gitea is running as and did the same cloing into /tmp as in the editor above. This is working so it is not a user permission problem but has to be a coding problem somehow.

@LordOfDragons commented on GitHub (Dec 9, 2020): 1) I plan to test pushing LFS files but the initial test has been about getting GIT repository working using a regular file commit 2) About upgrading I don't know: https://packages.gentoo.org/packages/www-apps/gitea . Version is 1.12.6 . 9999 gives GIT master. 3) Init using the web interface is not working. See below 3b) I tried now cloning the repo, calling "git init", commit a file and pushing. Still failing with the same error. 4) Yes, it happens with any repository I tried creating. ### Web Interface Git Init Problem Results in the following error: ``` An error has occurred: initRepository: prepareRepoCommit: git clone: exit status 128 - Cloning into '/tmp/gitea-testrepo981834641'... fatal: unable to access '/tmp/gitea-testrepo981834641/.git/config': Permission denied Gitea Version: 1.12.6 ``` A directory "gitea-testrepo981834641" does not exist in /tmp . /tmp is owned root:root with permission drwxrwxrwt. Maybe Gitea does not use a correct way to access tmp-files? Can this path be specified to avoid problems? EDIT: gitea.log contains something for the first time ``` 2020/12/09 10:05:02 ...s/repository/init.go:43:prepareRepoCommit() [E] Failed to clone from &{5 1 dragonlord 0xc000374280 testrepo testrepo 0 0 0 0 0 0 0 0 0 0 0 0 0 true false false false <nil> 0 map[] map[] [] <nil> false 0 <nil> false 0 <nil> 0 <nil> <nil> true false [] 0 0} into /tmp/gitea-testrepo981834641: stdout: Error: exit status 128 - Cloning into '/tmp/gitea-testrepo981834641'... fatal: unable to access '/tmp/gitea-testrepo981834641/.git/config': Permission denied 2020/12/09 10:05:02 ...sitory/repository.go:23:CreateRepository() [E] Rollback deleteRepository: repository does not exist [id: 5, uid: 1, owner_name: , name: ] 2020/12/09 10:05:02 routers/repo/repo.go:176:handleCreateError() [E] CreatePost: initRepository: prepareRepoCommit: git clone: exit status 128 - Cloning into '/tmp/gitea-testrepo981834641'... fatal: unable to access '/tmp/gitea-testrepo981834641/.git/config': Permission denied ``` EDIT EDIT: I don't know if this has an influence on /tmp but I'm running a hardened kernel on the server. EDIT EDIT EDIT: I cloned the git repo as regular user into /tmp which is working. Gitea has to do something wrong to get permission denied where I as regular user do not get it. EDIT EDIT EDIT EDIT: I sudo-ed into the "git" user, which is what Gitea is running as and did the same cloing into /tmp as in the editor above. This is working so it is not a user permission problem but has to be a coding problem somehow.

GiteaMirror commented

@lunny commented on GitHub (Dec 9, 2020):

@LordOfDragons Is it work to just push to Gitea server but not with Apache so that we can know if it's a problem related apache configuration or Gitea itself.

Gitea should be run in user git so if it visit a path root owned, it may return permission denied.

@lunny commented on GitHub (Dec 9, 2020): @LordOfDragons Is it work to just push to Gitea server but not with Apache so that we can know if it's a problem related apache configuration or Gitea itself. Gitea should be run in user git so if it visit a path root owned, it may return permission denied.

GiteaMirror commented

@LordOfDragons commented on GitHub (Dec 9, 2020):

@lunny As mentioned above I also tried on the server itself going directly to localhost:3000 but the result is the same. Under Gentoo Gitea runs as user git as far as I can see. I tried accessing /tmp using the git user and this works so it should not be a permission problem pertinent to the user.

To be on the safe side I tried right now cloning from localhost:3000 using git user into /tmp . This works. when I go into "lfstest" directory and call "git init" it tells me it "re-initialized" the repository.

@LordOfDragons commented on GitHub (Dec 9, 2020): @lunny As mentioned above I also tried on the server itself going directly to localhost:3000 but the result is the same. Under Gentoo Gitea runs as user git as far as I can see. I tried accessing /tmp using the git user and this works so it should not be a permission problem pertinent to the user. To be on the safe side I tried right now cloning from localhost:3000 using git user into /tmp . This works. when I go into "lfstest" directory and call "git init" it tells me it "re-initialized" the repository.

GiteaMirror commented

@LordOfDragons commented on GitHub (Dec 16, 2020):

Any news on the issue? This issues is a hard road block to use Gitea.

@LordOfDragons commented on GitHub (Dec 16, 2020): Any news on the issue? This issues is a hard road block to use Gitea.

GiteaMirror commented

@lunny commented on GitHub (Dec 16, 2020):

From the log it's obvious that it's a folder permission problem about /tmp. what's the file mode of /tmp ?

@lunny commented on GitHub (Dec 16, 2020): From the log it's obvious that it's a folder permission problem about /tmp. what's the file mode of /tmp ?

GiteaMirror commented

@LordOfDragons commented on GitHub (Dec 16, 2020):

Output from "ls -a /tmp".

drwxrwxrwt 71 root    root     458752 16. Dec 18:19 .

As mentioned above if I su into "git" user I can clone git into "/tmp" and call "git init" without problem. So it can not be a permission problem on "/tmp".

@LordOfDragons commented on GitHub (Dec 16, 2020): Output from "ls -a /tmp". ``` drwxrwxrwt 71 root root 458752 16. Dec 18:19 . ``` As mentioned above if I su into "git" user I can clone git into "/tmp" and call "git init" without problem. So it can not be a permission problem on "/tmp".

GiteaMirror commented

@lunny commented on GitHub (Dec 17, 2020):

@LordOfDragons Could you try to open /tmp/gitea-testrepo981834641/.git/config and check the file/dir mode when the error is reproducing?

@lunny commented on GitHub (Dec 17, 2020): @LordOfDragons Could you try to open `/tmp/gitea-testrepo981834641/.git/config` and check the file/dir mode when the error is reproducing?

GiteaMirror commented

@zeripath commented on GitHub (Dec 17, 2020):

As mentioned above if I su into "git" user I can clone git into "/tmp" and call "git init" without problem. So it can not be a permission problem on "/tmp".

That's not true.

Just because an interactive user can sudo as another user from a shell and run things from that sudo'd shell fine does not mean that the gitea process is able to.

You do not say how you are running gitea - but from what you are saying I do not believe that you are running it from an interactive user's shell, (and you shouldn't have to.)

Check your privileges.

I cannot be more explicit as to how to go about that because privilege granting and mapping is somewhat black magic and I find them poorly documented in general, but I would not be surprised if you setup was such that child processes might not be allowed access to tmp from other processes or something like that.

Consider running Gitea with a different environment set TMPDIR which may not have the same permissions/privilege protection.

@zeripath commented on GitHub (Dec 17, 2020): > As mentioned above if I su into "git" user I can clone git into "/tmp" and call "git init" without problem. So it can not be a permission problem on "/tmp". That's not true. Just because an interactive user can sudo as another user from a shell and run things from that sudo'd shell fine does not mean that the gitea process is able to. You do not say how you are running gitea - but from what you are saying I do not believe that you are running it from an interactive user's shell, (and you shouldn't have to.) Check your privileges. I cannot be more explicit as to how to go about that because privilege granting and mapping is somewhat black magic and I find them poorly documented in general, but I would not be surprised if you setup was such that child processes might not be allowed access to tmp from other processes or something like that. Consider running Gitea with a different environment set TMPDIR which may not have the same permissions/privilege protection.

GiteaMirror commented

@zeripath commented on GitHub (Dec 17, 2020):

For an example of another unique situation that was caused by a hardened system see #9792

In that case the systemd capabilities and privileges were set up in such a way that if a service or process asked to do something they were not allowed to, instead of receiving an EPERM - they would simply die.

Let me just demonstrate how appalling and impossible to debug that could be - a process that just wants to check if stdin is a tty in order to see if they're running in an interactive manner, would, if they didn't have permission to read the tty just die. That means that calls to cat just died - immediately and without any information being returned.

No-one could figure out what was happening - it was like files just didn't exist. Who would ever consider that cat would fail.

@zeripath commented on GitHub (Dec 17, 2020): For an example of another unique situation that was caused by a hardened system see #9792 In that case the systemd capabilities and privileges were set up in such a way that if a service or process asked to do something they were not allowed to, instead of receiving an EPERM - they would simply die. Let me just demonstrate how appalling and impossible to debug that could be - a process that just wants to check if stdin is a tty in order to see if they're running in an interactive manner, would, if they didn't have permission to read the tty just die. That means that calls to `cat` just died - immediately and without any information being returned. No-one could figure out what was happening - it was like files just didn't exist. Who would ever consider that `cat` would fail.

GiteaMirror commented

@LordOfDragons commented on GitHub (Dec 17, 2020):

@LordOfDragons Could you try to open /tmp/gitea-testrepo981834641/.git/config and check the file/dir mode when the error is reproducing?

I tried it and the directory named in the 500 error page ( /tmp/gitea-testrepo981834641/.git/config ) does not exist. Hence the directory itself can not be created.

@LordOfDragons commented on GitHub (Dec 17, 2020): > @LordOfDragons Could you try to open `/tmp/gitea-testrepo981834641/.git/config` and check the file/dir mode when the error is reproducing? I tried it and the directory named in the 500 error page ( /tmp/gitea-testrepo981834641/.git/config ) does not exist. Hence the directory itself can not be created.

GiteaMirror commented

@LordOfDragons commented on GitHub (Dec 17, 2020):

As mentioned above if I su into "git" user I can clone git into "/tmp" and call "git init" without problem. So it can not be a permission problem on "/tmp".

That's not true.

Just because an interactive user can sudo as another user from a shell and run things from that sudo'd shell fine does not mean that the gitea process is able to.

You do not say how you are running gitea - but from what you are saying I do not believe that you are running it from an interactive user's shell, (and you shouldn't have to.)

Check your privileges.

I cannot be more explicit as to how to go about that because privilege granting and mapping is somewhat black magic and I find them poorly documented in general, but I would not be surprised if you setup was such that child processes might not be allowed access to tmp from other processes or something like that.

Consider running Gitea with a different environment set TMPDIR which may not have the same permissions/privilege protection.

I said "su" not "sudo". Not quite the same but I get your concern there. Personally I have no objection against making gittea use a different tmp directory but this init problem here is not the initial problem why I created the issue in the first place. The initial problem is that gittea did allow clone but not push to a directory owned by the same user gittea is running as.

That said I did another check to verify something. The running process is listed like this:

/usr/bin/gitea web --config /etc/gitea/app.ini

The app.ini states to use RUN_USER=git . So far this is fine. I have though noticed something which I missed so far. This is the permission listing of the git repository directory:

dir -l /var/lib/gitea/gitea-repositories
drwxr-xr-x 3 root git 4096  3. Dec 02:33 dragondreams

This is actually quite interesting. This combination of user/group/permission is strange. It means Gittea web interface somehow managed to create a repository directory with permission 755 with root:git . Actually the entire directories and files underneath do not have group write permission.

What exactly is going wrong here? Does Gittea assign the wrong permissions (maybe wrong umask)? Did Gitea not properly drop priviledges to access the file system?

Knowing a bit more on how Gittea creates repositories could help figure out the appropriate cure for this. I don't want to just chown around in the repositories just to have Gitea ending up the wrong way the next time something is commited.

@LordOfDragons commented on GitHub (Dec 17, 2020): > > As mentioned above if I su into "git" user I can clone git into "/tmp" and call "git init" without problem. So it can not be a permission problem on "/tmp". > > That's not true. > > Just because an interactive user can sudo as another user from a shell and run things from that sudo'd shell fine does not mean that the gitea process is able to. > > You do not say how you are running gitea - but from what you are saying I do not believe that you are running it from an interactive user's shell, (and you shouldn't have to.) > > Check your privileges. > > I cannot be more explicit as to how to go about that because privilege granting and mapping is somewhat black magic and I find them poorly documented in general, but I would not be surprised if you setup was such that child processes might not be allowed access to tmp from other processes or something like that. > > Consider running Gitea with a different environment set TMPDIR which may not have the same permissions/privilege protection. I said "su" not "sudo". Not quite the same but I get your concern there. Personally I have no objection against making gittea use a different tmp directory but this init problem here is not the initial problem why I created the issue in the first place. The initial problem is that gittea did allow clone but not push to a directory owned by the same user gittea is running as. That said I did another check to verify something. The running process is listed like this: ``` /usr/bin/gitea web --config /etc/gitea/app.ini ``` The app.ini states to use RUN_USER=git . So far this is fine. I have though noticed something which I missed so far. This is the permission listing of the git repository directory: ``` dir -l /var/lib/gitea/gitea-repositories drwxr-xr-x 3 root git 4096 3. Dec 02:33 dragondreams ``` This is actually quite interesting. This combination of user/group/permission is strange. It means Gittea web interface somehow managed to create a repository directory with permission 755 with root:git . Actually the entire directories and files underneath do not have group write permission. What exactly is going wrong here? Does Gittea assign the wrong permissions (maybe wrong umask)? Did Gitea not properly drop priviledges to access the file system? Knowing a bit more on how Gittea creates repositories could help figure out the appropriate cure for this. I don't want to just chown around in the repositories just to have Gitea ending up the wrong way the next time something is commited.

GiteaMirror commented

@LordOfDragons commented on GitHub (Dec 17, 2020):

This is the init script running gitea. I see nothing wrong here:

#!/sbin/openrc-run
# Copyright 2016-2019 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2

description="Gitea, a self-hosted Git service"

: ${GITEA_CONF:=/etc/gitea/app.ini}
: ${GITEA_USER:=git}
: ${GITEA_GROUP:=git}
: ${GITEA_WORK_DIR:=/var/lib/gitea}
: ${GITEA_CUSTOM:=${GITEA_WORK_DIR}/custom}

command="/usr/bin/gitea web"
command_args="--config ${GITEA_CONF}"
command_background="true"
command_user="${GITEA_USER}:${GITEA_GROUP}"
error_log="/var/log/${RC_SVCNAME}/${RC_SVCNAME}.err"
pidfile="/run/${RC_SVCNAME}.pid"
required_files="${GITEA_CONF}"
start_stop_daemon_args="-d ${GITEA_WORK_DIR}"
start_stop_daemon_args="${start_stop_daemon_args} -e GITEA_WORK_DIR=${GITEA_WORK_DIR}"
start_stop_daemon_args="${start_stop_daemon_args} -e GITEA_CUSTOM=${GITEA_CUSTOM}"

@LordOfDragons commented on GitHub (Dec 17, 2020): This is the init script running gitea. I see nothing wrong here: ``` #!/sbin/openrc-run # Copyright 2016-2019 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 description="Gitea, a self-hosted Git service" : ${GITEA_CONF:=/etc/gitea/app.ini} : ${GITEA_USER:=git} : ${GITEA_GROUP:=git} : ${GITEA_WORK_DIR:=/var/lib/gitea} : ${GITEA_CUSTOM:=${GITEA_WORK_DIR}/custom} command="/usr/bin/gitea web" command_args="--config ${GITEA_CONF}" command_background="true" command_user="${GITEA_USER}:${GITEA_GROUP}" error_log="/var/log/${RC_SVCNAME}/${RC_SVCNAME}.err" pidfile="/run/${RC_SVCNAME}.pid" required_files="${GITEA_CONF}" start_stop_daemon_args="-d ${GITEA_WORK_DIR}" start_stop_daemon_args="${start_stop_daemon_args} -e GITEA_WORK_DIR=${GITEA_WORK_DIR}" start_stop_daemon_args="${start_stop_daemon_args} -e GITEA_CUSTOM=${GITEA_CUSTOM}" ```

GiteaMirror commented

@LordOfDragons commented on GitHub (Dec 17, 2020):

I've found the following bug report: https://bugs.gentoo.org/750218

The finding in this bug report is that Gitea is installed owned by "root" with the suid flag set. It is questioned why this is the case and if it is intended or not. I'll try the mentioned fix to see if I get past my problems.

@LordOfDragons commented on GitHub (Dec 17, 2020): I've found the following bug report: https://bugs.gentoo.org/750218 The finding in this bug report is that Gitea is installed owned by "root" with the suid flag set. It is questioned why this is the case and if it is intended or not. I'll try the mentioned fix to see if I get past my problems.

GiteaMirror commented

@zeripath commented on GitHub (Dec 17, 2020):

ugh. Gitea should not need SUID.

@zeripath commented on GitHub (Dec 17, 2020): ugh. Gitea should not need SUID.

GiteaMirror commented