github-starred/gitea
2
0
Fork 0
mirror of https://github.com/go-gitea/gitea.git synced 2026-03-09 12:46:42 -05:00
Code Issues 2.6k Packages Projects Releases 100 Wiki Activity

Git push fails with "Permission denied" on tmp_objdir in rootless Gitea on Kubernetes with old Longhorn CSI driver #14737

New Issue
Open
opened 2025-11-02 11:21:36 -06:00 by GiteaMirror · 6 comments
GiteaMirror commented 2025-11-02 11:21:36 -06:00
Owner
Copy Link

Originally created by @PatrickHuetter on GitHub (Jul 10, 2025).

Description

Environment

  • Gitea Version: 1.24.2-rootless
  • Deployment Method: Official Helm Chart v12.1.1
  • Kubernetes: K3s
  • Storage: Longhorn volumes
  • Container Runtime: Running as uid=1000(git) gid=1000(git)

Problem Description

When pushing repositories with more than ~30 files via HTTPS, git push consistently fails with:

remote: fatal: could not open '/data/git/gitea-repositories/myorg/myrepo.git/./objects/tmp_objdir-incoming-[random]/pack/tmp_pack_[random]' for reading: Permission denied
error: unpack failed: index-pack abnormal exit
To https://git.example.com/myorg/myrepo.git
 ! [remote rejected] main -> main (unpacker error)
error: failed to push some refs to 'https://git.example.com/myorg/myrepo.git'

Small commits (1-5 files) usually work without issues. The error occurs consistently with larger pushes.

What I've Tried

1. Security Context Configuration

Added comprehensive security contexts to ensure proper permissions:

podSecurityContext:
  fsGroup: 1000
  fsGroupChangePolicy: "Always"  # Also tried "OnRootMismatch"
  runAsUser: 1000
  runAsGroup: 1000
  runAsNonRoot: true

containerSecurityContext:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
      - ALL
  privileged: false
  readOnlyRootFilesystem: false
  runAsGroup: 1000
  runAsNonRoot: true
  runAsUser: 1000

2. Init Container for Permission Fixes

Attempted to fix permissions via initPreScript:

initPreScript: |
  # Create required directories with proper permissions
  mkdir -p /data/gitea/tmp || true
  chmod 1777 /data/gitea/tmp || true
  mkdir -p /tmp/gitea || true
  chmod 1777 /tmp/gitea || true
  # Fix repository permissions if needed
  find /data/git/gitea-repositories -type d -name objects -exec chmod 775 {} \; 2>/dev/null || true

3. Resource Limits

Increased resources to rule out memory/CPU constraints:

resources:
  limits:
    cpu: 2000m
    memory: 4Gi
  requests:
    cpu: 500m
    memory: 1Gi

4. Repository Configuration

Tried setting custom temp path:

gitea:
  config:
    repository:
      TEMP_PATH: /data/gitea/tmp

Observations

  1. The pod is running with correct user/group (1000:1000)
  2. Directory permissions look correct: 
    drwxrwsr-x    5 git      git           4096 Jul 10 12:11 /data/git
drwxrwsr-x   43 git      git           4096 Apr 27 16:34 /data/git/gitea-repositories
drwxr-sr-x    7 git      git           4096 Jul  9 09:46 /data/git/gitea-repositories/myorg/myrepo.git
drwxr-sr-x    4 git      git           4096 Jul 10 12:14 /data/git/gitea-repositories/myorg/myrepo.git/objects
  3. The error occurs when git-receive-pack creates temporary directories during object unpacking
  4. The temporary directories (tmp_objdir-incoming-*) appear to be created with incorrect permissions

Questions

  1. Is this a known issue with the rootless image when handling large git pushes?
  2. Are there specific volume mount options or security contexts required for rootless Gitea on Kubernetes?
  3. Would switching to the non-rootless image be the recommended solution?
  4. Are there any git configuration options that could help with the temporary directory permissions?

Additional Information

  • The volume is a Longhorn PVC mounted at /data
  • The issue is reproducible across pod restarts
  • SSH git operations have not been tested yet
  • No relevant error messages in Gitea logs besides the git-receive-pack output

Any help or guidance would be greatly appreciated!

Gitea Version

1.24.2

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

No response

Operating System

NixOS with k3s (Kubernetes) with Longhorn and gitea as container

How are you running Gitea?

NixOS with k3s (Kubernetes) with Longhorn and gitea as container

Database

PostgreSQL

Originally created by @PatrickHuetter on GitHub (Jul 10, 2025). ### Description ## Environment - **Gitea Version**: 1.24.2-rootless - **Deployment Method**: Official Helm Chart v12.1.1 - **Kubernetes**: K3s - **Storage**: Longhorn volumes - **Container Runtime**: Running as uid=1000(git) gid=1000(git) ## Problem Description When pushing repositories with more than ~30 files via HTTPS, git push consistently fails with: ``` remote: fatal: could not open '/data/git/gitea-repositories/myorg/myrepo.git/./objects/tmp_objdir-incoming-[random]/pack/tmp_pack_[random]' for reading: Permission denied error: unpack failed: index-pack abnormal exit To https://git.example.com/myorg/myrepo.git ! [remote rejected] main -> main (unpacker error) error: failed to push some refs to 'https://git.example.com/myorg/myrepo.git' ``` Small commits (1-5 files) usually work without issues. The error occurs consistently with larger pushes. ## What I've Tried ### 1. Security Context Configuration Added comprehensive security contexts to ensure proper permissions: ```yaml podSecurityContext: fsGroup: 1000 fsGroupChangePolicy: "Always" # Also tried "OnRootMismatch" runAsUser: 1000 runAsGroup: 1000 runAsNonRoot: true containerSecurityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL privileged: false readOnlyRootFilesystem: false runAsGroup: 1000 runAsNonRoot: true runAsUser: 1000 ``` ### 2. Init Container for Permission Fixes Attempted to fix permissions via initPreScript: ```yaml initPreScript: | # Create required directories with proper permissions mkdir -p /data/gitea/tmp || true chmod 1777 /data/gitea/tmp || true mkdir -p /tmp/gitea || true chmod 1777 /tmp/gitea || true # Fix repository permissions if needed find /data/git/gitea-repositories -type d -name objects -exec chmod 775 {} \; 2>/dev/null || true ``` ### 3. Resource Limits Increased resources to rule out memory/CPU constraints: ```yaml resources: limits: cpu: 2000m memory: 4Gi requests: cpu: 500m memory: 1Gi ``` ### 4. Repository Configuration Tried setting custom temp path: ```yaml gitea: config: repository: TEMP_PATH: /data/gitea/tmp ``` ## Observations 1. The pod is running with correct user/group (1000:1000) 2. Directory permissions look correct: ``` drwxrwsr-x 5 git git 4096 Jul 10 12:11 /data/git drwxrwsr-x 43 git git 4096 Apr 27 16:34 /data/git/gitea-repositories drwxr-sr-x 7 git git 4096 Jul 9 09:46 /data/git/gitea-repositories/myorg/myrepo.git drwxr-sr-x 4 git git 4096 Jul 10 12:14 /data/git/gitea-repositories/myorg/myrepo.git/objects ``` 3. The error occurs when git-receive-pack creates temporary directories during object unpacking 4. The temporary directories (`tmp_objdir-incoming-*`) appear to be created with incorrect permissions ## Questions 1. Is this a known issue with the rootless image when handling large git pushes? 2. Are there specific volume mount options or security contexts required for rootless Gitea on Kubernetes? 3. Would switching to the non-rootless image be the recommended solution? 4. Are there any git configuration options that could help with the temporary directory permissions? ## Additional Information - The volume is a Longhorn PVC mounted at `/data` - The issue is reproducible across pod restarts - SSH git operations have not been tested yet - No relevant error messages in Gitea logs besides the git-receive-pack output Any help or guidance would be greatly appreciated! ### Gitea Version 1.24.2 ### Can you reproduce the bug on the Gitea demo site? No ### Log Gist _No response_ ### Screenshots _No response_ ### Git Version _No response_ ### Operating System NixOS with k3s (Kubernetes) with Longhorn and gitea as container ### How are you running Gitea? NixOS with k3s (Kubernetes) with Longhorn and gitea as container ### Database PostgreSQL
GiteaMirror added the issue/workaroundtype/upstream labels 2025-11-02 11:21:36 -06:00
GiteaMirror commented 2025-11-02 11:21:37 -06:00
Author
Owner
Copy Link

@wxiaoguang commented on GitHub (Jul 10, 2025):

IIRC some other users also reported similar problems, but there is still no conclusion or solution ....

Is this a known issue with the rootless image when handling large git pushes?

I think yes but I don't have more idea (I don't use rootless)

Would switching to the non-rootless image be the recommended solution?

"rootful" and "rootless" use different directory layout and config, I don't think it can be switched easily, unless you backup the database and files, and setup a clean "rootful" instance, and put the files back to correct locations.

Are there any git configuration options that could help with the temporary directory permissions?

No idea, but I feel that the "tmp_objdir" is controlled by git command, so no Gitea config option would affect it

@wxiaoguang commented on GitHub (Jul 10, 2025): IIRC some other users also reported similar problems, but there is still no conclusion or solution .... > Is this a known issue with the rootless image when handling large git pushes? I think yes but I don't have more idea (I don't use rootless) > Would switching to the non-rootless image be the recommended solution? "rootful" and "rootless" use different directory layout and config, I don't think it can be switched easily, unless you backup the database and files, and setup a clean "rootful" instance, and put the files back to correct locations. > Are there any git configuration options that could help with the temporary directory permissions? No idea, but I feel that the "tmp_objdir" is controlled by git command, so no Gitea config option would affect it
GiteaMirror commented 2025-11-02 11:21:37 -06:00
Author
Owner
Copy Link

@PatrickHuetter commented on GitHub (Jul 10, 2025):

I tested and checked so many things. I can't figure out what the problem is. It's massively blocking currently since we can't really push many files at once.

@PatrickHuetter commented on GitHub (Jul 10, 2025): I tested and checked so many things. I can't figure out what the problem is. It's massively blocking currently since we can't really push many files at once.
GiteaMirror commented 2025-11-02 11:21:37 -06:00
Author
Owner
Copy Link

@wxiaoguang commented on GitHub (Jul 10, 2025):

Some more questions:

  1. Is the problem reproducible in only one or two repositories, or it is reproducible in all repositories?
  2. If you deploy a clean local rootless instance, is it able to reproduce?
@wxiaoguang commented on GitHub (Jul 10, 2025): Some more questions: 1. Is the problem reproducible in only one or two repositories, or it is reproducible in all repositories? 2. If you deploy a clean local rootless instance, is it able to reproduce?
GiteaMirror commented 2025-11-02 11:21:38 -06:00
Author
Owner
Copy Link

@PatrickHuetter commented on GitHub (Jul 10, 2025):

Some more questions:

  1. Is the problem reproducible in only one or two repositories, or it is reproducible in all repositories?
  2. If you deploy a clean local rootless instance, is it able to reproduce?

  1. It's reproducible in up to 6 repositories i tested. Also today in a clean new repository. For example: I created a new repository and initalized an angular app with some components and files (some pictures). Pushing the first commit isn't possible. Pushing the same local git repository to another origin (for example gitlab) works. Also mirroring the gitlab repository to gitea afterwards, doesn't work.

  2. Didn't test this yet, since i only work on kubernetes currently. I'll look further the next days to test this or just create a new namespace with new gitea instance.

@PatrickHuetter commented on GitHub (Jul 10, 2025): > Some more questions: > > 1. Is the problem reproducible in only one or two repositories, or it is reproducible in all repositories? > 2. If you deploy a clean local rootless instance, is it able to reproduce? 1. It's reproducible in up to 6 repositories i tested. Also today in a clean new repository. For example: I created a new repository and initalized an angular app with some components and files (some pictures). Pushing the first commit isn't possible. Pushing the same local git repository to another origin (for example gitlab) works. Also mirroring the gitlab repository to gitea afterwards, doesn't work. 2. Didn't test this yet, since i only work on kubernetes currently. I'll look further the next days to test this or just create a new namespace with new gitea instance.
GiteaMirror commented 2025-11-02 11:21:38 -06:00
Author
Owner
Copy Link

@wxiaoguang commented on GitHub (Jul 10, 2025):

After googling about the "Permission denied tmp_objdir" error, I found that some users said that it is "filesystem/storage" related.

@wxiaoguang commented on GitHub (Jul 10, 2025): After googling about the "Permission denied tmp_objdir" error, I found that some users said that it is "filesystem/storage" related.
GiteaMirror commented 2025-11-02 11:21:38 -06:00
Author
Owner
Copy Link

@PatrickHuetter commented on GitHub (Jul 10, 2025):

Solution Found: Longhorn Storage Driver Bug

I've found the root cause and solution for this issue! 🎉

The Problem

This is not a Gitea bug, but rather a Longhorn storage driver issue that affects git operations creating temporary files during push/checkout operations.

The Solution

After extensive debugging and reading through release notes, I discovered that Longhorn had a known bug affecting git operations in containers. The issue is documented in [longhorn/longhorn#10975](https://github.com/longhorn/longhorn/issues/10975) and was fixed in recent Longhorn releases.

The problem occurs because:

  1. Git creates temporary directories (tmp_objdir-incoming-*) during large push operations
  2. Longhorn's CSI driver had permission issues when handling these temporary file operations
  3. This particularly affects rootless containers where permission handling is more strict

What Fixed It

Simply updating Longhorn to the latest version resolved the issue completely. The push operations that were failing with 30+ files now work perfectly without any changes to the Gitea configuration.

For Others Experiencing This Issue

If you're running Gitea on Kubernetes with Longhorn storage and experiencing similar permission errors:

  1. Check your Longhorn version - The bug was present in older versions
  2. Update Longhorn to the latest stable release
  3. No Gitea configuration changes are necessary

Why This Matters

This issue can be misleading because:

  • The error appears to be a Gitea permission problem
  • It only occurs with larger git operations (30+ files)
  • All directory permissions appear correct when inspected
  • The rootless setup makes it seem like a container permission issue

But it's actually the underlying storage driver failing to properly handle git's temporary file operations during the receive-pack process.

I hope this helps others who might encounter the same issue. The Gitea rootless image works perfectly fine - just make sure your storage driver is up to date!

Environment After Fix

  • Gitea: 1.24.2-rootless (unchanged)
  • Longhorn: Updated to latest version
  • No changes to security contexts, init containers, or Gitea configuration required

Feel free to close this issue as it's not a Gitea bug. Thanks for the great work on Gitea! 🚀

@PatrickHuetter commented on GitHub (Jul 10, 2025): ## Solution Found: Longhorn Storage Driver Bug I've found the root cause and solution for this issue! 🎉 ### The Problem This is **not a Gitea bug**, but rather a Longhorn storage driver issue that affects git operations creating temporary files during push/checkout operations. ### The Solution After extensive debugging and reading through release notes, I discovered that Longhorn had a known bug affecting git operations in containers. The issue is documented in [[longhorn/longhorn#10975](https://github.com/longhorn/longhorn/issues/10975)](https://github.com/longhorn/longhorn/issues/10975) and was fixed in recent Longhorn releases. The problem occurs because: 1. Git creates temporary directories (`tmp_objdir-incoming-*`) during large push operations 2. Longhorn's CSI driver had permission issues when handling these temporary file operations 3. This particularly affects rootless containers where permission handling is more strict ### What Fixed It Simply **updating Longhorn to the latest version** resolved the issue completely. The push operations that were failing with 30+ files now work perfectly without any changes to the Gitea configuration. ### For Others Experiencing This Issue If you're running Gitea on Kubernetes with Longhorn storage and experiencing similar permission errors: 1. **Check your Longhorn version** - The bug was present in older versions 2. **Update Longhorn** to the latest stable release 3. No Gitea configuration changes are necessary ### Why This Matters This issue can be misleading because: - The error appears to be a Gitea permission problem - It only occurs with larger git operations (30+ files) - All directory permissions appear correct when inspected - The rootless setup makes it seem like a container permission issue But it's actually the underlying storage driver failing to properly handle git's temporary file operations during the receive-pack process. I hope this helps others who might encounter the same issue. The Gitea rootless image works perfectly fine - just make sure your storage driver is up to date! ### Environment After Fix - Gitea: 1.24.2-rootless (unchanged) - Longhorn: Updated to latest version - No changes to security contexts, init containers, or Gitea configuration required Feel free to close this issue as it's not a Gitea bug. Thanks for the great work on Gitea! 🚀
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
$20
$250
$50
$500
backport/done
💎 Bounty
docs-update-needed
good first issue
hacktoberfest
issue/bounty
issue/confirmed
issue/critical
issue/duplicate
issue/needs-feedback
issue/not-a-bug
issue/regression
issue/stale
issue/workaround
lgtm/need 2
modifies/api
modifies/translation
outdated/backport/v1.18
outdated/theme/markdown
outdated/theme/timetracker
performance/bigrepo
performance/cpu
performance/memory
performance/speed
pr/breaking
proposal/accepted
proposal/rejected
pr/wip
pull-request
Mirrored from GitHub Pull Request
 reviewed/wontfix
💰 Rewarded
skip-changelog
status/blocked
topic/accessibility
topic/api
topic/authentication
topic/build
topic/code-linting
topic/commit-signing
topic/content-rendering
topic/deployment
topic/distribution
topic/federation
topic/gitea-actions
topic/issues
topic/lfs
topic/mobile
topic/moderation
topic/packages
topic/pr
topic/projects
topic/repo
topic/repo-migration
topic/security
topic/theme
topic/ui
topic/ui-interaction
topic/ux
topic/webhooks
topic/wiki
type/bug
type/deprecation
type/docs
type/enhancement
type/feature
type/miscellaneous
type/proposal
type/question
type/refactoring
type/summary
type/testing
type/upstream
No Label issue/workaround type/upstream
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/gitea#14737
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?