Gitea server should not checkout on upload #2692

New Issue

GiteaMirror · 2025-11-02T04:44:38-06:00

GiteaMirror commented

2025-11-02 04:44:38 -06:00

Originally created by @zeripath on GitHub (Dec 27, 2018).

Gitea version (or commit ref): master

Description

When performing an upload, the UpdateRepoFile function in models/repo_editor.go actually checkout's a copy of the target repository, moves the uploaded files to the correct places, commits and pushes back to the repository.git.

This is very wasteful (and I suspect possibly dangerous, especially on Windows but I don't know). Clearer it would be better to just push (or at least commit) directly to the repository.git

I know of two approaches that are likely to be better:

The most complete appears to be the approach taken in bozaro/git-as-svn which is a Java bridge presenting git repos as SVN. This doesn't checkout a copy of the repo but rather appears to create objects and trees directly in the repository.git by having a deep understanding of the git repository layout thus simulating a commit. It even knows how to manage LFS. I don't quite understand what it's doing but it may be possible to use our current go git libraries to do something similar directly.

A less do it your own way appears to be to use git fast-import. This is a well documented git-core tool for importing data from other software. You simply use the protocol defined in the help file to commit the whole modified files, git will then do the diffs and store the objects. Unfortunately this doesn't appear run any hooks, so it appears that if the object should be in LFS that needs dealing with first. Similarly the post-receive etc hooks would need to be dealt with first/afterwards - although this is likely to be less difficult to deal with.

I will do some more basic testing with git fast-import which I think should be quite easy to pull in. I might also contact the git-as-svn maintainer to see if they can enlighten me more on how their solution works.

Originally created by @zeripath on GitHub (Dec 27, 2018). - Gitea version (or commit ref): master ## Description When performing an upload, the UpdateRepoFile function in models/repo_editor.go actually checkout's a copy of the target repository, moves the uploaded files to the correct places, commits and pushes back to the repository.git. This is very wasteful (and I suspect possibly dangerous, especially on Windows but I don't know). Clearer it would be better to just push (or at least commit) directly to the repository.git I know of two approaches that are likely to be better: The most complete appears to be the approach taken in bozaro/git-as-svn which is a Java bridge presenting git repos as SVN. This doesn't checkout a copy of the repo but rather appears to create objects and trees directly in the repository.git by having a deep understanding of the git repository layout thus simulating a commit. It even knows how to manage LFS. I don't quite understand what it's doing but it may be possible to use our current go git libraries to do something similar directly. A less do it your own way appears to be to use git fast-import. This is a well documented git-core tool for importing data from other software. You simply use the protocol defined in the help file to commit the whole modified files, git will then do the diffs and store the objects. Unfortunately this doesn't appear run any hooks, so it appears that if the object should be in LFS that needs dealing with first. Similarly the post-receive etc hooks would need to be dealt with first/afterwards - although this is likely to be less difficult to deal with. I will do some more basic testing with git fast-import which I think should be quite easy to pull in. I might also contact the git-as-svn maintainer to see if they can enlighten me more on how their solution works.

GiteaMirror added the type/enhancement label 2025-11-02 04:44:38 -06:00

GiteaMirror closed this issue

2025-11-02 04:44:38 -06:00

GiteaMirror commented

2025-11-02 04:44:39 -06:00

@typeless commented on GitHub (Dec 27, 2018):

A solution is to use shared-clone and sparse-checkout, which is similar to https://github.com/go-gitea/gitea/pull/4921 does.

@typeless commented on GitHub (Dec 27, 2018): A solution is to use shared-clone and sparse-checkout, which is similar to https://github.com/go-gitea/gitea/pull/4921 does.

GiteaMirror commented

2025-11-02 04:44:39 -06:00

@zeripath commented on GitHub (Dec 27, 2018):

We checkout elsewhere too? I don't think we should need to do any checking out - The data is all in the repository, just indirectly addressed. I would be very surprised if GitHub were checking out to do merges.

@zeripath commented on GitHub (Dec 27, 2018): We checkout elsewhere too? I don't think we should need to do any checking out - The data is all in the repository, just indirectly addressed. I would be very surprised if GitHub were checking out to do merges.

GiteaMirror commented

2025-11-02 04:44:39 -06:00

@typeless commented on GitHub (Dec 27, 2018):

This article https://githubengineering.com/move-fast/ has good insight.

The problem with manipulating the .git too-low-level-ish is that it is much tougher to guarantee the correctness without comprehensive testing and would risk the integrity of the repositories. The best option would be we have that implemented in the upstream git tool and to enjoy their free test suite.
That's a reason why we still do checkouts (or at least partially) besides being quick-and-dirty.

@typeless commented on GitHub (Dec 27, 2018): This article https://githubengineering.com/move-fast/ has good insight. The problem with manipulating the .git too-low-level-ish is that it is much tougher to guarantee the correctness without comprehensive testing and would risk the integrity of the repositories. The best option would be we have that implemented in the upstream `git` tool and to enjoy their free test suite. That's a reason why we still do checkouts (or at least partially) besides being quick-and-dirty.

GiteaMirror commented

2025-11-02 04:44:39 -06:00

@zeripath commented on GitHub (Dec 27, 2018):

Yeah agreed - I too would be extremely cautious about reimplementing git - however I suspect that we can use some of the plumbing a bit more cleverly to get the best of both worlds. We're already using a go reimplementation of git so it might be that that can help too.

For example the git fast-import plumbing seems like it would be very easy to use - if we are happy we understand what it does.

@zeripath commented on GitHub (Dec 27, 2018): Yeah agreed - I too would be extremely cautious about reimplementing git - however I suspect that we can use some of the plumbing a bit more cleverly to get the best of both worlds. We're already using a go reimplementation of git so it might be that that can help too. For example the git fast-import plumbing seems like it would be very easy to use - if we are happy we understand what it does.

GiteaMirror commented

2025-11-02 04:44:40 -06:00

@zeripath commented on GitHub (Jan 2, 2019):

OK, fast-import is not available on our current docker. However, you can achieve the same things using the plumbing commands available in git. My comments in #601 (https://github.com/go-gitea/gitea/issues/601#issuecomment-450194075) present an approach that uses bare-repos and index files.

@zeripath commented on GitHub (Jan 2, 2019): OK, fast-import is not available on our current docker. However, you can achieve the same things using the plumbing commands available in git. My comments in #601 (https://github.com/go-gitea/gitea/issues/601#issuecomment-450194075) present an approach that uses bare-repos and index files.

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/gitea#2692