pull-mirror sync seems unnecessarily slow #8388

New Issue

GiteaMirror · 2025-11-02T08:04:33-06:00

GiteaMirror commented

2025-11-02 08:04:33 -06:00

Originally created by @petergardfjall on GitHub (Jan 21, 2022).

Gitea Version

6c7084cbe

Git Version

2.25.1

Operating System

Linux (Ubuntu 20.04)

How are you running Gitea?

Built from source, run locally on the host.

Database

PostgreSQL

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Description

A mirror-sync operation not only runs git remote update, but also tries to sync any repo releases with the available repo tags. For large repositories with many tags, this can be a costly operation, both in time and computational resources.

I noticed that when doing a "plain git mirror" (which doesn't include any releases) of a big repo (such as Kubernetes with about 900 tags), the synchronize operation (SyncReleasesWithTags) spent a lot of time (about six minutes) listing/syncing tags with releases.

This appears to be caused by repetitive calls like (one for each repo tag):

git show-ref --tags -- v8.2.4477
git cat-file -t 29ab6ce9f36660cffaad3c8789e71162e5db5d2f
git cat-file -p 29ab6ce9f36660cffaad3c8789e71162e5db5d2f
git rev-list --count 29ab6ce9f36660cffaad3c8789e71162e5db5d2f

In particular git rev-list --count' can be heavy for large repos with many commits.

It seems like there is an opportunity to improve performance and reduce resource use by making this procedure more efficient for pull-mirrors.

Originally created by @petergardfjall on GitHub (Jan 21, 2022). ### Gitea Version 6c7084cbe ### Git Version 2.25.1 ### Operating System Linux (Ubuntu 20.04) ### How are you running Gitea? Built from source, run locally on the host. ### Database PostgreSQL ### Can you reproduce the bug on the Gitea demo site? No ### Log Gist _No response_ ### Description A mirror-sync operation not only runs `git remote update`, but also tries to sync any repo releases with the available repo tags. For large repositories with many tags, this can be a costly operation, both in time and computational resources. I noticed that when doing a "plain git mirror" (which doesn't include any releases) of a big repo (such as Kubernetes with about 900 tags), the synchronize operation (`SyncReleasesWithTags`) spent a lot of time (about six minutes) listing/syncing tags with releases. This appears to be caused by repetitive calls like (one for each repo tag): ```bash git show-ref --tags -- v8.2.4477 git cat-file -t 29ab6ce9f36660cffaad3c8789e71162e5db5d2f git cat-file -p 29ab6ce9f36660cffaad3c8789e71162e5db5d2f git rev-list --count 29ab6ce9f36660cffaad3c8789e71162e5db5d2f ``` In particular `git rev-list --count'` can be heavy for large repos with many commits. It seems like there is an opportunity to improve performance and reduce resource use by making this procedure more efficient for pull-mirrors.

GiteaMirror added the performance/speed type/bug labels 2025-11-02 08:04:33 -06:00

GiteaMirror closed this issue

2025-11-02 08:04:34 -06:00

GiteaMirror commented

2025-11-02 08:04:34 -06:00

@petergardfjall commented on GitHub (Jan 21, 2022):

An attempt to address this issue can be found here: https://github.com/go-gitea/gitea/pull/18351

@petergardfjall commented on GitHub (Jan 21, 2022): An attempt to address this issue can be found here: https://github.com/go-gitea/gitea/pull/18351

GiteaMirror commented

2025-11-02 08:04:35 -06:00

@petergardfjall commented on GitHub (Mar 31, 2022):

Closing this issue since https://github.com/go-gitea/gitea/pull/19125 is now merged.

@petergardfjall commented on GitHub (Mar 31, 2022): Closing this issue since https://github.com/go-gitea/gitea/pull/19125 is now merged.

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/gitea#8388