github-starred/gitea
2
0
Fork 0
mirror of https://github.com/go-gitea/gitea.git synced 2026-08-02 06:25:51 -05:00
Code Issues 2.6k Packages Projects Releases 100 Wiki Activity

Automatically clean up docker images in the registry without a tag pointing to them #9774

New Issue
Open
opened 2025-11-02 08:49:10 -06:00 by GiteaMirror · 21 comments
GiteaMirror commented 2025-11-02 08:49:10 -06:00
Owner
Copy Link
Copy Source

Originally created by @kolaente on GitHub (Nov 3, 2022).

Originally assigned to: @lunny on GitHub.

When pushing new docker images for an existing tag, the old image still exists and uses up storage one the server. While you can use images just by pointing to their sha, I've yet to find someone who actively uses that. For my own registry (portus) I have a cron job to automatically remove everything that does not have a tag pointing to it. Docker even has a command for this.

Having a cleanup job like that would allow to keep old versions but still solve the storage space problem.

@KN4CK3R in https://github.com/go-gitea/gitea/issues/21658#issuecomment-1301794468:

No, only if it's "older than" or not included in the "keep pattern". But it should be no problem to add a special logic here because there is already the custom Version == "latest" for containers.

Gitlab has an automatic garbage collection process for this: https://docs.gitlab.com/ee/administration/packages/container_registry.html#removing-untagged-manifests-and-unreferenced-layers

I think it's best to discuss this before implementing, mostly regarding these open questions:

  1. Should this be enabled automatically?
  2. Should this be a repo/org setting or a global config one?
Originally created by @kolaente on GitHub (Nov 3, 2022). Originally assigned to: @lunny on GitHub. When pushing new docker images for an existing tag, the old image still exists and uses up storage one the server. While you can use images just by pointing to their sha, I've yet to find someone who actively uses that. For my own registry (portus) I have a cron job to automatically remove everything that does not have a tag pointing to it. Docker even has a [command for this](https://distribution.github.io/distribution/about/garbage-collection/). Having a cleanup job like that would allow to keep old versions but still solve the storage space problem. @KN4CK3R in https://github.com/go-gitea/gitea/issues/21658#issuecomment-1301794468: > No, only if it's "older than" or not included in the "keep pattern". But it should be no problem to add a special logic here because there is already the custom `Version == "latest"` for containers. Gitlab has an automatic garbage collection process for this: https://docs.gitlab.com/ee/administration/packages/container_registry.html#removing-untagged-manifests-and-unreferenced-layers I think it's best to discuss this before implementing, mostly regarding these open questions: 1. Should this be enabled automatically? 2. Should this be a repo/org setting or a global config one?
GiteaMirror added the topic/packagesproposal/acceptedtype/feature labels 2025-11-02 08:49:10 -06:00
GiteaMirror commented 2025-11-02 08:49:10 -06:00
Author
Owner
Copy Link
Copy Source

@KN4CK3R commented on GitHub (Nov 3, 2022):

Just for clarification, a repo has no impact on packages:

Should this be a repouser/org setting or a global config one?

I checked again how I implemented this and currently there are no untagged images in the container registry! (Exception: If you upload a multiarch image, the different arches are untagged images) If you tag and push an image you can later pull that image with the tag and its hash. If a tag gets pushed again the old tag/version gets removed and that deletes the hash reference too. So after that operation there is no untagged image available anymore.

https://github.com/go-gitea/gitea/blob/f17edfaf5a31ea3f4e9152424b75c2c4986acbe3/routers/api/packages/container/manifest.go#L309-L312

So at the moment the cleanup does not need to remove untagged images because there are none. The question should first be "Should Gitea keep untagged version?"

@KN4CK3R commented on GitHub (Nov 3, 2022): Just for clarification, a repo has no impact on packages: > Should this be a ~~repo~~user/org setting or a global config one? I checked again how I implemented this and currently there are no untagged images in the container registry! (Exception: If you upload a multiarch image, the different arches are untagged images) If you tag and push an image you can later pull that image with the tag and its hash. If a tag gets pushed again the old tag/version gets removed and that deletes the hash reference too. So after that operation there is no untagged image available anymore. https://github.com/go-gitea/gitea/blob/f17edfaf5a31ea3f4e9152424b75c2c4986acbe3/routers/api/packages/container/manifest.go#L309-L312 So at the moment the cleanup does not need to remove untagged images because there are none. The question should first be "Should Gitea keep untagged version?"
GiteaMirror commented 2025-11-02 08:49:11 -06:00
Author
Owner
Copy Link
Copy Source

@silverwind commented on GitHub (Nov 3, 2022):

Use case sounds pretty similar to git gc which we already automatically run as a cron IIRC.

Should this be enabled automatically?

If it's stable, I'd say so.

Should this be a repo/org setting or a global config one?

I think global is sufficient. Ideally it should just be another cron to cleanup orphaned images, like we already do for orphaned git commits via git gc.

@silverwind commented on GitHub (Nov 3, 2022): Use case sounds pretty similar to `git gc` which we already automatically run as a cron IIRC. > Should this be enabled automatically? If it's stable, I'd say so. > Should this be a repo/org setting or a global config one? I think global is sufficient. Ideally it should just be another cron to cleanup orphaned images, like we already do for orphaned git commits via `git gc`.
GiteaMirror commented 2025-11-02 08:49:11 -06:00
Author
Owner
Copy Link
Copy Source

@theodiem commented on GitHub (Jan 5, 2023):

I've came across this issue after experiencing the same effect.
Building multiarch images when only the manifest is tagged, left me with lots of "packages" behind with only the digest (the manifest had only one copy since it was tagged).

Tagging each arch so it gets overwritten makes the "details" tab a bit impractical when you have too much different arch and versions (for matrix builds).

Should this be a repo/org setting or a global config one?

In my case, I would be happy with the exact same global mechanism described (similar to the cron that runs git gc)

@theodiem commented on GitHub (Jan 5, 2023): I've came across this issue after experiencing the same effect. Building multiarch images when only the manifest is tagged, left me with lots of "packages" behind with only the digest (the manifest had only one copy since it was tagged). Tagging each arch so it gets overwritten makes the "details" tab a bit impractical when you have too much different arch and versions (for matrix builds). > Should this be a repo/org setting or a global config one? In my case, I would be happy with the exact same global mechanism described (similar to the cron that runs `git gc`)
GiteaMirror commented 2025-11-02 08:49:11 -06:00
Author
Owner
Copy Link
Copy Source

@salasrod commented on GitHub (Feb 3, 2023):

I am also looking for a similar feature, going out of my way to manually prune images is painful.

@salasrod commented on GitHub (Feb 3, 2023): I am also looking for a similar feature, going out of my way to manually prune images is painful.
GiteaMirror commented 2025-11-02 08:49:11 -06:00
Author
Owner
Copy Link
Copy Source

@lunny commented on GitHub (Feb 3, 2023):

Doesn't #21658 resolved the issue?

@lunny commented on GitHub (Feb 3, 2023): Doesn't #21658 resolved the issue?
GiteaMirror commented 2025-11-02 08:49:11 -06:00
Author
Owner
Copy Link
Copy Source

@kolaente commented on GitHub (Feb 3, 2023):

@lunny I didn't test it but I don't think so. The PR allows to configure rules for removal of tags, I just want to remove every image layer not associated with a tag.

@kolaente commented on GitHub (Feb 3, 2023): @lunny I didn't test it but I don't think so. The PR allows to configure rules for removal of tags, I just want to remove every image layer not associated with a tag.
GiteaMirror commented 2025-11-02 08:49:12 -06:00
Author
Owner
Copy Link
Copy Source

@peiwenxu commented on GitHub (Sep 19, 2023):

Is this still happening?

@peiwenxu commented on GitHub (Sep 19, 2023): Is this still happening?
GiteaMirror commented 2025-11-02 08:49:12 -06:00
Author
Owner
Copy Link
Copy Source

@jum commented on GitHub (Sep 20, 2023):

No, I have 1.20.4 running and it does not happen.

Am 19. September 2023 14:06:27 MESZ schrieb Peiwen Xu @.***>:

Is this still happening?

--
Reply to this email directly or view it on GitHub:
https://github.com/go-gitea/gitea/issues/21673#issuecomment-1725380028
You are receiving this because you are subscribed to this thread.

Message ID: @.***>

@jum commented on GitHub (Sep 20, 2023): No, I have 1.20.4 running and it does not happen. Am 19. September 2023 14:06:27 MESZ schrieb Peiwen Xu ***@***.***>: >Is this still happening? > >-- >Reply to this email directly or view it on GitHub: >https://github.com/go-gitea/gitea/issues/21673#issuecomment-1725380028 >You are receiving this because you are subscribed to this thread. > >Message ID: ***@***.***>
GiteaMirror commented 2025-11-02 08:49:12 -06:00
Author
Owner
Copy Link
Copy Source

@silverwind commented on GitHub (Sep 21, 2023):

No one has implemented this yet, but it's definitely a vital feature to conserve disk space.

Maybe it should be disabled by default to support pulling image by hash, which is a rare, but valid use case.

@silverwind commented on GitHub (Sep 21, 2023): No one has implemented this yet, but it's definitely a vital feature to conserve disk space. Maybe it should be disabled by default to support pulling image by hash, which is a rare, but valid use case.
GiteaMirror commented 2025-11-02 08:49:12 -06:00
Author
Owner
Copy Link
Copy Source

@c521wy commented on GitHub (Jan 27, 2024):

Does anyone tried this cleanup rule?

image
@c521wy commented on GitHub (Jan 27, 2024): Does anyone tried this cleanup rule? <img width="616" alt="image" src="https://github.com/go-gitea/gitea/assets/4578684/5d6fafad-9eb6-43bc-82af-b69c1f583715">
GiteaMirror commented 2025-11-02 08:49:13 -06:00
Author
Owner
Copy Link
Copy Source

@kolaente commented on GitHub (Jan 30, 2024):

Does anyone tried this cleanup rule?
image

Using that and then checking with the preview yields no results, does not look like its working.

@kolaente commented on GitHub (Jan 30, 2024): > Does anyone tried this cleanup rule? > <img alt="image" width="616" src="https://private-user-images.githubusercontent.com/4578684/300176333-5d6fafad-9eb6-43bc-82af-b69c1f583715.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDY2MTgyMTYsIm5iZiI6MTcwNjYxNzkxNiwicGF0aCI6Ii80NTc4Njg0LzMwMDE3NjMzMy01ZDZmYWZhZC05ZWI2LTQzYmMtODJhZi1iNjljMWY1ODM3MTUucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDEzMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDAxMzBUMTIzMTU2WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ZjcyZTIyZDQwOGE4OWMwZWJkNDU4NDQ2YzVhNjJmMWZjYzU1YTEyMjVhMjFiNmUyOTVhMDZjY2QwOWFlNzU3MCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.te_yy1ApkXAiIXAYsbISy4rZZM41vjTho7R6R42P6ss"> Using that and then checking with the preview yields no results, does not look like its working.
GiteaMirror commented 2025-11-02 08:49:13 -06:00
Author
Owner
Copy Link
Copy Source

@kolaente commented on GitHub (Jan 30, 2024):

It looks like the official docker registry implementation uses this function to find and remove all untagged layers, as described here.

@KN4CK3R As far as I understood from glancing over the code, Gitea does not just "embed" the official registry package, so it's not as easy as just copying or calling that function?

@kolaente commented on GitHub (Jan 30, 2024): It looks like the official docker registry implementation uses [this function](https://github.com/distribution/distribution/blob/9b3eac8f08207cc088270cf4acabb6ebeee1e7a0/registry/storage/garbagecollect.go#L31) to find and remove all untagged layers, [as described here](https://distribution.github.io/distribution/about/garbage-collection/). @KN4CK3R As far as I understood from glancing over the code, Gitea does not just "embed" the official registry package, so it's not as easy as just copying or calling that function?
GiteaMirror commented 2025-11-02 08:49:13 -06:00
Author
Owner
Copy Link
Copy Source

@mhkarimi1383 commented on GitHub (Mar 27, 2024):

I'm facing the same issue with the latest version of gitea

@mhkarimi1383 commented on GitHub (Mar 27, 2024): I'm facing the same issue with the latest version of gitea
GiteaMirror commented 2025-11-02 08:49:13 -06:00
Author
Owner
Copy Link
Copy Source

@ViRb3 commented on GitHub (Jul 27, 2024):

Does anyone tried this cleanup rule?

image

The following seems to work perfectly! It deletes all images that do not have an associated tag with them. I would just suggest using ^sha256:.+ instead, as you could otherwise match a tag that for some reason has sha256 in the middle.

@ViRb3 commented on GitHub (Jul 27, 2024): > Does anyone tried this cleanup rule? > > <img alt="image" width="616" src="https://private-user-images.githubusercontent.com/4578684/300176333-5d6fafad-9eb6-43bc-82af-b69c1f583715.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjIxMTU1NTEsIm5iZiI6MTcyMjExNTI1MSwicGF0aCI6Ii80NTc4Njg0LzMwMDE3NjMzMy01ZDZmYWZhZC05ZWI2LTQzYmMtODJhZi1iNjljMWY1ODM3MTUucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDcyNyUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA3MjdUMjEyMDUxWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ODQxN2ZhMTg1YTAxMTg5ZWI2MzY4N2M3MzYxY2VlZDdkOThjNTQ4NGMwZGFkN2MyMjM5YjhjNWUyZDUxNTkzYSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.v9MDHtRqdP8kTmnLAQloexE1Mpa41Z9en2TDnDY-HWA"> The following seems to work perfectly! It deletes all images that do not have an associated tag with them. I would just suggest using `^sha256:.+` instead, as you could otherwise match a tag that for some reason has sha256 in the middle.
GiteaMirror commented 2025-11-02 08:49:13 -06:00
Author
Owner
Copy Link
Copy Source

@KimonHoffmann commented on GitHub (Sep 5, 2024):

Be careful with this approach, when using multi platform images!
In this case the individual platform images might be untagged, but the images themselves may still be referenced (by the multi platform manifest that is). I'm currently trying to deal with this problem myself and have not yet found a way that does not require deeper insight into the relationships of the images involved.

If someone has something to suggest that'd be very welcome!

@KimonHoffmann commented on GitHub (Sep 5, 2024): Be careful with this approach, when using multi platform images! In this case the individual platform images might be *untagged*, but the images themselves may still be *referenced* (by the multi platform manifest that is). I'm currently trying to deal with this problem myself and have not yet found a way that does not require deeper insight into the relationships of the images involved. If someone has something to suggest that'd be very welcome!
GiteaMirror commented 2025-11-02 08:49:14 -06:00
Author
Owner
Copy Link
Copy Source

@gjung56 commented on GitHub (Sep 5, 2024):

Yes, the cleanup rule delete platform variants images.

Until we can find a integrated solution, I ended up with an external cronjob that prune old images in my self-hosted instance.

I fetched the registry api and used the gitea golang sdk, in a hacky way but It's working.

gitea_registry_prune.go.txt

@gjung56 commented on GitHub (Sep 5, 2024): Yes, the cleanup rule delete platform variants images. Until we can find a integrated solution, I ended up with an external cronjob that prune old images in my self-hosted instance. I fetched the registry api and used the gitea golang sdk, in a hacky way but It's working. [gitea_registry_prune.go.txt](https://github.com/user-attachments/files/17001316/gitea_registry_prune.go.txt)
GiteaMirror commented 2025-11-02 08:49:14 -06:00
Author
Owner
Copy Link
Copy Source

@stuzer05 commented on GitHub (Dec 12, 2024):

Yes, the cleanup rule delete platform variants images.

Until we can find a integrated solution, I ended up with an external cronjob that prune old images in my self-hosted instance.

I fetched the registry api and used the gitea golang sdk, in a hacky way but It's working.

gitea_registry_prune.go.txt

Thank you, that works perfectly as needed!

@lunny will there be a solution to cleanup orphan registry images? Space grows on server so this manual hack is the only way cleanup space.

@stuzer05 commented on GitHub (Dec 12, 2024): > Yes, the cleanup rule delete platform variants images. > > Until we can find a integrated solution, I ended up with an external cronjob that prune old images in my self-hosted instance. > > I fetched the registry api and used the gitea golang sdk, in a hacky way but It's working. > > [gitea_registry_prune.go.txt](https://github.com/user-attachments/files/17001316/gitea_registry_prune.go.txt) Thank you, that works perfectly as needed! @lunny will there be a solution to cleanup orphan registry images? Space grows on server so this manual hack is the only way cleanup space.
GiteaMirror commented 2025-11-02 08:49:14 -06:00
Author
Owner
Copy Link
Copy Source

@lunny commented on GitHub (Dec 13, 2024):

I don't think so. I will take a look at this problem.

@lunny commented on GitHub (Dec 13, 2024): I don't think so. I will take a look at this problem.
GiteaMirror commented 2025-11-02 08:49:14 -06:00
Author
Owner
Copy Link
Copy Source

@stuzer05 commented on GitHub (Dec 13, 2024):

I don't think so. I will take a look at this problem.

Orphan images occur when pushing with the same label (e.g. latest) to container registry. Those "behind" images are the problem

@stuzer05 commented on GitHub (Dec 13, 2024): > I don't think so. I will take a look at this problem. Orphan images occur when pushing with the same label (e.g. latest) to container registry. Those "behind" images are the problem
GiteaMirror commented 2025-11-02 08:49:14 -06:00
Author
Owner
Copy Link
Copy Source

@philkunz commented on GitHub (Feb 7, 2025):

Any updates on this one? We just ran into this problem, pushing a lot of latest images for internal build tools on our code.foss.global instance.

@philkunz commented on GitHub (Feb 7, 2025): Any updates on this one? We just ran into this problem, pushing a lot of latest images for internal build tools on our code.foss.global instance.
GiteaMirror commented 2025-11-02 08:49:15 -06:00
Author
Owner
Copy Link
Copy Source

@stuzer05 commented on GitHub (Feb 7, 2025):

Any updates on this one? We just ran into this problem, pushing a lot of latest images for internal build tools on our code.foss.global instance.

Try this for now https://gitea.stuzer.link/stuzer05/gitea-docker-registry-prune

@stuzer05 commented on GitHub (Feb 7, 2025): > Any updates on this one? We just ran into this problem, pushing a lot of latest images for internal build tools on our code.foss.global instance. Try this for now https://gitea.stuzer.link/stuzer05/gitea-docker-registry-prune
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
$20
$250
$50
$500
backport/done
💎 Bounty
docs-update-needed
good first issue
hacktoberfest
issue/bounty
issue/confirmed
issue/critical
issue/duplicate
issue/needs-feedback
issue/not-a-bug
issue/regression
issue/stale
issue/workaround
lgtm/need 2
modifies/api
modifies/translation
outdated/backport/v1.18
outdated/theme/markdown
outdated/theme/timetracker
performance/bigrepo
performance/cpu
performance/memory
performance/speed
pr/breaking
proposal/accepted
proposal/rejected
pr/wip
pull-request
Mirrored from GitHub Pull Request
 reviewed/wontfix
💰 Rewarded
skip-changelog
status/blocked
topic/accessibility
topic/api
topic/authentication
topic/build
topic/code-linting
topic/commit-signing
topic/content-rendering
topic/deployment
topic/distribution
topic/federation
topic/gitea-actions
topic/issues
topic/lfs
topic/mobile
topic/moderation
topic/packages
topic/pr
topic/projects
topic/repo
topic/repo-migration
topic/security
topic/theme
topic/ui
topic/ui-interaction
topic/ux
topic/webhooks
topic/wiki
type/bug
type/deprecation
type/docs
type/enhancement
type/feature
type/miscellaneous
type/proposal
type/question
type/refactoring
type/summary
type/testing
type/upstream
No labels proposal/accepted topic/packages type/feature
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
GiteaMirror ninjasurge (Kaehlen Rog)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/gitea#9774