mirror of
https://github.com/go-gitea/gitea.git
synced 2026-03-25 07:56:03 -05:00
Container registry - layer drops #8899
Closed
opened 2025-11-02 08:22:27 -06:00 by GiteaMirror
·
26 comments
No Branch/Tag Specified
main
release/v1.25
release/v1.24
release/v1.23
release/v1.22
release/v1.21
release/v1.20
release/v1.19
release/v1.18
release/v1.17
release/v1.16
release/v1.15
release/v1.14
release/v1.13
release/v1.12
release/v1.11
release/v1.10
release/v1.9
release/v1.8
v1.25.3
v1.25.2
v1.25.1
v1.25.0
v1.24.7
v1.25.0-rc0
v1.26.0-dev
v1.24.6
v1.24.5
v1.24.4
v1.24.3
v1.24.2
v1.24.1
v1.24.0
v1.23.8
v1.24.0-rc0
v1.25.0-dev
v1.23.7
v1.23.6
v1.23.5
v1.23.4
v1.23.3
v1.23.2
v1.23.1
v1.23.0
v1.23.0-rc0
v1.24.0-dev
v1.22.6
v1.22.5
v1.22.4
v1.22.3
v1.22.2
v1.22.1
v1.22.0
v1.23.0-dev
v1.22.0-rc1
v1.21.11
v1.22.0-rc0
v1.21.10
v1.21.9
v1.21.8
v1.21.7
v1.21.6
v1.21.5
v1.21.4
v1.21.3
v1.21.2
v1.20.6
v1.21.1
v1.21.0
v1.21.0-rc2
v1.21.0-rc1
v1.20.5
v1.22.0-dev
v1.21.0-rc0
v1.20.4
v1.20.3
v1.20.2
v1.20.1
v1.20.0
v1.19.4
v1.21.0-dev
v1.20.0-rc2
v1.20.0-rc1
v1.20.0-rc0
v1.19.3
v1.19.2
v1.19.1
v1.19.0
v1.19.0-rc1
v1.20.0-dev
v1.19.0-rc0
v1.18.5
v1.18.4
v1.18.3
v1.18.2
v1.18.1
v1.18.0
v1.17.4
v1.18.0-rc1
v1.19.0-dev
v1.18.0-rc0
v1.17.3
v1.17.2
v1.17.1
v1.17.0
v1.17.0-rc2
v1.16.9
v1.17.0-rc1
v1.18.0-dev
v1.16.8
v1.16.7
v1.16.6
v1.16.5
v1.16.4
v1.16.3
v1.16.2
v1.16.1
v1.16.0
v1.15.11
v1.17.0-dev
v1.16.0-rc1
v1.15.10
v1.15.9
v1.15.8
v1.15.7
v1.15.6
v1.15.5
v1.15.4
v1.15.3
v1.15.2
v1.15.1
v1.14.7
v1.15.0
v1.15.0-rc3
v1.14.6
v1.15.0-rc2
v1.14.5
v1.16.0-dev
v1.15.0-rc1
v1.14.4
v1.14.3
v1.14.2
v1.14.1
v1.14.0
v1.13.7
v1.14.0-rc2
v1.13.6
v1.13.5
v1.14.0-rc1
v1.15.0-dev
v1.13.4
v1.13.3
v1.13.2
v1.13.1
v1.13.0
v1.12.6
v1.13.0-rc2
v1.14.0-dev
v1.13.0-rc1
v1.12.5
v1.12.4
v1.12.3
v1.12.2
v1.12.1
v1.11.8
v1.12.0
v1.11.7
v1.12.0-rc2
v1.11.6
v1.12.0-rc1
v1.13.0-dev
v1.11.5
v1.11.4
v1.11.3
v1.10.6
v1.12.0-dev
v1.11.2
v1.10.5
v1.11.1
v1.10.4
v1.11.0
v1.11.0-rc2
v1.10.3
v1.11.0-rc1
v1.10.2
v1.10.1
v1.10.0
v1.9.6
v1.9.5
v1.10.0-rc2
v1.11.0-dev
v1.10.0-rc1
v1.9.4
v1.9.3
v1.9.2
v1.9.1
v1.9.0
v1.9.0-rc2
v1.10.0-dev
v1.9.0-rc1
v1.8.3
v1.8.2
v1.8.1
v1.8.0
v1.8.0-rc3
v1.7.6
v1.8.0-rc2
v1.7.5
v1.8.0-rc1
v1.9.0-dev
v1.7.4
v1.7.3
v1.7.2
v1.7.1
v1.7.0
v1.7.0-rc3
v1.6.4
v1.7.0-rc2
v1.6.3
v1.7.0-rc1
v1.7.0-dev
v1.6.2
v1.6.1
v1.6.0
v1.6.0-rc2
v1.5.3
v1.6.0-rc1
v1.6.0-dev
v1.5.2
v1.5.1
v1.5.0
v1.5.0-rc2
v1.5.0-rc1
v1.5.0-dev
v1.4.3
v1.4.2
v1.4.1
v1.4.0
v1.4.0-rc3
v1.4.0-rc2
v1.3.3
v1.4.0-rc1
v1.3.2
v1.3.1
v1.3.0
v1.3.0-rc2
v1.3.0-rc1
v1.2.3
v1.2.2
v1.2.1
v1.2.0
v1.2.0-rc3
v1.2.0-rc2
v1.1.4
v1.2.0-rc1
v1.1.3
v1.1.2
v1.1.1
v1.1.0
v1.0.2
v1.0.1
v1.0.0
v0.9.99
Labels
Clear labels
$20
$250
$50
$500
backport/done
💎 Bounty
docs-update-needed
good first issue
hacktoberfest
issue/bounty
issue/confirmed
issue/critical
issue/duplicate
issue/needs-feedback
issue/not-a-bug
issue/regression
issue/stale
issue/workaround
lgtm/need 2
modifies/api
modifies/translation
outdated/backport/v1.18
outdated/theme/markdown
outdated/theme/timetracker
performance/bigrepo
performance/cpu
performance/memory
performance/speed
pr/breaking
proposal/accepted
proposal/rejected
pr/wip
pull-request
reviewed/wontfix
💰 Rewarded
skip-changelog
status/blocked
topic/accessibility
topic/api
topic/authentication
topic/build
topic/code-linting
topic/commit-signing
topic/content-rendering
topic/deployment
topic/distribution
topic/federation
topic/gitea-actions
topic/issues
topic/lfs
topic/mobile
topic/moderation
topic/packages
topic/pr
topic/projects
topic/repo
topic/repo-migration
topic/security
topic/theme
topic/ui
topic/ui-interaction
topic/ux
topic/webhooks
topic/wiki
type/bug
type/deprecation
type/docs
type/enhancement
type/feature
type/miscellaneous
type/proposal
type/question
type/refactoring
type/summary
type/testing
type/upstream
Mirrored from GitHub Pull Request
Milestone
No items
No Milestone
Projects
Clear projects
No project
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: github-starred/gitea#8899
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @PavelMXFox on GitHub (May 2, 2022).
Description
Hi!
The layer
4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1is constantly disappearing, it contains 1024 zero-bytes. The 'gitea/packages/4f/4f' folder remains empty. If I upload an archived version of a layer, everything works until any image containing this layer is deleted, and after that the layer disappears. At this time in the gitea logsI attach a copy of the layer.
Gitea Version
1.17.0+dev-511-g71bafa026
Can you reproduce the bug on the Gitea demo site?
No
Log Gist
No response
Screenshots
No response
Git Version
No response
Operating System
No response
How are you running Gitea?
in docker
Database
MySQL
@PavelMXFox commented on GitHub (May 2, 2022):
Update - this repeated with other blobs, when pushing other image, contains this blob. Blob's file was removed from FS, but in DB table package_blob row with it presents, and because it blob cant'be re-uploaded with docker push
@PavelMXFox commented on GitHub (May 12, 2022):
Can't reproduce on [1.17.0+dev-568-gcbd45471b] it seems to be fixed.
@PavelMXFox commented on GitHub (May 18, 2022):
Gitea [1.17.0+dev-587-g9ea920640]
Image layer has been dropeed by registry.
Directory
packages/2f/f2is empty.if i restore
2ff2e940e73e30a29749f6cdec49203aa1d1a94b30b59988ce6288e14b836fcdfile from backup - all work fine.It was when i build && push image with
docker buildxDocker's log
Gitea log
@PavelMXFox commented on GitHub (May 18, 2022):
I see series of HTTP500, usually after some layers drop. Now I can't find if something layers are missing, maybe it's a new bug. Just in case, I attach a full push log.
#13 3.925 error: failed commit on ref "layer-sha256:fe18631e174dd7b4a8a9da2f7a7f1f3fb8f40a658bda2c3934612b08d01a1aac": unexpected status: 500 Internal Server Error@lunny commented on GitHub (May 19, 2022):
Could you enable dev mode in app.ini so that more detail could be displayed.
@wxiaoguang commented on GitHub (May 19, 2022):
The problem is caused by concurrency requests to database.
In TryInsertPackage/GetOrInsertVersion: Get -> Insert.
Then
insertcan triggerError 1062: Duplicate entry '10-_upload' for key 'UQE_package_version_s'error.@PavelMXFox commented on GitHub (May 19, 2022):
Ok
@PavelMXFox commented on GitHub (May 21, 2022):
Hi! With mode=dev.
This appears every time while building cross-platform images with
docker buildx. In this build i can't find if any layes drop yet.Builder log:
Gitea log:
@ChromoX commented on GitHub (Jul 12, 2022):
I'm experiencing this problem on v1.17.0-rc1, running on Kubernetes.
I have this symptom: https://github.com/go-gitea/gitea/issues/19586#issuecomment-1131194984
@markrexwinkel commented on GitHub (Jul 18, 2022):
I also experienced this issue with v1.17.0-rc1. When configuring Docker or Podman to use just one thread it works fine, as soon as you configure more parallel threads to push the container layers, error 500 appears. So it seems like a concurrency issue.
@Taha-Firoz commented on GitHub (Sep 1, 2022):
nope not a concurrency issue, just tried this,
max_concurrent_uploads: 1still leads to 500 error@zeripath commented on GitHub (Sep 4, 2022):
I think this might solve the issue but I'm not sure if it's gonna cause the whole transaction to fail instead.
Another option is to parse the error returned back.
@zeripath commented on GitHub (Sep 4, 2022):
It IS a concurrency issue - just an internal concurrency issue to do with transactional isolation etc.
The
TryInsert*functions are incorrect depending on the transactional isolation level within the database.@Taha-Firoz commented on GitHub (Sep 5, 2022):
But why won't any subsequent pushes work? I've tried many times and basically every second or third image pushed has a dropped layer in it. The database reports the layer exists but the layer doesn't exist in the FS, shouldn't there be some amount of validation.
@davidhiendl commented on GitHub (Sep 9, 2022):
I've experienced this problem when testing a new Gitea instance in different configurations:
The result appears to be the same: Image pushes mostly fail, retrying them sometimes works. This occurred for a variety of public and private images.
Is there a workaround for this issue available? Would building Gitea + the TryInsert PR help?
@fcwoknhenuxdfiyv commented on GitHub (Sep 29, 2022):
It seems some layers are being silently dropped when pushing. When pulling a recently pushed image it kept getting stuck retrying one particular layer.
I checked the contents of the named volume and there is nothing in the
bl/ob/blobhashtree. Not even thebldirectory exists.I did a push to an instance of the official Docker
registry:2and then copied the layer tobl/ob/blobhashand pulled successfully from GT.@PavelMXFox commented on GitHub (Sep 29, 2022):
Hi, I did the same - recover layer file from backup. Image pull become sucessfull but after some pushes with this layer - it drops again.
@Taha-Firoz commented on GitHub (Sep 29, 2022):
I'm not sure but it's probably because it doesn't actually check the fs but just checks the database if this layer was ever added. If the database has it recorded it assumes it exists. There really should be a fs check added in their too along with the database one to make sure this problem doesn't come up again.
@KN4CK3R commented on GitHub (Nov 16, 2022):
Copy from https://codeberg.org/woodpecker-plugins/plugin-docker-buildx/issues/42
--
What I think the problem with two uploads uploading the same file is:
Now we have valid database entries from Upload 2 but the blob in the file storage is missing. Every upload afterwards is skipped because the deduplication checks if the blob hash is already present. This does not check the file storage layer because there should be never a missing blob. So a reupload is only possible after all packages which reference this blob are deleted and the registry clean up job removes the unreferenced blob entries.
@Fogapod commented on GitHub (Dec 9, 2022):
I've got this issue again today: https://github.com/go-gitea/gitea/issues/21736
Gitea 1.17.3, local installation, postgres.
gitea[123269]: 2022/12/09 06:23:46 ...ntainer/container.go:84:apiError() [E] [6392d472] open /var/lib/gitea/data/packages/c5/1d/c51da6ab853721f149e9dea36102d7aabe02e0958ad199661066c851e391ede2: no such file or directoryWhat are the steps to fix this? Last time I had to write a script to delete all docker packages in instance because web ui on 1.17 only allows deleting a single version. Is there a way to tell which version hash
c51da6ab853721f149e9dea36102d7aabe02e0958ad199661066c851e391ede2corresponds to?Luckily there were only a few versions and I could delete them by hand then run package cleanup. But if other packages referenced same has it would be much harder to fix@Fogapod commented on GitHub (Dec 9, 2022):
Deleting blob from postgres table as suggested in my issue did work after package was pushed again:
@Fogapod commented on GitHub (Dec 9, 2022):
I don't think this was caused by concurrent pushes. I am running a small instance with only a few docker containers. All containers are pushed by drone.
Corrupted repository has this setting in release pipeline:
and a few steps before / after docker push. Also none other docker builds were running at that time in drone
@KN4CK3R commented on GitHub (Dec 9, 2022):
is a pipeline setting and not a docker setting.
@Fogapod commented on GitHub (Dec 9, 2022):
it is. my point is that it could never push concurrently. and after checking all runs around that time there were gaps in 10/20 mins between any docker pushes
@johnmarksilly commented on GitHub (Apr 13, 2023):
@KN4CK3R What is the fix for this? I get this all the time and on push, some layers show a retry at least once.
@KN4CK3R commented on GitHub (Apr 13, 2023):
This is fixed since Gitea 1.17.4 (#21862)