github-starred/gitea
2
0
Fork 0
mirror of https://github.com/go-gitea/gitea.git synced 2026-03-22 06:24:14 -05:00
Code Issues 2.6k Packages Projects Releases 100 Wiki Activity

Manifest push fails when using Gitea container registry with Docker BuildKit caching #12402

New Issue
Closed
opened 2025-11-02 10:08:43 -06:00 by GiteaMirror · 4 comments
GiteaMirror commented 2025-11-02 10:08:43 -06:00
Owner
Copy Link

Originally created by @julmb on GitHub (Jan 29, 2024).

Description

I am trying to use a Gitea container registry with Docker BuildKit caching (https://docs.docker.com/build/cache/). More specifically, I am trying to use the registry cache backend (https://docs.docker.com/build/cache/backends/registry/). Here, intermediate layers produced during a build are uploaded to the registry under a separate tag to be reused in subsequent builds.

I am using the following command:

docker buildx build --push --tag try.gitea.io/jules/suicune:latest --cache-to type=registry,ref=try.gitea.io/jules/suicune:build,mode=max .

This builds the image and successfully pushes it to the registry. It then starts pushing the cache layers to the registry. Pushing the layers works, but the final step of pushing the manifest fails:

#26 exporting cache to registry
#26 writing layer sha256:bab2cfbf1e9bee2409320e78b3cb9c3c2b9f2c54089bd89f42d011ad9bec699e 1.1s done
#26 writing layer sha256:c540ddb30ad1b73ba05d74a94bae43ced662b72ccca916592ab2037ae9d5cdd8
#26 writing layer sha256:c540ddb30ad1b73ba05d74a94bae43ced662b72ccca916592ab2037ae9d5cdd8 0.3s done
#26 writing layer sha256:ccb8642b226bb4d4492d41ea386268ca14457afe7eeafa621ae5dc701d39d01b
#26 writing layer sha256:ccb8642b226bb4d4492d41ea386268ca14457afe7eeafa621ae5dc701d39d01b 0.1s done
#26 writing layer sha256:dfab772f9f48504aee8bc3eed3f7f99a585e6e3d4c63d21a075521a2d796a90c
#26 writing layer sha256:dfab772f9f48504aee8bc3eed3f7f99a585e6e3d4c63d21a075521a2d796a90c 0.3s done
#26 writing layer sha256:e1a7258d27e602de318bdf87fdc38ef8c33c7c832728f727fad2a4db8b39300e
#26 writing layer sha256:e1a7258d27e602de318bdf87fdc38ef8c33c7c832728f727fad2a4db8b39300e 0.4s done
#26 writing layer sha256:e32bc7a16024c28d44887fdf36e51dea3448ffa1488fcc6f0892e49f53b0ab0c
#26 writing layer sha256:e32bc7a16024c28d44887fdf36e51dea3448ffa1488fcc6f0892e49f53b0ab0c 0.3s done
#26 writing layer sha256:ee12cf5233c9d50764aee90249dfbe32f3322a2d952963250fcf4c454858c5e3
#26 writing layer sha256:ee12cf5233c9d50764aee90249dfbe32f3322a2d952963250fcf4c454858c5e3 0.3s done
#26 writing layer sha256:f20bbea5142a083a99655706934104b8ee7664639d4c8ebc0ff403161e543ca2
#26 writing layer sha256:f20bbea5142a083a99655706934104b8ee7664639d4c8ebc0ff403161e543ca2 57.1s done
#26 ...

#29 [auth] *:: jules/suicune:pull,push token for try.gitea.io
#29 DONE 0.0s

#26 exporting cache to registry
#26 writing config sha256:daab72fe0a1a74867c48380ab59eae3d459e6c2e3b92227504bb991021adbd43
#26 writing config sha256:daab72fe0a1a74867c48380ab59eae3d459e6c2e3b92227504bb991021adbd43 1.0s done
#26 writing cache manifest sha256:f6ac364184a1cc56a1d30179846230533ec50f23e594596ac49f0f45e02b2bfc
#26 preparing build cache for export 220.1s done
#26 writing cache manifest sha256:f6ac364184a1cc56a1d30179846230533ec50f23e594596ac49f0f45e02b2bfc 0.2s done
#26 ERROR: error writing manifest blob: failed commit on ref "sha256:f6ac364184a1cc56a1d30179846230533ec50f23e594596ac49f0f45e02b2bfc": unexpected status from PUT request to https://try.gitea.io/v2/jules/suicune/manifests/build: 400 Bad Request

The Gitea logs (LEVEL = trace) show the following:

2024/01/29 08:00:27 ...s/process/manager.go:188:Add() [T] Start 65b75b1b-4: HEAD: /v2/jules/suicune/manifests/build (request)
2024/01/29 08:00:27 ...eb/routing/logger.go:47:func1() [T] router: started   HEAD /v2/jules/suicune/manifests/build for 172.25.0.1:33980
2024/01/29 08:00:27 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/jules/suicune/manifests/build for 172.25.0.1:33980, 404 Not Found in 0.9ms @ container/container.go:612(container.HeadManifest)
2024/01/29 08:00:27 ...s/process/manager.go:231:remove() [T] Done 65b75b1b-4: HEAD: /v2/jules/suicune/manifests/build
2024/01/29 08:00:27 ...s/process/manager.go:188:Add() [T] Start 65b75b1b-5: PUT: /v2/jules/suicune/manifests/build (request)
2024/01/29 08:00:27 ...eb/routing/logger.go:47:func1() [T] router: started   PUT /v2/jules/suicune/manifests/build for 172.25.0.1:33992
2024/01/29 08:00:27 ...eb/routing/logger.go:102:func1() [I] router: completed PUT /v2/jules/suicune/manifests/build for 172.25.0.1:33992, 400 Bad Request in 1.0ms @ container/container.go:527(container.UploadManifest)
2024/01/29 08:00:27 ...s/process/manager.go:231:remove() [T] Done 65b75b1b-5: PUT: /v2/jules/suicune/manifests/build

This happens both with my own instance and the one at https://try.gitea.io/. The exact same command works fine if Docker Hub is used as the registry.

I have looked at container/container.go:527 but the line numbers seem to be inaccurate so I cannot tell exactly where the 400 Bad Request comes from.

I have also come across https://forum.gitea.com/t/multiarch-docker-images-with-buildkit-and-cache-upload-fail/6213, which might be related.

Gitea Version

1.21.4

Can you reproduce the bug on the Gitea demo site?

Yes

Log Gist

No response

Screenshots

No response

Git Version

No response

Operating System

No response

How are you running Gitea?

I am running Gitea using the official gitea/gitea Docker image.

Database

None

Originally created by @julmb on GitHub (Jan 29, 2024). ### Description I am trying to use a Gitea container registry with Docker BuildKit caching (https://docs.docker.com/build/cache/). More specifically, I am trying to use the registry cache backend (https://docs.docker.com/build/cache/backends/registry/). Here, intermediate layers produced during a build are uploaded to the registry under a separate tag to be reused in subsequent builds. I am using the following command: ```sh docker buildx build --push --tag try.gitea.io/jules/suicune:latest --cache-to type=registry,ref=try.gitea.io/jules/suicune:build,mode=max . ``` This builds the image and successfully pushes it to the registry. It then starts pushing the cache layers to the registry. Pushing the layers works, but the final step of pushing the manifest fails: ```log #26 exporting cache to registry #26 writing layer sha256:bab2cfbf1e9bee2409320e78b3cb9c3c2b9f2c54089bd89f42d011ad9bec699e 1.1s done #26 writing layer sha256:c540ddb30ad1b73ba05d74a94bae43ced662b72ccca916592ab2037ae9d5cdd8 #26 writing layer sha256:c540ddb30ad1b73ba05d74a94bae43ced662b72ccca916592ab2037ae9d5cdd8 0.3s done #26 writing layer sha256:ccb8642b226bb4d4492d41ea386268ca14457afe7eeafa621ae5dc701d39d01b #26 writing layer sha256:ccb8642b226bb4d4492d41ea386268ca14457afe7eeafa621ae5dc701d39d01b 0.1s done #26 writing layer sha256:dfab772f9f48504aee8bc3eed3f7f99a585e6e3d4c63d21a075521a2d796a90c #26 writing layer sha256:dfab772f9f48504aee8bc3eed3f7f99a585e6e3d4c63d21a075521a2d796a90c 0.3s done #26 writing layer sha256:e1a7258d27e602de318bdf87fdc38ef8c33c7c832728f727fad2a4db8b39300e #26 writing layer sha256:e1a7258d27e602de318bdf87fdc38ef8c33c7c832728f727fad2a4db8b39300e 0.4s done #26 writing layer sha256:e32bc7a16024c28d44887fdf36e51dea3448ffa1488fcc6f0892e49f53b0ab0c #26 writing layer sha256:e32bc7a16024c28d44887fdf36e51dea3448ffa1488fcc6f0892e49f53b0ab0c 0.3s done #26 writing layer sha256:ee12cf5233c9d50764aee90249dfbe32f3322a2d952963250fcf4c454858c5e3 #26 writing layer sha256:ee12cf5233c9d50764aee90249dfbe32f3322a2d952963250fcf4c454858c5e3 0.3s done #26 writing layer sha256:f20bbea5142a083a99655706934104b8ee7664639d4c8ebc0ff403161e543ca2 #26 writing layer sha256:f20bbea5142a083a99655706934104b8ee7664639d4c8ebc0ff403161e543ca2 57.1s done #26 ... #29 [auth] *:: jules/suicune:pull,push token for try.gitea.io #29 DONE 0.0s #26 exporting cache to registry #26 writing config sha256:daab72fe0a1a74867c48380ab59eae3d459e6c2e3b92227504bb991021adbd43 #26 writing config sha256:daab72fe0a1a74867c48380ab59eae3d459e6c2e3b92227504bb991021adbd43 1.0s done #26 writing cache manifest sha256:f6ac364184a1cc56a1d30179846230533ec50f23e594596ac49f0f45e02b2bfc #26 preparing build cache for export 220.1s done #26 writing cache manifest sha256:f6ac364184a1cc56a1d30179846230533ec50f23e594596ac49f0f45e02b2bfc 0.2s done #26 ERROR: error writing manifest blob: failed commit on ref "sha256:f6ac364184a1cc56a1d30179846230533ec50f23e594596ac49f0f45e02b2bfc": unexpected status from PUT request to https://try.gitea.io/v2/jules/suicune/manifests/build: 400 Bad Request ``` The Gitea logs (LEVEL = trace) show the following: ```log 2024/01/29 08:00:27 ...s/process/manager.go:188:Add() [T] Start 65b75b1b-4: HEAD: /v2/jules/suicune/manifests/build (request) 2024/01/29 08:00:27 ...eb/routing/logger.go:47:func1() [T] router: started HEAD /v2/jules/suicune/manifests/build for 172.25.0.1:33980 2024/01/29 08:00:27 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/jules/suicune/manifests/build for 172.25.0.1:33980, 404 Not Found in 0.9ms @ container/container.go:612(container.HeadManifest) 2024/01/29 08:00:27 ...s/process/manager.go:231:remove() [T] Done 65b75b1b-4: HEAD: /v2/jules/suicune/manifests/build 2024/01/29 08:00:27 ...s/process/manager.go:188:Add() [T] Start 65b75b1b-5: PUT: /v2/jules/suicune/manifests/build (request) 2024/01/29 08:00:27 ...eb/routing/logger.go:47:func1() [T] router: started PUT /v2/jules/suicune/manifests/build for 172.25.0.1:33992 2024/01/29 08:00:27 ...eb/routing/logger.go:102:func1() [I] router: completed PUT /v2/jules/suicune/manifests/build for 172.25.0.1:33992, 400 Bad Request in 1.0ms @ container/container.go:527(container.UploadManifest) 2024/01/29 08:00:27 ...s/process/manager.go:231:remove() [T] Done 65b75b1b-5: PUT: /v2/jules/suicune/manifests/build ``` This happens both with my own instance and the one at https://try.gitea.io/. The exact same command works fine if Docker Hub is used as the registry. I have looked at `container/container.go:527` but the line numbers seem to be inaccurate so I cannot tell exactly where the `400 Bad Request` comes from. I have also come across https://forum.gitea.com/t/multiarch-docker-images-with-buildkit-and-cache-upload-fail/6213, which might be related. ### Gitea Version 1.21.4 ### Can you reproduce the bug on the Gitea demo site? Yes ### Log Gist _No response_ ### Screenshots _No response_ ### Git Version _No response_ ### Operating System _No response_ ### How are you running Gitea? I am running Gitea using the official `gitea/gitea` Docker image. ### Database None
GiteaMirror added the topic/packagestype/bug labels 2025-11-02 10:08:43 -06:00
GiteaMirror commented 2025-11-02 10:08:44 -06:00
Author
Owner
Copy Link

@bcbnz commented on GitHub (Feb 2, 2024):

I just hit the same problem trying to push Docker images from an Actions build with 1.21.5. The manifest push fails with buildx, but using the (deprecated) legacy build command without buildx installed works. This means you cannot use the docker/build-push-action in a workflow as it relies on buildx being available. The easy workaround is to call docker build and docker push:

    steps:
      - name: Check out repository
        uses: actions/checkout@v3

      - name: Log in to local registry
        uses: docker/login-action@v3
        with:
          registry: my.gitea.com
          username: ${{ gitea.repository_owner }}
          password: ${{ secrets.REGISTRY_TOKEN }} # gitea.token doesn't work for push, provide a personal access token

      - name: Build Docker image
        run: docker build -t my.gitea.com/user/img1 img1

      - name: Push Docker image
        run: docker push my.gitea.com/user/img1
@bcbnz commented on GitHub (Feb 2, 2024): I just hit the same problem trying to push Docker images from an Actions build with 1.21.5. The manifest push fails with buildx, but using the [(deprecated)](https://docs.docker.com/engine/deprecated/#legacy-builder-for-linux-images) legacy build command without buildx installed works. This means you cannot use the `docker/build-push-action` in a workflow as it relies on buildx being available. The easy workaround is to call `docker build` and `docker push`: ```yaml steps: - name: Check out repository uses: actions/checkout@v3 - name: Log in to local registry uses: docker/login-action@v3 with: registry: my.gitea.com username: ${{ gitea.repository_owner }} password: ${{ secrets.REGISTRY_TOKEN }} # gitea.token doesn't work for push, provide a personal access token - name: Build Docker image run: docker build -t my.gitea.com/user/img1 img1 - name: Push Docker image run: docker push my.gitea.com/user/img1 ```
GiteaMirror commented 2025-11-02 10:08:44 -06:00
Author
Owner
Copy Link

@julmb commented on GitHub (Feb 2, 2024):

I just hit the same problem trying to push Docker images from an Actions build with 1.21.5. The manifest push fails with buildx, but using the (deprecated) legacy build command without buildx installed works. This means you cannot use the docker/build-push-action in a workflow as it relies on buildx being available.

This is not the problem I am having. Both docker buildx build on the command line as well as the docker/build-push-action Github/Gitea action work just fine for me. The problem only occurs when using the --cache-to directive to push intermediate layers to a Gitea registry for caching.

@julmb commented on GitHub (Feb 2, 2024): > I just hit the same problem trying to push Docker images from an Actions build with 1.21.5. The manifest push fails with buildx, but using the [(deprecated)](https://docs.docker.com/engine/deprecated/#legacy-builder-for-linux-images) legacy build command without buildx installed works. This means you cannot use the `docker/build-push-action` in a workflow as it relies on buildx being available. This is not the problem I am having. Both `docker buildx build` on the command line as well as the `docker/build-push-action` Github/Gitea action work just fine for me. The problem only occurs when using the `--cache-to` directive to push intermediate layers to a Gitea registry for caching.
GiteaMirror commented 2025-11-02 10:08:44 -06:00
Author
Owner
Copy Link

@ItsJustRuby commented on GitHub (Feb 23, 2024):

@julmb Applying the fix described in https://stackoverflow.com/a/77766002 fixed this issue for me:

The solution was to add these options to --cache-to:
mode=max,image-manifest=true,oci-mediatypes=true,type=registry,ref=

@ItsJustRuby commented on GitHub (Feb 23, 2024): @julmb Applying the fix described in https://stackoverflow.com/a/77766002 fixed this issue for me: > The solution was to add these options to --cache-to: > mode=max,image-manifest=true,oci-mediatypes=true,type=registry,ref=
GiteaMirror commented 2025-11-02 10:08:44 -06:00
Author
Owner
Copy Link

@julmb commented on GitHub (Mar 17, 2024):

Thank you so much for this hint! It really works!

Apparently there is a lot of information on it out there that I somehow never came across:
https://github.com/moby/buildkit/issues/2251
https://github.com/moby/buildkit/pull/3724
https://gitlab.com/gitlab-org/container-registry/-/issues/407

From reading some of this, it sounds like the default behavior of BuildKit is to produce cache images that are not compliant with the OCI Image Spec. If this is the case, then I think the Gitea registry is correct to reject these. GitLab has apparently been accepting these noncompliant images due to a missing check and now that is causing issues.

The correct usage pattern here seems to be using the new image-manifest option to make BuildKit produce compliant cache images, which has solved my issue and allowed me to push cache images to the gitea registry.

Unless someone with more expertise on the subject wants to chime in, I would consider this resolved.

@julmb commented on GitHub (Mar 17, 2024): Thank you so much for this hint! It really works! Apparently there is a lot of information on it out there that I somehow never came across: https://github.com/moby/buildkit/issues/2251 https://github.com/moby/buildkit/pull/3724 https://gitlab.com/gitlab-org/container-registry/-/issues/407 From reading some of this, it sounds like the default behavior of BuildKit is to produce cache images that are not compliant with the OCI Image Spec. If this is the case, then I think the Gitea registry is correct to reject these. GitLab has apparently been accepting these noncompliant images due to a missing check and now that is causing issues. The correct usage pattern here seems to be using the new `image-manifest` option to make BuildKit produce compliant cache images, which has solved my issue and allowed me to push cache images to the gitea registry. Unless someone with more expertise on the subject wants to chime in, I would consider this resolved.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
$20
$250
$50
$500
backport/done
💎 Bounty
docs-update-needed
good first issue
hacktoberfest
issue/bounty
issue/confirmed
issue/critical
issue/duplicate
issue/needs-feedback
issue/not-a-bug
issue/regression
issue/stale
issue/workaround
lgtm/need 2
modifies/api
modifies/translation
outdated/backport/v1.18
outdated/theme/markdown
outdated/theme/timetracker
performance/bigrepo
performance/cpu
performance/memory
performance/speed
pr/breaking
proposal/accepted
proposal/rejected
pr/wip
pull-request
Mirrored from GitHub Pull Request
 reviewed/wontfix
💰 Rewarded
skip-changelog
status/blocked
topic/accessibility
topic/api
topic/authentication
topic/build
topic/code-linting
topic/commit-signing
topic/content-rendering
topic/deployment
topic/distribution
topic/federation
topic/gitea-actions
topic/issues
topic/lfs
topic/mobile
topic/moderation
topic/packages
topic/pr
topic/projects
topic/repo
topic/repo-migration
topic/security
topic/theme
topic/ui
topic/ui-interaction
topic/ux
topic/webhooks
topic/wiki
type/bug
type/deprecation
type/docs
type/enhancement
type/feature
type/miscellaneous
type/proposal
type/question
type/refactoring
type/summary
type/testing
type/upstream
No Label topic/packages type/bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/gitea#12402
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?