github-starred/gitea
2
0
Fork 0
mirror of https://github.com/go-gitea/gitea.git synced 2026-03-21 14:06:26 -05:00
Code Issues 2.6k Packages Projects Releases 100 Wiki Activity

[BUG] Ongoing problems with Docker-Image Repo #13995

New Issue
Closed
opened 2025-11-02 10:59:32 -06:00 by GiteaMirror · 10 comments
GiteaMirror commented 2025-11-02 10:59:32 -06:00
Owner
Copy Link

Originally created by @the-hotmann on GitHub (Jan 15, 2025).

Description

Related Issues/PRs:

@wxiaoguang thanks for your fix here: https://github.com/go-gitea/gitea/issues/33010#issuecomment-2572760404
But apparently I can not test it as I already am on "nightly", which is basically "1.24-nightly" and therefore newer than "1.23-nightly".

Here some info about my situation:

  1. I use the current nightly build 1.24.0+dev-179-gf6dbf0e7b3
  2. I use Debian Testing as Host

This is my docker-build command: (censored)

docker buildx build -f Dockerfile --pull --push --provenance=false \
    \
    --build-arg ALPINE_TAG="$TAG" \
    --platform linux/amd64/v1,linux/amd64/v2,linux/amd64/v3,linux/arm64/v8 \
    -t sub.domain.tld/repo/image:v${VERSION} \
    -t sub.domain.tld/repo/image \
    .

This is the error I get:

 > exporting to image:
3.174 error: failed commit on ref "layer-sha256:bba02a3b73b4653e6165c781254a34247ad12dcebc38e638a8b7606ddb719fab": unexpected status from PUT request to https://sub.domain.tld/v2/repo/image/blobs/uploads/n5iojjlsxfxsnobkf5g9gaae3?digest=sha256%3Abba02a3b73b4653e6165c781254a34247ad12dcebc38e638a8b7606ddb719fab: 500 Internal Server Error
3.174 retrying in 1s
5.005 error: failed commit on ref "layer-sha256:bba02a3b73b4653e6165c781254a34247ad12dcebc38e638a8b7606ddb719fab": unexpected status from PUT request to https://sub.domain.tld/v2/repo/image/blobs/uploads/b5xta9fiofskiwdd0yy2nalna?digest=sha256%3Abba02a3b73b4653e6165c781254a34247ad12dcebc38e638a8b7606ddb719fab: 500 Internal Server Error
5.005 retrying in 2s
7.815 error: failed commit on ref "layer-sha256:bba02a3b73b4653e6165c781254a34247ad12dcebc38e638a8b7606ddb719fab": unexpected status from PUT request to https://sub.domain.tld/v2/repo/image/blobs/uploads/fvt2wi7tsrbtbpsmtv3dohkjn?digest=sha256%3Abba02a3b73b4653e6165c781254a34247ad12dcebc38e638a8b7606ddb719fab: 500 Internal Server Error
7.815 retrying in 4s
12.61 error: failed commit on ref "layer-sha256:bba02a3b73b4653e6165c781254a34247ad12dcebc38e638a8b7606ddb719fab": unexpected status from PUT request to https://sub.domain.tld/v2/repo/image/blobs/uploads/6emxck8jbooxwgju3vvatlruo?digest=sha256%3Abba02a3b73b4653e6165c781254a34247ad12dcebc38e638a8b7606ddb719fab: 500 Internal Server Error

First when I got this error, I navigated to https://sub.domain.tld/-/admin/packages and deleted the unreferenced packages. Then the upload worked again. I have to do this all 4-5 times I PUSH to the repo and this is very annoying, but since I use Gitea for my own, this is not the biggest problem.

Genenrally speaking Gitea is the coolest, cleanest, resource-saving Repo-Manager, with at the same time, the worst Docker-Image Repo I have ever tested. Since I set it up about half a year ago I just had trouble with it and the Docker-Image Repo was not working stable out of the box.

I would appreciate a fix for

Gitea Version

1.24.0+dev-179-gf6dbf0e7b3

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

No response

Operating System

Debian Testing (12.9)

How are you running Gitea?

I run Gitea in a selfhosted docker-container.

Database

PostgreSQL

Originally created by @the-hotmann on GitHub (Jan 15, 2025). ### Description Related Issues/PRs: - https://github.com/go-gitea/gitea/issues/33010 - https://github.com/go-gitea/gitea/issues/31215 - https://github.com/go-gitea/gitea/issues/29563 - https://github.com/go-gitea/gitea/issues/28973 - https://github.com/go-gitea/gitea/pull/33053 - https://github.com/go-gitea/gitea/pull/32022 @wxiaoguang thanks for your fix here: https://github.com/go-gitea/gitea/issues/33010#issuecomment-2572760404 But apparently I can not test it as I already am on "nightly", which is basically "1.24-nightly" and therefore newer than "1.23-nightly". Here some info about my situation: 1. I use the current nightly build `1.24.0+dev-179-gf6dbf0e7b3` 2. I use Debian Testing as Host This is my docker-build command: (censored) ``` docker buildx build -f Dockerfile --pull --push --provenance=false \ \ --build-arg ALPINE_TAG="$TAG" \ --platform linux/amd64/v1,linux/amd64/v2,linux/amd64/v3,linux/arm64/v8 \ -t sub.domain.tld/repo/image:v${VERSION} \ -t sub.domain.tld/repo/image \ . ``` This is the error I get: ``` > exporting to image: 3.174 error: failed commit on ref "layer-sha256:bba02a3b73b4653e6165c781254a34247ad12dcebc38e638a8b7606ddb719fab": unexpected status from PUT request to https://sub.domain.tld/v2/repo/image/blobs/uploads/n5iojjlsxfxsnobkf5g9gaae3?digest=sha256%3Abba02a3b73b4653e6165c781254a34247ad12dcebc38e638a8b7606ddb719fab: 500 Internal Server Error 3.174 retrying in 1s 5.005 error: failed commit on ref "layer-sha256:bba02a3b73b4653e6165c781254a34247ad12dcebc38e638a8b7606ddb719fab": unexpected status from PUT request to https://sub.domain.tld/v2/repo/image/blobs/uploads/b5xta9fiofskiwdd0yy2nalna?digest=sha256%3Abba02a3b73b4653e6165c781254a34247ad12dcebc38e638a8b7606ddb719fab: 500 Internal Server Error 5.005 retrying in 2s 7.815 error: failed commit on ref "layer-sha256:bba02a3b73b4653e6165c781254a34247ad12dcebc38e638a8b7606ddb719fab": unexpected status from PUT request to https://sub.domain.tld/v2/repo/image/blobs/uploads/fvt2wi7tsrbtbpsmtv3dohkjn?digest=sha256%3Abba02a3b73b4653e6165c781254a34247ad12dcebc38e638a8b7606ddb719fab: 500 Internal Server Error 7.815 retrying in 4s 12.61 error: failed commit on ref "layer-sha256:bba02a3b73b4653e6165c781254a34247ad12dcebc38e638a8b7606ddb719fab": unexpected status from PUT request to https://sub.domain.tld/v2/repo/image/blobs/uploads/6emxck8jbooxwgju3vvatlruo?digest=sha256%3Abba02a3b73b4653e6165c781254a34247ad12dcebc38e638a8b7606ddb719fab: 500 Internal Server Error ``` First when I got this error, I navigated to `https://sub.domain.tld/-/admin/packages` and deleted the unreferenced packages. Then the upload worked again. I have to do this all 4-5 times I PUSH to the repo and this is very annoying, but since I use Gitea for my own, this is not the biggest problem. Genenrally speaking Gitea is the coolest, cleanest, resource-saving Repo-Manager, with at the same time, the worst Docker-Image Repo I have ever tested. Since I set it up about half a year ago I just had trouble with it and the Docker-Image Repo was not working stable out of the box. I would appreciate a fix for ### Gitea Version 1.24.0+dev-179-gf6dbf0e7b3 ### Can you reproduce the bug on the Gitea demo site? No ### Log Gist _No response_ ### Screenshots _No response_ ### Git Version _No response_ ### Operating System Debian Testing (12.9) ### How are you running Gitea? I run Gitea in a selfhosted docker-container. ### Database PostgreSQL
GiteaMirror added the type/bug label 2025-11-02 10:59:32 -06:00
GiteaMirror commented 2025-11-02 10:59:34 -06:00
Author
Owner
Copy Link

@the-hotmann commented on GitHub (Jan 15, 2025):

I also figured:

When I PUSH very small layers (test-files in a seperate layer) it takes way longer to run into this problem.

@the-hotmann commented on GitHub (Jan 15, 2025): I also figured: When I PUSH very small layers (test-files in a seperate layer) it takes way longer to run into this problem.
GiteaMirror commented 2025-11-02 10:59:34 -06:00
Author
Owner
Copy Link

@the-hotmann commented on GitHub (Jan 15, 2025):

I also noticed, that if I push to: sub.domain.tld/repo/image and get the error unexpected status from PUT request. I can just delete the WHOLE "image" from Gitea and push again, then it works.

The problem here: deleting an Image/Package must be done seperately for EVERY layer .. which is unconventional. Beeing able to delete (and directly purge) a package should be something that makes handeling Docker-Images in Gitea much better.
(but this could be a seperate Feature-Request)

@the-hotmann commented on GitHub (Jan 15, 2025): I also noticed, that if I push to: `sub.domain.tld/repo/image` and get the error `unexpected status from PUT request`. I can just delete the WHOLE "`image`" from Gitea and push again, then it works. The problem here: deleting an Image/Package must be done seperately for EVERY layer .. which is unconventional. Beeing able to delete (and directly purge) a package should be something that makes handeling Docker-Images in Gitea much better. (but this could be a seperate Feature-Request)
GiteaMirror commented 2025-11-02 10:59:34 -06:00
Author
Owner
Copy Link

@wxiaoguang commented on GitHub (Jan 15, 2025):

@wxiaoguang thanks for your fix here: #33010 (comment)

That fix is in 1.23.1 and main-nightly (1.24) now.

Other problems might be new ones (there are still many TODOs in the container package registry)

@wxiaoguang commented on GitHub (Jan 15, 2025): > @wxiaoguang thanks for your fix here: [#33010 (comment)](https://github.com/go-gitea/gitea/issues/33010#issuecomment-2572760404) That fix is in 1.23.1 and main-nightly (1.24) now. Other problems might be new ones (there are still many TODOs in the container package registry)
GiteaMirror commented 2025-11-02 10:59:34 -06:00
Author
Owner
Copy Link

@the-hotmann commented on GitHub (Jan 15, 2025):

Since I am running on nightly I assume I already have the fix?

@the-hotmann commented on GitHub (Jan 15, 2025): Since I am running on `nightly` I assume I already have the fix?
GiteaMirror commented 2025-11-02 10:59:35 -06:00
Author
Owner
Copy Link

@wxiaoguang commented on GitHub (Jan 15, 2025):

Since I am running on nightly I assume I already have the fix?

nightly are built from different commits, so it depends on which nightly you are using, you can check it by the version if it is an official build.

I use the current nightly build 1.24.0+dev-179-gf6dbf0e7b3

f6dbf0e7b3 is the latest nightly, so it includes that fix.

And if you see 500 errors, usually there should be backend logs, please provide the Gitea's logs, it would help a lot.

@wxiaoguang commented on GitHub (Jan 15, 2025): > Since I am running on `nightly` I assume I already have the fix? `nightly` are built from different commits, so it depends on which `nightly` you are using, you can check it by the version if it is an official build. > I use the current nightly build 1.24.0+dev-179-gf6dbf0e7b3 `f6dbf0e7b3` is the latest nightly, so it includes that fix. ---- And if you see 500 errors, usually there should be backend logs, please provide the Gitea's logs, it would help a lot.
GiteaMirror commented 2025-11-02 10:59:35 -06:00
Author
Owner
Copy Link

@the-hotmann commented on GitHub (Jan 15, 2025):

If you help me where I can find them, I will provide them.
I assumed they are in /data/gitea/log/, but there is nothing:

gitea:/data/gitea# ls -l
total 16
drwxr-xr-x    2 git      git              6 Nov 26 10:28 actions_artifacts
drwxr-xr-x    2 git      git              6 Nov 26 10:28 actions_log
drwxr-xr-x    2 git      git              6 Nov 26 10:28 attachments
drwxr-xr-x    3 git      git           4096 Nov 29 23:04 avatars
drwxr-xr-x    2 git      git             21 Nov 26 08:23 conf
drwxr-xr-x    2 git      git             24 Nov 26 10:28 home
drwx------    3 git      git             26 Jan 15 01:17 indexers
drwxr-xr-x    2 git      git             25 Nov 26 10:28 jwt
drwxr-xr-x    2 git      git              6 Nov 26 08:23 log
drwxr-xr-x  259 git      git           8192 Dec 11 23:05 packages
drwxr-xr-x    3 git      git             20 Nov 26 10:28 queues
drwxr-xr-x    2 git      git              6 Nov 26 10:28 repo-archive
drwxr-xr-x    2 git      git              6 Nov 26 10:28 repo-avatars
drwx------   18 git      git            150 Nov 29 22:58 sessions
drwxr-xr-x    3 git      git             28 Dec 14 20:45 tmp
gitea:/data/gitea# ls -la log/
total 4
drwxr-xr-x    2 git      git              6 Nov 26 08:23 .
drwxr-xr-x   17 git      git           4096 Nov 26 10:28 ..
gitea:/data/gitea#

I run giteea with this compose:

  gitea:
    image: gitea/gitea:nightly
    container_name: gitea
    hostname: gitea
    environment:
      USER_UID: "850"
      USER_GID: "850"
      GITEA__database__DB_TYPE: "postgres"
      GITEA__database__HOST: "postgres_17:5432"
      GITEA__database__NAME: "gitea"
      GITEA__database__USER: "gitea"
      GITEA__database__PASSWD: "CENSORED"
      DISABLE_REGISTRATION: true
    volumes:
      - "/var/docker/data/gitea/:/data/:rw"
      - "/etc/timezone:/etc/timezone:ro"
      - "/etc/localtime:/etc/localtime:ro"
    ports:
      - "127.0.0.1:3000:3000"
#      - "222:22"
    depends_on:
      - postgres_17
    deploy:
      resources:
        limits:
          memory: 2G
    restart: unless-stopped

  postgres_17:
    image: postgres:17-alpine
    container_name: postgres_17
    hostname: postgres_17
    environment:
      POSTGRES_PASSWORD: "CENSORED"
      POSTGRES_USER: "admin"
    logging:
      driver: "none"
    volumes:
      - "/var/docker/data/postgres_17/:/var/lib/postgresql/data/:rw"
    deploy:
      resources:
        limits:
          memory: 1G
    restart: unless-stopped

Just noticed I forgot the healthcheck and dependency with healthcheck, but this is very likely not the problem.

@the-hotmann commented on GitHub (Jan 15, 2025): If you help me where I can find them, I will provide them. I assumed they are in `/data/gitea/log/`, but there is nothing: ``` gitea:/data/gitea# ls -l total 16 drwxr-xr-x 2 git git 6 Nov 26 10:28 actions_artifacts drwxr-xr-x 2 git git 6 Nov 26 10:28 actions_log drwxr-xr-x 2 git git 6 Nov 26 10:28 attachments drwxr-xr-x 3 git git 4096 Nov 29 23:04 avatars drwxr-xr-x 2 git git 21 Nov 26 08:23 conf drwxr-xr-x 2 git git 24 Nov 26 10:28 home drwx------ 3 git git 26 Jan 15 01:17 indexers drwxr-xr-x 2 git git 25 Nov 26 10:28 jwt drwxr-xr-x 2 git git 6 Nov 26 08:23 log drwxr-xr-x 259 git git 8192 Dec 11 23:05 packages drwxr-xr-x 3 git git 20 Nov 26 10:28 queues drwxr-xr-x 2 git git 6 Nov 26 10:28 repo-archive drwxr-xr-x 2 git git 6 Nov 26 10:28 repo-avatars drwx------ 18 git git 150 Nov 29 22:58 sessions drwxr-xr-x 3 git git 28 Dec 14 20:45 tmp gitea:/data/gitea# ls -la log/ total 4 drwxr-xr-x 2 git git 6 Nov 26 08:23 . drwxr-xr-x 17 git git 4096 Nov 26 10:28 .. gitea:/data/gitea# ``` I run giteea with this compose: ``` gitea: image: gitea/gitea:nightly container_name: gitea hostname: gitea environment: USER_UID: "850" USER_GID: "850" GITEA__database__DB_TYPE: "postgres" GITEA__database__HOST: "postgres_17:5432" GITEA__database__NAME: "gitea" GITEA__database__USER: "gitea" GITEA__database__PASSWD: "CENSORED" DISABLE_REGISTRATION: true volumes: - "/var/docker/data/gitea/:/data/:rw" - "/etc/timezone:/etc/timezone:ro" - "/etc/localtime:/etc/localtime:ro" ports: - "127.0.0.1:3000:3000" # - "222:22" depends_on: - postgres_17 deploy: resources: limits: memory: 2G restart: unless-stopped postgres_17: image: postgres:17-alpine container_name: postgres_17 hostname: postgres_17 environment: POSTGRES_PASSWORD: "CENSORED" POSTGRES_USER: "admin" logging: driver: "none" volumes: - "/var/docker/data/postgres_17/:/var/lib/postgresql/data/:rw" deploy: resources: limits: memory: 1G restart: unless-stopped ``` Just noticed I forgot the healthcheck and dependency with healthcheck, but this is very likely not the problem.
GiteaMirror commented 2025-11-02 10:59:35 -06:00
Author
Owner
Copy Link

@wxiaoguang commented on GitHub (Jan 15, 2025):

Use docker logs to collect logs

https://docs.gitea.com/help/support

@wxiaoguang commented on GitHub (Jan 15, 2025): Use `docker logs` to collect logs https://docs.gitea.com/help/support
GiteaMirror commented 2025-11-02 10:59:36 -06:00
Author
Owner
Copy Link

@the-hotmann commented on GitHub (Jan 15, 2025):

Indeed .. there was the hint.
I already checked these logs, but since they print so much (as the logs here are very verbose) I missed the small hints:

2025/01/15 16:33:00 ...ntainer/container.go:92:apiError() [E] write /data/gitea/tmp/package-upload/pbsgu8gap03bmz9zbzrkf60v0: no space left on device

I moved this docker-mount to another volume, now everything was able to be pushed.

I still have a last question:
How can I go back from nightly to stable?

Is there a chance of doing so? I was hoping the tool is also able to migrate down to lower versions, but this seems not to be the case.

Anyway thanks for your great help!

@the-hotmann commented on GitHub (Jan 15, 2025): Indeed .. there was the hint. I already checked these logs, but since they print so much (as the logs here are very verbose) I missed the small hints: ``` 2025/01/15 16:33:00 ...ntainer/container.go:92:apiError() [E] write /data/gitea/tmp/package-upload/pbsgu8gap03bmz9zbzrkf60v0: no space left on device ``` I moved this docker-mount to another volume, now everything was able to be pushed. I still have a last question: How can I go back from nightly to stable? Is there a chance of doing so? I was hoping the tool is also able to migrate down to lower versions, but this seems not to be the case. Anyway thanks for your great help!
GiteaMirror commented 2025-11-02 10:59:36 -06:00
Author
Owner
Copy Link

@wxiaoguang commented on GitHub (Jan 15, 2025):

How can I go back from nightly to stable?

At the moment, you could manually update the "version" table to the 1.23's version 312 and use a 1.23 release. I have checked that it should work, but do not do so in the future unless you have confirmed the database is safe to be used for an old release.

@wxiaoguang commented on GitHub (Jan 15, 2025): > How can I go back from nightly to stable? At the moment, you could manually update the "version" table to the 1.23's version `312` and use a 1.23 release. I have checked that it should work, but do not do so in the future unless you have confirmed the database is safe to be used for an old release.
GiteaMirror commented 2025-11-02 10:59:36 -06:00
Author
Owner
Copy Link

@the-hotmann commented on GitHub (Jan 15, 2025):

Ok thanks, I will do so.

@the-hotmann commented on GitHub (Jan 15, 2025): Ok thanks, I will do so.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
$20
$250
$50
$500
backport/done
💎 Bounty
docs-update-needed
good first issue
hacktoberfest
issue/bounty
issue/confirmed
issue/critical
issue/duplicate
issue/needs-feedback
issue/not-a-bug
issue/regression
issue/stale
issue/workaround
lgtm/need 2
modifies/api
modifies/translation
outdated/backport/v1.18
outdated/theme/markdown
outdated/theme/timetracker
performance/bigrepo
performance/cpu
performance/memory
performance/speed
pr/breaking
proposal/accepted
proposal/rejected
pr/wip
pull-request
Mirrored from GitHub Pull Request
 reviewed/wontfix
💰 Rewarded
skip-changelog
status/blocked
topic/accessibility
topic/api
topic/authentication
topic/build
topic/code-linting
topic/commit-signing
topic/content-rendering
topic/deployment
topic/distribution
topic/federation
topic/gitea-actions
topic/issues
topic/lfs
topic/mobile
topic/moderation
topic/packages
topic/pr
topic/projects
topic/repo
topic/repo-migration
topic/security
topic/theme
topic/ui
topic/ui-interaction
topic/ux
topic/webhooks
topic/wiki
type/bug
type/deprecation
type/docs
type/enhancement
type/feature
type/miscellaneous
type/proposal
type/question
type/refactoring
type/summary
type/testing
type/upstream
No Label type/bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/gitea#13995
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?