Docker push to registry gets 500 error for some layers #13178

Closed
opened 2025-11-02 10:33:46 -06:00 by GiteaMirror · 2 comments
Owner

Originally created by @junzhengca on GitHub (Jun 19, 2024).

Description

When pushing Docker images to container registry, I get 500 errors for some layers after automated retries.

5f70bf18a086: Layer already exists 
adfb1d834c14: Layer already exists 
245b4b04eefc: Layer already exists 
075c1f9ddfb2: Pushing [==================================================>]  2.995MB/2.995MB
d4fa65af1494: Waiting 
284b272e4c5d: Waiting 
aa49f4877180: Waiting 
1ef96cc3773b: Waiting 
0fbf9681f063: Waiting 
4b778def1fd6: Waiting 
6d7bfc74a78d: Waiting 
df87cf2cbe5a: Waiting 
c03b855bf7ce: Waiting 
5a578fe88ffc: Waiting 
ecbda56a4ecd: Waiting 
2ca469df2d7a: Waiting 
72c690143394: Waiting 
received unexpected HTTP status: 500 Internal Server Error

What I have tried:

  • Reducing concurrent pushes to 1 layer at a time: does nothing, still same error
  • Increasing Nginx reverse proxy body size: does nothing, Nginx doesn't report any errors, I am using the official reverse proxy configuration https://docs.gitea.com/administration/reverse-proxies with certbot for SSL.
  • Looking at Gitea logs doesn't raise any immidiate red flags
2024/06/19 21:26:43 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/<REDACTED_REGISTRY_PATH>/blobs/sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 for 173.35.35.243:0, 200 OK in 5.1ms @ container/container.go:486(container.HeadBlob)
2024/06/19 21:26:43 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/<REDACTED_REGISTRY_PATH>/blobs/sha256:c4f956cad6340013f20d5a3e6487b341ee703a1f2f6d6bccae93fffe7dd63854 for 173.35.35.243:0, 200 OK in 5.1ms @ container/container.go:486(container.HeadBlob)
2024/06/19 21:26:44 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/<REDACTED_REGISTRY_PATH>/blobs/sha256:f134d980ef08b059da2509fe2b1c86c755f951c5ecfedc76d46fb55a566779ae for 173.35.35.243:0, 200 OK in 10.6ms @ container/container.go:486(container.HeadBlob)
2024/06/19 21:26:44 ...eb/routing/logger.go:102:func1() [I] router: completed POST /v2/<REDACTED_REGISTRY_PATH>/blobs/uploads/ for 173.35.35.243:0, 202 Accepted in 21.0ms @ container/container.go:218(container.InitiateUploadBlob)
2024/06/19 21:26:49 ...eb/routing/logger.go:102:func1() [I] router: completed POST /v2/<REDACTED_REGISTRY_PATH>/blobs/uploads/ for 173.35.35.243:0, 202 Accepted in 20.0ms @ container/container.go:218(container.InitiateUploadBlob)
2024/06/19 21:27:00 ...eb/routing/logger.go:102:func1() [I] router: completed POST /v2/<REDACTED_REGISTRY_PATH>/blobs/uploads/ for 173.35.35.243:0, 202 Accepted in 16.1ms @ container/container.go:218(container.InitiateUploadBlob)
2024/06/19 21:27:16 ...eb/routing/logger.go:102:func1() [I] router: completed POST /v2/<REDACTED_REGISTRY_PATH>/blobs/uploads/ for 173.35.35.243:0, 202 Accepted in 14.0ms @ container/container.go:218(container.InitiateUploadBlob)

Gitea Version

1.22.0

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

Assuming this doesn't matter as I am running the official Docker image

Operating System

Debian GNU/Linux 11 (bullseye) aarch64

How are you running Gitea?

I am running Gitea using a Docker compose file

version: "3"

networks:
  gitea:
    external: false

services:
  server:
    image: gitea/gitea:1.22.0
    container_name: gitea
    environment:
      - USER_UID=1000
      - USER_GID=1000
    restart: always
    networks:
      - gitea
    volumes:
      - ./gitea:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "3000:3000"
      - "222:22"

Database

SQLite

Originally created by @junzhengca on GitHub (Jun 19, 2024). ### Description When pushing Docker images to container registry, I get 500 errors for some layers after automated retries. ``` 5f70bf18a086: Layer already exists adfb1d834c14: Layer already exists 245b4b04eefc: Layer already exists 075c1f9ddfb2: Pushing [==================================================>] 2.995MB/2.995MB d4fa65af1494: Waiting 284b272e4c5d: Waiting aa49f4877180: Waiting 1ef96cc3773b: Waiting 0fbf9681f063: Waiting 4b778def1fd6: Waiting 6d7bfc74a78d: Waiting df87cf2cbe5a: Waiting c03b855bf7ce: Waiting 5a578fe88ffc: Waiting ecbda56a4ecd: Waiting 2ca469df2d7a: Waiting 72c690143394: Waiting received unexpected HTTP status: 500 Internal Server Error ``` What I have tried: - Reducing concurrent pushes to 1 layer at a time: does nothing, still same error - Increasing Nginx reverse proxy body size: does nothing, Nginx doesn't report any errors, I am using the official reverse proxy configuration https://docs.gitea.com/administration/reverse-proxies with certbot for SSL. - Looking at Gitea logs doesn't raise any immidiate red flags ``` 2024/06/19 21:26:43 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/<REDACTED_REGISTRY_PATH>/blobs/sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 for 173.35.35.243:0, 200 OK in 5.1ms @ container/container.go:486(container.HeadBlob) 2024/06/19 21:26:43 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/<REDACTED_REGISTRY_PATH>/blobs/sha256:c4f956cad6340013f20d5a3e6487b341ee703a1f2f6d6bccae93fffe7dd63854 for 173.35.35.243:0, 200 OK in 5.1ms @ container/container.go:486(container.HeadBlob) 2024/06/19 21:26:44 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/<REDACTED_REGISTRY_PATH>/blobs/sha256:f134d980ef08b059da2509fe2b1c86c755f951c5ecfedc76d46fb55a566779ae for 173.35.35.243:0, 200 OK in 10.6ms @ container/container.go:486(container.HeadBlob) 2024/06/19 21:26:44 ...eb/routing/logger.go:102:func1() [I] router: completed POST /v2/<REDACTED_REGISTRY_PATH>/blobs/uploads/ for 173.35.35.243:0, 202 Accepted in 21.0ms @ container/container.go:218(container.InitiateUploadBlob) 2024/06/19 21:26:49 ...eb/routing/logger.go:102:func1() [I] router: completed POST /v2/<REDACTED_REGISTRY_PATH>/blobs/uploads/ for 173.35.35.243:0, 202 Accepted in 20.0ms @ container/container.go:218(container.InitiateUploadBlob) 2024/06/19 21:27:00 ...eb/routing/logger.go:102:func1() [I] router: completed POST /v2/<REDACTED_REGISTRY_PATH>/blobs/uploads/ for 173.35.35.243:0, 202 Accepted in 16.1ms @ container/container.go:218(container.InitiateUploadBlob) 2024/06/19 21:27:16 ...eb/routing/logger.go:102:func1() [I] router: completed POST /v2/<REDACTED_REGISTRY_PATH>/blobs/uploads/ for 173.35.35.243:0, 202 Accepted in 14.0ms @ container/container.go:218(container.InitiateUploadBlob) ``` ### Gitea Version 1.22.0 ### Can you reproduce the bug on the Gitea demo site? No ### Log Gist _No response_ ### Screenshots _No response_ ### Git Version Assuming this doesn't matter as I am running the official Docker image ### Operating System Debian GNU/Linux 11 (bullseye) aarch64 ### How are you running Gitea? I am running Gitea using a Docker compose file ```yaml version: "3" networks: gitea: external: false services: server: image: gitea/gitea:1.22.0 container_name: gitea environment: - USER_UID=1000 - USER_GID=1000 restart: always networks: - gitea volumes: - ./gitea:/data - /etc/timezone:/etc/timezone:ro - /etc/localtime:/etc/localtime:ro ports: - "3000:3000" - "222:22" ``` ### Database SQLite
GiteaMirror added the issue/needs-feedbacktype/bug labels 2025-11-02 10:33:46 -06:00
Author
Owner

@techknowlogick commented on GitHub (Jun 19, 2024):

The logs you have provided are solely for the router, are you able to grab all the logs? If Gitea is providing a 500 then it will be logged if the logs for non-router are also enabled.
You can find details onto how to enable those logs here: https://docs.gitea.com/help/support

@techknowlogick commented on GitHub (Jun 19, 2024): The logs you have provided are solely for the router, are you able to grab all the logs? If Gitea is providing a 500 then it will be logged if the logs for non-router are also enabled. You can find details onto how to enable those logs here: https://docs.gitea.com/help/support
Author
Owner

@GiteaBot commented on GitHub (Jul 21, 2024):

We close issues that need feedback from the author if there were no new comments for a month. 🍵

@GiteaBot commented on GitHub (Jul 21, 2024): We close issues that need feedback from the author if there were no new comments for a month. :tea:
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/gitea#13178