mirror of
https://github.com/go-gitea/gitea.git
synced 2026-03-12 02:24:21 -05:00
Docker images have certain layers stuck in retrying during push #14156
Closed
opened 2025-11-02 11:04:44 -06:00 by GiteaMirror
·
8 comments
No Branch/Tag Specified
main
release/v1.25
release/v1.24
release/v1.23
release/v1.22
release/v1.21
release/v1.20
release/v1.19
release/v1.18
release/v1.17
release/v1.16
release/v1.15
release/v1.14
release/v1.13
release/v1.12
release/v1.11
release/v1.10
release/v1.9
release/v1.8
v1.25.3
v1.25.2
v1.25.1
v1.25.0
v1.24.7
v1.25.0-rc0
v1.26.0-dev
v1.24.6
v1.24.5
v1.24.4
v1.24.3
v1.24.2
v1.24.1
v1.24.0
v1.23.8
v1.24.0-rc0
v1.25.0-dev
v1.23.7
v1.23.6
v1.23.5
v1.23.4
v1.23.3
v1.23.2
v1.23.1
v1.23.0
v1.23.0-rc0
v1.24.0-dev
v1.22.6
v1.22.5
v1.22.4
v1.22.3
v1.22.2
v1.22.1
v1.22.0
v1.23.0-dev
v1.22.0-rc1
v1.21.11
v1.22.0-rc0
v1.21.10
v1.21.9
v1.21.8
v1.21.7
v1.21.6
v1.21.5
v1.21.4
v1.21.3
v1.21.2
v1.20.6
v1.21.1
v1.21.0
v1.21.0-rc2
v1.21.0-rc1
v1.20.5
v1.22.0-dev
v1.21.0-rc0
v1.20.4
v1.20.3
v1.20.2
v1.20.1
v1.20.0
v1.19.4
v1.21.0-dev
v1.20.0-rc2
v1.20.0-rc1
v1.20.0-rc0
v1.19.3
v1.19.2
v1.19.1
v1.19.0
v1.19.0-rc1
v1.20.0-dev
v1.19.0-rc0
v1.18.5
v1.18.4
v1.18.3
v1.18.2
v1.18.1
v1.18.0
v1.17.4
v1.18.0-rc1
v1.19.0-dev
v1.18.0-rc0
v1.17.3
v1.17.2
v1.17.1
v1.17.0
v1.17.0-rc2
v1.16.9
v1.17.0-rc1
v1.18.0-dev
v1.16.8
v1.16.7
v1.16.6
v1.16.5
v1.16.4
v1.16.3
v1.16.2
v1.16.1
v1.16.0
v1.15.11
v1.17.0-dev
v1.16.0-rc1
v1.15.10
v1.15.9
v1.15.8
v1.15.7
v1.15.6
v1.15.5
v1.15.4
v1.15.3
v1.15.2
v1.15.1
v1.14.7
v1.15.0
v1.15.0-rc3
v1.14.6
v1.15.0-rc2
v1.14.5
v1.16.0-dev
v1.15.0-rc1
v1.14.4
v1.14.3
v1.14.2
v1.14.1
v1.14.0
v1.13.7
v1.14.0-rc2
v1.13.6
v1.13.5
v1.14.0-rc1
v1.15.0-dev
v1.13.4
v1.13.3
v1.13.2
v1.13.1
v1.13.0
v1.12.6
v1.13.0-rc2
v1.14.0-dev
v1.13.0-rc1
v1.12.5
v1.12.4
v1.12.3
v1.12.2
v1.12.1
v1.11.8
v1.12.0
v1.11.7
v1.12.0-rc2
v1.11.6
v1.12.0-rc1
v1.13.0-dev
v1.11.5
v1.11.4
v1.11.3
v1.10.6
v1.12.0-dev
v1.11.2
v1.10.5
v1.11.1
v1.10.4
v1.11.0
v1.11.0-rc2
v1.10.3
v1.11.0-rc1
v1.10.2
v1.10.1
v1.10.0
v1.9.6
v1.9.5
v1.10.0-rc2
v1.11.0-dev
v1.10.0-rc1
v1.9.4
v1.9.3
v1.9.2
v1.9.1
v1.9.0
v1.9.0-rc2
v1.10.0-dev
v1.9.0-rc1
v1.8.3
v1.8.2
v1.8.1
v1.8.0
v1.8.0-rc3
v1.7.6
v1.8.0-rc2
v1.7.5
v1.8.0-rc1
v1.9.0-dev
v1.7.4
v1.7.3
v1.7.2
v1.7.1
v1.7.0
v1.7.0-rc3
v1.6.4
v1.7.0-rc2
v1.6.3
v1.7.0-rc1
v1.7.0-dev
v1.6.2
v1.6.1
v1.6.0
v1.6.0-rc2
v1.5.3
v1.6.0-rc1
v1.6.0-dev
v1.5.2
v1.5.1
v1.5.0
v1.5.0-rc2
v1.5.0-rc1
v1.5.0-dev
v1.4.3
v1.4.2
v1.4.1
v1.4.0
v1.4.0-rc3
v1.4.0-rc2
v1.3.3
v1.4.0-rc1
v1.3.2
v1.3.1
v1.3.0
v1.3.0-rc2
v1.3.0-rc1
v1.2.3
v1.2.2
v1.2.1
v1.2.0
v1.2.0-rc3
v1.2.0-rc2
v1.1.4
v1.2.0-rc1
v1.1.3
v1.1.2
v1.1.1
v1.1.0
v1.0.2
v1.0.1
v1.0.0
v0.9.99
Labels
Clear labels
$20
$250
$50
$500
backport/done
💎 Bounty
docs-update-needed
good first issue
hacktoberfest
issue/bounty
issue/confirmed
issue/critical
issue/duplicate
issue/needs-feedback
issue/not-a-bug
issue/regression
issue/stale
issue/workaround
lgtm/need 2
modifies/api
modifies/translation
outdated/backport/v1.18
outdated/theme/markdown
outdated/theme/timetracker
performance/bigrepo
performance/cpu
performance/memory
performance/speed
pr/breaking
proposal/accepted
proposal/rejected
pr/wip
pull-request
reviewed/wontfix
💰 Rewarded
skip-changelog
status/blocked
topic/accessibility
topic/api
topic/authentication
topic/build
topic/code-linting
topic/commit-signing
topic/content-rendering
topic/deployment
topic/distribution
topic/federation
topic/gitea-actions
topic/issues
topic/lfs
topic/mobile
topic/moderation
topic/packages
topic/pr
topic/projects
topic/repo
topic/repo-migration
topic/security
topic/theme
topic/ui
topic/ui-interaction
topic/ux
topic/webhooks
topic/wiki
type/bug
type/deprecation
type/docs
type/enhancement
type/feature
type/miscellaneous
type/proposal
type/question
type/refactoring
type/summary
type/testing
type/upstream
Mirrored from GitHub Pull Request
No Label
type/bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: github-starred/gitea#14156
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @JustKato on GitHub (Feb 19, 2025).
Description
Description
The current issue I am facing is the fact that I can not use the container registry at all, I have tried all possible combinations and the results are always the same:
docker loginwith my authentication token with my username ( and _token username ) and when I push the image I always get the same response ( can be seen below at the logs section )Notes
I have tried changing the org name, the org altogether, make it a personal repo, to try a new account, to try a new name format, to delete the repo and re-create it, etc.. Nothing worked and I would just get 401 Unauthorized and Retrying for all layers.
It asked for my GPG password from my key ( I didn't configure anything like this on my gitea ) at one point, I completed it, and after that it simply pushed SOME of the layers succesfully and I am getting stuck at this latest erorr.
Setup
General idea of the setup:
I have an NGINX running on systemd that is doing a reverse proxy of all requests handling domains and whatnot to my docker compose containers.
I have multiple services working using this method, and I have been using this setup many many times and it has proven itself.
https nginx -> reverse proxy -> http://0.0.0.0:8000 local docker container
nginx config
docker-compose.yml
Gitea config .ini
Logs
Command Line ( client side sending the request )
Gitea docker log
docker logs -f giteaGitea Version
1.23.4
Can you reproduce the bug on the Gitea demo site?
No
Log Gist
No response
Screenshots
No response
Git Version
git version 2.47.2
Operating System
Container Alpine Linux v3.21 | Host OS: Debian 12
How are you running Gitea?
Through a reverse proxy with nginx and a docker-compose.yml that is running the container.
Database
PostgreSQL
@JustKato commented on GitHub (Feb 19, 2025):
I would appreciate any kind of tips or help, it's been really infuriating trying to deal with this, what is most annoying is that the same docker-compose is running on my personal server which doesn't have these issues with the container registry... I really want to find out how I have gotten to this point, I would even agree to send a .zip of my docker-compose.yml to devs.
@wxiaoguang commented on GitHub (Feb 19, 2025):
Just a guess, would setting
max-concurrent-uploadsto 1 resolve the problem? IIRC there are still many pending bugs in the container/docker registry implementation.also @KN4CK3R
@JustKato commented on GitHub (Feb 19, 2025):
Update
About
Managed to reproduce the issue from pushing from another machine, I have tried doing another container push by:
docker login git.mydomain.com⚠ The exact same behaviour is present when using a PAT ⚠
Client command output
Server Log
@JustKato commented on GitHub (Feb 19, 2025):
Update
Description
I have tried to create a completely new DockerFile and push that as a completely standalone package and I am getting exacrtly the same issues, it has 4 layers and on the 4th layer it gets stuck in retrying...
logs
Dockerfile
Client Logs
Server Logs
@JustKato commented on GitHub (Feb 19, 2025):
Where could I find this to set? Is it in the postgresql or in the gitea config?
@KN4CK3R commented on GitHub (Feb 19, 2025):
It's a Docker client setting for the build/push command.
@JustKato commented on GitHub (Feb 19, 2025):
Doesn't seem to fix the issue, I have applied it in the daemon and other places. Is this because of postgresql? My other mariadb instance is working flawlessly and I use it daily, over 50 repos working just fine to auto-build into docker images.
Is there no fix for this?
@JustKato commented on GitHub (Feb 19, 2025):
Update
After re-creating and reproducing this issue on multiple servers from multiple clients ( definitely the server-side issue ), I have finally figured it out, how? I let the retry command actually finish. What did it say?
Now if you are not an idiot like me, you wait for commands to finish their retries to tell you the error, but me being a smartass I thought that the authorization issues were a deeper rooted issue...
I have now found out that, it is not a gitea issue...
Fixed 🍾🎉
The issue was the nginx configuration for the site, you will notice that I do not mention anywhere what the request size can be, hence it defaults to a quite reasonable but small size ( for repository layers )...
If you are having this issue, please make sure you set the
client_max_body_size 0;or to whatever big size like10Ginside of yourserver { HERE }body in the nginx config!Thanks @KN4CK3R and @wxiaoguang for even humoring my stupid issue...