Server misbehaving when trying to push/pull/login to docker registry #13002

@wxiaoguang commented on GitHub (May 17, 2024):

Related to #30885

How do you deploy the servers? Is there a reverse proxy, and are the "Host" / "X-Forwarded-Proto" headers correctly set?
What does the "Self Check" page say in the admin panel (nightly)?

@wxiaoguang commented on GitHub (May 17, 2024): Related to #30885 * How do you deploy the servers? Is there a reverse proxy, and are the "Host" / "X-Forwarded-Proto" headers correctly set? * What does the "Self Check" page say in the admin panel (nightly)?

GiteaMirror commented

@jessielw commented on GitHub (May 17, 2024):

Related to #30885

How do you deploy the servers? Is there a reverse proxy, and are the "Host" / "X-Forwarded-Proto" headers correctly set?

What does the "Self Check" page say in the admin panel (nightly)?

I deploy with docker/nginx (swag). I have the nginx config setup properly (as advised in the docs) and haven't modified any of that. I did double check all of that though just to be sure.

I did have some database warnings in the health page that I used the doctor to resolve.

For MySQL/MariaDB users, you could use the "gitea doctor convert" command to fix the collation problems, or you could also fix the problem by "ALTER ... COLLATE ..." SQLs manually.

This was present on nightly before I rolled back. Fixing it didn't resolve the issue.

@jessielw commented on GitHub (May 17, 2024): > Related to #30885 > > * How do you deploy the servers? Is there a reverse proxy, and are the "Host" / "X-Forwarded-Proto" headers correctly set? > * What does the "Self Check" page say in the admin panel (nightly)? I deploy with docker/nginx (swag). I have the nginx config setup properly (as advised in the docs) and haven't modified any of that. I did double check all of that though just to be sure. I did have some database warnings in the health page that I used the doctor to resolve. ``` For MySQL/MariaDB users, you could use the "gitea doctor convert" command to fix the collation problems, or you could also fix the problem by "ALTER ... COLLATE ..." SQLs manually. ``` This was present on nightly before I rolled back. Fixing it didn't resolve the issue.

GiteaMirror commented

@wxiaoguang commented on GitHub (May 17, 2024):

Could you share your nginx config?

I did a quick test:

upstream gitea {
  server 127.0.0.1:3000;
}

server {
  server_name this-host.local;
  listen 80;
  location / {
    proxy_pass http://gitea;
    ## proxy_set_header Host $host;
    ## proxy_set_header X-Forwarded-Proto $scheme;
  }
}

Without these proxy_set_header:

$ curl -v http://this-host.local/v2
< Www-Authenticate: Bearer realm="http://gitea/v2/token",service="container_registry",scope="*"

With these proxy_set_header (uncomment):

$ curl -v http://this-host.local/v2
< Www-Authenticate: Bearer realm="http://this-host.local/v2/token",service="container_registry",scope="*"

I think these are expected behaviors.

@wxiaoguang commented on GitHub (May 17, 2024): Could you share your nginx config? ---- I did a quick test: ``` upstream gitea { server 127.0.0.1:3000; } server { server_name this-host.local; listen 80; location / { proxy_pass http://gitea; ## proxy_set_header Host $host; ## proxy_set_header X-Forwarded-Proto $scheme; } } ``` Without these `proxy_set_header`: ``` $ curl -v http://this-host.local/v2 < Www-Authenticate: Bearer realm="http://gitea/v2/token",service="container_registry",scope="*" ``` With these `proxy_set_header` (uncomment): ``` $ curl -v http://this-host.local/v2 < Www-Authenticate: Bearer realm="http://this-host.local/v2/token",service="container_registry",scope="*" ``` I think these are expected behaviors.

GiteaMirror commented

@wxiaoguang commented on GitHub (May 17, 2024):

And one more thing, I can see that you are using Gitea in a sub-path.

So have you checked/changed your /v2 config section in your nginx to make sure the headers are correctly set?

@wxiaoguang commented on GitHub (May 17, 2024): And one more thing, I can see that you are using Gitea in a sub-path. So have you checked/changed your `/v2` config section in your nginx to make sure the headers are correctly set?

GiteaMirror commented

@wxiaoguang commented on GitHub (May 17, 2024):

And one more thing, I can see that you are using Gitea in a sub-path.

So have you checked/changed your /v2 config section in your nginx to make sure the headers are correctly set?

Hmm, I can see one new problem here.

By design (required by the "container" standard ....), the container /v2 path should always be in the root. But AppURL (and the new GuessCurrentAppURL) always have the "sub-path". I will make a new PR to remove the sub-path from "realm".

@wxiaoguang commented on GitHub (May 17, 2024): > And one more thing, I can see that you are using Gitea in a sub-path. > > So have you checked/changed your `/v2` config section in your nginx to make sure the headers are correctly set? Hmm, I can see one new problem here. By design (required by the "container" standard ....), the container `/v2` path should always be in the root. But `AppURL` (and the new `GuessCurrentAppURL`) always have the "sub-path". I will make a new PR to remove the sub-path from "realm".

GiteaMirror commented

@jessielw commented on GitHub (May 17, 2024):

This is the configuration I've been running for a while. I followed the documentation/an issue on the tracker here to set it up like so. It's been working really well until that commit as far as I am aware. Here is my nginx.

location /gitea {
    return 301 $scheme://$host/gitea/;
}

location /gitea/ {
    client_max_body_size 512M;

    # make nginx use unescaped URI, keep "%2F" as is
    rewrite ^ $request_uri;
    rewrite ^/gitea(/.*) $1 break;
    proxy_pass http://gitea:3000$uri;

    proxy_set_header Connection $http_connection;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
}

# This forwards docker traffic to gitea
location /v2/ {
    client_max_body_size 10G;
    proxy_pass http://gitea:3000/v2/;
}

Let me know if anything should be changed.

Yes I am using gitea in a sub-path. Thanks for the help so far!

@jessielw commented on GitHub (May 17, 2024): This is the configuration I've been running for a while. I followed the documentation/an issue on the tracker here to set it up like so. It's been working really well until that commit as far as I am aware. Here is my nginx. ``` location /gitea { return 301 $scheme://$host/gitea/; } location /gitea/ { client_max_body_size 512M; # make nginx use unescaped URI, keep "%2F" as is rewrite ^ $request_uri; rewrite ^/gitea(/.*) $1 break; proxy_pass http://gitea:3000$uri; proxy_set_header Connection $http_connection; proxy_set_header Upgrade $http_upgrade; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; } # This forwards docker traffic to gitea location /v2/ { client_max_body_size 10G; proxy_pass http://gitea:3000/v2/; } ``` Let me know if anything should be changed. Yes I am using gitea in a sub-path. Thanks for the help so far!

GiteaMirror commented

@wxiaoguang commented on GitHub (May 17, 2024):

# This forwards docker traffic to gitea
location /v2/ {
    client_max_body_size 10G;
    proxy_pass http://gitea:3000/v2/;
}

So, please add proxy_set_header in this section too. 🎉

(just copy these from the location /gitea/ section)

@wxiaoguang commented on GitHub (May 17, 2024): ``` # This forwards docker traffic to gitea location /v2/ { client_max_body_size 10G; proxy_pass http://gitea:3000/v2/; } ``` So, please add `proxy_set_header` in this section too. 🎉 (just copy these from the `location /gitea/` section)

GiteaMirror commented

@jessielw commented on GitHub (May 17, 2024):

I will add that as well when I can get back on my computer in the morning. Thanks for the advice and looking into the bug. I didn't realize that was supposed to go in that block as well.

I'll be back on in the AM. Thanks again!

@jessielw commented on GitHub (May 17, 2024): I will add that as well when I can get back on my computer in the morning. Thanks for the advice and looking into the bug. I didn't realize that was supposed to go in that block as well. I'll be back on in the AM. Thanks again!

GiteaMirror commented

@KN4CK3R commented on GitHub (May 17, 2024):

Wasn't there a section in https://docs.gitea.com/administration/reverse-proxies which described the additional /v2/ route? I'm missing that part in the current docs.

@KN4CK3R commented on GitHub (May 17, 2024): Wasn't there a section in https://docs.gitea.com/administration/reverse-proxies which described the additional `/v2/` route? I'm missing that part in the current docs.

GiteaMirror commented

@wxiaoguang commented on GitHub (May 17, 2024):

Wasn't there a section in https://docs.gitea.com/administration/reverse-proxies which described the additional /v2/ route? I'm missing that part in the current docs.

That's why I proposed: Improve reverse proxy documents #31003

@wxiaoguang commented on GitHub (May 17, 2024): > Wasn't there a section in https://docs.gitea.com/administration/reverse-proxies which described the additional `/v2/` route? I'm missing that part in the current docs. That's why I proposed: Improve reverse proxy documents #31003

GiteaMirror commented

@KN4CK3R commented on GitHub (May 17, 2024):

Found the docs. It's only available in the "next" version, so with 1.22 docs. https://docs.gitea.com/next/administration/reverse-proxies#docker--container-registry

@KN4CK3R commented on GitHub (May 17, 2024): Found the docs. It's only available in the "next" version, so with 1.22 docs. https://docs.gitea.com/next/administration/reverse-proxies#docker--container-registry

GiteaMirror commented

@wxiaoguang commented on GitHub (May 17, 2024):

Found the docs. It's only available in the "next" version, so with 1.22 docs. https://docs.gitea.com/next/administration/reverse-proxies#docker--container-registry

The problem is that the section is too simple, it doesn't mention that the header & URI decoding requirements.

@wxiaoguang commented on GitHub (May 17, 2024): > Found the docs. It's only available in the "next" version, so with 1.22 docs. https://docs.gitea.com/next/administration/reverse-proxies#docker--container-registry The problem is that the section is too simple, it doesn't mention that the header & URI decoding requirements.

GiteaMirror commented