Gitea Freezing on FreeBSD Using Jails and a Possible Way to Fix It When Using a Reverse Proxy to Serve #8395

New Issue

GiteaMirror · 2025-11-02T08:04:45-06:00

GiteaMirror commented

2025-11-02 08:04:45 -06:00

Originally created by @thearchivalone on GitHub (Jan 23, 2022).

Gitea Version

1.15.10

Git Version

2.34.1

Operating System

FreeBSD 13.0-p6

How are you running Gitea?

I was running Gitea and Postgres in the same Bastille jail and configured Gitea to listen to the local port Postgres was running on. With some guidance in #18180, it was narrowed down that my particular issue dealt with how I was reverse proxying and possibly running out of local ports too quickly (port exhaustion) for Caddy and other software to interact with. My server is going to be hosting substantially more services so I needed to quickly nip this in the bud. The solution is to use Unix Domain Sockets (UDS from here on out) that can interact with each other between jails. I'm not sure what the security risks are, so your mileage will vary here.

Taken from this guide, you'll need to do a few things:

Backup any important data from within your jails.
Create a folder in a jail that will house your sockets. I use /var/run/jail but you can name it anything as long as it's unique.
Stop it.
Create the directories you're going to mount your sockets to.
Set their permissions correctly if you needed to create users within your jails. The UID and GID's must be unique for this to work properly.
Add the directories to this jail's fstab file making sure any that will be writing sockets to are set to rw (read-write) while others are ro (read-only).
Test your jail by starting it. If there's no errors, move on to the next step.
Inside it, set your application to listen to the socket directory created in Step 2. Most apps require giving your socket file a name here. If you can't write to it, check your fstab and check if the current user owns it (especially if you created a non-root user).
Restart your app.
Check if the socket file exists in the host's socket folder.
Repeat these steps for any jail that will be using that jail's socket.

UDS aren't officially supported in Bastille but can be great if you run a large number of applications on your server that need to be sent across the web (such as a blog or your current git software like Gitea).

Database

PostgreSQL

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Description

No response

Screenshots

No response

Originally created by @thearchivalone on GitHub (Jan 23, 2022). ### Gitea Version 1.15.10 ### Git Version 2.34.1 ### Operating System FreeBSD 13.0-p6 ### How are you running Gitea? I was running Gitea and Postgres in the same Bastille jail and configured Gitea to listen to the local port Postgres was running on. With some guidance in #18180, it was narrowed down that my particular issue dealt with how I was reverse proxying and possibly running out of local ports too quickly (port exhaustion) for Caddy and other software to interact with. My server is going to be hosting substantially more services so I needed to quickly nip this in the bud. The solution is to use Unix Domain Sockets (UDS from here on out) that can interact with each other between jails. I'm not sure what the security risks are, so your mileage will vary here. Taken from this [guide](https://www.ohreally.nl/2021/02/08/freebsd-jails-a-complete-example/#Implementation), you'll need to do a few things: 1) Backup any important data from within your jails. 2) Create a folder in a jail that will house your sockets. I use /var/run/jail but you can name it anything as long as it's unique. 3) Stop it. 4) Create the directories you're going to mount your sockets to. 5) Set their permissions correctly if you needed to create users within your jails. The UID and GID's must be unique for this to work properly. 6) Add the directories to this jail's fstab file making sure any that will be writing sockets to are set to rw (read-write) while others are ro (read-only). 7) Test your jail by starting it. If there's no errors, move on to the next step. 8) Inside it, set your application to listen to the socket directory created in Step 2. Most apps require giving your socket file a name here. If you can't write to it, check your fstab and check if the current user owns it (especially if you created a non-root user). 9) Restart your app. 10) Check if the socket file exists in the host's socket folder. 11) Repeat these steps for any jail that will be using that jail's socket. UDS aren't officially supported in Bastille but can be great if you run a large number of applications on your server that need to be sent across the web (such as a blog or your current git software like Gitea). ### Database PostgreSQL ### Can you reproduce the bug on the Gitea demo site? No ### Log Gist _No response_ ### Description _No response_ ### Screenshots _No response_

GiteaMirror closed this issue

2025-11-02 08:04:45 -06:00

GiteaMirror commented

2025-11-02 08:04:46 -06:00

@zeripath commented on GitHub (Jan 25, 2022):

I think we need to be clear here that this is only a solution for port exhaustion.

Port exhaustion is always a potential result of http proxying: https://www.nginx.com/blog/overcoming-ephemeral-port-exhaustion-nginx-plus/ . It may be that Gitea is making this worse but I'm not certain how to find a cause.

Similarly there's an issue with port exhaustion from the db connections - we experienced this problem on MySQL earlier and we added some options to help reduce the numbers of ports opened when communicating with MySQL. It may be that we need a similar thing for Postgres.

@zeripath commented on GitHub (Jan 25, 2022): I think we need to be clear here that this is only a solution for port exhaustion. Port exhaustion is always a potential result of http proxying: https://www.nginx.com/blog/overcoming-ephemeral-port-exhaustion-nginx-plus/ . It may be that Gitea is making this worse but I'm not certain how to find a cause. Similarly there's an issue with port exhaustion from the db connections - we experienced this problem on MySQL earlier and we added some options to help reduce the numbers of ports opened when communicating with MySQL. It may be that we need a similar thing for Postgres.

GiteaMirror commented

2025-11-02 08:04:46 -06:00

@tsowa commented on GitHub (Jan 25, 2022):

In #18180 I showed my configuration for gitea, I have never had a problem with jails and gitea.

@tsowa commented on GitHub (Jan 25, 2022): In [#18180](https://github.com/go-gitea/gitea/issues/18180#issuecomment-1020672183) I showed my configuration for gitea, I have never had a problem with jails and gitea.

GiteaMirror commented

2025-11-02 08:04:46 -06:00

@thearchivalone commented on GitHub (Jan 25, 2022):

I think we need to be clear here that this is only a solution for port exhaustion.

Port exhaustion is always a potential result of http proxying: https://www.nginx.com/blog/overcoming-ephemeral-port-exhaustion-nginx-plus/ . It may be that Gitea is making this worse but I'm certain how to find a cause.

Similarly there's an issue with port exhaustion from the db connections - we experienced this problem on MySQL earlier and we added some options to help reduce the numbers of ports opened when communicating with MySQL. It may be that we need a similar thing for Postgres.

Thanks for getting back to me. I agree that this is for a specific situation and mostly posted it for anyone else that wanted to run Unix Sockets like this. My previous server I ran using them wherever able without even bothering using localhost ports, so I never experienced the issue until recently. Mind you, I also wasn't using any kind of containers at the time but instead went custom and built almost everything from source, but that's an entirely different subject altogether.

@tsowa my current firewall settings are slightly modified from the Bastille docs example with some extra anchors added for specific IP addresses and whatnot. It's not as advanced as yours but gets the job done. Much of what you have in your nginx config is handled through macros in caddy; outside of using external snippets (short pieces of information that can be loaded from a file), I didn't see much different.

<my git domain> {
    import bgssl <my ssl certs for all subdomains of my main site>
    reverse_proxy <container IP and port>
    <logging code>
}

@thearchivalone commented on GitHub (Jan 25, 2022): > I think we need to be clear here that this is only a solution for port exhaustion. > > Port exhaustion is always a potential result of http proxying: https://www.nginx.com/blog/overcoming-ephemeral-port-exhaustion-nginx-plus/ . It may be that Gitea is making this worse but I'm certain how to find a cause. > > Similarly there's an issue with port exhaustion from the db connections - we experienced this problem on MySQL earlier and we added some options to help reduce the numbers of ports opened when communicating with MySQL. It may be that we need a similar thing for Postgres. Thanks for getting back to me. I agree that this is for a specific situation and mostly posted it for anyone else that wanted to run Unix Sockets like this. My previous server I ran using them wherever able without even bothering using localhost ports, so I never experienced the issue until recently. Mind you, I also wasn't using any kind of containers at the time but instead went custom and built almost everything from source, but that's an entirely different subject altogether. @tsowa my current firewall settings are slightly modified from the Bastille docs example with some extra anchors added for specific IP addresses and whatnot. It's not as advanced as yours but gets the job done. Much of what you have in your nginx config is handled through macros in caddy; outside of using external snippets (short pieces of information that can be loaded from a file), I didn't see much different. ``` <my git domain> { import bgssl <my ssl certs for all subdomains of my main site> reverse_proxy <container IP and port> <logging code> } ```

GiteaMirror commented

2025-11-02 08:04:46 -06:00

@zeripath commented on GitHub (Jan 25, 2022):

@tsowa any time you are using a http proxy and a db on localhost or same IP, depending on how they handles time_waits and connection closing and db connection scaling there is always a risk of port exhaustion and Gitea blocking due to port exhaustion.
Less drastic cases of port exhaustion can occur in any situation where you bridge the world to one IP alone. Using unix ports completely eradicates this as a possible issue.

The problem is that there is a fundamentally a limited number of ports for TCP connections and every connection has to have a unique port-address pair. (In fact if SO_REUSEADDR is not set every TCP connection has to come from a unique port.)

Outside facing services are essentially prevented from running out of ports by allowing ports to be reused so long as they're pointing to different IP aaddresses (this works unless of course you get more than 65535 connections from the same IP). However, as soon as you have a proxy all of that traffic now has to go from that proxy to your Gitea server, e.g. localhost-localhost. If you add in a db communicating over TCP on localhost then that also takes ports.

The problem is then made worse by db connection pool scaling where contention for db connections means that more connections are opened and then potentially held open in a pool before being closed - (although being held open isn't necessarily the problem - it's the closing and reopening. Everytime a client port is closed it may fall in to a TIME_WAIT state meaning you can't use that port again for up to two minutes!)

This means under load even more ports are taken and the reverse proxy can't get to the ports. Worse, you can end up with a situation whereby http requests are blocked waiting for a db port but all ports are in use by http requests or are in TIME_WAIT. (And it is at this point that Gitea can completely block because as we currently don't pass context down to the db cancelling the http request won't cancel the db request.)

MySQL had a particular problem here so we explicitly created options to tell the connections not to open too many connections. It's likely we need to do the same thing for postgres. It's also possible to change the TIME_WAIT length. (Eventually we will also finally pass context down to db requests and we should be able to cancel requests.)

Docker setups are less badly affected because the db and the http proxy are on different internal IPs meaning that neither should block the other and if the http proxy is blocked waiting for a port that doesn't block the db so one will come soon enough.

However if you're simply using localhost and you can use them, unix ports are probably the best as they do not suffer this problem.

@zeripath commented on GitHub (Jan 25, 2022): @tsowa any time you are using a http proxy and a db on localhost or same IP, depending on how they handles time_waits and connection closing and db connection scaling there is always a risk of port exhaustion and Gitea blocking due to port exhaustion. Less drastic cases of port exhaustion can occur in any situation where you bridge the world to one IP alone. Using unix ports completely eradicates this as a possible issue. The problem is that there is a fundamentally a limited number of ports for TCP connections and every connection has to have a unique port-address pair. (In fact if SO_REUSEADDR is not set every TCP connection has to come from a unique port.) Outside facing services are essentially prevented from running out of ports by allowing ports to be reused so long as they're pointing to different IP aaddresses (this works unless of course you get more than 65535 connections from the same IP). However, as soon as you have a proxy all of that traffic now has to go from that proxy to your Gitea server, e.g. localhost-localhost. If you add in a db communicating over TCP on localhost then that also takes ports. The problem is then made worse by db connection pool scaling where contention for db connections means that more connections are opened and then potentially held open in a pool before being closed - (although being held open isn't necessarily the problem - it's the closing and reopening. Everytime a client port is closed it may fall in to a TIME_WAIT state meaning you can't use that port again for up to two minutes!) This means under load even more ports are taken and the reverse proxy can't get to the ports. Worse, you can end up with a situation whereby http requests are blocked waiting for a db port but all ports are in use by http requests or are in TIME_WAIT. (And it is at this point that Gitea can completely block because as we currently don't pass context down to the db cancelling the http request won't cancel the db request.) MySQL had a particular problem here so we explicitly created options to tell the connections not to open too many connections. It's likely we need to do the same thing for postgres. It's also possible to change the TIME_WAIT length. (Eventually we will also finally pass context down to db requests and we should be able to cancel requests.) Docker setups are less badly affected because the db and the http proxy are on different internal IPs meaning that neither should block the other and if the http proxy is blocked waiting for a port that doesn't block the db so one will come soon enough. However if you're simply using localhost and you can use them, unix ports are probably the best as they do not suffer this problem.

GiteaMirror commented

2025-11-02 08:04:47 -06:00

@wxiaoguang commented on GitHub (Apr 16, 2022):

It seems not to be a real issue.

Feel free to reopen if necessary.

@wxiaoguang commented on GitHub (Apr 16, 2022): It seems not to be a real issue. Feel free to reopen if necessary.

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/gitea#8395