Unable to access interface from local IP #8

New Issue

2025-11-13T11:47:11-06:00

GiteaMirror commented

2025-11-13 11:47:11 -06:00

Originally created by @sequence711 on GitHub (Jan 9, 2025).

Hello,

I opened an issue 12 hours or so ago but it doesn't seem to be visible to anyone except my own account (not this one). I'll repost the issue here so it's actually visible while I'm trying to fix my account :)

I'm trying to set up Pangolin on a Raspberry Pi on the same network as a few other servers (and my workstation).

I'm encountering a problem when trying to connect via the subdomain I set, but only when connected to the same network as Pangolin. If I turn on a VPN or connect from my phone (on mobile data) it works fine.

I've tried to troubleshoot it, but I'm still unsure exactly what causes this problem. It's not impossible there's something in my network acting up.

However, I'm able to spin up Caddy and reverse proxy to the same subdomain with another service on the Raspberry Pi and other servers with no problem what so ever.

I have no experience with Traefik so I tried looking through the configuration and modifying it but it made no difference.

Any idea what could be going wrong? I would send logs if I could but I haven't seen any log reacting at all to my connection attempts (successful and unsuccessful).

Originally created by @sequence711 on GitHub (Jan 9, 2025). Hello, I opened an issue 12 hours or so ago but it doesn't seem to be visible to anyone except my own account (not this one). I'll repost the issue here so it's actually visible while I'm trying to fix my account :) -------------- I'm trying to set up Pangolin on a Raspberry Pi on the same network as a few other servers (and my workstation). I'm encountering a problem when trying to connect via the subdomain I set, but only when connected to the same network as Pangolin. If I turn on a VPN or connect from my phone (on mobile data) it works fine. I've tried to troubleshoot it, but I'm still unsure exactly what causes this problem. It's not impossible there's something in my network acting up. However, I'm able to spin up Caddy and reverse proxy to the same subdomain with another service on the Raspberry Pi and other servers with no problem what so ever. I have no experience with Traefik so I tried looking through the configuration and modifying it but it made no difference. Any idea what could be going wrong? I would send logs if I could but I haven't seen any log reacting at all to my connection attempts (successful and unsuccessful).

GiteaMirror closed this issue

2025-11-13 11:47:11 -06:00

GiteaMirror commented

2025-11-13 11:47:12 -06:00

@oschwartz10612 commented on GitHub (Jan 9, 2025):

Hi!

If I understand correctly you are saying:

You are able to connect from your WAN to the Pangolin UI through Traefik
You are unable to connect from your LAN but while still using the DNS and not the local IP

Is this right?

If so I would suspect that your DNS is resolving the WAN (public) IP of your network - which would make sense and is how it should work - but your internal router is not doing "hairpin NAT" to loop back your request. I think this is simply a routing problem that Caddy might handle better than Traefik.

I would try to look into hairpin NAT on your router. What is your router model? Or maybe split DNS? Take a look at this Reddit post as well and see if it can help at all: https://www.reddit.com/r/selfhosted/comments/ozound/traefik_2_lets_encrypt_hairpin_internal_lan_not/

@oschwartz10612 commented on GitHub (Jan 9, 2025): Hi! If I understand correctly you are saying: - You are able to connect from your WAN to the Pangolin UI through Traefik - You are unable to connect from your LAN but while still using the DNS and not the local IP Is this right? If so I would suspect that your DNS is resolving the WAN (public) IP of your network - which would make sense and is how it should work - but your internal router is not doing "hairpin NAT" to loop back your request. I think this is simply a routing problem that Caddy might handle better than Traefik. I would try to look into hairpin NAT on your router. What is your router model? Or maybe split DNS? Take a look at this Reddit post as well and see if it can help at all: https://www.reddit.com/r/selfhosted/comments/ozound/traefik_2_lets_encrypt_hairpin_internal_lan_not/

GiteaMirror commented

2025-11-13 11:47:12 -06:00

@sequence711 commented on GitHub (Jan 9, 2025):

Hello!

You are able to connect from your WAN to the Pangolin UI through Traefik
You are unable to connect from your LAN but while still using the DNS and not the local IP

Correct, I can reach the interface via IP and port but I'm not able to login, it redirects me to the interface again.

I also thought it was NAT hairpinning at first but it is enabled and working for everything except Pangolin. I also tried split DNS through Adguard Home but when I try to connect it loads forever. Also tried a combination of those but it does not work.

My router is a EdgeRouter 4! I've tried searching for solutions both for Traefik and the router but I've come up empty handed :/.

I do find it a little odd that it would work flawlessly with Caddy with hairpinning enabled, and if I disable hairpinning it stops working, so I think hairpinning is working as intended.

I see a comment about Traefik and SSL, perhaps that is the issue? Usually I get an error message saying there's something wrong with the certs though.

@sequence711 commented on GitHub (Jan 9, 2025): Hello! > You are able to connect from your WAN to the Pangolin UI through Traefik You are unable to connect from your LAN but while still using the DNS and not the local IP Correct, I can reach the interface via IP and port but I'm not able to login, it redirects me to the interface again. I also thought it was NAT hairpinning at first but it is enabled and working for everything except Pangolin. I also tried split DNS through Adguard Home but when I try to connect it loads forever. Also tried a combination of those but it does not work. My router is a EdgeRouter 4! I've tried searching for solutions both for Traefik and the router but I've come up empty handed :/. I do find it a little odd that it would work flawlessly with Caddy with hairpinning enabled, and if I disable hairpinning it stops working, so I *think* hairpinning is working as intended. I see a comment about Traefik and SSL, perhaps that is the issue? Usually I get an error message saying there's something wrong with the certs though.

GiteaMirror commented

2025-11-13 11:47:12 -06:00

@sequence711 commented on GitHub (Jan 9, 2025):

I'm pretty sure this isn't solely a networking issue but I'm not entirely sure what else to check.
So far I've tried the following:

Split DNS (on/off)
Hairpin NAT (on/off)
Different operating systems (Linux, Android and Windows)
Using IP for "next-service" instead of docker service name
Adding my computers and routers local IP as a ClientIP rule.
Disabling redirect-to-https middleware.
Rebooting (everything)
Removing Badger from traefik_config.yml
Re-installing with Pangolin binary
Turning firewall off and on

I also created a separate Traefik instance to see if I could reverse proxy to other services - which worked with no problems. The services are hosted on the same host in another Docker container.
This worked with and without the automatic HTTPS.

And while writing this I came up with other things to try that just might work and I just found it.

I had used the default configuration generated from the installer (checking through the compose.yml and the other configuration files) and nothing looked odd except Traefik using network_mode: service:gerbil which I hadn't seen yet, but assumed this was necessary for some reason.

I only had to move the ports 80 and 443 from Gerbil to Traefik and now it's working locally as well.

But that makes me wonder, does Gerbil need to have access to the ports as well? Will it work with the ports on Traefik instead of Gerbil?

@sequence711 commented on GitHub (Jan 9, 2025): I'm pretty sure this isn't solely a networking issue but I'm not entirely sure what else to check. So far I've tried the following: * Split DNS (on/off) * Hairpin NAT (on/off) * Different operating systems (Linux, Android and Windows) * Using IP for "next-service" instead of docker service name * Adding my computers and routers local IP as a ClientIP rule. * Disabling redirect-to-https middleware. * Rebooting (everything) * Removing Badger from traefik_config.yml * Re-installing with Pangolin binary * Turning firewall off and on I also created a separate Traefik instance to see if I could reverse proxy to other services - which worked with no problems. The services are hosted on the same host in another Docker container. This worked with and without the automatic HTTPS. --- And while writing this I came up with other things to try that just *might* work and I just found it. I had used the default configuration generated from the installer (checking through the compose.yml and the other configuration files) and nothing looked odd except Traefik using ```network_mode: service:gerbil``` which I hadn't seen yet, but assumed this was necessary for some reason. I only had to move the ports 80 and 443 from Gerbil to Traefik and now it's working locally as well. But that makes me wonder, does Gerbil need to have access to the ports as well? Will it work with the ports on Traefik instead of Gerbil?

GiteaMirror commented

2025-11-13 11:47:13 -06:00

@sequence711 commented on GitHub (Jan 10, 2025):

Alright I figured out that it does not work if you move 80 and 443 to Traefik.
I also found out everything was caused by overlapping subnets (Wireguard 10.0.0.0/8 overlapped with my internal network). I have no idea why it worked after moving the ports but I've moved everything back and it's working now.

I can't believe it took me days to figure this out, I've never even opened an issue before..

Anyhow, thanks for the product, I'm sure this will grow big in no time. It looks very polished already 😄
This is exacly what I've been looking for!

@sequence711 commented on GitHub (Jan 10, 2025): Alright I figured out that it does not work if you move 80 and 443 to Traefik. I also found out everything was caused by overlapping subnets (Wireguard 10.0.0.0/8 overlapped with my internal network). I have no idea why it worked after moving the ports but I've moved everything back and it's working now. I can't believe it took me days to figure this out, I've never even opened an issue before.. Anyhow, thanks for the product, I'm sure this will grow big in no time. It looks very polished already 😄 This is exacly what I've been looking for!

GiteaMirror commented

2025-11-13 11:47:13 -06:00

@miloschwartz commented on GitHub (Jan 10, 2025):

Glad you got it working! We're going to document this in the docs in case anyone else has this issue. Thanks

@miloschwartz commented on GitHub (Jan 10, 2025): Glad you got it working! We're going to document this in the docs in case anyone else has this issue. Thanks

GiteaMirror commented

2025-11-13 11:47:13 -06:00

@polarathene commented on GitHub (Mar 17, 2025):

We're going to document this in the docs in case anyone else has this issue.

Was the usage of network_mode: service:gerbil documented? I didn't seem to have luck finding anything in the docs about it and git blame provides no context on the repo compose example that introduced it.

It appears to share the same bridge network IP assigned to the referenced service. Was that necessary or beneficial for anything in particular? (EDIT: It also shares the same loopback address)

# compose.yaml

services:
  hello:
    image: caddy:2.9
    container_name: hello
    environment:
      CADDY_HOST: hello
    configs:
      - source: caddy-config
        target: /etc/caddy/Caddyfile

  world:
    image: caddy:2.9
    container_name: world
    network_mode: service:hello
    # `network_mode` prevents setting hostname (inherited from network source)
    environment:
      CADDY_HOST: world
    configs:
      - source: caddy-config
        target: /etc/caddy/Caddyfile

configs:
  caddy-config:
    content: |
      {
        auto_https off
      }

      :80 {
        respond <<HEREDOC
          Caddy ({env.CADDY_HOST}) received:
          - Host (Server): {host}
          - Remote (Client IP): {remote_host}

          HEREDOC
      }

docker compose up -d --force-recreate

# Query `localhost` / `127.0.0.1` or `hello` (bridge IP instead of loopback):
# (`world` is not valid due to `network_mode`):
docker exec caddy hello -qO - 127.0.0.1

The above query can be done multiple times and you'll find the response alternates between both containers (recognizable via the CADDY_HOST env distinction) listening on the same shared interfaces (default bind is 0.0.0.0), since Caddy supports that.

The Pangolin Docker DNS docs mention that localhost would only refer to the container itself, which is generally true, but this is perhaps something to be mindful of.

I haven't seen network_mode used like this before, so I'm curious if it was intentional or just what worked well at the time (possibly without knowing about other alternatives).

@polarathene commented on GitHub (Mar 17, 2025): > We're going to document this in the docs in case anyone else has this issue. Was the usage of `network_mode: service:gerbil` documented? I didn't seem to have luck finding anything in the docs about it and `git blame` provides no context on the repo compose example that introduced it. It appears to share the same bridge network IP assigned to the referenced service. Was that necessary or beneficial for anything in particular? (_**EDIT:** It also shares the same loopback address_) ```yaml # compose.yaml services: hello: image: caddy:2.9 container_name: hello environment: CADDY_HOST: hello configs: - source: caddy-config target: /etc/caddy/Caddyfile world: image: caddy:2.9 container_name: world network_mode: service:hello # `network_mode` prevents setting hostname (inherited from network source) environment: CADDY_HOST: world configs: - source: caddy-config target: /etc/caddy/Caddyfile configs: caddy-config: content: | { auto_https off } :80 { respond <<HEREDOC Caddy ({env.CADDY_HOST}) received: - Host (Server): {host} - Remote (Client IP): {remote_host} HEREDOC } ``` ```bash docker compose up -d --force-recreate # Query `localhost` / `127.0.0.1` or `hello` (bridge IP instead of loopback): # (`world` is not valid due to `network_mode`): docker exec caddy hello -qO - 127.0.0.1 ``` The above query can be done multiple times and you'll find the response alternates between both containers (_recognizable via the `CADDY_HOST` env distinction_) listening on the same shared interfaces (_default bind is `0.0.0.0`_), since Caddy supports that. The [Pangolin Docker DNS docs](https://docs.fossorial.io/Getting%20Started/dns-networking#notes-on-docker) mention that `localhost` would only refer to the container itself, which is generally true, but this is perhaps something to be mindful of. I haven't seen `network_mode` used like this before, so I'm curious if it was intentional or just what worked well at the time (_possibly without knowing about other alternatives_).

GiteaMirror commented

2025-11-13 11:47:14 -06:00

@polarathene commented on GitHub (Mar 17, 2025):

An additional heads-up regarding the docs here:

For "Local" sites running in Docker in the same compose as Pangolin: You usually want to address the host machine.
One way to do this is by using the special address: 172.17.0.1.

That's the default docker0 bridge network gateway and it'll vary based on the subnet address pools used when letting compose create a bridged network.

services:
  example:
    extra_hosts:
      - "host.docker.internal:host-gateway"

The above snippet adds that 172.17.0.1 IP (or equivalent) to the containers /etc/hosts as a convenience (DNS tools won't resolve it as a result):

There can be caveats as the 2nd link notes when you rely on host-gateway (or slightly worse 172.17.0.1) vs setting the extra_hosts mapping to the appropriate gateway IP (_which you can control with static assignment in the networks config (like the 2nd link details, although you shouldn't need to set ip_range + gateway attributes, just the subnet if keeping it simple).

For the actual use-case described in the docs (to reach sites running within the same compose container network)... you could leverage the internal DNS docker manages within containers (127.0.0.11:53).

It'll be the default DNS configured in a container, and you can query the service name, container name, hostname, or network aliases configured. Just set the container hostname to the public domain name if it doesn't need to go through the proxy layer.

In my earlier Caddy example, if network_mode: service:gerbil wasn't set, you could use wget -qO - world to get the expected response in a deterministic manner, just like you would then get within the hello container instance. You could use the example query above, but run it from within the world container to query hello and it'll resolve that to the private network IP the container got from the default bridge network compose created.

The only other scenario when you do need some indirection, such as reaching the reverse proxy instead of the container directly, you can do that via network aliases:

services:
  hello:
    image: caddy:2.9
    container_name: hello
    hostname: reverse-proxy
    # Additional DNS names to resolve to this container IP
    # for the default bridge network Compose creates:
    networks:
      default:
        aliases:
          - world.internal
    configs:
      - source: caddy-router
        target: /etc/caddy/Caddyfile

  world:
    image: caddy:2.9
    container_name: world
    hostname: some-website
    configs:
      - source: caddy-website
        target: /etc/caddy/Caddyfile

configs:
  caddy-router:
    content: |
      http://world.internal {
        reverse_proxy world:80
      }

  caddy-website:
    content: |
      :80 {
        respond <<HEREDOC
          Caddy ({env.HOSTNAME}) received:
          - Host (Server): {host}
          - Remote (Client IP): {remote_host}
          - Forwarded From (via a reverse-proxy): {header.X-Forwarded-For}

          HEREDOC
      }

# Making the default network created a little easier to demonstrate:
networks:
  default:
    name: example

Any container in that compose network can then use the alias to connect to the reverse proxy instead of a direct connection, similar to public access. The aliases could match the public domains provided there's no concerns with that.

$ docker compose down
$ docker compose up -d --force-recreate

# Deterministic: world.internal (fedora @ 172.19.0.4) => hello:80 (reverse-proxy @ 172.19.0.2) => world:80 (website)
$ docker run --rm -it --network example fedora curl world.internal
Caddy (some-website) received:
- Host (Server): world.internal
- Remote (Client IP): 172.19.0.2
- Forwarded From (via a reverse-proxy): 172.19.0.4

# `world` or `some-website` can both be used to connect to the container directly:
$ docker run --rm --network example -it fedora curl world
Caddy (some-website) received:
- Host (Server): world
- Remote (Client IP): 172.19.0.4
- Forwarded From (via a reverse-proxy):

Another option instead of aliases is to change the DNS default to intercept your domains and set the IP to the container one instead of the public IP via a conditional rewrite (blocky does that well).

Also be mindful of Docker defaulting to userland-proxy: true which can be set to false in /etc/docker/daemon.json. That handles NAT hairpinning on the Docker host IIRC, you'll find a notable performance improvement if benching network throughput (I recall the proxy halved the iperf3 performance last I checked), it's mostly there for localhost routing convenience (but I don't think it's necessary for that either today, [I have a table to compare routing differences, although it may be a bit outdated by now).

@polarathene commented on GitHub (Mar 17, 2025): An additional heads-up [regarding the docs here](https://docs.fossorial.io/Getting%20Started/dns-networking#notes-on-docker): > _For "Local" sites running in Docker in the same compose as Pangolin: You usually want to address the host machine._ > _One way to do this is by using the special address: `172.17.0.1`._ That's the default `docker0` bridge network gateway and it'll vary based on the subnet address pools used when letting compose create a bridged network. ```yaml services: example: extra_hosts: - "host.docker.internal:host-gateway" ``` The above snippet adds that `172.17.0.1` IP (or equivalent) to the containers `/etc/hosts` as a convenience (_DNS tools won't resolve it as a result_): - https://stackoverflow.com/questions/48546124/what-is-the-linux-equivalent-of-host-docker-internal - https://stackoverflow.com/questions/70725881/what-is-the-equivalent-of-add-host-host-docker-internalhost-gateway-in-a-comp/70725882#70725882 (more detailed) There can be caveats as the 2nd link notes when you rely on `host-gateway` (_or slightly worse `172.17.0.1`_) vs setting the `extra_hosts` mapping to the appropriate gateway IP (_which you can control with static assignment in the `networks` config (_like the 2nd link details, although you shouldn't need to set `ip_range` + `gateway` attributes, just the `subnet` if keeping it simple_). --- For the actual use-case described in the docs (_to reach sites running within the same compose container network_)... you could leverage the internal DNS docker manages within containers (`127.0.0.11:53`). It'll be the default DNS configured in a container, and you can query the service name, container name, hostname, or network aliases configured. Just set the container `hostname` to the public domain name if it doesn't need to go through the proxy layer. In my earlier Caddy example, if `network_mode: service:gerbil` wasn't set, you could use `wget -qO - world` to get the expected response in a deterministic manner, just like you would then get within the `hello` container instance. You could use the example query above, but run it from within the `world` container to query `hello` and it'll resolve that to the private network IP the container got from the default bridge network compose created. The only other scenario when you do need some indirection, such as reaching the reverse proxy instead of the container directly, you can do that via network aliases: ```yaml services: hello: image: caddy:2.9 container_name: hello hostname: reverse-proxy # Additional DNS names to resolve to this container IP # for the default bridge network Compose creates: networks: default: aliases: - world.internal configs: - source: caddy-router target: /etc/caddy/Caddyfile world: image: caddy:2.9 container_name: world hostname: some-website configs: - source: caddy-website target: /etc/caddy/Caddyfile configs: caddy-router: content: | http://world.internal { reverse_proxy world:80 } caddy-website: content: | :80 { respond <<HEREDOC Caddy ({env.HOSTNAME}) received: - Host (Server): {host} - Remote (Client IP): {remote_host} - Forwarded From (via a reverse-proxy): {header.X-Forwarded-For} HEREDOC } # Making the default network created a little easier to demonstrate: networks: default: name: example ``` Any container in that compose network can then use the alias to connect to the reverse proxy instead of a direct connection, similar to public access. The aliases could match the public domains provided there's no concerns with that. ```console $ docker compose down $ docker compose up -d --force-recreate # Deterministic: world.internal (fedora @ 172.19.0.4) => hello:80 (reverse-proxy @ 172.19.0.2) => world:80 (website) $ docker run --rm -it --network example fedora curl world.internal Caddy (some-website) received: - Host (Server): world.internal - Remote (Client IP): 172.19.0.2 - Forwarded From (via a reverse-proxy): 172.19.0.4 # `world` or `some-website` can both be used to connect to the container directly: $ docker run --rm --network example -it fedora curl world Caddy (some-website) received: - Host (Server): world - Remote (Client IP): 172.19.0.4 - Forwarded From (via a reverse-proxy): ``` Another option instead of aliases is to change the DNS default to intercept your domains and set the IP to the container one instead of the public IP via a conditional rewrite (blocky does that well). --- Also be mindful of Docker defaulting to `userland-proxy: true` which can be set to `false` in `/etc/docker/daemon.json`. That handles NAT hairpinning on the Docker host IIRC, you'll find a notable performance improvement if benching network throughput (_I recall the proxy halved the iperf3 performance last I checked_), it's mostly there for localhost routing convenience (_but I don't think it's necessary for that either today, [I have a [table to compare routing differences](https://github.com/docker/docs/issues/17312#issuecomment-1736446266), although it may be a bit outdated by now_).

GiteaMirror commented

2025-11-13 11:47:14 -06:00

@oschwartz10612 commented on GitHub (Mar 17, 2025):

Thanks for all of the great information. We will try to update the docs - or feel free to contribute if you have time. https://github.com/fosrl/docs

network_mode: service:gerbil is used to put the Traefik container in to the Gerbil container's network namespace. This is so it can traverse over the wireguard interface inside of the Gerbil network namespace easily. This is documented from docker here: https://docs.docker.com/reference/compose-file/services/#network_mode

@oschwartz10612 commented on GitHub (Mar 17, 2025): Thanks for all of the great information. We will try to update the docs - or feel free to contribute if you have time. https://github.com/fosrl/docs `network_mode: service:gerbil` is used to put the Traefik container in to the Gerbil container's network namespace. This is so it can traverse over the wireguard interface inside of the Gerbil network namespace easily. This is documented from docker here: https://docs.docker.com/reference/compose-file/services/#network_mode

GiteaMirror commented

2025-11-13 11:47:14 -06:00

@polarathene commented on GitHub (Mar 17, 2025):

This is documented from docker here

Yeah I saw those docs and it didn't help me understand what it was doing that well hence my reproduction above.

At first I thought it was establishing a network that was isolated between the two containers, rather than sharing the same network(s).

This is so it can traverse over the wireguard interface inside of the Gerbil network namespace easily.

I'm not too familiar with what's going on under the hood, but I'm assuming the difference is you can just refer to ports or localhost:1234 instead of gerbil:1234 / traefik:1234? Probably a little more efficient on network routing too that way, but I've no idea if there's a meaningful amount of overhead avoided 😅

We will try to update the docs

Sometimes for more niche and technical topics it's simpler to keep it brief on the docs and just link to related discussions like here where more detailed info can be found. I do this when updating docs for topics users have brought up but the concerns aren't directly related to what the project itself manages/does, as is the case here I think with Docker / Compose specific networking features and caveats.

@polarathene commented on GitHub (Mar 17, 2025): > This is documented from docker here Yeah I saw those docs and it didn't help me understand what it was doing that well hence my reproduction above. At first I thought it was establishing a network that was isolated between the two containers, rather than sharing the same network(s). > This is so it can traverse over the wireguard interface inside of the Gerbil network namespace easily. I'm not too familiar with what's going on under the hood, but I'm assuming the difference is you can just refer to ports or `localhost:1234` instead of `gerbil:1234` / `traefik:1234`? Probably a little more efficient on network routing too that way, but I've no idea if there's a meaningful amount of overhead avoided 😅 > We will try to update the docs Sometimes for more niche and technical topics it's simpler to keep it brief on the docs and just link to related discussions like here where more detailed info can be found. I do this when updating docs for topics users have brought up but the concerns aren't directly related to what the project itself manages/does, as is the case here I think with Docker / Compose specific networking features and caveats.