github-starred/pangolin
2
0
Fork 0
mirror of https://github.com/fosrl/pangolin.git synced 2026-08-27 06:47:02 -05:00
Code Issues 1.6k Packages Projects Releases 79 Wiki Activity

Newt container unable to make connections byound the docker host #702

New Issue
Open
opened 2025-11-13 12:08:12 -06:00 by GiteaMirror · 10 comments
GiteaMirror commented 2025-11-13 12:08:12 -06:00
Owner
Copy Link
Copy Source

Originally created by @corvus-ch on GitHub (Sep 28, 2025).

I have a strange issue where the newt container is unable to make connections beyond the hosts' boundary.

The leading indicator is, that all DNS lookups fail. Doing some digging with ping, requests to the docker host are successful, while requests to the router are not. Using a different image on the same docker host, works just fine in terms of DNS lookup and network connectivity.

Content of docker-compose.yml:

services:
  newt:
    image: fosrl/newt
    container_name: newt
    restart: unless-stopped
    environment:
      - PANGOLIN_ENDPOINT=https://pangolin.example.com
      - NEWT_ID=…
      - NEWT_SECRET=…
      - DOCKER_SOCKET=/var/run/docker.sock
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

networks:
  default:
    driver: bridge
    name: pangolin

Output of docker-compose logs -f:

newt  | INFO: 2025/09/28 05:10:35 Newt version 1.5.0
newt  | ERROR: 2025/09/28 05:10:43 Error checking for updates: failed to fetch release info: Get "https://api.github.com/repos/fosrl/newt/releases/latest": dial tcp: lookup api.github.com on 127.0.0.11:53: server misbehaving
newt  | ERROR: 2025/09/28 05:10:51 Failed to connect: failed to get token: failed to request new token: Post "https://pangolin.example.com/api/v1/auth/newt/get-token": dial tcp: lookup pangolin.example.com on 127.0.0.11:53: server misbehaving. Retrying in 3s...
…

Where the last line is repeating over and over again.

Output of docker-compose exec newt ip addr:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0@if27: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
    link/ether 2a:1d:fd:f5:c9:0f brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.2/16 brd 172.18.255.255 scope global eth0
       valid_lft forever preferred_lft forever

Output of docker-compose exec newt ping -c 1 172.18.0.1:

64 bytes from 172.18.0.1: seq=0 ttl=64 time=0.213 ms

--- 172.18.0.1 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.213/0.213/0.213 ms

Output of docker-compose exec newt ping -c 1 172.16.42.147 where 172.16.42.147 is the hosts IP address:

PING 172.16.42.147 (172.16.42.147): 56 data bytes
64 bytes from 172.16.42.147: seq=0 ttl=64 time=0.154 ms

--- 172.16.42.147 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.154/0.154/0.154 ms

Output of docker-compose exec newt ping -c 1 172.16.42.1 where 172.16.42.1 is the routers IP address:

PING 172.16.42.1 (172.16.42.1): 56 data bytes

--- 172.16.42.1 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss

However, using another image, pinging the router or even an external site works quite fine.

Output of docker run --rm -it alpine ping -c 1 172.16.42.1

PING 172.16.42.1 (172.16.42.1): 56 data bytes
64 bytes from 172.16.42.1: seq=0 ttl=63 time=1004.398 ms

--- 172.16.42.1 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 1004.398/1004.398/1004.398 ms

Output of docker run --rm -it alpine ping -c 1 example.com

PING example.com (23.220.75.245): 56 data bytes
64 bytes from 23.220.75.245: seq=0 ttl=38 time=168.658 ms

--- example.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 168.658/168.658/168.658 ms

What is preventing the newt image from reaching network nodes beyond the docker host?

Images used:

REPOSITORY   TAG       DIGEST                                                                    IMAGE ID
fosrl/newt   latest    sha256:659c7cefac5e0678d0bf1e87c9b65d1f841ab0ac21e75f9507dfc67a42ef34da   015dfeed2246
alpine       latest    sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1   02f8efbefad6
Originally created by @corvus-ch on GitHub (Sep 28, 2025). I have a strange issue where the newt container is unable to make connections beyond the hosts' boundary. The leading indicator is, that all DNS lookups fail. Doing some digging with ping, requests to the docker host are successful, while requests to the router are not. Using a different image on the same docker host, works just fine in terms of DNS lookup and network connectivity. Content of `docker-compose.yml`: ```yaml services: newt: image: fosrl/newt container_name: newt restart: unless-stopped environment: - PANGOLIN_ENDPOINT=https://pangolin.example.com - NEWT_ID=… - NEWT_SECRET=… - DOCKER_SOCKET=/var/run/docker.sock volumes: - /var/run/docker.sock:/var/run/docker.sock networks: default: driver: bridge name: pangolin ``` Output of `docker-compose logs -f`: ``` newt | INFO: 2025/09/28 05:10:35 Newt version 1.5.0 newt | ERROR: 2025/09/28 05:10:43 Error checking for updates: failed to fetch release info: Get "https://api.github.com/repos/fosrl/newt/releases/latest": dial tcp: lookup api.github.com on 127.0.0.11:53: server misbehaving newt | ERROR: 2025/09/28 05:10:51 Failed to connect: failed to get token: failed to request new token: Post "https://pangolin.example.com/api/v1/auth/newt/get-token": dial tcp: lookup pangolin.example.com on 127.0.0.11:53: server misbehaving. Retrying in 3s... … ``` Where the last line is repeating over and over again. Output of `docker-compose exec newt ip addr`: ``` 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0@if27: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP link/ether 2a:1d:fd:f5:c9:0f brd ff:ff:ff:ff:ff:ff inet 172.18.0.2/16 brd 172.18.255.255 scope global eth0 valid_lft forever preferred_lft forever ``` Output of `docker-compose exec newt ping -c 1 172.18.0.1`: ```PING 172.18.0.1 (172.18.0.1): 56 data bytes 64 bytes from 172.18.0.1: seq=0 ttl=64 time=0.213 ms --- 172.18.0.1 ping statistics --- 1 packets transmitted, 1 packets received, 0% packet loss round-trip min/avg/max = 0.213/0.213/0.213 ms ``` Output of `docker-compose exec newt ping -c 1 172.16.42.147` where `172.16.42.147` is the hosts IP address: ``` PING 172.16.42.147 (172.16.42.147): 56 data bytes 64 bytes from 172.16.42.147: seq=0 ttl=64 time=0.154 ms --- 172.16.42.147 ping statistics --- 1 packets transmitted, 1 packets received, 0% packet loss round-trip min/avg/max = 0.154/0.154/0.154 ms ``` Output of `docker-compose exec newt ping -c 1 172.16.42.1` where `172.16.42.1` is the routers IP address: ``` PING 172.16.42.1 (172.16.42.1): 56 data bytes --- 172.16.42.1 ping statistics --- 1 packets transmitted, 0 packets received, 100% packet loss ``` However, using another image, pinging the router or even an external site works quite fine. Output of `docker run --rm -it alpine ping -c 1 172.16.42.1` ``` PING 172.16.42.1 (172.16.42.1): 56 data bytes 64 bytes from 172.16.42.1: seq=0 ttl=63 time=1004.398 ms --- 172.16.42.1 ping statistics --- 1 packets transmitted, 1 packets received, 0% packet loss round-trip min/avg/max = 1004.398/1004.398/1004.398 ms ``` Output of `docker run --rm -it alpine ping -c 1 example.com` ``` PING example.com (23.220.75.245): 56 data bytes 64 bytes from 23.220.75.245: seq=0 ttl=38 time=168.658 ms --- example.com ping statistics --- 1 packets transmitted, 1 packets received, 0% packet loss round-trip min/avg/max = 168.658/168.658/168.658 ms ``` What is preventing the newt image from reaching network nodes beyond the docker host? Images used: ``` REPOSITORY TAG DIGEST IMAGE ID fosrl/newt latest sha256:659c7cefac5e0678d0bf1e87c9b65d1f841ab0ac21e75f9507dfc67a42ef34da 015dfeed2246 alpine latest sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1 02f8efbefad6 ```
GiteaMirror added the stale label 2025-11-13 12:08:12 -06:00
GiteaMirror commented 2025-11-13 12:08:13 -06:00
Author
Owner
Copy Link
Copy Source

@hhftechnology commented on GitHub (Sep 28, 2025):

force the container to use a resolver instead of docker one:

services:
  newt:
    image: fosrl/newt
    dns:
      - 1.1.1.1
      - 8.8.8.8

and then try to inspect

docker exec -it newt cat /etc/resolv.conf

If it only shows 127.0.0.11, thats the issue. Overriding DNS (above) should fix it.

And how is your infra deployed?
Pangolin on VPS and tunneled to home?

@hhftechnology commented on GitHub (Sep 28, 2025): force the container to use a resolver instead of docker one: ```yaml services: newt: image: fosrl/newt dns: - 1.1.1.1 - 8.8.8.8 ``` and then try to inspect ```cli docker exec -it newt cat /etc/resolv.conf ``` If it only shows 127.0.0.11, thats the issue. Overriding DNS (above) should fix it. And how is your infra deployed? Pangolin on VPS and tunneled to home?
GiteaMirror commented 2025-11-13 12:08:13 -06:00
Author
Owner
Copy Link
Copy Source

@corvus-ch commented on GitHub (Sep 28, 2025):

Thanks, @hhftechnology for giving this some thoughts.

I already tried setting explicit DNS servers. But DNS resolution is not the root cause. DNS resolutions fails, because the newt container can not do IP connections beyond the docker host. Thus, it can not reach the DNS server located in my LAN, and it also can not reach any DNS server on the internet for it can not even reach the router.

Other containers on the same docker host have no issues in reaching any system as well as doing DNS resolution.

And how is your infra deployed? Pangolin on VPS and tunneled to home?

Pangolin is hosted inside a VM on a cloud provider and has a public IP with a DNS record pointing to that IP.

This works without issues for newt running on another host in the same LAN.

@corvus-ch commented on GitHub (Sep 28, 2025): Thanks, @hhftechnology for giving this some thoughts. I already tried setting explicit DNS servers. But DNS resolution is not the root cause. DNS resolutions fails, because the newt container can not do IP connections beyond the docker host. Thus, it can not reach the DNS server located in my LAN, and it also can not reach any DNS server on the internet for it can not even reach the router. Other containers on the same docker host have no issues in reaching any system as well as doing DNS resolution. > And how is your infra deployed? Pangolin on VPS and tunneled to home? Pangolin is hosted inside a VM on a cloud provider and has a public IP with a DNS record pointing to that IP. This works without issues for newt running on another host in the same LAN.
GiteaMirror commented 2025-11-13 12:08:13 -06:00
Author
Owner
Copy Link
Copy Source

@hhftechnology commented on GitHub (Sep 28, 2025):

some issue with your local networking.
try docker run --rm --network host fosrl/newt:latest ping -c1 api.github.com in you affected host

@hhftechnology commented on GitHub (Sep 28, 2025): some issue with your local networking. try `docker run --rm --network host fosrl/newt:latest ping -c1 api.github.com` in you affected host
GiteaMirror commented 2025-11-13 12:08:13 -06:00
Author
Owner
Copy Link
Copy Source

@corvus-ch commented on GitHub (Sep 28, 2025):

The plot thickens. Looks like the network created by docker compose is messed up.

  • Working:
    • docker run --rm fosrl/newt:latest ping -c1 api.github.com
    • docker run --rm fosrl/newt:latest ping -c1 172.16.42.1
    • docker run --rm --network host fosrl/newt:latest ping -c1 api.github.com
    • docker run --rm --network host fosrl/newt:latest ping -c1 172.16.42.1
    • docker run --rm --network bridge fosrl/newt:latest ping -c1 api.github.com
    • docker run --rm --network bridge fosrl/newt:latest ping -c1 172.16.42.1
  • Failing:
    • docker run --rm --network pangolin fosrl/newt:latest ping -c1 api.github.com
    • docker run --rm --network pangolin fosrl/newt:latest ping -c1 172.16.42.1

List of docker networks (docker network ls):

NETWORK ID     NAME       DRIVER    SCOPE
5615d88fa21f   bridge     bridge    local
475e600dbd0f   host       host      local
e52aa2959a42   none       null      local
87bbd895a97f   pangolin   bridge    local

Routing table (ip route):

default via 172.16.42.1 dev wlan0  src 172.16.42.147  metric 600
172.16.42.0/24 dev wlan0 scope link  src 172.16.42.147  metric 600
172.17.0.0/16 dev docker0 scope link  src 172.17.0.1
172.18.0.0/16 dev br-87bbd895a97f scope link  src 172.18.0.1

Firewall rules (iptables --list --table nat):

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
DOCKER     all  --  anywhere            !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE  all  --  172.18.0.0/16        anywhere
MASQUERADE  all  --  172.17.0.0/16        anywhere

Chain DOCKER (2 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

I fail to see, what the difference is between the default bridge network created by docker and the pangolin bridge network created by docker compose.

@corvus-ch commented on GitHub (Sep 28, 2025): The plot thickens. Looks like the network created by docker compose is messed up. * Working: * `docker run --rm fosrl/newt:latest ping -c1 api.github.com` * `docker run --rm fosrl/newt:latest ping -c1 172.16.42.1` * `docker run --rm --network host fosrl/newt:latest ping -c1 api.github.com` * `docker run --rm --network host fosrl/newt:latest ping -c1 172.16.42.1` * `docker run --rm --network bridge fosrl/newt:latest ping -c1 api.github.com` * `docker run --rm --network bridge fosrl/newt:latest ping -c1 172.16.42.1` * Failing: * `docker run --rm --network pangolin fosrl/newt:latest ping -c1 api.github.com` * `docker run --rm --network pangolin fosrl/newt:latest ping -c1 172.16.42.1` List of docker networks (`docker network ls`): ``` NETWORK ID NAME DRIVER SCOPE 5615d88fa21f bridge bridge local 475e600dbd0f host host local e52aa2959a42 none null local 87bbd895a97f pangolin bridge local ``` Routing table (`ip route`): ``` default via 172.16.42.1 dev wlan0 src 172.16.42.147 metric 600 172.16.42.0/24 dev wlan0 scope link src 172.16.42.147 metric 600 172.17.0.0/16 dev docker0 scope link src 172.17.0.1 172.18.0.0/16 dev br-87bbd895a97f scope link src 172.18.0.1 ``` Firewall rules (`iptables --list --table nat`): ``` Chain PREROUTING (policy ACCEPT) target prot opt source destination DOCKER all -- anywhere anywhere ADDRTYPE match dst-type LOCAL Chain INPUT (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination DOCKER all -- anywhere !127.0.0.0/8 ADDRTYPE match dst-type LOCAL Chain POSTROUTING (policy ACCEPT) target prot opt source destination MASQUERADE all -- 172.18.0.0/16 anywhere MASQUERADE all -- 172.17.0.0/16 anywhere Chain DOCKER (2 references) target prot opt source destination RETURN all -- anywhere anywhere RETURN all -- anywhere anywhere ``` I fail to see, what the difference is between the default bridge network created by docker and the pangolin bridge network created by docker compose.
GiteaMirror commented 2025-11-13 12:08:13 -06:00
Author
Owner
Copy Link
Copy Source

@corvus-ch commented on GitHub (Sep 28, 2025):

Does not seem docker compose related. Any additional bridge network seems to fail.

docker network create --driver bridge custom-net
docker run --rm --network custom-net fosrl/newt:latest ping -c1 172.16.42.1

Results in the same failure behaviour.

@corvus-ch commented on GitHub (Sep 28, 2025): Does not seem docker compose related. Any additional bridge network seems to fail. ``` docker network create --driver bridge custom-net docker run --rm --network custom-net fosrl/newt:latest ping -c1 172.16.42.1 ``` Results in the same failure behaviour.
GiteaMirror commented 2025-11-13 12:08:14 -06:00
Author
Owner
Copy Link
Copy Source

@github-actions[bot] commented on GitHub (Oct 13, 2025):

This issue has been automatically marked as stale due to 14 days of inactivity. It will be closed in 14 days if no further activity occurs.

@github-actions[bot] commented on GitHub (Oct 13, 2025): This issue has been automatically marked as stale due to 14 days of inactivity. It will be closed in 14 days if no further activity occurs.
GiteaMirror commented 2025-11-13 12:08:14 -06:00
Author
Owner
Copy Link
Copy Source

@AstralDestiny commented on GitHub (Oct 19, 2025):

Are you by chance on Ubuntu? @corvus-ch If you can can you create a container with a changed dns entry like 1.1.1.1 for the container's dns via --dns 1.1.1.1 --dns 1.0.0.1

services:
 newt:
  dns:
   - 1.1.1.1
   - 1.0.0.1

I've seen stuff like this on ubuntu hosts also make sure you're not using snap docker,

which docker
(Should return /usr/bin/docker)
snap list | grep docker
If that returns anything uninstall snap docker as those are decades old.. if not check your nameserver config on the host.

@AstralDestiny commented on GitHub (Oct 19, 2025): Are you by chance on Ubuntu? @corvus-ch If you can can you create a container with a changed dns entry like 1.1.1.1 for the container's dns via --dns 1.1.1.1 --dns 1.0.0.1 ```yml services: newt: dns: - 1.1.1.1 - 1.0.0.1 ``` I've seen stuff like this on ubuntu hosts also make sure you're not using snap docker, which docker (Should return /usr/bin/docker) snap list | grep docker If that returns anything uninstall snap docker as those are decades old.. if not check your nameserver config on the host.
GiteaMirror commented 2025-11-13 12:08:14 -06:00
Author
Owner
Copy Link
Copy Source

@corvus-ch commented on GitHub (Oct 20, 2025):

Why is everybody obsessed with DNS? Yes, yes, I know the memes.

In my case however, the container is unable to make any IP connection that goes beyond the docker host. So it does not matter what DNS server I configure, the newt container can not reach it.

I could host a DNS service on the same host, which would resolve DNS resolution. But then, newt would be unable to contact gerbil for the exact same reason, it currently fails to resolve DNS.

I am running this on PostmarketOS running on a Fairphone.

@corvus-ch commented on GitHub (Oct 20, 2025): Why is everybody obsessed with DNS? Yes, yes, I know the memes. In my case however, the container is unable to make any IP connection that goes beyond the docker host. So it does not matter what DNS server I configure, the newt container can not reach it. I could host a DNS service on the same host, which would resolve DNS resolution. But then, newt would be unable to contact gerbil for the exact same reason, it currently fails to resolve DNS. I am running this on PostmarketOS running on a Fairphone.
GiteaMirror commented 2025-11-13 12:08:14 -06:00
Author
Owner
Copy Link
Copy Source

@AstralDestiny commented on GitHub (Oct 24, 2025):

Docker will still use 127.0.0.11:53 as that's the internal docker dns but sometimes it gets broken if the host has something funky or stuff like pihole or whatnot on top of it. Oh some phone thing no idea. Could be any numerous things then. making a vm for this already looks annoying.

@AstralDestiny commented on GitHub (Oct 24, 2025): Docker will still use 127.0.0.11:53 as that's the internal docker dns but sometimes it gets broken if the host has something funky or stuff like pihole or whatnot on top of it. Oh some phone thing no idea. Could be any numerous things then. making a vm for this already looks annoying.
GiteaMirror commented 2025-11-13 12:08:14 -06:00
Author
Owner
Copy Link
Copy Source

@github-actions[bot] commented on GitHub (Nov 8, 2025):

This issue has been automatically marked as stale due to 14 days of inactivity. It will be closed in 14 days if no further activity occurs.

@github-actions[bot] commented on GitHub (Nov 8, 2025): This issue has been automatically marked as stale due to 14 days of inactivity. It will be closed in 14 days if no further activity occurs.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
api
authentication
bug
config
dependencies
docker
documentation
enhancement
good first issue
help wanted
Improvement
Look Into
needs investigating
networking
new feature
non-critical bug
potential bug
pull-request
Mirrored from GitHub Pull Request
 question
reverse proxy
Security
stale
ui
wontfix
No labels stale
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
GiteaMirror ninjasurge (Kaehlen Rog)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/pangolin#702