github-starred/pangolin
2
0
Fork 0
mirror of https://github.com/fosrl/pangolin.git synced 2026-08-26 09:04:44 -05:00
Code Issues 1.6k Packages Projects Releases 79 Wiki Activity

[GH-ISSUE #617] Newt cannot ping Wireguard server IP after clean installation #3472

New Issue
Closed
opened 2026-04-20 07:26:11 -05:00 by GiteaMirror · 4 comments
GiteaMirror commented 2026-04-20 07:26:11 -05:00
Owner
Copy Link
Copy Source

Originally created by @mkadrlik on GitHub (Apr 29, 2025).
Original GitHub issue: https://github.com/fosrl/pangolin/issues/617

Hello!

I have been attempting to find a solution for this issue on my own, but I believe I have hit an impasse, as I can't explain why Newt is unable to ping the Gerbil Wireguard server IP address. It should be noted that this has worked in the past flawlessly, but stopped in the last month or so (I've been working on it intermittently). I have done the following to try and remediate on my own:

  • Complete uninstall and reinstallation of Pangolin following these instructions
  • Changed IP range to multiple different IP addressed either within the subnet range of the Docker network, or using the 100...* range that comes OOTB
  • Tried running newt from terminal and Docker compose files
  • Changed port # to a new # and back to the original #
  • Walked versions back down through beta
  • Tried with both Badger running and not
  • Tried different networks for all Docker components

Below are the Docker compose files I am using:

name: pangolin
services:
  pangolin:
    image: fosrl/pangolin:1.2.0
    container_name: pangolin
    restart: unless-stopped
    volumes:
      - ./config:/app/config
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3001/api/v1/"]
      interval: "3s"
      timeout: "3s"
      retries: 5

  gerbil:
    image: fosrl/gerbil:1.0.0
    container_name: gerbil
    restart: unless-stopped
    depends_on:
      pangolin:
        condition: service_healthy
    command:
      - --reachableAt=http://gerbil:3003
      - --generateAndSaveKeyTo=/var/config/key
      - --remoteConfig=http://pangolin:3001/api/v1/gerbil/get-config
      - --reportBandwidthTo=http://pangolin:3001/api/v1/gerbil/receive-bandwidth
      - --log-level=DEBUG
    volumes:
      - ./config/:/var/config
    cap_add:
      - NET_ADMIN
      - SYS_MODULE
    ports:
      - 51820:51820/udp
      - 443:443 # Port for traefik because of the network_mode
      - 80:80 # Port for traefik because of the network_mode

  traefik:
    image: traefik:v3.3.3
    container_name: traefik
    restart: unless-stopped

    network_mode: service:gerbil # Ports appear on the gerbil service

    depends_on:
      pangolin:
        condition: service_healthy
    command:
      - --configFile=/etc/traefik/traefik_config.yml
    volumes:
      - ./config/traefik:/etc/traefik:ro # Volume to store the Traefik configuration
      - ./config/letsencrypt:/letsencrypt # Volume to store the Let's Encrypt certificates
      - ./config/traefik/logs:/var/log/traefik # Volume to store Traefik logs

networks:
  default:
    driver: bridge
    name: pangolin

Newt:

services:
  newt:
    image: fosrl/newt
    container_name: newt
    restart: unless-stopped
    environment:
      - PANGOLIN_ENDPOINT=<REDACTED>
      - NEWT_ID=<REDACTED>
      - NEWT_SECRET=<REDACTED>
      - LOG_LEVEL=DEBUG

networks:
  default:
    driver: bridge
    name: pangolin
    external: true

As it stands, I just can't get good enough logging to truly understand the issue, so I'm not certain where to go next. Here are a collection of the logs I have for you all to review, by Docker application:

Newt:

DEBUG: 2025/04/29 17:43:59 Public key: <REDACTED>
INFO: 2025/04/29 17:43:59 Sent registration message
INFO: 2025/04/29 17:43:59 Received registration message
INFO: 2025/04/29 17:43:59 Received: {Type:newt/wg/connect Data:map[endpoint:<REDACTED>:51820 publicKey:<REDACTED> serverIP:172.50.8.1 targets:map[tcp:[] udp:[]] tunnelIP:172.50.8.4]}
DEBUG: wireguard: 2025/04/29 17:43:59 Routine: decryption worker 2 - started
DEBUG: wireguard: 2025/04/29 17:43:59 Routine: encryption worker 4 - started
DEBUG: wireguard: 2025/04/29 17:43:59 Routine: TUN reader - started
DEBUG: wireguard: 2025/04/29 17:43:59 Routine: encryption worker 3 - started
DEBUG: wireguard: 2025/04/29 17:43:59 Routine: decryption worker 1 - started
DEBUG: wireguard: 2025/04/29 17:43:59 Routine: encryption worker 1 - started
DEBUG: wireguard: 2025/04/29 17:43:59 Routine: event worker - started
DEBUG: wireguard: 2025/04/29 17:43:59 Interface up requested
DEBUG: wireguard: 2025/04/29 17:43:59 Routine: handshake worker 1 - started
DEBUG: wireguard: 2025/04/29 17:43:59 Routine: encryption worker 2 - started
DEBUG: wireguard: 2025/04/29 17:43:59 Routine: handshake worker 4 - started
DEBUG: wireguard: 2025/04/29 17:43:59 Routine: handshake worker 3 - started
DEBUG: wireguard: 2025/04/29 17:43:59 Routine: decryption worker 3 - started
DEBUG: wireguard: 2025/04/29 17:43:59 Routine: decryption worker 4 - started
DEBUG: wireguard: 2025/04/29 17:43:59 Routine: handshake worker 2 - started
DEBUG: wireguard: 2025/04/29 17:43:59 UDP bind has been updated
DEBUG: wireguard: 2025/04/29 17:43:59 Interface state was Down, requested Up, now Up
DEBUG: wireguard: 2025/04/29 17:43:59 Routine: receive incoming v4 - started
DEBUG: wireguard: 2025/04/29 17:43:59 Routine: receive incoming v6 - started
DEBUG: wireguard: 2025/04/29 17:43:59 UAPI: Updating private key
DEBUG: wireguard: 2025/04/29 17:43:59 peer(9Moy…Y10g) - UAPI: Created
DEBUG: wireguard: 2025/04/29 17:43:59 peer(9Moy…Y10g) - UAPI: Adding allowedip
DEBUG: wireguard: 2025/04/29 17:43:59 peer(9Moy…Y10g) - UAPI: Updating endpoint
DEBUG: wireguard: 2025/04/29 17:43:59 peer(9Moy…Y10g) - UAPI: Updating persistent keepalive interval
DEBUG: wireguard: 2025/04/29 17:43:59 peer(9Moy…Y10g) - Starting
DEBUG: wireguard: 2025/04/29 17:43:59 peer(9Moy…Y10g) - Sending keepalive packet
DEBUG: wireguard: 2025/04/29 17:43:59 peer(9Moy…Y10g) - Sending handshake initiation
DEBUG: wireguard: 2025/04/29 17:43:59 peer(9Moy…Y10g) - Routine: sequential sender - started
DEBUG: wireguard: 2025/04/29 17:43:59 peer(9Moy…Y10g) - Routine: sequential receiver - started
INFO: 2025/04/29 17:43:59 WireGuard device created. Lets ping the server now...
INFO: 2025/04/29 17:43:59 Ping attempt 1
INFO: 2025/04/29 17:43:59 Pinging 172.50.8.1
DEBUG: wireguard: 2025/04/29 17:44:04 peer(9Moy…Y10g) - Sending handshake initiation
WARN: 2025/04/29 17:44:09 Ping attempt 1 failed: failed to read ICMP packet: i/o timeout
INFO: 2025/04/29 17:44:09 Starting ping check
INFO: 2025/04/29 17:44:09 Ping attempt 2
INFO: 2025/04/29 17:44:09 Pinging 172.50.8.1
DEBUG: wireguard: 2025/04/29 17:44:09 peer(9Moy…Y10g) - Sending handshake initiation
DEBUG: wireguard: 2025/04/29 17:44:14 peer(9Moy…Y10g) - Sending handshake initiation
WARN: 2025/04/29 17:44:19 Ping attempt 2 failed: failed to read ICMP packet: i/o timeout
DEBUG: wireguard: 2025/04/29 17:44:19 peer(9Moy…Y10g) - Sending handshake initiation
INFO: 2025/04/29 17:44:21 Ping attempt 3
INFO: 2025/04/29 17:44:21 Pinging 172.50.8.1
DEBUG: wireguard: 2025/04/29 17:44:24 peer(9Moy…Y10g) - Sending handshake initiation
DEBUG: wireguard: 2025/04/29 17:44:29 peer(9Moy…Y10g) - Sending handshake initiation
WARN: 2025/04/29 17:44:31 Ping attempt 3 failed: failed to read ICMP packet: i/o timeout
INFO: 2025/04/29 17:44:33 Ping attempt 4
INFO: 2025/04/29 17:44:33 Pinging 172.50.8.1
DEBUG: wireguard: 2025/04/29 17:44:34 peer(9Moy…Y10g) - Sending handshake initiation
INFO: 2025/04/29 17:44:39 Pinging 172.50.8.1
DEBUG: wireguard: 2025/04/29 17:44:39 peer(9Moy…Y10g) - Sending handshake initiation
WARN: 2025/04/29 17:44:43 Ping attempt 4 failed: failed to read ICMP packet: i/o timeout
DEBUG: wireguard: 2025/04/29 17:44:44 peer(9Moy…Y10g) - Sending handshake initiation
INFO: 2025/04/29 17:44:45 Ping attempt 5
INFO: 2025/04/29 17:44:45 Pinging 172.50.8.1
WARN: 2025/04/29 17:44:49 Connection to server lost. Continuous reconnection attempts will be made.
WARN: 2025/04/29 17:44:49 Please check your internet connection and ensure the Pangolin server is online.
WARN: 2025/04/29 17:44:49 Newt will continue reconnection attempts automatically when connectivity is restored.
DEBUG: wireguard: 2025/04/29 17:44:49 peer(9Moy…Y10g) - Sending handshake initiation

Pangolin:

> @fosrl/pangolin@0.0.0 start
> NODE_OPTIONS=--enable-source-maps NODE_ENV=development ENVIRONMENT=prod sh -c 'node dist/migrations.mjs && node dist/server.mjs'
Starting migrations from version 1.2.0
Migrations to run: 
All migrations completed successfully
2025-04-29T17:08:14.167Z [warn]: Email SMTP configuration is missing. Emails will not be sent.
2025-04-29T17:08:14.325Z [debug]: Using CORS options {"origin":["<REDACTED>"],"methods":["GET","POST","PUT","DELETE","PATCH"],"credentials":false}
2025-04-29T17:08:14.604Z [info]: API server is running on http://localhost:3000
2025-04-29T17:08:14.605Z [info]: Internal server is running on http://localhost:3001
2025-04-29T17:08:15.035Z [info]: Next.js server is running on http://localhost:3002
2025-04-29T17:08:16.479Z [debug]: Sending config:  {"listenPort":51820,"ipAddress":"172.50.8.1/24","peers":[{"publicKey":"<REDACTED>","allowedIps":[]}]}
2025-04-29T17:09:36.738Z [debug]: POST /api/v1/auth/newt/get-token
2025-04-29T17:09:36.980Z [info]: Establishing websocket connection
2025-04-29T17:09:36.980Z [info]: Client added to tracking - Newt ID: xds2wjz4vn8zk0x, Total connections: 1
2025-04-29T17:09:36.980Z [info]: WebSocket connection established - Newt ID: xds2wjz4vn8zk0x
2025-04-29T17:09:36.996Z [info]: Handling register message!
2025-04-29T17:09:37.103Z [info]: Public key mismatch. Deleting old peer...
2025-04-29T17:09:37.134Z [info]: Peer deleted successfully: {"0":"P","1":"e","2":"e","3":"r","4":" ","5":"r","6":"e","7":"m","8":"o","9":"v","10":"e","11":"d","12":" ","13":"s","14":"u","15":"c","16":"c","17":"e","18":"s","19":"s","20":"f","21":"u","22":"l","23":"l","24":"y"}
2025-04-29T17:09:37.137Z [info]: Peer added successfully: {"0":"P","1":"e","2":"e","3":"r","4":" ","5":"a","6":"d","7":"d","8":"e","9":"d","10":" ","11":"s","12":"u","13":"c","14":"c","15":"e","16":"s","17":"s","18":"f","19":"u","20":"l","21":"l","22":"y"}

Traefik:

2025-04-29T17:08:17Z INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:107 > Traefik version 3.3.3 built on 2025-01-31T14:55:01Z version=3.3.3
api={"basePath":"/","dashboard":true,"insecure":true} certificatesResolvers={"letsencrypt":{"acme":{"caServer":"https://acme-v02.api.letsencrypt.org/directory","certificatesDuration":2160,"email":"<REDACTED>","httpChallenge":{"entryPoint":"web"},"keyType":"RSA4096","storage":"/letsencrypt/acme.json"}}} entryPoints={"traefik":{"address":":8080","forwardedHeaders":{},"http":{"maxHeaderBytes":1048576},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}},"web":{"address":":80","forwardedHeaders":{},"http":{"maxHeaderBytes":1048576},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}},"websecure":{"address":":443","forwardedHeaders":{},"http":{"maxHeaderBytes":1048576,"tls":{"certResolver":"letsencrypt"}},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"30m0s"}},"udp":{"timeout":"3s"}}} global={"checkNewVersion":true} log={"format":"common","level":"DEBUG"} providers={"file":{"filename":"/etc/traefik/dynamic_config.yml","watch":true},"http":{"endpoint":"http://pangolin:3001/api/v1/traefik-config","pollInterval":"30s","pollTimeout":"5s"},"providersThrottleDuration":"2s"} serversTransport={"insecureSkipVerify":true,"maxIdleConnsPerHost":200} tcpServersTransport={"dialKeepAlive":"15s","dialTimeout":"30s"}
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/cmd/traefik/traefik.go:114 > Static configuration loaded [json] staticConfiguration=
2025-04-29T17:08:17Z INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:633 > 
Stats collection is disabled.
Help us improve Traefik by turning this feature on :)
More details on: https://doc.traefik.io/traefik/contributing/data-collection/
2025-04-29T17:08:17Z INF github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:73 > Starting provider aggregator *aggregator.ProviderAggregator
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:231 > Starting TCP Server entryPointName=websecure
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:231 > Starting TCP Server entryPointName=web
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:231 > Starting TCP Server entryPointName=traefik
2025-04-29T17:08:17Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *file.Provider
filename=/etc/traefik/dynamic_config.yml watch=true
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *file.Provider provider configuration config=
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:122 > add watcher on: /etc/traefik
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:122 > add watcher on: /etc/traefik/dynamic_config.yml
2025-04-29T17:08:17Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *traefik.Provider
2025-04-29T17:08:17Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *http.Provider
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *traefik.Provider provider configuration config=
endpoint=http://pangolin:3001/api/v1/traefik-config pollInterval=30s pollTimeout=5s
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *http.Provider provider configuration config=
2025-04-29T17:08:17Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *acme.ChallengeTLSALPN
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *acme.ChallengeTLSALPN provider configuration config=
2025-04-29T17:08:17Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *acme.Provider
HTTPChallengeProvider={} ResolverName=letsencrypt TLSChallengeProvider={} caServer=https://acme-v02.api.letsencrypt.org/directory certificatesDuration=2160 email=<REDACTED> httpChallenge={"entryPoint":"web"} keyType=RSA4096 storage=/letsencrypt/acme.json store={}
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *acme.Provider provider configuration config=
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:232 > Attempt to renew certificates "720h0m0s" before expiry and check every "24h0m0s" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme
2025-04-29T17:08:17Z INF github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:884 > Testing certificate renew... acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme
http={"middlewares":{"redirect-to-https":{"redirectScheme":{"scheme":"https"}}},"routers":{"api-router":{"entryPoints":["websecure"],"rule":"Host(`<REDACTED>`) && PathPrefix(`/api/v1`)","service":"api-service","tls":{"certResolver":"letsencrypt"}},"main-app-router-redirect":{"entryPoints":["web"],"middlewares":["redirect-to-https"],"rule":"Host(`<REDACTED>`)","service":"next-service"},"next-router":{"entryPoints":["websecure"],"rule":"Host(`<REDACTED>`) && !PathPrefix(`/api/v1`)","service":"next-service","tls":{"certResolver":"letsencrypt"}},"ws-router":{"entryPoints":["websecure"],"rule":"Host(`<REDACTED>`)","service":"api-service","tls":{"certResolver":"letsencrypt"}}},"services":{"api-service":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://pangolin:3000"}]}},"next-service":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://pangolin:3002"}]}}}} tcp={} tls={} udp={}
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config=providerName=file
http={"middlewares":{"dashboard_redirect":{"redirectRegex":{"permanent":true,"regex":"^(http:\\/\\/(\\[[\\w:.]+\\]|[\\w\\._-]+)(:\\d+)?)\\/$","replacement":"${1}/dashboard/"}},"dashboard_stripprefix":{"stripPrefix":{"prefixes":["/dashboard/","/dashboard"]}}},"models":{"websecure":{"observability":{},"tls":{"certResolver":"letsencrypt"}}},"routers":{"acme-http":{"entryPoints":["web"],"priority":9223372036854776000,"rule":"PathPrefix(`/.well-known/acme-challenge/`)","ruleSyntax":"v3","service":"acme-http@internal"},"api":{"entryPoints":["traefik"],"priority":9223372036854776000,"rule":"PathPrefix(`/api`)","ruleSyntax":"v3","service":"api@internal"},"dashboard":{"entryPoints":["traefik"],"middlewares":["dashboard_redirect@internal","dashboard_stripprefix@internal"],"priority":9223372036854776000,"rule":"PathPrefix(`/`)","ruleSyntax":"v3","service":"dashboard@internal"}},"serversTransports":{"default":{"insecureSkipVerify":true,"maxIdleConnsPerHost":200}},"services":{"acme-http":{},"api":{},"dashboard":{},"noop":{}}} tcp={"serversTransports":{"default":{"dialKeepAlive":"15s","dialTimeout":"30s"}}} tls={} udp={}
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config=providerName=internal
http={} tcp={} tls={} udp={}
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config=providerName=letsencrypt.acme
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/tls/certificate.go:132 > Adding certificate for domain(s) <REDACTED>
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/tls/certificate.go:132 > Adding certificate for domain(s) <REDACTED>
http={} tcp={} tls={} udp={}
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config=providerName=http
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321 > No default certificate, fallback to the internal generated certificate tlsStoreName=default
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:313 > Creating load-balancer entryPointName=web routerName=main-app-router-redirect@file serviceName=next-service@file
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:350 > Creating server entryPointName=web routerName=main-app-router-redirect@file serverName=889f9773a10bf3fe serviceName=next-service@file target=http://pangolin:3002
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:29 > Creating middleware entryPointName=web middlewareName=redirect-to-https@file middlewareType=RedirectScheme routerName=main-app-router-redirect@file
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:30 > Setting up redirection to https  entryPointName=web middlewareName=redirect-to-https@file middlewareType=RedirectScheme routerName=main-app-router-redirect@file
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/stripprefix/strip_prefix.go:32 > Creating middleware entryPointName=traefik middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix routerName=dashboard@internal
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=traefik middlewareName=dashboard_stripprefix@internal routerName=dashboard@internal
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_regex.go:17 > Creating middleware entryPointName=traefik middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex routerName=dashboard@internal
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_regex.go:18 > Setting up redirection from ^(http:\/\/(\[[\w:.]+\]|[\w\._-]+)(:\d+)?)\/$ to ${1}/dashboard/ entryPointName=traefik middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex routerName=dashboard@internal
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=traefik middlewareName=dashboard_redirect@internal routerName=dashboard@internal
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:313 > Creating load-balancer entryPointName=websecure routerName=ws-router@file serviceName=api-service@file
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:350 > Creating server entryPointName=websecure routerName=ws-router@file serverName=889f9573a10bf098 serviceName=api-service@file target=http://pangolin:3000
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/router/tcp/manager.go:237 > Adding route for <REDACTED> with TLS options default entryPointName=websecure
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:470 > Trying to challenge certificate for domain [<REDACTED>] found in HostSNI rule ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme routerName=ws-router@file rule=Host(`<REDACTED>`)
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:470 > Trying to challenge certificate for domain [<REDACTED>] found in HostSNI rule ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme routerName=api-router@file rule="Host(`<REDACTED>`) && PathPrefix(`/api/v1`)"
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:940 > Looking for provided certificate(s) to validate ["<REDACTED>"]... ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme routerName=ws-router@file rule=Host(`<REDACTED>`)
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:470 > Trying to challenge certificate for domain [<REDACTED>] found in HostSNI rule ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme routerName=next-router@file rule="Host(`<REDACTED>`) && !PathPrefix(`/api/v1`)"
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:940 > Looking for provided certificate(s) to validate ["<REDACTED>"]... ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme routerName=next-router@file rule="Host(`<REDACTED>`) && !PathPrefix(`/api/v1`)"
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/tls/certificate.go:132 > Adding certificate for domain(s) <REDACTED>
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:940 > Looking for provided certificate(s) to validate ["<REDACTED>"]... ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme routerName=api-router@file rule="Host(`<REDACTED>`) && PathPrefix(`/api/v1`)"
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/tls/certificate.go:132 > Adding certificate for domain(s) <REDACTED>
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321 > No default certificate, fallback to the internal generated certificate tlsStoreName=default
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:984 > No ACME certificate generation required for domains ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory domains=["<REDACTED>"] providerName=letsencrypt.acme routerName=ws-router@file rule=Host(`<REDACTED>`)
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:984 > No ACME certificate generation required for domains ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory domains=["<REDACTED>"] providerName=letsencrypt.acme routerName=api-router@file rule="Host(`<REDACTED>`) && PathPrefix(`/api/v1`)"
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:984 > No ACME certificate generation required for domains ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory domains=["<REDACTED>"] providerName=letsencrypt.acme routerName=next-router@file rule="Host(`<REDACTED>`) && !PathPrefix(`/api/v1`)"
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:313 > Creating load-balancer entryPointName=web routerName=main-app-router-redirect@file serviceName=next-service@file
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:350 > Creating server entryPointName=web routerName=main-app-router-redirect@file serverName=889f9773a10bf3fe serviceName=next-service@file target=http://pangolin:3002
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:29 > Creating middleware entryPointName=web middlewareName=redirect-to-https@file middlewareType=RedirectScheme routerName=main-app-router-redirect@file
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:30 > Setting up redirection to https  entryPointName=web middlewareName=redirect-to-https@file middlewareType=RedirectScheme routerName=main-app-router-redirect@file
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/stripprefix/strip_prefix.go:32 > Creating middleware entryPointName=traefik middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix routerName=dashboard@internal
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=traefik middlewareName=dashboard_stripprefix@internal routerName=dashboard@internal
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_regex.go:17 > Creating middleware entryPointName=traefik middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex routerName=dashboard@internal
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_regex.go:18 > Setting up redirection from ^(http:\/\/(\[[\w:.]+\]|[\w\._-]+)(:\d+)?)\/$ to ${1}/dashboard/ entryPointName=traefik middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex routerName=dashboard@internal
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=traefik middlewareName=dashboard_redirect@internal routerName=dashboard@internal
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:313 > Creating load-balancer entryPointName=websecure routerName=ws-router@file serviceName=api-service@file
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:350 > Creating server entryPointName=websecure routerName=ws-router@file serverName=889f9573a10bf098 serviceName=api-service@file target=http://pangolin:3000
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/router/tcp/manager.go:237 > Adding route for <REDACTED> with TLS options default entryPointName=websecure
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:470 > Trying to challenge certificate for domain [<REDACTED>] found in HostSNI rule ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme routerName=api-router@file rule="Host(`<REDACTED>`) && PathPrefix(`/api/v1`)"
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:470 > Trying to challenge certificate for domain [<REDACTED>] found in HostSNI rule ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme routerName=next-router@file rule="Host(`<REDACTED>`) && !PathPrefix(`/api/v1`)"
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:940 > Looking for provided certificate(s) to validate ["<REDACTED>"]... ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme routerName=api-router@file rule="Host(`<REDACTED>`) && PathPrefix(`/api/v1`)"
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:470 > Trying to challenge certificate for domain [<REDACTED>] found in HostSNI rule ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme routerName=ws-router@file rule=Host(`<REDACTED>`)
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:940 > Looking for provided certificate(s) to validate ["<REDACTED>"]... ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme routerName=ws-router@file rule=Host(`<REDACTED>`)
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:984 > No ACME certificate generation required for domains ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory domains=["<REDACTED>"] providerName=letsencrypt.acme routerName=api-router@file rule="Host(`<REDACTED>`) && PathPrefix(`/api/v1`)"
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:940 > Looking for provided certificate(s) to validate ["<REDACTED>"]... ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme routerName=next-router@file rule="Host(`<REDACTED>`) && !PathPrefix(`/api/v1`)"
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:984 > No ACME certificate generation required for domains ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory domains=["<REDACTED>"] providerName=letsencrypt.acme routerName=ws-router@file rule=Host(`<REDACTED>`)
2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:984 > No ACME certificate generation required for domains ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory domains=["<REDACTED>"] providerName=letsencrypt.acme routerName=next-router@file rule="Host(`<REDACTED>`) && !PathPrefix(`/api/v1`)"
2025-04-29T17:09:36Z DBG github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:207 > Service selected by WRR: 889f9573a10bf098
2025-04-29T17:09:36Z DBG github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:207 > Service selected by WRR: 889f9573a10bf098

Gerbil:

INFO: 2025/04/29 17:08:16 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config
INFO: 2025/04/29 17:08:16 Created WireGuard interface wg0
INFO: 2025/04/29 17:08:16 Assigned IP address 172.50.8.1/24 to interface wg0
INFO: 2025/04/29 17:08:16 Attempting to delete existing MSS clamping rule for chain INPUT
DEBUG: 2025/04/29 17:08:16 Deletion stopped for chain INPUT: exit status 1 (output: iptables: Bad rule (does a matching rule exist in that chain?).
)
INFO: 2025/04/29 17:08:16 Attempting to delete existing MSS clamping rule for chain OUTPUT
DEBUG: 2025/04/29 17:08:16 Deletion stopped for chain OUTPUT: exit status 1 (output: iptables: Bad rule (does a matching rule exist in that chain?).
)
INFO: 2025/04/29 17:08:16 Attempting to delete existing MSS clamping rule for chain FORWARD
DEBUG: 2025/04/29 17:08:16 Deletion stopped for chain FORWARD: exit status 1 (output: iptables: Bad rule (does a matching rule exist in that chain?).
)
INFO: 2025/04/29 17:08:16 Adding MSS clamping rule for chain INPUT
INFO: 2025/04/29 17:08:16 Successfully added and verified MSS clamping rule for chain INPUT
INFO: 2025/04/29 17:08:16 Adding MSS clamping rule for chain OUTPUT
INFO: 2025/04/29 17:08:16 Successfully added and verified MSS clamping rule for chain OUTPUT
INFO: 2025/04/29 17:08:16 Adding MSS clamping rule for chain FORWARD
INFO: 2025/04/29 17:08:16 Successfully added and verified MSS clamping rule for chain FORWARD
INFO: 2025/04/29 17:08:16 WireGuard interface wg0 created and configured
INFO: 2025/04/29 17:08:16 Peer <REDACTED> added successfully
INFO: 2025/04/29 17:08:16 Starting server on :3003
INFO: 2025/04/29 17:09:37 Peer <REDACTED> removed successfully
INFO: 2025/04/29 17:09:37 Peer <REDACTED> added successfully

Machine: ZimaCube running ZimaOS (a fork of CasaOS)

Please let me know if there's any other information I can provide to help, or if there's any tests that anyone would like me to run to identify the RC for this issue. Thanks in advance!

Originally created by @mkadrlik on GitHub (Apr 29, 2025). Original GitHub issue: https://github.com/fosrl/pangolin/issues/617 Hello! I have been attempting to find a solution for this issue on my own, but I believe I have hit an impasse, as I can't explain why Newt is unable to ping the Gerbil Wireguard server IP address. It should be noted that this has worked in the past flawlessly, but stopped in the last month or so (I've been working on it intermittently). I have done the following to try and remediate on my own: - Complete uninstall and reinstallation of Pangolin following [these instructions](https://docs.fossorial.io/Getting%20Started/quick-install) - Changed IP range to multiple different IP addressed either within the subnet range of the Docker network, or using the 100.*.*.* range that comes OOTB - Tried running newt from terminal and Docker compose files - Changed port # to a new # and back to the original # - Walked versions back down through beta - Tried with both Badger running and not - Tried different networks for all Docker components Below are the Docker compose files I am using: ``` yaml name: pangolin services: pangolin: image: fosrl/pangolin:1.2.0 container_name: pangolin restart: unless-stopped volumes: - ./config:/app/config healthcheck: test: ["CMD", "curl", "-f", "http://localhost:3001/api/v1/"] interval: "3s" timeout: "3s" retries: 5 gerbil: image: fosrl/gerbil:1.0.0 container_name: gerbil restart: unless-stopped depends_on: pangolin: condition: service_healthy command: - --reachableAt=http://gerbil:3003 - --generateAndSaveKeyTo=/var/config/key - --remoteConfig=http://pangolin:3001/api/v1/gerbil/get-config - --reportBandwidthTo=http://pangolin:3001/api/v1/gerbil/receive-bandwidth - --log-level=DEBUG volumes: - ./config/:/var/config cap_add: - NET_ADMIN - SYS_MODULE ports: - 51820:51820/udp - 443:443 # Port for traefik because of the network_mode - 80:80 # Port for traefik because of the network_mode traefik: image: traefik:v3.3.3 container_name: traefik restart: unless-stopped network_mode: service:gerbil # Ports appear on the gerbil service depends_on: pangolin: condition: service_healthy command: - --configFile=/etc/traefik/traefik_config.yml volumes: - ./config/traefik:/etc/traefik:ro # Volume to store the Traefik configuration - ./config/letsencrypt:/letsencrypt # Volume to store the Let's Encrypt certificates - ./config/traefik/logs:/var/log/traefik # Volume to store Traefik logs networks: default: driver: bridge name: pangolin ``` Newt: ``` yaml services: newt: image: fosrl/newt container_name: newt restart: unless-stopped environment: - PANGOLIN_ENDPOINT=<REDACTED> - NEWT_ID=<REDACTED> - NEWT_SECRET=<REDACTED> - LOG_LEVEL=DEBUG networks: default: driver: bridge name: pangolin external: true ``` As it stands, I just can't get good enough logging to truly understand the issue, so I'm not certain where to go next. Here are a collection of the logs I have for you all to review, by Docker application: Newt: ``` DEBUG: 2025/04/29 17:43:59 Public key: <REDACTED> INFO: 2025/04/29 17:43:59 Sent registration message INFO: 2025/04/29 17:43:59 Received registration message INFO: 2025/04/29 17:43:59 Received: {Type:newt/wg/connect Data:map[endpoint:<REDACTED>:51820 publicKey:<REDACTED> serverIP:172.50.8.1 targets:map[tcp:[] udp:[]] tunnelIP:172.50.8.4]} DEBUG: wireguard: 2025/04/29 17:43:59 Routine: decryption worker 2 - started DEBUG: wireguard: 2025/04/29 17:43:59 Routine: encryption worker 4 - started DEBUG: wireguard: 2025/04/29 17:43:59 Routine: TUN reader - started DEBUG: wireguard: 2025/04/29 17:43:59 Routine: encryption worker 3 - started DEBUG: wireguard: 2025/04/29 17:43:59 Routine: decryption worker 1 - started DEBUG: wireguard: 2025/04/29 17:43:59 Routine: encryption worker 1 - started DEBUG: wireguard: 2025/04/29 17:43:59 Routine: event worker - started DEBUG: wireguard: 2025/04/29 17:43:59 Interface up requested DEBUG: wireguard: 2025/04/29 17:43:59 Routine: handshake worker 1 - started DEBUG: wireguard: 2025/04/29 17:43:59 Routine: encryption worker 2 - started DEBUG: wireguard: 2025/04/29 17:43:59 Routine: handshake worker 4 - started DEBUG: wireguard: 2025/04/29 17:43:59 Routine: handshake worker 3 - started DEBUG: wireguard: 2025/04/29 17:43:59 Routine: decryption worker 3 - started DEBUG: wireguard: 2025/04/29 17:43:59 Routine: decryption worker 4 - started DEBUG: wireguard: 2025/04/29 17:43:59 Routine: handshake worker 2 - started DEBUG: wireguard: 2025/04/29 17:43:59 UDP bind has been updated DEBUG: wireguard: 2025/04/29 17:43:59 Interface state was Down, requested Up, now Up DEBUG: wireguard: 2025/04/29 17:43:59 Routine: receive incoming v4 - started DEBUG: wireguard: 2025/04/29 17:43:59 Routine: receive incoming v6 - started DEBUG: wireguard: 2025/04/29 17:43:59 UAPI: Updating private key DEBUG: wireguard: 2025/04/29 17:43:59 peer(9Moy…Y10g) - UAPI: Created DEBUG: wireguard: 2025/04/29 17:43:59 peer(9Moy…Y10g) - UAPI: Adding allowedip DEBUG: wireguard: 2025/04/29 17:43:59 peer(9Moy…Y10g) - UAPI: Updating endpoint DEBUG: wireguard: 2025/04/29 17:43:59 peer(9Moy…Y10g) - UAPI: Updating persistent keepalive interval DEBUG: wireguard: 2025/04/29 17:43:59 peer(9Moy…Y10g) - Starting DEBUG: wireguard: 2025/04/29 17:43:59 peer(9Moy…Y10g) - Sending keepalive packet DEBUG: wireguard: 2025/04/29 17:43:59 peer(9Moy…Y10g) - Sending handshake initiation DEBUG: wireguard: 2025/04/29 17:43:59 peer(9Moy…Y10g) - Routine: sequential sender - started DEBUG: wireguard: 2025/04/29 17:43:59 peer(9Moy…Y10g) - Routine: sequential receiver - started INFO: 2025/04/29 17:43:59 WireGuard device created. Lets ping the server now... INFO: 2025/04/29 17:43:59 Ping attempt 1 INFO: 2025/04/29 17:43:59 Pinging 172.50.8.1 DEBUG: wireguard: 2025/04/29 17:44:04 peer(9Moy…Y10g) - Sending handshake initiation WARN: 2025/04/29 17:44:09 Ping attempt 1 failed: failed to read ICMP packet: i/o timeout INFO: 2025/04/29 17:44:09 Starting ping check INFO: 2025/04/29 17:44:09 Ping attempt 2 INFO: 2025/04/29 17:44:09 Pinging 172.50.8.1 DEBUG: wireguard: 2025/04/29 17:44:09 peer(9Moy…Y10g) - Sending handshake initiation DEBUG: wireguard: 2025/04/29 17:44:14 peer(9Moy…Y10g) - Sending handshake initiation WARN: 2025/04/29 17:44:19 Ping attempt 2 failed: failed to read ICMP packet: i/o timeout DEBUG: wireguard: 2025/04/29 17:44:19 peer(9Moy…Y10g) - Sending handshake initiation INFO: 2025/04/29 17:44:21 Ping attempt 3 INFO: 2025/04/29 17:44:21 Pinging 172.50.8.1 DEBUG: wireguard: 2025/04/29 17:44:24 peer(9Moy…Y10g) - Sending handshake initiation DEBUG: wireguard: 2025/04/29 17:44:29 peer(9Moy…Y10g) - Sending handshake initiation WARN: 2025/04/29 17:44:31 Ping attempt 3 failed: failed to read ICMP packet: i/o timeout INFO: 2025/04/29 17:44:33 Ping attempt 4 INFO: 2025/04/29 17:44:33 Pinging 172.50.8.1 DEBUG: wireguard: 2025/04/29 17:44:34 peer(9Moy…Y10g) - Sending handshake initiation INFO: 2025/04/29 17:44:39 Pinging 172.50.8.1 DEBUG: wireguard: 2025/04/29 17:44:39 peer(9Moy…Y10g) - Sending handshake initiation WARN: 2025/04/29 17:44:43 Ping attempt 4 failed: failed to read ICMP packet: i/o timeout DEBUG: wireguard: 2025/04/29 17:44:44 peer(9Moy…Y10g) - Sending handshake initiation INFO: 2025/04/29 17:44:45 Ping attempt 5 INFO: 2025/04/29 17:44:45 Pinging 172.50.8.1 WARN: 2025/04/29 17:44:49 Connection to server lost. Continuous reconnection attempts will be made. WARN: 2025/04/29 17:44:49 Please check your internet connection and ensure the Pangolin server is online. WARN: 2025/04/29 17:44:49 Newt will continue reconnection attempts automatically when connectivity is restored. DEBUG: wireguard: 2025/04/29 17:44:49 peer(9Moy…Y10g) - Sending handshake initiation ``` Pangolin: ``` > @fosrl/pangolin@0.0.0 start > NODE_OPTIONS=--enable-source-maps NODE_ENV=development ENVIRONMENT=prod sh -c 'node dist/migrations.mjs && node dist/server.mjs' Starting migrations from version 1.2.0 Migrations to run: All migrations completed successfully 2025-04-29T17:08:14.167Z [warn]: Email SMTP configuration is missing. Emails will not be sent. 2025-04-29T17:08:14.325Z [debug]: Using CORS options {"origin":["<REDACTED>"],"methods":["GET","POST","PUT","DELETE","PATCH"],"credentials":false} 2025-04-29T17:08:14.604Z [info]: API server is running on http://localhost:3000 2025-04-29T17:08:14.605Z [info]: Internal server is running on http://localhost:3001 2025-04-29T17:08:15.035Z [info]: Next.js server is running on http://localhost:3002 2025-04-29T17:08:16.479Z [debug]: Sending config: {"listenPort":51820,"ipAddress":"172.50.8.1/24","peers":[{"publicKey":"<REDACTED>","allowedIps":[]}]} 2025-04-29T17:09:36.738Z [debug]: POST /api/v1/auth/newt/get-token 2025-04-29T17:09:36.980Z [info]: Establishing websocket connection 2025-04-29T17:09:36.980Z [info]: Client added to tracking - Newt ID: xds2wjz4vn8zk0x, Total connections: 1 2025-04-29T17:09:36.980Z [info]: WebSocket connection established - Newt ID: xds2wjz4vn8zk0x 2025-04-29T17:09:36.996Z [info]: Handling register message! 2025-04-29T17:09:37.103Z [info]: Public key mismatch. Deleting old peer... 2025-04-29T17:09:37.134Z [info]: Peer deleted successfully: {"0":"P","1":"e","2":"e","3":"r","4":" ","5":"r","6":"e","7":"m","8":"o","9":"v","10":"e","11":"d","12":" ","13":"s","14":"u","15":"c","16":"c","17":"e","18":"s","19":"s","20":"f","21":"u","22":"l","23":"l","24":"y"} 2025-04-29T17:09:37.137Z [info]: Peer added successfully: {"0":"P","1":"e","2":"e","3":"r","4":" ","5":"a","6":"d","7":"d","8":"e","9":"d","10":" ","11":"s","12":"u","13":"c","14":"c","15":"e","16":"s","17":"s","18":"f","19":"u","20":"l","21":"l","22":"y"} ``` Traefik: ``` 2025-04-29T17:08:17Z INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:107 > Traefik version 3.3.3 built on 2025-01-31T14:55:01Z version=3.3.3 api={"basePath":"/","dashboard":true,"insecure":true} certificatesResolvers={"letsencrypt":{"acme":{"caServer":"https://acme-v02.api.letsencrypt.org/directory","certificatesDuration":2160,"email":"<REDACTED>","httpChallenge":{"entryPoint":"web"},"keyType":"RSA4096","storage":"/letsencrypt/acme.json"}}} entryPoints={"traefik":{"address":":8080","forwardedHeaders":{},"http":{"maxHeaderBytes":1048576},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}},"web":{"address":":80","forwardedHeaders":{},"http":{"maxHeaderBytes":1048576},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}},"websecure":{"address":":443","forwardedHeaders":{},"http":{"maxHeaderBytes":1048576,"tls":{"certResolver":"letsencrypt"}},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"30m0s"}},"udp":{"timeout":"3s"}}} global={"checkNewVersion":true} log={"format":"common","level":"DEBUG"} providers={"file":{"filename":"/etc/traefik/dynamic_config.yml","watch":true},"http":{"endpoint":"http://pangolin:3001/api/v1/traefik-config","pollInterval":"30s","pollTimeout":"5s"},"providersThrottleDuration":"2s"} serversTransport={"insecureSkipVerify":true,"maxIdleConnsPerHost":200} tcpServersTransport={"dialKeepAlive":"15s","dialTimeout":"30s"} 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/cmd/traefik/traefik.go:114 > Static configuration loaded [json] staticConfiguration= 2025-04-29T17:08:17Z INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:633 > Stats collection is disabled. Help us improve Traefik by turning this feature on :) More details on: https://doc.traefik.io/traefik/contributing/data-collection/ 2025-04-29T17:08:17Z INF github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:73 > Starting provider aggregator *aggregator.ProviderAggregator 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:231 > Starting TCP Server entryPointName=websecure 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:231 > Starting TCP Server entryPointName=web 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:231 > Starting TCP Server entryPointName=traefik 2025-04-29T17:08:17Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *file.Provider filename=/etc/traefik/dynamic_config.yml watch=true 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *file.Provider provider configuration config= 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:122 > add watcher on: /etc/traefik 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:122 > add watcher on: /etc/traefik/dynamic_config.yml 2025-04-29T17:08:17Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *traefik.Provider 2025-04-29T17:08:17Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *http.Provider 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *traefik.Provider provider configuration config= endpoint=http://pangolin:3001/api/v1/traefik-config pollInterval=30s pollTimeout=5s 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *http.Provider provider configuration config= 2025-04-29T17:08:17Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *acme.ChallengeTLSALPN 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *acme.ChallengeTLSALPN provider configuration config= 2025-04-29T17:08:17Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *acme.Provider HTTPChallengeProvider={} ResolverName=letsencrypt TLSChallengeProvider={} caServer=https://acme-v02.api.letsencrypt.org/directory certificatesDuration=2160 email=<REDACTED> httpChallenge={"entryPoint":"web"} keyType=RSA4096 storage=/letsencrypt/acme.json store={} 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *acme.Provider provider configuration config= 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:232 > Attempt to renew certificates "720h0m0s" before expiry and check every "24h0m0s" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme 2025-04-29T17:08:17Z INF github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:884 > Testing certificate renew... acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme http={"middlewares":{"redirect-to-https":{"redirectScheme":{"scheme":"https"}}},"routers":{"api-router":{"entryPoints":["websecure"],"rule":"Host(`<REDACTED>`) && PathPrefix(`/api/v1`)","service":"api-service","tls":{"certResolver":"letsencrypt"}},"main-app-router-redirect":{"entryPoints":["web"],"middlewares":["redirect-to-https"],"rule":"Host(`<REDACTED>`)","service":"next-service"},"next-router":{"entryPoints":["websecure"],"rule":"Host(`<REDACTED>`) && !PathPrefix(`/api/v1`)","service":"next-service","tls":{"certResolver":"letsencrypt"}},"ws-router":{"entryPoints":["websecure"],"rule":"Host(`<REDACTED>`)","service":"api-service","tls":{"certResolver":"letsencrypt"}}},"services":{"api-service":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://pangolin:3000"}]}},"next-service":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://pangolin:3002"}]}}}} tcp={} tls={} udp={} 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config=providerName=file http={"middlewares":{"dashboard_redirect":{"redirectRegex":{"permanent":true,"regex":"^(http:\\/\\/(\\[[\\w:.]+\\]|[\\w\\._-]+)(:\\d+)?)\\/$","replacement":"${1}/dashboard/"}},"dashboard_stripprefix":{"stripPrefix":{"prefixes":["/dashboard/","/dashboard"]}}},"models":{"websecure":{"observability":{},"tls":{"certResolver":"letsencrypt"}}},"routers":{"acme-http":{"entryPoints":["web"],"priority":9223372036854776000,"rule":"PathPrefix(`/.well-known/acme-challenge/`)","ruleSyntax":"v3","service":"acme-http@internal"},"api":{"entryPoints":["traefik"],"priority":9223372036854776000,"rule":"PathPrefix(`/api`)","ruleSyntax":"v3","service":"api@internal"},"dashboard":{"entryPoints":["traefik"],"middlewares":["dashboard_redirect@internal","dashboard_stripprefix@internal"],"priority":9223372036854776000,"rule":"PathPrefix(`/`)","ruleSyntax":"v3","service":"dashboard@internal"}},"serversTransports":{"default":{"insecureSkipVerify":true,"maxIdleConnsPerHost":200}},"services":{"acme-http":{},"api":{},"dashboard":{},"noop":{}}} tcp={"serversTransports":{"default":{"dialKeepAlive":"15s","dialTimeout":"30s"}}} tls={} udp={} 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config=providerName=internal http={} tcp={} tls={} udp={} 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config=providerName=letsencrypt.acme 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/tls/certificate.go:132 > Adding certificate for domain(s) <REDACTED> 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/tls/certificate.go:132 > Adding certificate for domain(s) <REDACTED> http={} tcp={} tls={} udp={} 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config=providerName=http 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321 > No default certificate, fallback to the internal generated certificate tlsStoreName=default 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:313 > Creating load-balancer entryPointName=web routerName=main-app-router-redirect@file serviceName=next-service@file 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:350 > Creating server entryPointName=web routerName=main-app-router-redirect@file serverName=889f9773a10bf3fe serviceName=next-service@file target=http://pangolin:3002 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:29 > Creating middleware entryPointName=web middlewareName=redirect-to-https@file middlewareType=RedirectScheme routerName=main-app-router-redirect@file 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:30 > Setting up redirection to https entryPointName=web middlewareName=redirect-to-https@file middlewareType=RedirectScheme routerName=main-app-router-redirect@file 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/stripprefix/strip_prefix.go:32 > Creating middleware entryPointName=traefik middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix routerName=dashboard@internal 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=traefik middlewareName=dashboard_stripprefix@internal routerName=dashboard@internal 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_regex.go:17 > Creating middleware entryPointName=traefik middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex routerName=dashboard@internal 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_regex.go:18 > Setting up redirection from ^(http:\/\/(\[[\w:.]+\]|[\w\._-]+)(:\d+)?)\/$ to ${1}/dashboard/ entryPointName=traefik middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex routerName=dashboard@internal 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=traefik middlewareName=dashboard_redirect@internal routerName=dashboard@internal 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:313 > Creating load-balancer entryPointName=websecure routerName=ws-router@file serviceName=api-service@file 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:350 > Creating server entryPointName=websecure routerName=ws-router@file serverName=889f9573a10bf098 serviceName=api-service@file target=http://pangolin:3000 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/router/tcp/manager.go:237 > Adding route for <REDACTED> with TLS options default entryPointName=websecure 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:470 > Trying to challenge certificate for domain [<REDACTED>] found in HostSNI rule ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme routerName=ws-router@file rule=Host(`<REDACTED>`) 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:470 > Trying to challenge certificate for domain [<REDACTED>] found in HostSNI rule ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme routerName=api-router@file rule="Host(`<REDACTED>`) && PathPrefix(`/api/v1`)" 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:940 > Looking for provided certificate(s) to validate ["<REDACTED>"]... ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme routerName=ws-router@file rule=Host(`<REDACTED>`) 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:470 > Trying to challenge certificate for domain [<REDACTED>] found in HostSNI rule ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme routerName=next-router@file rule="Host(`<REDACTED>`) && !PathPrefix(`/api/v1`)" 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:940 > Looking for provided certificate(s) to validate ["<REDACTED>"]... ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme routerName=next-router@file rule="Host(`<REDACTED>`) && !PathPrefix(`/api/v1`)" 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/tls/certificate.go:132 > Adding certificate for domain(s) <REDACTED> 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:940 > Looking for provided certificate(s) to validate ["<REDACTED>"]... ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme routerName=api-router@file rule="Host(`<REDACTED>`) && PathPrefix(`/api/v1`)" 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/tls/certificate.go:132 > Adding certificate for domain(s) <REDACTED> 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321 > No default certificate, fallback to the internal generated certificate tlsStoreName=default 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:984 > No ACME certificate generation required for domains ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory domains=["<REDACTED>"] providerName=letsencrypt.acme routerName=ws-router@file rule=Host(`<REDACTED>`) 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:984 > No ACME certificate generation required for domains ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory domains=["<REDACTED>"] providerName=letsencrypt.acme routerName=api-router@file rule="Host(`<REDACTED>`) && PathPrefix(`/api/v1`)" 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:984 > No ACME certificate generation required for domains ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory domains=["<REDACTED>"] providerName=letsencrypt.acme routerName=next-router@file rule="Host(`<REDACTED>`) && !PathPrefix(`/api/v1`)" 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:313 > Creating load-balancer entryPointName=web routerName=main-app-router-redirect@file serviceName=next-service@file 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:350 > Creating server entryPointName=web routerName=main-app-router-redirect@file serverName=889f9773a10bf3fe serviceName=next-service@file target=http://pangolin:3002 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:29 > Creating middleware entryPointName=web middlewareName=redirect-to-https@file middlewareType=RedirectScheme routerName=main-app-router-redirect@file 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:30 > Setting up redirection to https entryPointName=web middlewareName=redirect-to-https@file middlewareType=RedirectScheme routerName=main-app-router-redirect@file 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/stripprefix/strip_prefix.go:32 > Creating middleware entryPointName=traefik middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix routerName=dashboard@internal 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=traefik middlewareName=dashboard_stripprefix@internal routerName=dashboard@internal 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_regex.go:17 > Creating middleware entryPointName=traefik middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex routerName=dashboard@internal 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_regex.go:18 > Setting up redirection from ^(http:\/\/(\[[\w:.]+\]|[\w\._-]+)(:\d+)?)\/$ to ${1}/dashboard/ entryPointName=traefik middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex routerName=dashboard@internal 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=traefik middlewareName=dashboard_redirect@internal routerName=dashboard@internal 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:313 > Creating load-balancer entryPointName=websecure routerName=ws-router@file serviceName=api-service@file 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:350 > Creating server entryPointName=websecure routerName=ws-router@file serverName=889f9573a10bf098 serviceName=api-service@file target=http://pangolin:3000 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/server/router/tcp/manager.go:237 > Adding route for <REDACTED> with TLS options default entryPointName=websecure 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:470 > Trying to challenge certificate for domain [<REDACTED>] found in HostSNI rule ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme routerName=api-router@file rule="Host(`<REDACTED>`) && PathPrefix(`/api/v1`)" 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:470 > Trying to challenge certificate for domain [<REDACTED>] found in HostSNI rule ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme routerName=next-router@file rule="Host(`<REDACTED>`) && !PathPrefix(`/api/v1`)" 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:940 > Looking for provided certificate(s) to validate ["<REDACTED>"]... ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme routerName=api-router@file rule="Host(`<REDACTED>`) && PathPrefix(`/api/v1`)" 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:470 > Trying to challenge certificate for domain [<REDACTED>] found in HostSNI rule ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme routerName=ws-router@file rule=Host(`<REDACTED>`) 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:940 > Looking for provided certificate(s) to validate ["<REDACTED>"]... ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme routerName=ws-router@file rule=Host(`<REDACTED>`) 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:984 > No ACME certificate generation required for domains ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory domains=["<REDACTED>"] providerName=letsencrypt.acme routerName=api-router@file rule="Host(`<REDACTED>`) && PathPrefix(`/api/v1`)" 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:940 > Looking for provided certificate(s) to validate ["<REDACTED>"]... ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme routerName=next-router@file rule="Host(`<REDACTED>`) && !PathPrefix(`/api/v1`)" 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:984 > No ACME certificate generation required for domains ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory domains=["<REDACTED>"] providerName=letsencrypt.acme routerName=ws-router@file rule=Host(`<REDACTED>`) 2025-04-29T17:08:17Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:984 > No ACME certificate generation required for domains ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory domains=["<REDACTED>"] providerName=letsencrypt.acme routerName=next-router@file rule="Host(`<REDACTED>`) && !PathPrefix(`/api/v1`)" 2025-04-29T17:09:36Z DBG github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:207 > Service selected by WRR: 889f9573a10bf098 2025-04-29T17:09:36Z DBG github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:207 > Service selected by WRR: 889f9573a10bf098 ``` Gerbil: ``` INFO: 2025/04/29 17:08:16 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config INFO: 2025/04/29 17:08:16 Created WireGuard interface wg0 INFO: 2025/04/29 17:08:16 Assigned IP address 172.50.8.1/24 to interface wg0 INFO: 2025/04/29 17:08:16 Attempting to delete existing MSS clamping rule for chain INPUT DEBUG: 2025/04/29 17:08:16 Deletion stopped for chain INPUT: exit status 1 (output: iptables: Bad rule (does a matching rule exist in that chain?). ) INFO: 2025/04/29 17:08:16 Attempting to delete existing MSS clamping rule for chain OUTPUT DEBUG: 2025/04/29 17:08:16 Deletion stopped for chain OUTPUT: exit status 1 (output: iptables: Bad rule (does a matching rule exist in that chain?). ) INFO: 2025/04/29 17:08:16 Attempting to delete existing MSS clamping rule for chain FORWARD DEBUG: 2025/04/29 17:08:16 Deletion stopped for chain FORWARD: exit status 1 (output: iptables: Bad rule (does a matching rule exist in that chain?). ) INFO: 2025/04/29 17:08:16 Adding MSS clamping rule for chain INPUT INFO: 2025/04/29 17:08:16 Successfully added and verified MSS clamping rule for chain INPUT INFO: 2025/04/29 17:08:16 Adding MSS clamping rule for chain OUTPUT INFO: 2025/04/29 17:08:16 Successfully added and verified MSS clamping rule for chain OUTPUT INFO: 2025/04/29 17:08:16 Adding MSS clamping rule for chain FORWARD INFO: 2025/04/29 17:08:16 Successfully added and verified MSS clamping rule for chain FORWARD INFO: 2025/04/29 17:08:16 WireGuard interface wg0 created and configured INFO: 2025/04/29 17:08:16 Peer <REDACTED> added successfully INFO: 2025/04/29 17:08:16 Starting server on :3003 INFO: 2025/04/29 17:09:37 Peer <REDACTED> removed successfully INFO: 2025/04/29 17:09:37 Peer <REDACTED> added successfully ``` Machine: ZimaCube running ZimaOS (a fork of CasaOS) Please let me know if there's any other information I can provide to help, or if there's any tests that anyone would like me to run to identify the RC for this issue. Thanks in advance!
GiteaMirror commented 2026-04-20 07:26:12 -05:00
Author
Owner
Copy Link
Copy Source

@miloschwartz commented on GitHub (Apr 30, 2025):

Usually the ping failures are due to one or both:

  • UDP port 51820 is not open on the server
  • Something is obscuring the public IP of the server such as Cloudflare Proxy being enabled on the DNS records. You can check by making sure what you have entered in gerbil.base_endpoint in the config file resolves to the public IP of the server with dig or nslookup
<!-- gh-comment-id:2840685916 --> @miloschwartz commented on GitHub (Apr 30, 2025): Usually the ping failures are due to one or both: - UDP port 51820 is not open on the server - Something is obscuring the public IP of the server such as Cloudflare Proxy being enabled on the DNS records. You can check by making sure what you have entered in `gerbil.base_endpoint` in the config file resolves to the public IP of the server with dig or nslookup
GiteaMirror commented 2026-04-20 07:26:12 -05:00
Author
Owner
Copy Link
Copy Source

@mkadrlik commented on GitHub (May 1, 2025):

Usually the ping failures are due to one or both:

  • UDP port 51820 is not open on the server
  • Something is obscuring the public IP of the server such as Cloudflare Proxy being enabled on the DNS records. You can check by making sure what you have entered in gerbil.base_endpoint in the config file resolves to the public IP of the server with dig or nslookup

The cloudflare suggestion was solid - the proxy was enabled on all my A DNS records for the reverse proxy so I turned that off, with the same result. I did also check nslookup against the gerbil.base_endpoint and it did resolve correctly to the right public IP address. I then just did a fresh reinstall just to cover my bases, and I still am getting the same error.

I then checked to see if the UDP port is opened using netstat -tuplen | grep 51820, and received the following result:

udp        0      0 0.0.0.0:51820           0.0.0.0:*                           0          76667688   2450989/docker-prox 
udp6       0      0 :::51820                :::*                                0          76669287   2450996/docker-prox

Unless I'm incorrect, that looks how I believe it should. Now, my assumption is that newt should be running on the same network (pangolin) that gerbil is on - is that correct? I went ahead and did that anyways and still got the same error.

Is it possible that a pihole could interfere with this? That seems to be the only change that I've made to my network that I can recall in the last 3 months...

Any other thoughts you all might have?

<!-- gh-comment-id:2845555332 --> @mkadrlik commented on GitHub (May 1, 2025): > Usually the ping failures are due to one or both: > > * UDP port 51820 is not open on the server > * Something is obscuring the public IP of the server such as Cloudflare Proxy being enabled on the DNS records. You can check by making sure what you have entered in `gerbil.base_endpoint` in the config file resolves to the public IP of the server with dig or nslookup The cloudflare suggestion was solid - the proxy was enabled on all my `A` DNS records for the reverse proxy so I turned that off, with the same result. I did also check nslookup against the `gerbil.base_endpoint` and it did resolve correctly to the right public IP address. I then just did a fresh reinstall just to cover my bases, and I still am getting the same error. I then checked to see if the UDP port is opened using `netstat -tuplen | grep 51820`, and received the following result: ```bash udp 0 0 0.0.0.0:51820 0.0.0.0:* 0 76667688 2450989/docker-prox udp6 0 0 :::51820 :::* 0 76669287 2450996/docker-prox ``` Unless I'm incorrect, that looks how I believe it should. Now, my assumption is that `newt` should be running on the same network (`pangolin`) that `gerbil` is on - is that correct? I went ahead and did that anyways and still got the same error. Is it possible that a pihole could interfere with this? That seems to be the only change that I've made to my network that I can recall in the last 3 months... Any other thoughts you all might have?
GiteaMirror commented 2026-04-20 07:26:12 -05:00
Author
Owner
Copy Link
Copy Source

@oschwartz10612 commented on GitHub (May 2, 2025):

Hi sorry you are having this hassle!

You are doing some good troubleshooting steps. The next thing I would do is a tcpdump on the server to see if you are getting any traffic to it from newt. Could you do the following command on your pangolin instance and see if you capture any packets when you start the newt connection?

tcpdump -i any -n udp port 51820

If you are not getting packets it is likely something is blocking traffic. You can also try to run echo "poke" | nc -u <base_endpoint> 51820 from the machine running newt to see if you see the packets. If not it is even more likely something is blocking it.

<!-- gh-comment-id:2846263091 --> @oschwartz10612 commented on GitHub (May 2, 2025): Hi sorry you are having this hassle! You are doing some good troubleshooting steps. The next thing I would do is a `tcpdump` on the server to see if you are getting any traffic to it from newt. Could you do the following command on your pangolin instance and see if you capture any packets when you start the newt connection? `tcpdump -i any -n udp port 51820` If you are not getting packets it is likely something is blocking traffic. You can also try to run `echo "poke" | nc -u <base_endpoint> 51820` from the machine running newt to see if you see the packets. If not it is even more likely something is blocking it.
GiteaMirror commented 2026-04-20 07:26:13 -05:00
Author
Owner
Copy Link
Copy Source

@mkadrlik commented on GitHub (May 3, 2025):

Disregard - I found the issue past the proxy; the wrong port was opened on my router, which was causing all the problems. Thanks for all the help, everyone!

<!-- gh-comment-id:2848756355 --> @mkadrlik commented on GitHub (May 3, 2025): Disregard - I found the issue past the proxy; the wrong port was opened on my router, which was causing all the problems. Thanks for all the help, everyone!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
api
authentication
bug
config
dependencies
docker
documentation
enhancement
good first issue
help wanted
Improvement
Look Into
needs investigating
networking
new feature
non-critical bug
potential bug
pull-request
Mirrored from GitHub Pull Request
 question
reverse proxy
Security
stale
ui
wontfix
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
GiteaMirror ninjasurge (Kaehlen Rog)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/pangolin#3472