github-starred/pangolin
2
0
Fork 0
mirror of https://github.com/fosrl/pangolin.git synced 2026-07-18 07:34:42 -05:00
Code Issues 1.6k Packages Projects Releases 79 Wiki Activity

Traefik shutting down randomly in Azure #82

New Issue
Closed
opened 2025-11-13 11:49:22 -06:00 by GiteaMirror · 6 comments
GiteaMirror commented 2025-11-13 11:49:22 -06:00
Owner
Copy Link

Originally created by @jwhite on GitHub (Feb 12, 2025).

At random times (so far over night in our region) the Traefik server stops functioning and is not restarting.

Any ideas on how to debug?

pangolin  | Starting migrations from version 1.0.0-beta.12
pangolin  | Migrations to run:
pangolin  | All migrations completed successfully
pangolin  | 2025-02-12T15:15:58.049Z [warn]: Email SMTP configuration is missing. Emails will not be sent.
pangolin  | 2025-02-12T15:15:58.657Z [info]: Updated orgs with new domain (########.com)
pangolin  | 2025-02-12T15:15:58.851Z [info]: Server admin (#######) already exists
pangolin  | 2025-02-12T15:16:04.764Z [info]: API server is running on http://localhost:3000
pangolin  | 2025-02-12T15:16:04.766Z [info]: Internal server is running on http://localhost:3001
pangolin  | 2025-02-12T15:16:05.630Z [info]: Next.js server is running on http://localhost:3002
gerbil    | ERROR: 2025/02/12 15:15:54 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp 172.18.0.3:3001: connect: connection refused
gerbil    | FATAL: 2025/02/12 15:15:54 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp 172.18.0.3:3001: connect: connection refused
gerbil    | = 'gerbil' is not a Gerbil command: assuming shell execution.
gerbil    | ERROR: 2025/02/12 15:15:58 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp 172.18.0.3:3001: connect: connection refused
gerbil    | FATAL: 2025/02/12 15:15:58 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp 172.18.0.3:3001: connect: connection refused
gerbil    | = 'gerbil' is not a Gerbil command: assuming shell execution.
gerbil    | INFO: 2025/02/12 15:16:05 Created WireGuard interface wg0
gerbil    | INFO: 2025/02/12 15:16:05 Assigned IP address 100.89.128.1/24 to interface wg0
gerbil    | INFO: 2025/02/12 15:16:05 WireGuard interface wg0 created and configured
gerbil    | INFO: 2025/02/12 15:16:05 Starting server on :3003
traefik   | 2025-02-12T15:15:12Z INF I have to go...
traefik   | 2025-02-12T15:15:12Z INF Stopping server gracefully
traefik   | 2025-02-12T15:15:12Z ERR error="accept tcp [::]:8080: use of closed network connection" entryPointName=traefik
traefik   | 2025-02-12T15:15:12Z ERR Error while starting server error="accept tcp [::]:8080: use of closed network connection" entryPointName=traefik
traefik   | 2025-02-12T15:15:12Z ERR error="accept tcp [::]:80: use of closed network connection" entryPointName=web
traefik   | 2025-02-12T15:15:12Z ERR Error while starting server error="accept tcp [::]:80: use of closed network connection" entryPointName=web
traefik   | 2025-02-12T15:15:12Z ERR error="accept tcp [::]:443: use of closed network connection" entryPointName=websecure
traefik   | 2025-02-12T15:15:12Z ERR Error while starting server error="accept tcp [::]:443: use of closed network connection" entryPointName=websecure
traefik   | 2025-02-12T15:15:14Z INF Server stopped
traefik   | 2025-02-12T15:15:14Z INF Shutting down
Originally created by @jwhite on GitHub (Feb 12, 2025). At random times (so far over night in our region) the Traefik server stops functioning and is not restarting. Any ideas on how to debug? ``` pangolin | Starting migrations from version 1.0.0-beta.12 pangolin | Migrations to run: pangolin | All migrations completed successfully pangolin | 2025-02-12T15:15:58.049Z [warn]: Email SMTP configuration is missing. Emails will not be sent. pangolin | 2025-02-12T15:15:58.657Z [info]: Updated orgs with new domain (########.com) pangolin | 2025-02-12T15:15:58.851Z [info]: Server admin (#######) already exists pangolin | 2025-02-12T15:16:04.764Z [info]: API server is running on http://localhost:3000 pangolin | 2025-02-12T15:16:04.766Z [info]: Internal server is running on http://localhost:3001 pangolin | 2025-02-12T15:16:05.630Z [info]: Next.js server is running on http://localhost:3002 gerbil | ERROR: 2025/02/12 15:15:54 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp 172.18.0.3:3001: connect: connection refused gerbil | FATAL: 2025/02/12 15:15:54 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp 172.18.0.3:3001: connect: connection refused gerbil | = 'gerbil' is not a Gerbil command: assuming shell execution. gerbil | ERROR: 2025/02/12 15:15:58 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp 172.18.0.3:3001: connect: connection refused gerbil | FATAL: 2025/02/12 15:15:58 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp 172.18.0.3:3001: connect: connection refused gerbil | = 'gerbil' is not a Gerbil command: assuming shell execution. gerbil | INFO: 2025/02/12 15:16:05 Created WireGuard interface wg0 gerbil | INFO: 2025/02/12 15:16:05 Assigned IP address 100.89.128.1/24 to interface wg0 gerbil | INFO: 2025/02/12 15:16:05 WireGuard interface wg0 created and configured gerbil | INFO: 2025/02/12 15:16:05 Starting server on :3003 traefik | 2025-02-12T15:15:12Z INF I have to go... traefik | 2025-02-12T15:15:12Z INF Stopping server gracefully traefik | 2025-02-12T15:15:12Z ERR error="accept tcp [::]:8080: use of closed network connection" entryPointName=traefik traefik | 2025-02-12T15:15:12Z ERR Error while starting server error="accept tcp [::]:8080: use of closed network connection" entryPointName=traefik traefik | 2025-02-12T15:15:12Z ERR error="accept tcp [::]:80: use of closed network connection" entryPointName=web traefik | 2025-02-12T15:15:12Z ERR Error while starting server error="accept tcp [::]:80: use of closed network connection" entryPointName=web traefik | 2025-02-12T15:15:12Z ERR error="accept tcp [::]:443: use of closed network connection" entryPointName=websecure traefik | 2025-02-12T15:15:12Z ERR Error while starting server error="accept tcp [::]:443: use of closed network connection" entryPointName=websecure traefik | 2025-02-12T15:15:14Z INF Server stopped traefik | 2025-02-12T15:15:14Z INF Shutting down
GiteaMirror commented 2025-11-13 11:49:23 -06:00
Author
Owner
Copy Link

@hhftechnology commented on GitHub (Feb 13, 2025):

Check if Traefik is being killed by the OOM killer

dmesg | grep -i kill

journalctl -u docker.service --since "1 hour ago"

@hhftechnology commented on GitHub (Feb 13, 2025): #### Check if Traefik is being killed by the OOM killer `dmesg | grep -i kill` `journalctl -u docker.service --since "1 hour ago"`
GiteaMirror commented 2025-11-13 11:49:23 -06:00
Author
Owner
Copy Link

@jwhite commented on GitHub (Mar 20, 2025):

@hhftechnology - Any ideas here?

Seems to crash every 7 to 10 days now, might be less frequent with the latest version.

dmesg reports nothing (with kill.)

It went down sometime last night, so here are the last 24 hours from journalctl. I didn't request the 'stopping' so I'm wondering if something in Azure told the containers to go down. For upgrade maybe? The more interesting thing is they don't come back up.

Cheers!

(Sorry for the ugly logs... not sure how to format for github.)

`Mar 20 14:03:08 pangolin systemd[1]: Stopping Docker Application Container Engine...
Mar 20 14:03:08 pangolin dockerd[667]: time="2025-03-20T14:03:08.518017360Z" level=info msg="Processing signal 'terminated'"
Mar 20 14:03:08 pangolin dockerd[667]: time="2025-03-20T14:03:08.720500095Z" level=info msg="ignoring event" container=75b68eda5cb3c8432497c9bbe23d491f5df01f72aef8c950a8bf69910112f2>
Mar 20 14:03:08 pangolin dockerd[667]: time="2025-03-20T14:03:08.873546304Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=75b68eda5cb3c8432497>
Mar 20 14:03:09 pangolin dockerd[667]: time="2025-03-20T14:03:09.291591061Z" level=info msg="ignoring event" container=0a630647aa1d62dfe348b597b192fd4723ff13577937937b33dd92e753b430>
Mar 20 14:03:09 pangolin dockerd[667]: time="2025-03-20T14:03:09.955513940Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=0a630647aa1d62dfe348>
Mar 20 14:03:10 pangolin dockerd[667]: time="2025-03-20T14:03:10.296724290Z" level=info msg="ignoring event" container=6d768021ec8443dfde73ac167f2fb48061aa7d13f465d8b4d2ac8f57144f53>
Mar 20 14:03:10 pangolin dockerd[667]: time="2025-03-20T14:03:10.324254771Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=6d768021ec8443dfde73>
Mar 20 14:03:10 pangolin dockerd[667]: time="2025-03-20T14:03:10.910421137Z" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd na>
Mar 20 14:03:10 pangolin dockerd[667]: time="2025-03-20T14:03:10.919007294Z" level=info msg="Daemon shutdown complete"
Mar 20 14:03:10 pangolin systemd[1]: docker.service: Deactivated successfully.
Mar 20 14:03:10 pangolin systemd[1]: Stopped Docker Application Container Engine.
Mar 20 14:03:10 pangolin systemd[1]: docker.service: Consumed 12min 39.471s CPU time.
Mar 20 14:03:10 pangolin systemd[1]: Starting Docker Application Container Engine...
Mar 20 14:03:12 pangolin dockerd[2346782]: time="2025-03-20T14:03:12.060560128Z" level=info msg="Starting up"
Mar 20 14:03:12 pangolin dockerd[2346782]: time="2025-03-20T14:03:12.067801989Z" level=info msg="OTEL tracing is not configured, using no-op tracer provider"
Mar 20 14:03:12 pangolin dockerd[2346782]: time="2025-03-20T14:03:12.068502794Z" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /ru>
Mar 20 14:03:12 pangolin dockerd[2346782]: time="2025-03-20T14:03:12.088607762Z" level=info msg="Creating a containerd client" address=/run/containerd/containerd.sock timeout=1m0s
Mar 20 14:03:12 pangolin dockerd[2346782]: time="2025-03-20T14:03:12.296640092Z" level=info msg="[graphdriver] using prior storage driver: overlay2"
Mar 20 14:03:12 pangolin dockerd[2346782]: time="2025-03-20T14:03:12.574162601Z" level=info msg="Loading containers: start."
Mar 20 14:03:13 pangolin dockerd[2346782]: time="2025-03-20T14:03:13.656392805Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external ser>
Mar 20 14:03:13 pangolin dockerd[2346782]: time="2025-03-20T14:03:13.852053933Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external ser>
Mar 20 14:03:15 pangolin dockerd[2346782]: time="2025-03-20T14:03:15.147157972Z" level=info msg="ignoring event" container=75b68eda5cb3c8432497c9bbe23d491f5df01f72aef8c950a8bf699101>
Mar 20 14:03:15 pangolin dockerd[2346782]: time="2025-03-20T14:03:15.303977952Z" level=error msg="copy stream failed" error="reading from a closed fifo" stream=stderr
Mar 20 14:03:15 pangolin dockerd[2346782]: time="2025-03-20T14:03:15.306150869Z" level=error msg="copy stream failed" error="reading from a closed fifo" stream=stdout
Mar 20 14:03:15 pangolin dockerd[2346782]: time="2025-03-20T14:03:15.561317851Z" level=error msg="failed to start container" container=0a630647aa1d62dfe348b597b192fd4723ff1357793793>
Mar 20 14:03:15 pangolin dockerd[2346782]: time="2025-03-20T14:03:15.563177766Z" level=info msg="Loading containers: done."
Mar 20 14:03:15 pangolin dockerd[2346782]: time="2025-03-20T14:03:15.857684869Z" level=warning msg="Not using native diff for overlay2, this may cause degraded performance for build>
Mar 20 14:03:15 pangolin dockerd[2346782]: time="2025-03-20T14:03:15.860685093Z" level=info msg="Docker daemon" commit=bea4de2 containerd-snapshotter=false storage-driver=overlay2 v>
Mar 20 14:03:15 pangolin dockerd[2346782]: time="2025-03-20T14:03:15.935963907Z" level=info msg="Initializing buildkit"
Mar 20 14:03:16 pangolin dockerd[2346782]: time="2025-03-20T14:03:16.011136321Z" level=info msg="Completed buildkit initialization"
Mar 20 14:03:16 pangolin dockerd[2346782]: time="2025-03-20T14:03:16.022885317Z" level=info msg="Daemon has completed initialization"
Mar 20 14:03:16 pangolin dockerd[2346782]: time="2025-03-20T14:03:16.022929017Z" level=info msg="API listen on /run/docker.sock"
Mar 20 14:03:16 pangolin systemd[1]: Started Docker Application Container Engine.
Mar 20 14:03:16 pangolin dockerd[2346782]: time="2025-03-20T14:03:16.384217464Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external ser>
Mar 20 14:03:17 pangolin dockerd[2346782]: time="2025-03-20T14:03:17.323641429Z" level=info msg="ignoring event" container=75b68eda5cb3c8432497c9bbe23d491f5df01f72aef8c950a8bf699101>
Mar 20 14:03:18 pangolin dockerd[2346782]: time="2025-03-20T14:03:18.412543112Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external ser>
Mar 20 14:03:19 pangolin dockerd[2346782]: time="2025-03-20T14:03:19.730402764Z" level=info msg="ignoring event" container=75b68eda5cb3c8432497c9bbe23d491f5df01f72aef8c950a8bf699101>
Mar 20 14:03:22 pangolin dockerd[2346782]: time="2025-03-20T14:03:22.185788195Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external ser>
Mar 20 14:03:22 pangolin dockerd[2346782]: time="2025-03-20T14:03:22.846101737Z" level=info msg="ignoring event" container=75b68eda5cb3c8432497c9bbe23d491f5df01f72aef8c950a8bf699101>
Mar 20 14:03:24 pangolin dockerd[2346782]: time="2025-03-20T14:03:24.151439193Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external ser>
Mar 20 14:03:24 pangolin dockerd[2346782]: time="2025-03-20T14:03:24.930379580Z" level=info msg="ignoring event" container=75b68eda5cb3c8432497c9bbe23d491f5df01f72aef8c950a8bf699101>
Mar 20 14:03:27 pangolin dockerd[2346782]: time="2025-03-20T14:03:27.083951270Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external ser>
Mar 20 14:03:28 pangolin dockerd[2346782]: time="2025-03-20T14:03:28.401508228Z" level=info msg="ignoring event" container=75b68eda5cb3c8432497c9bbe23d491f5df01f72aef8c950a8bf699101>
Mar 20 14:03:32 pangolin dockerd[2346782]: time="2025-03-20T14:03:32.226553017Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external ser>

< I log in and docker compose down here....>

Mar 20 20:30:37 pangolin dockerd[2346782]: time="2025-03-20T20:30:37.117302184Z" level=info msg="ignoring event" container=75b68eda5cb3c8432497c9bbe23d491f5df01f72aef8c950a8bf699101>
Mar 20 20:30:37 pangolin dockerd[2346782]: time="2025-03-20T20:30:37.324560649Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=75b68eda5cb3c843>
Mar 20 20:30:38 pangolin dockerd[2346782]: time="2025-03-20T20:30:38.852778862Z" level=info msg="ignoring event" container=6d768021ec8443dfde73ac167f2fb48061aa7d13f465d8b4d2ac8f5714>
Mar 20 20:30:38 pangolin dockerd[2346782]: time="2025-03-20T20:30:38.876531164Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=6d768021ec8443df>`
@jwhite commented on GitHub (Mar 20, 2025): @hhftechnology - Any ideas here? Seems to crash every 7 to 10 days now, might be less frequent with the latest version. dmesg reports nothing (with kill.) It went down sometime last night, so here are the last 24 hours from journalctl. I didn't request the 'stopping' so I'm wondering if something in Azure told the containers to go down. For upgrade maybe? The more interesting thing is they don't come back up. Cheers! (Sorry for the ugly logs... not sure how to format for github.) ``` `Mar 20 14:03:08 pangolin systemd[1]: Stopping Docker Application Container Engine... Mar 20 14:03:08 pangolin dockerd[667]: time="2025-03-20T14:03:08.518017360Z" level=info msg="Processing signal 'terminated'" Mar 20 14:03:08 pangolin dockerd[667]: time="2025-03-20T14:03:08.720500095Z" level=info msg="ignoring event" container=75b68eda5cb3c8432497c9bbe23d491f5df01f72aef8c950a8bf69910112f2> Mar 20 14:03:08 pangolin dockerd[667]: time="2025-03-20T14:03:08.873546304Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=75b68eda5cb3c8432497> Mar 20 14:03:09 pangolin dockerd[667]: time="2025-03-20T14:03:09.291591061Z" level=info msg="ignoring event" container=0a630647aa1d62dfe348b597b192fd4723ff13577937937b33dd92e753b430> Mar 20 14:03:09 pangolin dockerd[667]: time="2025-03-20T14:03:09.955513940Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=0a630647aa1d62dfe348> Mar 20 14:03:10 pangolin dockerd[667]: time="2025-03-20T14:03:10.296724290Z" level=info msg="ignoring event" container=6d768021ec8443dfde73ac167f2fb48061aa7d13f465d8b4d2ac8f57144f53> Mar 20 14:03:10 pangolin dockerd[667]: time="2025-03-20T14:03:10.324254771Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=6d768021ec8443dfde73> Mar 20 14:03:10 pangolin dockerd[667]: time="2025-03-20T14:03:10.910421137Z" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd na> Mar 20 14:03:10 pangolin dockerd[667]: time="2025-03-20T14:03:10.919007294Z" level=info msg="Daemon shutdown complete" Mar 20 14:03:10 pangolin systemd[1]: docker.service: Deactivated successfully. Mar 20 14:03:10 pangolin systemd[1]: Stopped Docker Application Container Engine. Mar 20 14:03:10 pangolin systemd[1]: docker.service: Consumed 12min 39.471s CPU time. Mar 20 14:03:10 pangolin systemd[1]: Starting Docker Application Container Engine... Mar 20 14:03:12 pangolin dockerd[2346782]: time="2025-03-20T14:03:12.060560128Z" level=info msg="Starting up" Mar 20 14:03:12 pangolin dockerd[2346782]: time="2025-03-20T14:03:12.067801989Z" level=info msg="OTEL tracing is not configured, using no-op tracer provider" Mar 20 14:03:12 pangolin dockerd[2346782]: time="2025-03-20T14:03:12.068502794Z" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /ru> Mar 20 14:03:12 pangolin dockerd[2346782]: time="2025-03-20T14:03:12.088607762Z" level=info msg="Creating a containerd client" address=/run/containerd/containerd.sock timeout=1m0s Mar 20 14:03:12 pangolin dockerd[2346782]: time="2025-03-20T14:03:12.296640092Z" level=info msg="[graphdriver] using prior storage driver: overlay2" Mar 20 14:03:12 pangolin dockerd[2346782]: time="2025-03-20T14:03:12.574162601Z" level=info msg="Loading containers: start." Mar 20 14:03:13 pangolin dockerd[2346782]: time="2025-03-20T14:03:13.656392805Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external ser> Mar 20 14:03:13 pangolin dockerd[2346782]: time="2025-03-20T14:03:13.852053933Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external ser> Mar 20 14:03:15 pangolin dockerd[2346782]: time="2025-03-20T14:03:15.147157972Z" level=info msg="ignoring event" container=75b68eda5cb3c8432497c9bbe23d491f5df01f72aef8c950a8bf699101> Mar 20 14:03:15 pangolin dockerd[2346782]: time="2025-03-20T14:03:15.303977952Z" level=error msg="copy stream failed" error="reading from a closed fifo" stream=stderr Mar 20 14:03:15 pangolin dockerd[2346782]: time="2025-03-20T14:03:15.306150869Z" level=error msg="copy stream failed" error="reading from a closed fifo" stream=stdout Mar 20 14:03:15 pangolin dockerd[2346782]: time="2025-03-20T14:03:15.561317851Z" level=error msg="failed to start container" container=0a630647aa1d62dfe348b597b192fd4723ff1357793793> Mar 20 14:03:15 pangolin dockerd[2346782]: time="2025-03-20T14:03:15.563177766Z" level=info msg="Loading containers: done." Mar 20 14:03:15 pangolin dockerd[2346782]: time="2025-03-20T14:03:15.857684869Z" level=warning msg="Not using native diff for overlay2, this may cause degraded performance for build> Mar 20 14:03:15 pangolin dockerd[2346782]: time="2025-03-20T14:03:15.860685093Z" level=info msg="Docker daemon" commit=bea4de2 containerd-snapshotter=false storage-driver=overlay2 v> Mar 20 14:03:15 pangolin dockerd[2346782]: time="2025-03-20T14:03:15.935963907Z" level=info msg="Initializing buildkit" Mar 20 14:03:16 pangolin dockerd[2346782]: time="2025-03-20T14:03:16.011136321Z" level=info msg="Completed buildkit initialization" Mar 20 14:03:16 pangolin dockerd[2346782]: time="2025-03-20T14:03:16.022885317Z" level=info msg="Daemon has completed initialization" Mar 20 14:03:16 pangolin dockerd[2346782]: time="2025-03-20T14:03:16.022929017Z" level=info msg="API listen on /run/docker.sock" Mar 20 14:03:16 pangolin systemd[1]: Started Docker Application Container Engine. Mar 20 14:03:16 pangolin dockerd[2346782]: time="2025-03-20T14:03:16.384217464Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external ser> Mar 20 14:03:17 pangolin dockerd[2346782]: time="2025-03-20T14:03:17.323641429Z" level=info msg="ignoring event" container=75b68eda5cb3c8432497c9bbe23d491f5df01f72aef8c950a8bf699101> Mar 20 14:03:18 pangolin dockerd[2346782]: time="2025-03-20T14:03:18.412543112Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external ser> Mar 20 14:03:19 pangolin dockerd[2346782]: time="2025-03-20T14:03:19.730402764Z" level=info msg="ignoring event" container=75b68eda5cb3c8432497c9bbe23d491f5df01f72aef8c950a8bf699101> Mar 20 14:03:22 pangolin dockerd[2346782]: time="2025-03-20T14:03:22.185788195Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external ser> Mar 20 14:03:22 pangolin dockerd[2346782]: time="2025-03-20T14:03:22.846101737Z" level=info msg="ignoring event" container=75b68eda5cb3c8432497c9bbe23d491f5df01f72aef8c950a8bf699101> Mar 20 14:03:24 pangolin dockerd[2346782]: time="2025-03-20T14:03:24.151439193Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external ser> Mar 20 14:03:24 pangolin dockerd[2346782]: time="2025-03-20T14:03:24.930379580Z" level=info msg="ignoring event" container=75b68eda5cb3c8432497c9bbe23d491f5df01f72aef8c950a8bf699101> Mar 20 14:03:27 pangolin dockerd[2346782]: time="2025-03-20T14:03:27.083951270Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external ser> Mar 20 14:03:28 pangolin dockerd[2346782]: time="2025-03-20T14:03:28.401508228Z" level=info msg="ignoring event" container=75b68eda5cb3c8432497c9bbe23d491f5df01f72aef8c950a8bf699101> Mar 20 14:03:32 pangolin dockerd[2346782]: time="2025-03-20T14:03:32.226553017Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external ser> < I log in and docker compose down here....> Mar 20 20:30:37 pangolin dockerd[2346782]: time="2025-03-20T20:30:37.117302184Z" level=info msg="ignoring event" container=75b68eda5cb3c8432497c9bbe23d491f5df01f72aef8c950a8bf699101> Mar 20 20:30:37 pangolin dockerd[2346782]: time="2025-03-20T20:30:37.324560649Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=75b68eda5cb3c843> Mar 20 20:30:38 pangolin dockerd[2346782]: time="2025-03-20T20:30:38.852778862Z" level=info msg="ignoring event" container=6d768021ec8443dfde73ac167f2fb48061aa7d13f465d8b4d2ac8f5714> Mar 20 20:30:38 pangolin dockerd[2346782]: time="2025-03-20T20:30:38.876531164Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=6d768021ec8443df>`
GiteaMirror commented 2025-11-13 11:49:23 -06:00
Author
Owner
Copy Link

@oschwartz10612 commented on GitHub (Mar 20, 2025):

The logs show Docker was stopped and restarted at 14:03:08 on March 20. This appears to be a system-initiated action (systemd[1]: Stopping Docker Application Container Engine...).

Container restart failures: After Docker restarts, it tries to bring up the containers but encounters errors:

time="2025-03-20T14:03:15.561317851Z" level=error msg="failed to start container"

This could be related to Azure's automatic maintenance operations. Check your Azure portal for scheduled maintenance events or set up notifications for them. Do you have a machine type that gets stopped and started like AWS Spot instances?

Make sure you are not running out of resources triggering your machine to restart - memory or CPU.

@oschwartz10612 commented on GitHub (Mar 20, 2025): The logs show Docker was stopped and restarted at 14:03:08 on March 20. This appears to be a system-initiated action (systemd[1]: Stopping Docker Application Container Engine...). Container restart failures: After Docker restarts, it tries to bring up the containers but encounters errors: time="2025-03-20T14:03:15.561317851Z" level=error msg="failed to start container" This could be related to Azure's automatic maintenance operations. Check your Azure portal for scheduled maintenance events or set up notifications for them. Do you have a machine type that gets stopped and started like AWS Spot instances? Make sure you are not running out of resources triggering your machine to restart - memory or CPU.
GiteaMirror commented 2025-11-13 11:49:23 -06:00
Author
Owner
Copy Link

@jwhite commented on GitHub (Mar 21, 2025):

I'll check the resources.

No, this is not on spot instances, but is a low resource vm.

I'll try to see if there are scheduled events. Either way --- it should come back up.

How do I find out why?

@jwhite commented on GitHub (Mar 21, 2025): I'll check the resources. No, this is not on spot instances, but is a low resource vm. I'll try to see if there are scheduled events. Either way --- it should come back up. How do I find out why?
GiteaMirror commented 2025-11-13 11:49:24 -06:00
Author
Owner
Copy Link

@hhftechnology commented on GitHub (Mar 21, 2025):

this is definitely oom issues
Mar 20 14:03:08 pangolin systemd[1]: Stopping Docker Application Container Engine...
indicate Docker is being stopped by systemd
Try to increase resources a bit.

# check for OOM killer activity
dmesg | grep -i 'killed process'

# check Docker memory usage
docker stats --no-stream

# Check system resource utilization
free -m
df -h

Last resort if you don't increase resources. check your compose path. otherwise, script will not run.

# create a simple monitoring script
cat > /etc/cron.d/docker-watchdog << EOF
*/5 * * * * root /usr/bin/docker ps | grep -q traefik || /usr/bin/docker-compose -f /path/to/docker-compose.yml up -d traefik
EOF
@hhftechnology commented on GitHub (Mar 21, 2025): this is definitely oom issues ```Mar 20 14:03:08 pangolin systemd[1]: Stopping Docker Application Container Engine...``` indicate Docker is being stopped by systemd Try to increase resources a bit. ``` # check for OOM killer activity dmesg | grep -i 'killed process' # check Docker memory usage docker stats --no-stream # Check system resource utilization free -m df -h ``` Last resort if you don't increase resources. check your compose path. otherwise, script will not run. ``` # create a simple monitoring script cat > /etc/cron.d/docker-watchdog << EOF */5 * * * * root /usr/bin/docker ps | grep -q traefik || /usr/bin/docker-compose -f /path/to/docker-compose.yml up -d traefik EOF ```
GiteaMirror commented 2025-11-13 11:49:24 -06:00
Author
Owner
Copy Link

@jwhite commented on GitHub (Mar 24, 2025):

It was a memory constraint issue. I need at least 2gb on the host, or it will sit around 70% consumed, and then eventually get shut down. That seems a bit high, should I be adjusting my docker compose to set some better limits or has that already been taken into consideration?

@jwhite commented on GitHub (Mar 24, 2025): It was a memory constraint issue. I need at least 2gb on the host, or it will sit around 70% consumed, and then eventually get shut down. That seems a bit high, should I be adjusting my docker compose to set some better limits or has that already been taken into consideration?
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
api
authentication
bug
config
dependencies
docker
documentation
enhancement
good first issue
help wanted
Improvement
Look Into
needs investigating
networking
new feature
non-critical bug
potential bug
pull-request
Mirrored from GitHub Pull Request
 question
reverse proxy
Security
stale
ui
wontfix
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/pangolin#82
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?