[HELP] komodo-periphery memory leak #140

@rita7lopes commented on GitHub (Nov 27, 2024):

These are the only logs I see each time I restart the container:

docker logs komodo-periphery-1
2024-11-26T15:36:12.256762Z  INFO periphery: Komodo Periphery version: v1.16.11
2024-11-26T15:36:12.256799Z  INFO periphery: PeripheryConfig { port: 8120, repo_dir: "/etc/komodo/repos", stack_dir: "/etc/komodo/stacks", stats_polling_rate: FiveSeconds, legacy_compose_cli: false, logging: LogConfig { level: Info, stdio: Standard, otlp_endpoint: "", opentelemetry_service_name: "Komodo" }, allowed_ips: [], passkeys: ["##############"], include_disk_mounts: ["/etc/hostname"], exclude_disk_mounts: [], secrets: {}, git_providers: [], docker_registries: [], ssl_enabled: true, ssl_key_file: "/etc/komodo/ssl/key.pem", ssl_cert_file: "/etc/komodo/ssl/cert.pem" }
2024-11-26T15:36:12.256910Z  INFO periphery: 🔒 Periphery SSL Enabled
2024-11-26T15:36:12.256934Z  INFO periphery: Komodo Periphery starting on https://0.0.0.0:8120

Containers running: 23
Processes running: 242

I tried removing the /proc, and the issue seems to be gone. No memory leak.

I noticed something interesting.
There's no memory leak on the machine where I have Komodo core and periphery on the same compose file:

##################################
# 🦎 KOMODO COMPOSE - POSTGRES 🦎 #
##################################

## This compose file will deploy:
##   1. Postgres + FerretDB Mongo adapter
##   2. Komodo Core
##   3. Komodo Periphery

services:
  postgres:
    image: postgres
    labels:
      komodo.skip: # Prevent Komodo from stopping with StopAllContainers
    restart: unless-stopped
    logging:
      driver: ${COMPOSE_LOGGING_DRIVER:-local}
    networks:
      - default
    # ports:
    #   - 5432:5432
    volumes:
      - pg-data:/var/lib/postgresql/data
    environment:
      - POSTGRES_USER=${DB_USERNAME}
      - POSTGRES_PASSWORD=${DB_PASSWORD}
      - POSTGRES_DB=${KOMODO_DATABASE_DB_NAME:-komodo}

  ferretdb:
    image: ghcr.io/ferretdb/ferretdb
    labels:
      komodo.skip: # Prevent Komodo from stopping with StopAllContainers
    restart: unless-stopped
    depends_on:
      - postgres
    logging:
      driver: ${COMPOSE_LOGGING_DRIVER:-local}
    networks:
      - default
    ports:
      - 27017:27017
    environment:
      - FERRETDB_POSTGRESQL_URL=postgres://postgres:5432/${KOMODO_DATABASE_DB_NAME:-komodo}

  core:
    image: ghcr.io/mbecker20/komodo:${COMPOSE_KOMODO_IMAGE_TAG:-latest}
    labels:
      komodo.skip: # Prevent Komodo from stopping with StopAllContainers
    restart: unless-stopped
    depends_on:
      - ferretdb
    logging:
      driver: ${COMPOSE_LOGGING_DRIVER:-local}
    networks:
      - default
    ports:
      - 9120:9120
    env_file: ./compose.env
    environment:
      KOMODO_DATABASE_URI: mongodb://${DB_USERNAME}:${DB_PASSWORD}@ferretdb:27017/${KOMODO_DATABASE_DB_NAME:-komodo}?authMechanism=PLAIN
    volumes:
      ## Core cache for repos for latest commit hash / contents
      - repo-cache:/repo-cache
      ## Store sync files on server
      # - /path/to/syncs:/syncs
      ## Optionally mount a custom core.config.toml
      # - /path/to/core.config.toml:/config/config.toml
    ## Allows for systemd Periphery connection at
    ## "http://host.docker.internal:8120"
    # extra_hosts:
    #   - host.docker.internal:host-gateway

  ## Deploy Periphery container using this block,
  ## or deploy the Periphery binary with systemd using
  ## https://github.com/mbecker20/komodo/tree/main/scripts
  periphery:
    image: ghcr.io/mbecker20/periphery:${COMPOSE_KOMODO_IMAGE_TAG:-latest}
    labels:
      komodo.skip: # Prevent Komodo from stopping with StopAllContainers
    restart: unless-stopped
    logging:
      driver: ${COMPOSE_LOGGING_DRIVER:-local}
    networks:
      - default
    env_file: ./compose.env
    ports:
      - 8120:8120
    volumes:
      ## Mount external docker socket
      - /var/run/docker.sock:/var/run/docker.sock
      ## Allow Periphery to see processes outside of container
      - /proc:/proc
      ## use self signed certs in docker volume,
      ## or mount your own signed certs.
      - ssl-certs:/etc/komodo/ssl
      ## manage repos in a docker volume,
      ## or change it to an accessible host directory.
      - repos:/etc/komodo/repos
      ## manage stack files in a docker volume,
      ## or change it to an accessible host directory.
      - stacks:/etc/komodo/stacks
      ## Optionally mount a path to store compose files
      # - /path/to/compose:/host/compose

volumes:
  # Postgres
  pg-data:
  # Core
  repo-cache:
  # Periphery
  ssl-certs:
  repos:
  stacks:

networks:
  default: {}

And this machine has even more containers and processes running
Containers running: 29
Processes running: 880

And this containers has the /proc
So this most likely means, that it's not the komodo-periphery itself leaking memory, but some process and/or machine spec that's the problem.

Processes that are running in the "memory-leaking" server and not on the "good" server:
processes.txt
There's nothing specific I see, all seems kernel/OS dependent. But I decided to share nonetheless, maybe it helps.

The machines are 100% different. Different host, different OS, different kernel, different specs...
"memory-leaking" server:

OS:

PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

kernel:
5.10.0-33-amd64

"good" server:

OS:

PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

kernel:
4.19.0-26-amd64

@rita7lopes commented on GitHub (Nov 27, 2024): These are the only logs I see each time I restart the container: ``` docker logs komodo-periphery-1 2024-11-26T15:36:12.256762Z INFO periphery: Komodo Periphery version: v1.16.11 2024-11-26T15:36:12.256799Z INFO periphery: PeripheryConfig { port: 8120, repo_dir: "/etc/komodo/repos", stack_dir: "/etc/komodo/stacks", stats_polling_rate: FiveSeconds, legacy_compose_cli: false, logging: LogConfig { level: Info, stdio: Standard, otlp_endpoint: "", opentelemetry_service_name: "Komodo" }, allowed_ips: [], passkeys: ["##############"], include_disk_mounts: ["/etc/hostname"], exclude_disk_mounts: [], secrets: {}, git_providers: [], docker_registries: [], ssl_enabled: true, ssl_key_file: "/etc/komodo/ssl/key.pem", ssl_cert_file: "/etc/komodo/ssl/cert.pem" } 2024-11-26T15:36:12.256910Z INFO periphery: 🔒 Periphery SSL Enabled 2024-11-26T15:36:12.256934Z INFO periphery: Komodo Periphery starting on https://0.0.0.0:8120 ``` Containers running: 23 Processes running: 242 I tried removing the /proc, and the issue seems to be gone. No memory leak. <img width="1299" alt="Captura de ecrã 2024-11-27, às 17 11 56" src="https://github.com/user-attachments/assets/f6aacb3f-cbfb-4ed7-a3ab-d251b1415b9e"> I noticed something interesting. There's no memory leak on the machine where I have Komodo core and periphery on the same compose file: ``` ################################## # 🦎 KOMODO COMPOSE - POSTGRES 🦎 # ################################## ## This compose file will deploy: ## 1. Postgres + FerretDB Mongo adapter ## 2. Komodo Core ## 3. Komodo Periphery services: postgres: image: postgres labels: komodo.skip: # Prevent Komodo from stopping with StopAllContainers restart: unless-stopped logging: driver: ${COMPOSE_LOGGING_DRIVER:-local} networks: - default # ports: # - 5432:5432 volumes: - pg-data:/var/lib/postgresql/data environment: - POSTGRES_USER=${DB_USERNAME} - POSTGRES_PASSWORD=${DB_PASSWORD} - POSTGRES_DB=${KOMODO_DATABASE_DB_NAME:-komodo} ferretdb: image: ghcr.io/ferretdb/ferretdb labels: komodo.skip: # Prevent Komodo from stopping with StopAllContainers restart: unless-stopped depends_on: - postgres logging: driver: ${COMPOSE_LOGGING_DRIVER:-local} networks: - default ports: - 27017:27017 environment: - FERRETDB_POSTGRESQL_URL=postgres://postgres:5432/${KOMODO_DATABASE_DB_NAME:-komodo} core: image: ghcr.io/mbecker20/komodo:${COMPOSE_KOMODO_IMAGE_TAG:-latest} labels: komodo.skip: # Prevent Komodo from stopping with StopAllContainers restart: unless-stopped depends_on: - ferretdb logging: driver: ${COMPOSE_LOGGING_DRIVER:-local} networks: - default ports: - 9120:9120 env_file: ./compose.env environment: KOMODO_DATABASE_URI: mongodb://${DB_USERNAME}:${DB_PASSWORD}@ferretdb:27017/${KOMODO_DATABASE_DB_NAME:-komodo}?authMechanism=PLAIN volumes: ## Core cache for repos for latest commit hash / contents - repo-cache:/repo-cache ## Store sync files on server # - /path/to/syncs:/syncs ## Optionally mount a custom core.config.toml # - /path/to/core.config.toml:/config/config.toml ## Allows for systemd Periphery connection at ## "http://host.docker.internal:8120" # extra_hosts: # - host.docker.internal:host-gateway ## Deploy Periphery container using this block, ## or deploy the Periphery binary with systemd using ## https://github.com/mbecker20/komodo/tree/main/scripts periphery: image: ghcr.io/mbecker20/periphery:${COMPOSE_KOMODO_IMAGE_TAG:-latest} labels: komodo.skip: # Prevent Komodo from stopping with StopAllContainers restart: unless-stopped logging: driver: ${COMPOSE_LOGGING_DRIVER:-local} networks: - default env_file: ./compose.env ports: - 8120:8120 volumes: ## Mount external docker socket - /var/run/docker.sock:/var/run/docker.sock ## Allow Periphery to see processes outside of container - /proc:/proc ## use self signed certs in docker volume, ## or mount your own signed certs. - ssl-certs:/etc/komodo/ssl ## manage repos in a docker volume, ## or change it to an accessible host directory. - repos:/etc/komodo/repos ## manage stack files in a docker volume, ## or change it to an accessible host directory. - stacks:/etc/komodo/stacks ## Optionally mount a path to store compose files # - /path/to/compose:/host/compose volumes: # Postgres pg-data: # Core repo-cache: # Periphery ssl-certs: repos: stacks: networks: default: {} ``` And this machine has even more containers and processes running Containers running: 29 Processes running: 880 And this containers has the `/proc ` So this most likely means, that it's not the komodo-periphery itself leaking memory, but some process and/or machine spec that's the problem. Processes that are running in the "memory-leaking" server and not on the "good" server: [processes.txt](https://github.com/user-attachments/files/17937896/processes.txt) There's nothing specific I see, all seems kernel/OS dependent. But I decided to share nonetheless, maybe it helps. The machines are 100% different. Different host, different OS, different kernel, different specs... **"memory-leaking" server:** - OS: ``` PRETTY_NAME="Debian GNU/Linux 11 (bullseye)" NAME="Debian GNU/Linux" VERSION_ID="11" VERSION="11 (bullseye)" VERSION_CODENAME=bullseye ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/" ``` - kernel: `5.10.0-33-amd64` **"good" server:** - OS: ``` PRETTY_NAME="Debian GNU/Linux 10 (buster)" NAME="Debian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)" VERSION_CODENAME=buster ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/" ``` - kernel: `4.19.0-26-amd64`

GiteaMirror commented

@mbecker20 commented on GitHub (Nov 28, 2024):

The /proc mount was added so periphery can pick up the process info on the host. While it does seem to work under many conditions, /proc it may have unintended consequences, as these are not normal files but handled by the kernel: https://stackoverflow.com/questions/47072586/docker-access-host-proc

Maybe it affected the docker containers report of its own memory usage?

@mbecker20 commented on GitHub (Nov 28, 2024): The `/proc` mount was added so periphery can pick up the process info on the host. While it does seem to work under many conditions, `/proc` it may have unintended consequences, as these are not normal files but handled by the kernel: https://stackoverflow.com/questions/47072586/docker-access-host-proc Maybe it affected the docker containers report of its own memory usage?

GiteaMirror commented

@rita7lopes commented on GitHub (Nov 29, 2024):

I don't believe that to be the case @mbecker20
The server needed a reboot after the OOM. kswapd0 was visibly trying to handle the situation without success... Reboot was needed.
The memory is reporting accurately, it indeed starts memory leaking on that server if I add the /proc

This is not a blocker but something I believe to be worth sharing.
Keep up the good work.

@rita7lopes commented on GitHub (Nov 29, 2024): I don't believe that to be the case @mbecker20 The server needed a reboot after the OOM. kswapd0 was visibly trying to handle the situation without success... Reboot was needed. The memory is reporting accurately, it indeed starts memory leaking on that server if I add the `/proc` This is not a blocker but something I believe to be worth sharing. Keep up the good work.

GiteaMirror commented

@mbecker20 commented on GitHub (Dec 2, 2024):

I think I may have found something related to processes / memory usage. It looks like the method in sysinfo library used to refresh the process list in periphery memory will not remove dead processes by default: https://github.com/GuillaumeGomez/sysinfo/blob/main/src/common/system.rs#L104. See the second parameter passed to refresh_processes_specifics is always false here.

It looks like this will cause the in memory process list to grow indefinitely, and faster when your host has a lot of processes coming and going.

@mbecker20 commented on GitHub (Dec 2, 2024): I think I may have found something related to processes / memory usage. It looks like the method in `sysinfo` library used to refresh the process list in periphery memory will not remove dead processes by default: https://github.com/GuillaumeGomez/sysinfo/blob/main/src/common/system.rs#L104. See the second parameter passed to `refresh_processes_specifics` is always false here. It looks like this will cause the in memory process list to grow indefinitely, and faster when your host has a lot of processes coming and going.

GiteaMirror commented

@mbecker20 commented on GitHub (Dec 2, 2024):

Hi, I've made the change the change to clear the dead processes from the sysinfo cache on process refresh in 1.16.12.

Could you try mounting /proc again on the problem server when you update to 1.16.12, and see if the problem still happens?

@mbecker20 commented on GitHub (Dec 2, 2024): Hi, I've made the change the change to clear the dead processes from the `sysinfo` cache on process refresh in [1.16.12](https://github.com/mbecker20/komodo/releases/tag/v1.16.12). Could you try mounting `/proc` again on the problem server when you update to 1.16.12, and see if the problem still happens?

GiteaMirror commented

@rita7lopes commented on GitHub (Dec 3, 2024):

Hey @mbecker20 sorry for the delay.
I'm testing it now.
Just FYI this is how the memory consumption looks like for the container for the last 7 days (without the /proc mounted)

At 15:38 UTC today I updated the image to 1.16.12 and mounted the /proc again. As you can see below it had a spike of memory at the start but then stabilized and even decreased compared to before.

I will keep it running and update you tomorrow so that we can come to a better conclusion.

@rita7lopes commented on GitHub (Dec 3, 2024): Hey @mbecker20 sorry for the delay. I'm testing it now. Just FYI this is how the memory consumption looks like for the container for the last 7 days (**without** the `/proc` mounted) ![Captura de ecrã 2024-12-03, às 15 46 10](https://github.com/user-attachments/assets/c7d61253-a785-4d78-a53c-5c79b81685fd) At 15:38 UTC today I updated the image to 1.16.12 and mounted the `/proc` again. As you can see below it had a spike of memory at the start but then stabilized and even decreased compared to before. ![Captura de ecrã 2024-12-03, às 15 48 35](https://github.com/user-attachments/assets/73e80fd9-7a1a-4f60-bc65-bad14da20774) I will keep it running and update you tomorrow so that we can come to a better conclusion.

GiteaMirror commented

@rita7lopes commented on GitHub (Dec 6, 2024):

Hello @mbecker20
Here's the memory consumption during the last two days (since I restarted it with the new version and /proc mounted).

It's clear that now the memory consumption is much better. However, it's still slowly increasing constantly, as it was happening with the previous version without the /proc mounted.
I will leave it running to see if eventually it stabilises.

@rita7lopes commented on GitHub (Dec 6, 2024): Hello @mbecker20 Here's the memory consumption during the last two days (since I restarted it with the new version and `/proc` mounted). <img width="945" alt="Captura de ecrã 2024-12-06, às 12 06 16" src="https://github.com/user-attachments/assets/8d9eeee9-efb2-4e61-8549-bc5c6bf24084"> It's clear that now the memory consumption is much better. However, it's still slowly increasing constantly, as it was happening with the previous version without the `/proc` mounted. I will leave it running to see if eventually it stabilises.

GiteaMirror commented

@2q2code commented on GitHub (Dec 13, 2024):

Just had my main server (which never crashes) crash and I traced it to this periphery leak, and OOM killer subsequently killing everything in sight. I am just looking for docs on how to update periphery (komodo has been updated, but the periphery has not been - explicitly, anyway).

Note that my periphery install was done by portainer because I was migrating all and it was easier this way. I am now changing it to be done by docker compose, deploying by ansible, as I wasn't expecting to find that portainer was still running (along with all its agents).

It's worth noting it is a very slow leak, as it took 3 weeks to eat up all the memory I had available (an extra 2-3GB, I think).

Now updating all periphery and will set up memory watches.

@2q2code commented on GitHub (Dec 13, 2024): Just had my main server (which never crashes) crash and I traced it to this periphery leak, and OOM killer subsequently killing everything in sight. I am just looking for docs on how to update periphery (komodo has been updated, but the periphery has not been - explicitly, anyway). Note that my periphery install was done by `portainer` because I was migrating all and it was easier this way. I am now changing it to be done by `docker compose`, deploying by ansible, as I wasn't expecting to find that portainer was *still* running (along with all its agents). It's worth noting it is a very slow leak, as it took 3 weeks to eat up all the memory I had available (an extra 2-3GB, I think). Now updating all periphery and will set up memory watches.

GiteaMirror commented

@mbecker20 commented on GitHub (Dec 13, 2024):

@2q2code Sorry about the crash! Luckily I believe the latest release already fixes this. To update Periphery installed with container, use the latest or 1.16.12 version tags, and launch the container. If you check the container logs after launch, you should see log saying starting up, and print the version 1.16.12. You can also see the version currently running in the UI, navigate to the Server in UI and you will see the Periphery version of the Server in the header / info area.

If you were previous using latest-aarch64 or 1.16.11-aarch64, you should remove the -aarch64, Komodo now releases multi arch image. Both x86_64 and ARM hosts can use the latest tag now.

@mbecker20 commented on GitHub (Dec 13, 2024): @2q2code Sorry about the crash! Luckily I believe the latest release already fixes this. To update Periphery installed with container, use the `latest` or `1.16.12` version tags, and launch the container. If you check the container logs after launch, you should see log saying starting up, and print the version `1.16.12`. You can also see the version currently running in the UI, navigate to the Server in UI and you will see the Periphery version of the Server in the header / info area. If you were previous using `latest-aarch64` or `1.16.11-aarch64`, you should **remove** the `-aarch64`, Komodo now releases multi arch image. Both x86_64 and ARM hosts can use the `latest` tag now.

GiteaMirror commented

@mbecker20 commented on GitHub (Dec 13, 2024):

@rita7lopes Any updates on the memory usage? I see in the last graph it climbed from 36.5 MB to almost 39 MB, but thats still pretty tiny. All my servers steady under 100MB now, I think always-growing list of processes in that sysinfo crate were the culprit.

@mbecker20 commented on GitHub (Dec 13, 2024): @rita7lopes Any updates on the memory usage? I see in the last graph it climbed from `36.5 MB` to almost `39` MB, but thats still pretty tiny. All my servers steady under 100MB now, I think always-growing list of processes in that sysinfo crate were the culprit.

GiteaMirror commented

@2q2code commented on GitHub (Dec 13, 2024):

@2q2code Sorry about the crash! Luckily I believe the latest release already fixes this. To update Periphery installed with container, use the latest or 1.16.12 version tags, and launch the container. If you check the container logs after launch, you should see log saying starting up, and print the version 1.16.12. You can also see the version currently running in the UI, navigate to the Server in UI and you will see the Periphery version of the Server in the header / info area.

If you were previous using latest-aarch64 or 1.16.11-aarch64, you should remove the -aarch64, Komodo now releases multi arch image. Both x86_64 and ARM hosts can use the latest tag now.

I will do a network-wide ansible playbook to upgrade and redeploy all periphery containers, so I won't need this, but do you think it might be an idea to support bootstrap-updating Komodo's own periphery containers?

@2q2code commented on GitHub (Dec 13, 2024): > @2q2code Sorry about the crash! Luckily I believe the latest release already fixes this. To update Periphery installed with container, use the `latest` or `1.16.12` version tags, and launch the container. If you check the container logs after launch, you should see log saying starting up, and print the version `1.16.12`. You can also see the version currently running in the UI, navigate to the Server in UI and you will see the Periphery version of the Server in the header / info area. > > > > If you were previous using `latest-aarch64` or `1.16.11-aarch64`, you should **remove** the `-aarch64`, Komodo now releases multi arch image. Both x86_64 and ARM hosts can use the `latest` tag now. I will do a network-wide ansible playbook to upgrade and redeploy all periphery containers, so I won't need this, but do you think it might be an idea to support bootstrap-updating Komodo's own periphery containers?

GiteaMirror commented

@mbecker20 commented on GitHub (Dec 13, 2024):

@2q2code I've considered how to do self updating for years, but its not so easy. It seems like Komodo should be able to do it because it can launch containers etc. Why not launch itself.

In fact, I set up Komodo Core compose files as a Stack in Komodo after I spin it up from cli. And then can see Komodo Core logs from the UI. And it does work for Komodo to Deploy itself.

But my Komodo Core compose doesn't include Periphery. I run Periphery under systemd --user, outside container. It is Periphery that is able to complete the redeploy process for Core container. Core doesn't update itself, Periphery does that.

But I'm not sure of any open source project which has container able to do self update. Maybe the Nextcloud AIO monstrosity (sorry NC) which has you launch a master container, mount docker socket in there, which then spins up a bunch of other containers. But single container deployments? Don't think so.

This is the problem for the Komodo case: All deploying is done by Periphery spawning shell child processes which run docker commands. But if that child process happens to kill its parent Periphery process, say in the course of updating the container, the child process will be killed as well, and the operation will fail to complete. In this case, Periphery will just kill itself, it cannot resurrect.

I've considered whether its possible to spawn a temporary container with -d --rm flags, and perform the update from there. If all users set up their Periphery exactly the same, maybe it could be done. However then consider all the edge cases from different users specific setups, mainly systemd vs container installs.

Again, been considering this for a while, it may be possible its just not so easy, hopefully you can see why now.

I have used both Puppet and Ansible to manage the Periphery deployment fully automated via Github Webhook. I prefer Ansible, and have a really nice flow figured out using the alpine/ansible container. I structure a git repo using this simple following layout, and use webhook to Deploy it as a Komodo Stack on commit, which updates all Periphery in the network.

root folder
- hosts.ini
- playbook.yaml
- periphery.config.toml
- ansible.compose.yaml

In ansible.compose.yaml its something like:

services:

  ansible:
    image: alpine/ansible
    user: 1000:1000
    network_mode: host
    working_dir: /project
    command: ansible-playbook -i hosts.ini playbook.yaml --extra-vars "HOME=$HOME"
    volumes:
      - /etc/passwd:/etc/passwd
      - $HOME/.ssh:$HOME/.ssh
      - ./.ansible:$HOME/.ansible
      - ./hosts.ini:/project/hosts.ini
      - ./playbook.yaml:/project/playbook.yaml

@mbecker20 commented on GitHub (Dec 13, 2024): @2q2code I've considered how to do self updating for years, but its not so easy. It seems like Komodo should be able to do it because it can launch containers etc. Why not launch itself. In fact, I set up Komodo Core compose files as a Stack in Komodo after I spin it up from cli. And then can see Komodo Core logs *from the UI*. And it *does work* for Komodo to Deploy itself. But my Komodo Core compose doesn't include Periphery. I run Periphery under `systemd --user`, outside container. It is Periphery that is able to complete the redeploy process for Core container. Core doesn't update *itself*, Periphery does that. But I'm not sure of any open source project which has container able to do self update. Maybe the Nextcloud AIO monstrosity (sorry NC) which has you launch a master container, mount docker socket in there, which then spins up a bunch of other containers. But single container deployments? Don't think so. This is the problem for the Komodo case: All deploying is done by Periphery spawning shell child processes which run docker commands. But if that child process happens to kill its parent Periphery process, say in the course of updating the container, the *child process* will be killed as well, and the operation will fail to complete. In this case, Periphery will just kill itself, it cannot resurrect. I've considered whether its possible to spawn a temporary container with `-d --rm` flags, and perform the update from there. If all users set up their Periphery exactly the same, maybe it could be done. However then consider all the edge cases from different users specific setups, mainly `systemd` vs `container` installs. Again, been considering this for a while, it may be possible its just not so easy, hopefully you can see why now. I have used both Puppet and Ansible to manage the Periphery deployment fully automated via Github Webhook. I prefer Ansible, and have a really nice flow figured out using the `alpine/ansible` container. I structure a git repo using this simple following layout, and use webhook to Deploy it as a Komodo Stack on commit, which updates all Periphery in the network. - `root folder` - `hosts.ini` - `playbook.yaml` - `periphery.config.toml` - `ansible.compose.yaml` In ansible.compose.yaml its something like: ```yaml services: ansible: image: alpine/ansible user: 1000:1000 network_mode: host working_dir: /project command: ansible-playbook -i hosts.ini playbook.yaml --extra-vars "HOME=$HOME" volumes: - /etc/passwd:/etc/passwd - $HOME/.ssh:$HOME/.ssh - ./.ansible:$HOME/.ansible - ./hosts.ini:/project/hosts.ini - ./playbook.yaml:/project/playbook.yaml ```

GiteaMirror commented

@rita7lopes commented on GitHub (Dec 18, 2024):

Hello @mbecker20,
I've been testing the new version 1.16.12 for a while now
I can confidently report that the memory leak issue has been resolved on version 1.16.12

The machine I've used to test this version has faced OOM caused by other containers running, but the periphery never freaked out. Even with the machine and its processes OOM, the periphery kept memory under 100MB always—solid as a rock!

Great work @mbecker20!!

@2q2code This version 100% works for me and I tested it pretty much under fire, so if you use the latest version I believe you should be fine!

@rita7lopes commented on GitHub (Dec 18, 2024): Hello @mbecker20, I've been testing the new version `1.16.12` for a while now I can **confidently** report that the memory leak issue has been **resolved** on version `1.16.12` The machine I've used to test this version has faced OOM caused by other containers running, but the periphery never freaked out. Even with the machine and its processes OOM, the periphery kept memory under 100MB always—solid as a rock! Great work @mbecker20!! @2q2code This version 100% works for me and I tested it pretty much under fire, so if you use the latest version I believe you should be fine!

GiteaMirror commented