Failed to validate run directory on host after stack write (canonicalize error) #1339

New Issue

2026-03-22T17:22:33-05:00

GiteaMirror commented

2026-03-22 17:22:33 -05:00

Originally created by @ragchuck on GitHub (Aug 23, 2025).

I've described my use case in this https://github.com/moghtech/komodo/issues/758#issuecomment-3217274733.

I want to reuse my existing stacks directory (stack_dir = "/opt/stacks".
I gave the komodo:komodo user ownership of the directory.
The stacks directory lies outside of the currently configured root (root_directory = "/home/komodo")

Whenever I try to "Pull Images" I get the following error:

ERROR: 500 Internal Server Error

TRACE:
1: Failed to validate run directory on host after stack write (canonicalize error)
2: Permission denied (os error 13)

It's hard to track the error down.

From what I can see is that the canonicalize here is unable to access the file path.

Based on the canonicalize documentation, the Unix equivalent is realpath:
When I try to perform realpath /opt/stacks/repo_name with the komodo user it works.

Doesn't the service run with the users permissions? I mean the user is part of the docker group and it works.

/home/komodo/.config/systemd/user/periphery.service

[Unit]
Description=Agent to connect with Komodo Core

[Service]
Environment="HOME=/home/komodo"
ExecStart=/bin/sh -lc "/home/komodo/.local/bin/periphery --config-path /home/komodo/.config/komodo/periphery.config.toml"
Restart=on-failure
TimeoutStartSec=0

[Install]
WantedBy=default.target

Not sure what else I could try.

Sure, I could migrate everything to the /etc/komodo directory, but I my approach should also work, right? I also think, this is a situation to learn something new. Maybe somebody can me point me to the error in my thought process.

Originally created by @ragchuck on GitHub (Aug 23, 2025). I've described my use case in this https://github.com/moghtech/komodo/issues/758#issuecomment-3217274733. * I want to reuse my existing stacks directory (`stack_dir = "/opt/stacks"`. * I gave the `komodo:komodo` user ownership of the directory. * The stacks directory lies outside of the currently configured root (`root_directory = "/home/komodo"`) Whenever I try to "Pull Images" I get the following error: > ERROR: 500 Internal Server Error > > TRACE: > 1: Failed to validate run directory on host after stack write (canonicalize error) > 2: Permission denied (os error 13) It's hard to track the error down. From what I can see is that the [canonicalize here](https://github.com/moghtech/komodo/blob/cae80b43e5c35ed2d3836ba07df5cf9c0dbfbb99/bin/periphery/src/api/compose.rs#L350) is unable to access the file path. Based on the [canonicalize documentation](https://doc.rust-lang.org/beta/std/fs/fn.canonicalize.html), the Unix equivalent is `realpath`: When I try to perform `realpath /opt/stacks/repo_name` with the komodo user it works. Doesn't the service run with the users permissions? I mean the user is part of the docker group and it works. <details><summary>/home/komodo/.config/systemd/user/periphery.service</summary> <p> ``` [Unit] Description=Agent to connect with Komodo Core [Service] Environment="HOME=/home/komodo" ExecStart=/bin/sh -lc "/home/komodo/.local/bin/periphery --config-path /home/komodo/.config/komodo/periphery.config.toml" Restart=on-failure TimeoutStartSec=0 [Install] WantedBy=default.target ``` </p> </details> Not sure what else I could try. Sure, I could migrate everything to the `/etc/komodo` directory, but I my approach should also work, right? I also think, this is a situation to learn something new. Maybe somebody can me point me to the error in my thought process.

GiteaMirror closed this issue

2026-03-22 17:22:35 -05:00

GiteaMirror commented

2026-03-22 17:22:36 -05:00

@mbecker20 commented on GitHub (Aug 24, 2025):

It looks like an ownership issue, Komodo user will need rw permission under /opt/stack too. Did you apply ownership recursively (-R)?

@mbecker20 commented on GitHub (Aug 24, 2025): It looks like an ownership issue, Komodo user will need rw permission under /opt/stack too. Did you apply ownership recursively (-R)?

GiteaMirror commented

2026-03-22 17:22:37 -05:00

@ragchuck commented on GitHub (Aug 30, 2025):

Sorry for not responding earlier. Yes, all files under the directory in question have the owner and group set. Though, some files need to be for root, because some containers are not rootless (yet).

@ragchuck commented on GitHub (Aug 30, 2025): Sorry for not responding earlier. Yes, all files under the directory in question have the owner and group set. Though, some files need to be for root, because some containers are not rootless (yet).

GiteaMirror commented

2026-03-22 17:22:39 -05:00

@codeshell commented on GitHub (Sep 8, 2025):

The reason might be rather trivial.
@ragchuck it is slightly different because you hit error 13 instead of error 2, but it's worth a try, I guess.

I hit a similar roadblock when adding a new server via peripery running as (--user) service.

Task Error: Stage 1 of 1 | 0.0 seconds

ERROR: 500 Internal Server Error

TRACE:
	1: Failed to validate run directory on host after stack write (canonicalize error)
	2: No such file or directory (os error 2)

❌ Pull Stack: With Pull Images from the Stack in Komodo, the repos_dir folder was created but empty and the task failed.
❌ Deploy Stack: With Redeploy from the Stack in Komodo, the stacks_dir was successfully created but empty and the task failed.

Thus most likely not a permission problem.

Only then it occurred to me that the host system (UGOS in my example) might not have git installed.

komodo-agent@NAS:~$ git --version
bash: git: command not found

So for me this error was fixed by running the following on the server with the "seemingly misbehaving" periphery:

sudo apt-get install git

I double checked the Komodo documentation and the periphery install logging. There is no test for it as dependency or at least a warning that as long as git is missing on the remote server some features cannot be used. Maybe this could be included somewhere because it is so obvious that you might forget to check this on a non standard system :)

@codeshell commented on GitHub (Sep 8, 2025): The reason might be rather trivial. @ragchuck _it is slightly different because you hit error 13 instead of error 2, but it's worth a try, I guess._ I hit a similar roadblock when adding a new server via peripery running as (--user) service. Task Error: Stage 1 of 1 | 0.0 seconds ```stderr ERROR: 500 Internal Server Error TRACE: 1: Failed to validate run directory on host after stack write (canonicalize error) 2: No such file or directory (os error 2) ``` - ❌ Pull Stack: With `Pull Images` from the Stack in Komodo, the `repos_dir` folder was created but empty and the task failed. - ❌ Deploy Stack: With `Redeploy` from the Stack in Komodo, the `stacks_dir` was successfully created but empty and the task failed. Thus most likely _not_ a permission problem. Only then it occurred to me that the host system (UGOS in my example) might not have `git` installed. ```sh komodo-agent@NAS:~$ git --version bash: git: command not found ``` So for me this error was fixed by running the following on the server with the "seemingly misbehaving" periphery: ```sh sudo apt-get install git ``` _I double checked the Komodo documentation and the periphery install logging. There is no test for it as dependency or at least a warning that as long as `git` is missing on the remote server some features cannot be used. Maybe this could be included somewhere because it is so obvious that you might forget to check this on a non standard system :)_

GiteaMirror commented

2026-03-22 17:22:39 -05:00

@mbecker20 commented on GitHub (Sep 8, 2025):

@codeshell thanks for report, there is an issue open for this #797

@mbecker20 commented on GitHub (Sep 8, 2025): @codeshell thanks for report, there is an issue open for this #797

GiteaMirror commented

2026-03-22 17:22:40 -05:00

@ragchuck commented on GitHub (Sep 8, 2025):

It looks like an ownership issue, Komodo user will need rw permission under /opt/stack too. Did you apply ownership recursively (-R)?

So, it worked after I also changed the owner of the parent directory /opt/stacks.

It's not clear to me, why it also needs the parent's directory the same owner.

@ragchuck commented on GitHub (Sep 8, 2025): > It looks like an ownership issue, Komodo user will need rw permission under /opt/stack too. Did you apply ownership recursively (-R)? So, it worked after I also changed the owner of the parent directory /opt/stacks. It's not clear to me, why it also needs the parent's directory the same owner.

GiteaMirror commented

2026-03-22 17:22:42 -05:00

@mbecker20 commented on GitHub (Sep 8, 2025):

IIRC it needs write on parent folder in order to make a new folder inside it, this happens when you clone new stack.

@mbecker20 commented on GitHub (Sep 8, 2025): IIRC it needs write on parent folder in order to make a new folder inside it, this happens when you clone new stack.