[Feature] Procedure filters #26

Open
opened 2025-10-31 15:00:05 -05:00 by GiteaMirror · 15 comments
Owner

Originally created by @C0untZero on GitHub (Aug 23, 2024).

At the moment creating a procedure to, for instance, deploy or redeploy all stacks on a server is tedious - you have to specifically add a DeployStack action for each stack on the server. And this is not limited to stacks only, any procedure action with a target resource is the same.

Based on our discussion on Discord, filters could help with this. The idea is to create one action with a filter target which could then filter on any resource property, like name, server, tag, image registry, git provider/account etc. The target dropdown should have a use custom filter option at the top which when pressed shows a query building window. This window should have a list of properties we can filter on for the relevant resource type (based on selected action). Each property to filter on would support exact name matching, simple wildcard search, or advanced regex matching. An example toml for a filter would be any of the three options below:

[[filter]]
server = "server1"

[[filter]]
name = "$reg\arr$\"

[[filter]]
name = "*arr"

Once a filter is created and the query window is closed, the target column for the configured row should show the filter query instead of a specific resource name:
image
Another nice to have feature would also be to show resource names that matched the query when the filter is hovered over, or maybe in the query config window at the bottom or something.

A simple implementation for this would be wildcard matching multiple resources based on one property.
A more advanced implementation could be built from this to support regex as well as matching on multiple properties using logic operators AND, OR, NOT, etc. Something akin to JQL - KQL (Komodo Query Language) maybe? ;)

Originally created by @C0untZero on GitHub (Aug 23, 2024). At the moment creating a procedure to, for instance, deploy or redeploy all stacks on a server is tedious - you have to specifically add a DeployStack action for each stack on the server. And this is not limited to stacks only, any procedure action with a target resource is the same. Based on our discussion on Discord, filters could help with this. The idea is to create one action with a filter target which could then filter on any resource property, like name, server, tag, image registry, git provider/account etc. The target dropdown should have a use custom filter option at the top which when pressed shows a query building window. This window should have a list of properties we can filter on for the relevant resource type (based on selected action). Each property to filter on would support exact name matching, simple wildcard search, or advanced regex matching. An example toml for a filter would be any of the three options below: ``` [[filter]] server = "server1" [[filter]] name = "$reg\arr$\" [[filter]] name = "*arr" ``` Once a filter is created and the query window is closed, the target column for the configured row should show the filter query instead of a specific resource name: ![image](https://github.com/user-attachments/assets/37068d4c-dba6-482d-be26-2d4f43e1224d) Another nice to have feature would also be to show resource names that matched the query when the filter is hovered over, or maybe in the query config window at the bottom or something. A simple implementation for this would be wildcard matching multiple resources based on one property. A more advanced implementation could be built from this to support regex as well as matching on multiple properties using logic operators AND, OR, NOT, etc. Something akin to [JQL](https://www.atlassian.com/software/jira/guides/jql/overview) - KQL (Komodo Query Language) maybe? ;)
GiteaMirror added the enhancement label 2025-10-31 15:00:05 -05:00
Author
Owner

@mbecker20 commented on GitHub (Aug 23, 2024):

Thanks for the write up C0unt!

It's going to be a great feature and I'm really excited by the possibilities here. I'm not sure how I will include it in the upcoming versions at this point, maybe hesitantly in 1.15. For now I will leave it as an enhancement.

🦎

@mbecker20 commented on GitHub (Aug 23, 2024): Thanks for the write up C0unt! It's going to be a great feature and I'm really excited by the possibilities here. I'm not sure how I will include it in the upcoming versions at this point, maybe hesitantly in 1.15. For now I will leave it as an enhancement. 🦎
Author
Owner

@mbecker20 commented on GitHub (Aug 23, 2024):

Regex support at the library level will be using the already included regex crate.
In terms of "Wildcard" syntax, we can use the cloudflare standard handled by their wilcard crate.

@mbecker20 commented on GitHub (Aug 23, 2024): Regex support at the library level will be using the already included [regex crate](https://crates.io/crates/regex). In terms of "Wildcard" syntax, we can use the cloudflare standard handled by their [wilcard crate](https://crates.io/crates/wildcard).
Author
Owner

@mbecker20 commented on GitHub (Oct 28, 2024):

1.16.5 gets most of the way there. It doesn't allow you to match on anything other than the resource name though, and currently doesn't have any preview of which resources are matched.

@mbecker20 commented on GitHub (Oct 28, 2024): 1.16.5 gets most of the way there. It doesn't allow you to match on anything other than the resource name though, and currently doesn't have any preview of which resources are matched.
Author
Owner

@C0untZero commented on GitHub (Oct 28, 2024):

Looking good, thanks for the progress on this.

I'm not sure about the decision of introducing duplicate actions with batch in name. Kinda crowds the action list imo.
Is this just temporary until the full query builder interface is built?

@C0untZero commented on GitHub (Oct 28, 2024): Looking good, thanks for the progress on this. I'm not sure about the decision of introducing duplicate actions with batch in name. Kinda crowds the action list imo. Is this just temporary until the full query builder interface is built?
Author
Owner

@mbecker20 commented on GitHub (Oct 28, 2024):

Thanks for the quick feedback. There is a reason it was done this way, I can try to explain.

There is another issue #93 which request to be able to make executions on more than one Stacks / Containers by selecting multiple in the table. This is best accomplished by adding ability to eg DeployStack, but target more than one Stack in the call.

Looking at the API for DeployStack, you see it returns a single Update. This is an issue, because a batch call to this API will under the hood call DeployStack on the individual stacks, each will produce an Update.

If you call to DeployStack targeting multiple Stacks, the return type of the data should actually be something like Vec<Update>, a list of all of the Updates for each Stacks deploy. So either the return type of DeployStack needs to change, or it needs another api. I think DeployStack -> Update is still a good API to expose and want to keep it as it is, so BatchDeployStack and the other Batch executions were created.

They all have this return type: BatchExecutionResponse. It is basically Vec<Update> but with some error handling, each underlying Update in the batch call may instead be some Error case which needs to be returned to user.

With this Batch execution handling setup, the frontend can use it to handle the batch executions when user select multiple resources in a table, and also Actions can call this API directly.

For Procedures, I felt it was clearer to make this matching behavior explicit at API method level. This way, Procedures continue to map 1-to-1 with the behavior of the API when called outside of Procedures. Ie, it won't work to call the DeployStack api directly passing arr-* as the stack, but using that with BatchDeployStack will work. Procedures just follow this standard, users should use the Batch apis if they intend to do a batch execution targetting multiple underlying resources.

The only downside I can see is the crowding of action list. But there is a search feature there, its not too bad when you start typing deploy.

@mbecker20 commented on GitHub (Oct 28, 2024): Thanks for the quick feedback. There is a reason it was done this way, I can try to explain. There is another issue #93 which request to be able to make executions on more than one Stacks / Containers by selecting multiple in the table. This is best accomplished by adding ability to eg `DeployStack`, but target more than one Stack in the call. Looking at the API for [DeployStack](https://docs.rs/komodo_client/latest/komodo_client/api/execute/struct.DeployStack.html), you see it returns a single [Update](https://docs.rs/komodo_client/latest/komodo_client/entities/update/struct.Update.html). This is an issue, because a batch call to this API will under the hood call `DeployStack` on the individual stacks, each will produce an `Update`. If you call to `DeployStack` targeting multiple Stacks, the return type of the data should actually be something like `Vec<Update>`, a list of all of the Updates for each Stacks deploy. So either the return type of `DeployStack` needs to change, or it needs another api. I think `DeployStack` -> `Update` is still a good API to expose and want to keep it as it is, so `BatchDeployStack` and the other `Batch` executions were created. They all have this return type: [BatchExecutionResponse](https://docs.rs/komodo_client/latest/komodo_client/api/execute/type.BatchExecutionResponse.html). It is basically `Vec<Update>` but with some error handling, each underlying `Update` in the batch call may instead be some Error case which needs to be returned to user. With this `Batch` execution handling setup, the frontend can use it to handle the batch executions when user select multiple resources in a table, and also `Actions` can call this API directly. For `Procedures`, I felt it was clearer to make this matching behavior explicit at API method level. This way, `Procedures` continue to map 1-to-1 with the behavior of the API when called outside of Procedures. Ie, it won't work to call the `DeployStack` api directly passing `arr-*` as the `stack`, but using that with `BatchDeployStack` will work. Procedures just follow this standard, users should use the `Batch` apis if they intend to do a batch execution targetting multiple underlying resources. The only downside I can see is the crowding of action list. But there is a search feature there, its not too bad when you start typing `deploy`.
Author
Owner

@C0untZero commented on GitHub (Oct 29, 2024):

Ok, now that I understand the backend part it makes sense. The search feature does make it easy though, so I guess it's all good.

@C0untZero commented on GitHub (Oct 29, 2024): Ok, now that I understand the backend part it makes sense. The search feature does make it easy though, so I guess it's all good.
Author
Owner

@oblivioncth commented on GitHub (Jun 25, 2025):

This somewhat it's own issue, but overlaps with this an some others:

It would be nice to have a way to exclude stacks from the batch actions, or any procedure really, altogether. For example, I use BatchPullStack + BatchDeployStack in one procedure, and BatchDeployStackIfChanged in another to blanket update all of my containers; however, I have a couple stacks that I generally want to stay stopped as I only spin them up manually on occasion and then stop them when I'm done using them.

As things stand, unless I add some kind of static identifier to the name of all the stacks I want to update, like "-update-" (ugly), then those few stacks I want to stay stopped will be deployed every time the procedure runs and I have to go out of my way to keep stopping them.

@oblivioncth commented on GitHub (Jun 25, 2025): This somewhat it's own issue, but overlaps with this an some others: It would be nice to have a way to exclude stacks from the batch actions, or any procedure really, altogether. For example, I use `BatchPullStack` + `BatchDeployStack` in one procedure, and `BatchDeployStackIfChanged` in another to blanket update all of my containers; however, I have a couple stacks that I generally want to stay stopped as I only spin them up manually on occasion and then stop them when I'm done using them. As things stand, unless I add some kind of static identifier to the name of all the stacks I want to update, like "-update-" (ugly), then those few stacks I want to stay stopped will be deployed every time the procedure runs and I have to go out of my way to keep stopping them.
Author
Owner

@samumatic commented on GitHub (Jun 26, 2025):

This is the same for me, i was trying to get the stacks excluded via regex (^(?!.*\btemplate\b).*) but no success there. Especially with the new "Templates" feature in the newest release the BatchDeployStackIfChanged tries to start my template stack because of my * regex.

I'm using renovate to update my compose files in GitHub and then BatchDeployStackIfChanged every night to update my stacks on the servers.

@samumatic commented on GitHub (Jun 26, 2025): This is the same for me, i was trying to get the stacks excluded via regex (`^(?!.*\btemplate\b).*`) but no success there. Especially with the new "Templates" feature in the newest release the `BatchDeployStackIfChanged` tries to start my template stack because of my `*` regex. I'm using renovate to update my compose files in GitHub and then `BatchDeployStackIfChanged` every night to update my stacks on the servers.
Author
Owner

@oblivioncth commented on GitHub (Jun 26, 2025):

@samumatic

Temporary workaround until this is hopefully a built-in feature:

// Same as BatchDeployStacksIfChanged, but with an exclusion filter
console.log('Deploying all non-excluded stacks...\n');

const excludedStacks = (await komodo.read("GetVariable", {
  name: "ADHOC_STACKS" // YOUR VARIABLE NAME HERE
}))?.value?.split(/\s+/).filter(Boolean) ?? [];

if (!excludedStacks.length)
  console.log('Exclusion filter is empty!\n');

const stacks = await komodo.read("ListStacks", {});

for(const stack of stacks) {
  const stackName = stack.name;

  if(excludedStacks.includes(stackName)) {
    console.log('Skipping stack', stackName, '(excuded)')
    continue;
  }

  console.log('Deploying', stackName)
  await komodo.execute("DeployStackIfChanged", {
   stack: stackName,
  });
}

console.log('');

Add this as an Action, then add a Variable called ADHOC_STACKS (or whatever you want and change it in the script) to your instance, containing a whitespace delimited list of stacks you don't want deployed.

Then just run this action in your procedure instead of BatchDeployStackIfChanged. You can also swap the call in the script to simply DeployStack if you wanted to use that for routine/unconditional redeploys also.

I also use renovate, as well as do weekly redeploys for the stacks with "latest" tags, hence why the variable, so it can be used in both actions.

I'd like to use a variable with newlines for this, but it causes a minor bug with syncs if you use those, so for now I'm just using plain spaces to separate the name (you could change the script to use another delimiter if you like as well): https://github.com/moghtech/komodo/issues/627

@oblivioncth commented on GitHub (Jun 26, 2025): @samumatic Temporary workaround until this is hopefully a built-in feature: ```ts // Same as BatchDeployStacksIfChanged, but with an exclusion filter console.log('Deploying all non-excluded stacks...\n'); const excludedStacks = (await komodo.read("GetVariable", { name: "ADHOC_STACKS" // YOUR VARIABLE NAME HERE }))?.value?.split(/\s+/).filter(Boolean) ?? []; if (!excludedStacks.length) console.log('Exclusion filter is empty!\n'); const stacks = await komodo.read("ListStacks", {}); for(const stack of stacks) { const stackName = stack.name; if(excludedStacks.includes(stackName)) { console.log('Skipping stack', stackName, '(excuded)') continue; } console.log('Deploying', stackName) await komodo.execute("DeployStackIfChanged", { stack: stackName, }); } console.log(''); ``` Add this as an Action, then add a Variable called `ADHOC_STACKS` (or whatever you want and change it in the script) to your instance, containing a whitespace delimited list of stacks you don't want deployed. Then just run this action in your procedure instead of BatchDeployStackIfChanged. You can also swap the call in the script to simply `DeployStack` if you wanted to use that for routine/unconditional redeploys also. I also use renovate, as well as do weekly redeploys for the stacks with "latest" tags, hence why the variable, so it can be used in both actions. I'd like to use a variable with newlines for this, but it causes a minor bug with syncs if you use those, so for now I'm just using plain spaces to separate the name (you could change the script to use another delimiter if you like as well): https://github.com/moghtech/komodo/issues/627
Author
Owner

@oblivioncth commented on GitHub (Jun 27, 2025):

It's in the linked issue, but for clarity I just wanted to add this here:

@mbecker20 indirectly suggested using tags for this, which I do think can be a lot better depending on your preference. It looks like this:

// Same as BatchDeployStacksIfChanged, but with an exclusion filter
console.log('Deploying all non-excluded stacks...\n');

// Get Tag Id. The API should throw if it's not found, but we employ safety checks just in case
const exclusionTagId = (await komodo.read("GetTag", { tag: "ad-hoc" }))?._id?.$oid ?? null; // YOUR TAG HERE
if (!exclusionTagId)
  console.log('Exclusion tag not found!\n');

// Deploy all stacks, minus the excluded ones
const allStacks = await komodo.read("ListStacks", {});
for(const stack of allStacks) {
  const name = stack.name;
  const id = stack.id;
  const tags = stack.tags;

  if(tags.includes(exclusionTagId)) {
    console.log('Skipping stack', name, '(excuded)')
    continue;
  }

  console.log('Deploying', name)
  await komodo.execute("DeployStackIfChanged", {
   stack: id,
  });
}

console.log('');

Just make the matching tag and tag your stacks with it.

@oblivioncth commented on GitHub (Jun 27, 2025): It's in the linked issue, but for clarity I just wanted to add this here: @mbecker20 indirectly suggested using tags for this, which I do think can be a lot better depending on your preference. It looks like this: ```ts // Same as BatchDeployStacksIfChanged, but with an exclusion filter console.log('Deploying all non-excluded stacks...\n'); // Get Tag Id. The API should throw if it's not found, but we employ safety checks just in case const exclusionTagId = (await komodo.read("GetTag", { tag: "ad-hoc" }))?._id?.$oid ?? null; // YOUR TAG HERE if (!exclusionTagId) console.log('Exclusion tag not found!\n'); // Deploy all stacks, minus the excluded ones const allStacks = await komodo.read("ListStacks", {}); for(const stack of allStacks) { const name = stack.name; const id = stack.id; const tags = stack.tags; if(tags.includes(exclusionTagId)) { console.log('Skipping stack', name, '(excuded)') continue; } console.log('Deploying', name) await komodo.execute("DeployStackIfChanged", { stack: id, }); } console.log(''); ``` Just make the matching tag and tag your stacks with it.
Author
Owner

@shadybraden commented on GitHub (Aug 29, 2025):

It's in the linked issue, but for clarity I just wanted to add this here:

@mbecker20 indirectly suggested using tags for this, which I do think can be a lot better depending on your preference. It looks like this:

// Same as BatchDeployStacksIfChanged, but with an exclusion filter
console.log('Deploying all non-excluded stacks...\n');

// Get Tag Id. The API should throw if it's not found, but we employ safety checks just in case
const exclusionTagId = (await komodo.read("GetTag", { tag: "ad-hoc" }))?._id?.$oid ?? null; // YOUR TAG HERE
if (!exclusionTagId)
console.log('Exclusion tag not found!\n');

// Deploy all stacks, minus the excluded ones
const allStacks = await komodo.read("ListStacks", {});
for(const stack of allStacks) {
const name = stack.name;
const id = stack.id;
const tags = stack.tags;

if(tags.includes(exclusionTagId)) {
console.log('Skipping stack', name, '(excuded)')
continue;
}

console.log('Deploying', name)
await komodo.execute("DeployStackIfChanged", {
stack: id,
});
}

console.log('');
Just make the matching tag and tag your stacks with it.

This works, but the issue I am having now is that everytime a DeployStackIfChanged runs my CPU pins for ~15 seconds as it runs deploy on every container (except ad-hoc) even if there are no changes.

Any clues on how to debug this?

@shadybraden commented on GitHub (Aug 29, 2025): > It's in the linked issue, but for clarity I just wanted to add this here: > > [@mbecker20](https://github.com/mbecker20) indirectly suggested using tags for this, which I do think can be a lot better depending on your preference. It looks like this: > > // Same as BatchDeployStacksIfChanged, but with an exclusion filter > console.log('Deploying all non-excluded stacks...\n'); > > // Get Tag Id. The API should throw if it's not found, but we employ safety checks just in case > const exclusionTagId = (await komodo.read("GetTag", { tag: "ad-hoc" }))?._id?.$oid ?? null; // YOUR TAG HERE > if (!exclusionTagId) > console.log('Exclusion tag not found!\n'); > > // Deploy all stacks, minus the excluded ones > const allStacks = await komodo.read("ListStacks", {}); > for(const stack of allStacks) { > const name = stack.name; > const id = stack.id; > const tags = stack.tags; > > if(tags.includes(exclusionTagId)) { > console.log('Skipping stack', name, '(excuded)') > continue; > } > > console.log('Deploying', name) > await komodo.execute("DeployStackIfChanged", { > stack: id, > }); > } > > console.log(''); > Just make the matching tag and tag your stacks with it. This works, but the issue I am having now is that everytime a `DeployStackIfChanged` runs my CPU pins for ~15 seconds as it runs `deploy` on every container (except ad-hoc) even if there are no changes. Any clues on how to debug this?
Author
Owner

@mbecker20 commented on GitHub (Aug 29, 2025):

@shadybraden
Is it definitely every time?

I should have mentioned this in the notes on this. The first deploy after you make a configuration change affecting the files, they still use old deploy cache files. So immediately this triggers deploy on all stacks, because there is Komodo cache diff.

After the first deploy, the caches will be updated, and subsequent deploy if changed will respect only doing configured executions with actual diffs to the files.

@mbecker20 commented on GitHub (Aug 29, 2025): @shadybraden Is it definitely every time? I should have mentioned this in the notes on this. The first deploy after you make a configuration change affecting the files, they still use old deploy cache files. So immediately this triggers deploy on all stacks, because there is Komodo cache diff. After the first deploy, the caches will be updated, and subsequent deploy if changed will respect only doing configured executions with actual diffs to the files.
Author
Owner

@shadybraden commented on GitHub (Aug 30, 2025):

Yup. Every time I run the following procedure or action:

[[procedure]]
name = "webhook_stacks_main"

[[procedure.config.stage]]
name = "git pull main"
enabled = true
executions = [
  { execution.type = "PullRepo", execution.params.repo = "compose_main", enabled = true }
]

[[procedure.config.stage]]
name = "DeployOnChanges"
enabled = true
executions = [
  { execution.type = "BatchDeployStackIfChanged", execution.params.pattern = "*", enabled = true }
]
[[action]]
name = "ac-hoc"
[action.config]
file_contents = """
// Same as BatchDeployStacksIfChanged, but with an exclusion filter
console.log('Deploying all non-excluded stacks...\n');

// Get Tag Id. The API should throw if it's not found, but we employ safety checks just in case
const exclusionTagId = (await komodo.read("GetTag", { tag: "ad-hoc" }))?._id?.$oid ?? null; // YOUR TAG HERE
if (!exclusionTagId)
  console.log('Exclusion tag not found!\n');

// Deploy all stacks, minus the excluded ones
const allStacks = await komodo.read("ListStacks", {});
for(const stack of allStacks) {
  const name = stack.name;
  const id = stack.id;
  const tags = stack.tags;

  if(tags.includes(exclusionTagId)) {
    console.log('Skipping stack', name, '(excuded)')
    continue;
  }

  console.log('Deploying', name)
  await komodo.execute("DeployStackIfChanged", {
   stack: id,
  });
}

console.log('');
"""

Here's a quick video of what I'm seeing:

https://github.com/user-attachments/assets/76706045-ea14-4bd4-bb38-04a4e266dae0

@shadybraden commented on GitHub (Aug 30, 2025): Yup. Every time I run the following procedure or action: ``` [[procedure]] name = "webhook_stacks_main" [[procedure.config.stage]] name = "git pull main" enabled = true executions = [ { execution.type = "PullRepo", execution.params.repo = "compose_main", enabled = true } ] [[procedure.config.stage]] name = "DeployOnChanges" enabled = true executions = [ { execution.type = "BatchDeployStackIfChanged", execution.params.pattern = "*", enabled = true } ] ``` ``` [[action]] name = "ac-hoc" [action.config] file_contents = """ // Same as BatchDeployStacksIfChanged, but with an exclusion filter console.log('Deploying all non-excluded stacks...\n'); // Get Tag Id. The API should throw if it's not found, but we employ safety checks just in case const exclusionTagId = (await komodo.read("GetTag", { tag: "ad-hoc" }))?._id?.$oid ?? null; // YOUR TAG HERE if (!exclusionTagId) console.log('Exclusion tag not found!\n'); // Deploy all stacks, minus the excluded ones const allStacks = await komodo.read("ListStacks", {}); for(const stack of allStacks) { const name = stack.name; const id = stack.id; const tags = stack.tags; if(tags.includes(exclusionTagId)) { console.log('Skipping stack', name, '(excuded)') continue; } console.log('Deploying', name) await komodo.execute("DeployStackIfChanged", { stack: id, }); } console.log(''); """ ``` Here's a quick video of what I'm seeing: https://github.com/user-attachments/assets/76706045-ea14-4bd4-bb38-04a4e266dae0
Author
Owner

@mbecker20 commented on GitHub (Aug 31, 2025):

@shadybraden I think it would help you figure out what is going on if you break it down and make sure it works one stack at a time. For example, have webhook only trigger DeployStackIfChanged on a single Stack, and try committing to files that are associated with that stack, and ones that are not associated with that stack. You should find that commits to files not associated with that stack do not trigger deploy.

@mbecker20 commented on GitHub (Aug 31, 2025): @shadybraden I think it would help you figure out what is going on if you break it down and make sure it works one stack at a time. For example, have webhook only trigger DeployStackIfChanged on a **single** Stack, and try committing to files that are associated with that stack, and ones that are not associated with that stack. You should find that commits to files not associated with that stack do not trigger deploy.
Author
Owner

@shadybraden commented on GitHub (Aug 31, 2025):

Hmm. I did a bunch more troubleshooting, and got nowhere. Then I stepped away and came back, and it began working exactly as expected...So perhaps it was a cache issue...time to take a backup

@shadybraden commented on GitHub (Aug 31, 2025): Hmm. I did a bunch more troubleshooting, and got nowhere. Then I stepped away and came back, and it began working exactly as expected...So perhaps it was a cache issue...time to take a backup
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/komodo#26