ts client 1.17.0

* resolver v3

add new ec2 instance types

clean up testing config

document the libraries a bit

clean up main

update sysinfo and otel

update client resolver 3.0

resolver v3 prog

clean up gitignore

implement periphery resolver v3

clean up

core read api v3

more prog

execute api

missing apis

compiling

1.16.13

work on more granular traits

prog on crud

* fmt

* format

* resource2 not really a benefit

* axum to 0.8

* bump aws deps

* just make it 1.17.0

* clean up cors

* the komodo env file should be highest priority over additional files

* add entities / message for test alerter

* test alert implementation

* rust 1.84.0

* axum update :param to {param} syntax

* fix last axum updates

* Add test alerter button

* higher quality / colored icons

* komodo-logo

* simplify network stats

* rename Test Alerter button

* escape incoming sync backslashes (BREAKING)

* clean up rust client websocket subscription

* finish oidc comment

* show update available stack table

* update available deployment table

* feature: use the repo path instead of name in GetLatestCommit (#282)

* Update repo path handling in commit fetching

- Changed `name` to `path` for repository identification.
- Updated cache update function to use the new path field.
- Improved error message for non-directory repo paths.

* feat: use optional name and path in GetLatestCommit

* review: don't use optional for name

* review: use helper

* review: remove redundant to_string()

* 1.17.0-dev

* feature: add post_deploy command (#288)

* feature: add post_deploy command

* review: do not run post_deploy if deploy failed

* feature: interpolate secrets in custom alerter (#289)

* feature: interpolate secrets in custom alerter

* fix rust warning

* review: sanitize errors

* review: sanitize error message

* Remove .git from remote_url (#299)

Remove .git from remote_url

Co-authored-by: Deon Marshall <dmarshall@ccp.com.au>

* mbecker20 -> moghtech

* remove example from cargo toml workspace

* dev-1

* fix login screen logo

* more legible favicon

* fix new compose images

* docs new organization

* typescript subscribe_to_update_websocket

* add donate button docsite

* add config save button in desktop sidebar navigator

* add save button to config bottom

* feature: allow docker image text to overflow in table (#301)

* feature: allow docker image text to overflow in table

* review: use break-words

* wip: revert line break in css file

* feature: update devcontainer node release

* improve First Login docs

* FIx PullStack re #302 and record docker compose config on stack deploy

* requery alerts more often

* improve update indicator style and also put on home screen

* Add all services stack log

* 1.17.0-dev-2

* fix api name chnage

* choose which stack services to include in logs

* feature: improve tables quick actions on mobile (#312)

* feature: improve tables quick actions on mobile

* review: fix gap4

* review: use flex-wrap

* improve pull to git init on existing folder without .git

* Fix unclear ComposePull log re #244

* use komodo_client.subscribe_to_update_websocket, and click indicator to reconnect

* dev-3

* ServerTemplate description

* improve WriteComposeContentsToHost instrument fields

* give server stat charts labels

* filters wrap

* show provider usernames from config file

* Stack: Fix git repo new compose file initialization

* init sync file new repo

* set branch on git init folder

* ResourceSync: pending view toggle between "Execute" vs "Commit" sync direction

* Improve resource sync Execute / Pending view selector

* standardize running commands with interpolation / output sanitizations

* fix all clippy lints

* fix rand

* lock certain users username / password, prevent demo creds from being changed.

* revert to login screen whenever the call to check login fails

* ResourceSync state resolution refinement

* make sure parent directories exist whenever writing files

* don't prune images if server not enabled

* update most deps

* update openidconnect dependency, and use reqwest rustls-tls-native-roots

* dev-4

* resource sync only add escaping on toml between the """

* Stacks executions take list of services -- Auto update only redeploys services with update

* auto update all service deploy option

* dev-5 fix the stack service executions

* clean up service_args

* rust 1.85

* store sync edits on localstorage

* stack edits on localstorage and show last deployed config

* add yarn install to runfile

* Fix actions when core on https

* add update_available query parameter to filter for only stacks /deployments with available update

* rust 2024 and fmt

* rename test.compose.yaml to dev.compose.yaml, and update runfile

* update .devcontainer / dev docs for updated runfile

* use png in topbar logo, svg quality sometimes bad

* OIDC: Support PKCE auth (secret optional)

* update docs on OIDC and client secret

* cycle the oidc client on interval to ensure up to date JWKs

* add KOMODO_LOCK_LOGIN_CREDENTIALS_FOR in config doc

* update deps

* resource sync toggle resource / variable / user group inclusion independantly

* use jsonwebtoken

* improve variable value table overflow

* colored tags

* fix sync summary count ok

* default new tag colors to grey

* soften tag opacity a bit

* Update config.tsx (#358)

* isolate stacks / deployments with pending updates

* update some deps

* use Tooltip component instead of HoverCard for mobile compatibility

* batch Build builds

* link to typescript client in the intro

* add link to main docs from client docs

* doc tweaks

* use moghtech/komodo-core and moghtech/komodo-periphery as images

* remove unnecessary explicit network

* periphery.compose.yaml

* clean up periphery compose

* add link to config

* update periphery container compose config

* rust 1.85.1

* update sync docs

* 1.17.0

---------

Co-authored-by: unsync <1211591+unsync@users.noreply.github.com>
Co-authored-by: Deon Marshall <dmarshall@ccp.com.au>
Co-authored-by: komodo <komodo@komo.do>
Co-authored-by: wlatic <jamesoh@gmail.com>

.devcontainer/dev.compose.yaml

@@ -23,7 +23,7 @@ services:
 
   db:
     extends:
-      file: ../test.compose.yaml
+      file: ../dev.compose.yaml
       service: ferretdb
 
 volumes:

Cargo.toml

@@ -8,8 +8,8 @@ members = [
 ]
 
 [workspace.package]
-version = "1.17.0-dev-4"
-edition = "2021"
+version = "1.17.0"
+edition = "2024"
 authors = ["mbecker20 <becker.maxh@gmail.com>"]
 license = "GPL-3.0-or-later"
 repository = "https://github.com/moghtech/komodo"
@@ -44,22 +44,23 @@ mungos = "3.2.0"
 svi = "1.0.1"
 
 # ASYNC
-reqwest = { version = "0.12.12", default-features = false, features = ["json", "rustls-tls-native-roots"] }
-tokio = { version = "1.43.0", features = ["full"] }
-tokio-util = "0.7.13"
+reqwest = { version = "0.12.15", default-features = false, features = ["json", "rustls-tls-native-roots"] }
+tokio = { version = "1.44.1", features = ["full"] }
+tokio-util = "0.7.14"
 futures = "0.3.31"
 futures-util = "0.3.31"
+arc-swap = "1.7.1"
 
 # SERVER
 axum-extra = { version = "0.10.0", features = ["typed-header"] }
 tower-http = { version = "0.6.2", features = ["fs", "cors"] }
-axum-server = { version = "0.7.1", features = ["tls-rustls"] }
+axum-server = { version = "0.7.2", features = ["tls-rustls"] }
 axum = { version = "0.8.1", features = ["ws", "json", "macros"] }
 tokio-tungstenite = "0.26.2"
 
 # SER/DE
 ordered_hash_map = { version = "0.4.0", features = ["serde"] }
-serde = { version = "1.0.218", features = ["derive"] }
+serde = { version = "1.0.219", features = ["derive"] }
 strum = { version = "0.27.1", features = ["derive"] }
 serde_json = "1.0.140"
 serde_yaml = "0.9.34"
@@ -70,31 +71,31 @@ anyhow = "1.0.97"
 thiserror = "2.0.12"
 
 # LOGGING
-opentelemetry-otlp = { version = "0.28.0", features = ["tls-roots", "reqwest-rustls"] }
-opentelemetry_sdk = { version = "0.28.0", features = ["rt-tokio"] }
+opentelemetry-otlp = { version = "0.29.0", features = ["tls-roots", "reqwest-rustls"] }
+opentelemetry_sdk = { version = "0.29.0", features = ["rt-tokio"] }
 tracing-subscriber = { version = "0.3.19", features = ["json"] }
-opentelemetry-semantic-conventions = "0.28.0"
-tracing-opentelemetry = "0.29.0"
-opentelemetry = "0.28.0"
+opentelemetry-semantic-conventions = "0.29.0"
+tracing-opentelemetry = "0.30.0"
+opentelemetry = "0.29.0"
 tracing = "0.1.41"
 
 # CONFIG
-clap = { version = "4.5.31", features = ["derive"] }
+clap = { version = "4.5.32", features = ["derive"] }
 dotenvy = "0.15.7"
 envy = "0.4.2"
 
 # CRYPTO / AUTH
-uuid = { version = "1.15.1", features = ["v4", "fast-rng", "serde"] }
+uuid = { version = "1.16.0", features = ["v4", "fast-rng", "serde"] }
+jsonwebtoken = { version = "9.3.1", default-features = false }
 openidconnect = "4.0.0"
 urlencoding = "2.1.3"
 nom_pem = "4.0.0"
 bcrypt = "0.17.0"
 base64 = "0.22.1"
-rustls = "0.23.23"
+rustls = "0.23.25"
 hmac = "0.12.1"
 sha2 = "0.10.8"
 rand = "0.9.0"
-jwt = "0.16.0"
 hex = "0.4.3"
 
 # SYSTEM
@@ -102,15 +103,16 @@ bollard = "0.18.1"
 sysinfo = "0.33.1"
 
 # CLOUD
-aws-config = "1.5.17"
-aws-sdk-ec2 = "1.114.0"
+aws-config = "1.6.0"
+aws-sdk-ec2 = "1.118.1"
+aws-credential-types = "1.2.2"
 
 # MISC
 derive_builder = "0.20.2"
 typeshare = "1.0.4"
-octorust = "0.9.0"
+octorust = "0.10.0"
 dashmap = "6.1.0"
 wildcard = "0.3.0"
 colored = "3.0.0"
 regex = "1.11.1"
-bson = "2.13.0"
+bson = "2.14.0"

bin/binaries.Dockerfile

@@ -1,7 +1,7 @@
 ## Builds the Komodo Core and Periphery binaries
 ## for a specific architecture.
 
-FROM rust:1.84.1-bullseye AS builder
+FROM rust:1.85.1-bullseye AS builder
 
 WORKDIR /builder
 COPY Cargo.toml Cargo.lock ./

bin/core/Cargo.toml

@@ -37,9 +37,10 @@ mungos.workspace = true
 slack.workspace = true
 svi.workspace = true
 # external
-axum-server.workspace = true
+aws-credential-types.workspace = true
 ordered_hash_map.workspace = true
 openidconnect.workspace = true
+axum-server.workspace = true
 urlencoding.workspace = true
 aws-sdk-ec2.workspace = true
 aws-config.workspace = true
@@ -51,6 +52,7 @@ serde_yaml.workspace = true
 typeshare.workspace = true
 octorust.workspace = true
 wildcard.workspace = true
+arc-swap.workspace = true
 dashmap.workspace = true
 tracing.workspace = true
 reqwest.workspace = true
@@ -71,5 +73,5 @@ envy.workspace = true
 rand.workspace = true
 hmac.workspace = true
 sha2.workspace = true
-jwt.workspace = true
+jsonwebtoken.workspace = true
 hex.workspace = true

bin/core/aio.Dockerfile

@@ -1,7 +1,7 @@
 ## All in one, multi stage compile + runtime Docker build for your architecture.
 
 # Build Core
-FROM rust:1.84.1-bullseye AS core-builder
+FROM rust:1.85.1-bullseye AS core-builder
 
 WORKDIR /builder
 COPY Cargo.toml Cargo.lock ./

bin/core/src/alert/discord.rs

@@ -94,7 +94,9 @@ pub async fn send_alert(
     } => {
       let link = resource_link(ResourceTargetVariant::Deployment, id);
       let to = fmt_docker_container_state(to);
-      format!("📦 Deployment **{name}** is now **{to}**\nserver: **{server_name}**\nprevious: **{from}**\n{link}")
+      format!(
+        "📦 Deployment **{name}** is now **{to}**\nserver: **{server_name}**\nprevious: **{from}**\n{link}"
+      )
     }
     AlertData::DeploymentImageUpdateAvailable {
       id,
@@ -104,7 +106,9 @@ pub async fn send_alert(
       image,
     } => {
       let link = resource_link(ResourceTargetVariant::Deployment, id);
-      format!("⬆ Deployment **{name}** has an update available\nserver: **{server_name}**\nimage: **{image}**\n{link}")
+      format!(
+        "⬆ Deployment **{name}** has an update available\nserver: **{server_name}**\nimage: **{image}**\n{link}"
+      )
     }
     AlertData::DeploymentAutoUpdated {
       id,
@@ -114,7 +118,9 @@ pub async fn send_alert(
       image,
     } => {
       let link = resource_link(ResourceTargetVariant::Deployment, id);
-      format!("⬆ Deployment **{name}** was updated automatically ⏫\nserver: **{server_name}**\nimage: **{image}**\n{link}")
+      format!(
+        "⬆ Deployment **{name}** was updated automatically ⏫\nserver: **{server_name}**\nimage: **{image}**\n{link}"
+      )
     }
     AlertData::StackStateChange {
       id,
@@ -126,7 +132,9 @@ pub async fn send_alert(
     } => {
       let link = resource_link(ResourceTargetVariant::Stack, id);
       let to = fmt_stack_state(to);
-      format!("🥞 Stack **{name}** is now {to}\nserver: **{server_name}**\nprevious: **{from}**\n{link}")
+      format!(
+        "🥞 Stack **{name}** is now {to}\nserver: **{server_name}**\nprevious: **{from}**\n{link}"
+      )
     }
     AlertData::StackImageUpdateAvailable {
       id,
@@ -137,7 +145,9 @@ pub async fn send_alert(
       image,
     } => {
       let link = resource_link(ResourceTargetVariant::Stack, id);
-      format!("⬆ Stack **{name}** has an update available\nserver: **{server_name}**\nservice: **{service}**\nimage: **{image}**\n{link}")
+      format!(
+        "⬆ Stack **{name}** has an update available\nserver: **{server_name}**\nservice: **{service}**\nimage: **{image}**\n{link}"
+      )
     }
     AlertData::StackAutoUpdated {
       id,
@@ -150,13 +160,17 @@ pub async fn send_alert(
       let images_label =
         if images.len() > 1 { "images" } else { "image" };
       let images = images.join(", ");
-      format!("⬆ Stack **{name}** was updated automatically ⏫\nserver: **{server_name}**\n{images_label}: **{images}**\n{link}")
+      format!(
+        "⬆ Stack **{name}** was updated automatically ⏫\nserver: **{server_name}**\n{images_label}: **{images}**\n{link}"
+      )
     }
     AlertData::AwsBuilderTerminationFailed {
       instance_id,
       message,
     } => {
-      format!("{level} | Failed to terminated AWS builder instance\ninstance id: **{instance_id}**\n{message}")
+      format!(
+        "{level} | Failed to terminated AWS builder instance\ninstance id: **{instance_id}**\n{message}"
+      )
     }
     AlertData::ResourceSyncPendingUpdates { id, name } => {
       let link =
@@ -167,7 +181,9 @@ pub async fn send_alert(
     }
     AlertData::BuildFailed { id, name, version } => {
       let link = resource_link(ResourceTargetVariant::Build, id);
-      format!("{level} | Build **{name}** failed\nversion: **v{version}**\n{link}")
+      format!(
+        "{level} | Build **{name}** failed\nversion: **v{version}**\n{link}"
+      )
     }
     AlertData::RepoBuildFailed { id, name } => {
       let link = resource_link(ResourceTargetVariant::Repo, id);

bin/core/src/alert/mod.rs

@@ -1,21 +1,21 @@
-use std::collections::HashSet;
 use ::slack::types::Block;
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use derive_variants::ExtractVariant;
 use futures::future::join_all;
 use komodo_client::entities::{
+  ResourceTargetVariant,
   alert::{Alert, AlertData, AlertDataVariant, SeverityLevel},
   alerter::*,
   deployment::DeploymentState,
   stack::StackState,
-  ResourceTargetVariant,
 };
 use mungos::{find::find_collect, mongodb::bson::doc};
+use std::collections::HashSet;
 use tracing::Instrument;
 
-use crate::{config::core_config, state::db_client};
 use crate::helpers::interpolate::interpolate_variables_secrets_into_string;
 use crate::helpers::query::get_variables_and_secrets;
+use crate::{config::core_config, state::db_client};
 
 mod discord;
 mod slack;
@@ -136,7 +136,6 @@ async fn send_custom_alert(
   url: &str,
   alert: &Alert,
 ) -> anyhow::Result<()> {
-
   let vars_and_secrets = get_variables_and_secrets().await?;
   let mut global_replacers = HashSet::new();
   let mut secret_replacers = HashSet::new();
@@ -156,9 +155,14 @@ async fn send_custom_alert(
     .send()
     .await
     .map_err(|e| {
-      let replacers = secret_replacers.into_iter().collect::<Vec<_>>();
-      let sanitized_error = svi::replace_in_string(&format!("{e:?}"), &replacers);
-      anyhow::Error::msg(format!("Error with request: {}", sanitized_error))
+      let replacers =
+        secret_replacers.into_iter().collect::<Vec<_>>();
+      let sanitized_error =
+        svi::replace_in_string(&format!("{e:?}"), &replacers);
+      anyhow::Error::msg(format!(
+        "Error with request: {}",
+        sanitized_error
+      ))
     })
     .context("failed at post request to alerter")?;
   let status = res.status();

bin/core/src/alert/slack.rs

@@ -73,7 +73,9 @@ pub async fn send_alert(
       let region = fmt_region(region);
       match alert.level {
         SeverityLevel::Ok => {
-          let text = format!("{level} | *{name}*{region} cpu usage at *{percentage:.1}%*");
+          let text = format!(
+            "{level} | *{name}*{region} cpu usage at *{percentage:.1}%*"
+          );
           let blocks = vec![
             Block::header(level),
             Block::section(format!(
@@ -87,7 +89,9 @@ pub async fn send_alert(
           (text, blocks.into())
         }
         _ => {
-          let text = format!("{level} | *{name}*{region} cpu usage at *{percentage:.1}%* 📈");
+          let text = format!(
+            "{level} | *{name}*{region} cpu usage at *{percentage:.1}%* 📈"
+          );
           let blocks = vec![
             Block::header(level),
             Block::section(format!(
@@ -113,7 +117,9 @@ pub async fn send_alert(
       let percentage = 100.0 * used_gb / total_gb;
       match alert.level {
         SeverityLevel::Ok => {
-          let text = format!("{level} | *{name}*{region} memory usage at *{percentage:.1}%* 💾");
+          let text = format!(
+            "{level} | *{name}*{region} memory usage at *{percentage:.1}%* 💾"
+          );
           let blocks = vec![
             Block::header(level),
             Block::section(format!(
@@ -130,7 +136,9 @@ pub async fn send_alert(
           (text, blocks.into())
         }
         _ => {
-          let text = format!("{level} | *{name}*{region} memory usage at *{percentage:.1}%* 💾");
+          let text = format!(
+            "{level} | *{name}*{region} memory usage at *{percentage:.1}%* 💾"
+          );
           let blocks = vec![
             Block::header(level),
             Block::section(format!(
@@ -160,7 +168,9 @@ pub async fn send_alert(
       let percentage = 100.0 * used_gb / total_gb;
       match alert.level {
         SeverityLevel::Ok => {
-          let text = format!("{level} | *{name}*{region} disk usage at *{percentage:.1}%* | mount point: *{path:?}* 💿");
+          let text = format!(
+            "{level} | *{name}*{region} disk usage at *{percentage:.1}%* | mount point: *{path:?}* 💿"
+          );
           let blocks = vec![
             Block::header(level),
             Block::section(format!(
@@ -169,12 +179,17 @@ pub async fn send_alert(
             Block::section(format!(
               "mount point: {path:?} | using *{used_gb:.1} GiB* / *{total_gb:.1} GiB*"
             )),
-            Block::section(resource_link(ResourceTargetVariant::Server, id)),
+            Block::section(resource_link(
+              ResourceTargetVariant::Server,
+              id,
+            )),
           ];
           (text, blocks.into())
         }
         _ => {
-          let text = format!("{level} | *{name}*{region} disk usage at *{percentage:.1}%* | mount point: *{path:?}* 💿");
+          let text = format!(
+            "{level} | *{name}*{region} disk usage at *{percentage:.1}%* | mount point: *{path:?}* 💿"
+          );
           let blocks = vec![
             Block::header(level),
             Block::section(format!(
@@ -183,7 +198,10 @@ pub async fn send_alert(
             Block::section(format!(
               "mount point: {path:?} | using *{used_gb:.1} GiB* / *{total_gb:.1} GiB*"
             )),
-            Block::section(resource_link(ResourceTargetVariant::Server, id)),
+            Block::section(resource_link(
+              ResourceTargetVariant::Server,
+              id,
+            )),
           ];
           (text, blocks.into())
         }

bin/core/src/api/auth.rs

@@ -1,6 +1,6 @@
 use std::{sync::OnceLock, time::Instant};
 
-use axum::{http::HeaderMap, routing::post, Router};
+use axum::{Router, http::HeaderMap, routing::post};
 use derive_variants::{EnumVariants, ExtractVariant};
 use komodo_client::{api::auth::*, entities::user::User};
 use resolver_api::Resolve;
@@ -105,8 +105,7 @@ fn login_options_reponse() -> &'static GetLoginOptionsResponse {
         && !config.google_oauth.secret.is_empty(),
       oidc: config.oidc_enabled
         && !config.oidc_provider.is_empty()
-        && !config.oidc_client_id.is_empty()
-        && !config.oidc_client_secret.is_empty(),
+        && !config.oidc_client_id.is_empty(),
       registration_disabled: config.disable_user_registration,
     }
   })

bin/core/src/api/execute/action.rs

@@ -120,11 +120,22 @@ impl Resolve<ExecuteArgs> for RunAction {
       format!("Failed to write action file to {path:?}")
     })?;
 
+    let CoreConfig { ssl_enabled, .. } = core_config();
+
+    let https_cert_flag = if *ssl_enabled {
+      " --unsafely-ignore-certificate-errors=localhost"
+    } else {
+      ""
+    };
+
     let mut res = run_komodo_command(
       // Keep this stage name as is, the UI will find the latest update log by matching the stage name
       "Execute Action",
       None,
-      format!("deno run --allow-all {}", path.display()),
+      format!(
+        "deno run --allow-all{https_cert_flag} {}",
+        path.display()
+      ),
     )
     .await;
 
@@ -306,8 +317,8 @@ fn delete_file(
         if name == file {
           if let Err(e) = fs::remove_file(entry.path()).await {
             warn!(
-            "Failed to clean up generated file after action execution | {e:#}"
-          );
+              "Failed to clean up generated file after action execution | {e:#}"
+            );
           };
           return true;
         }

bin/core/src/api/execute/build.rs

@@ -1,6 +1,6 @@
 use std::{collections::HashSet, future::IntoFuture, time::Duration};
 
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use formatting::format_serror;
 use futures::future::join_all;
 use komodo_client::{
@@ -560,7 +560,9 @@ impl Resolve<ExecuteArgs> for CancelBuild {
       )
       .await
       {
-        warn!("failed to set CancelBuild Update status Complete after timeout | {e:#}")
+        warn!(
+          "failed to set CancelBuild Update status Complete after timeout | {e:#}"
+        )
       }
     });
 

bin/core/src/api/execute/deployment.rs

@@ -1,21 +1,21 @@
 use std::{collections::HashSet, sync::OnceLock};
 
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use cache::TimeoutCache;
 use formatting::format_serror;
 use komodo_client::{
   api::execute::*,
   entities::{
+    Version,
     build::{Build, ImageRegistryConfig},
     deployment::{
-      extract_registry_domain, Deployment, DeploymentImage,
+      Deployment, DeploymentImage, extract_registry_domain,
     },
     get_image_name, komodo_timestamp, optional_string,
     permission::PermissionLevel,
     server::Server,
     update::{Log, Update},
     user::User,
-    Version,
   },
 };
 use periphery_client::api;

bin/core/src/api/execute/mod.rs

@@ -1,17 +1,17 @@
 use std::{pin::Pin, time::Instant};
 
 use anyhow::Context;
-use axum::{middleware, routing::post, Extension, Router};
-use axum_extra::{headers::ContentType, TypedHeader};
+use axum::{Extension, Router, middleware, routing::post};
+use axum_extra::{TypedHeader, headers::ContentType};
 use derive_variants::{EnumVariants, ExtractVariant};
 use formatting::format_serror;
 use futures::future::join_all;
 use komodo_client::{
   api::execute::*,
   entities::{
+    Operation,
     update::{Log, Update},
     user::User,
-    Operation,
   },
 };
 use mungos::by_id::find_one_by_id;
@@ -25,7 +25,7 @@ use uuid::Uuid;
 use crate::{
   auth::auth_request,
   helpers::update::{init_execution_update, update_update},
-  resource::{list_full_for_user_using_pattern, KomodoResource},
+  resource::{KomodoResource, list_full_for_user_using_pattern},
   state::db_client,
 };
 

bin/core/src/api/execute/procedure.rs

@@ -1,6 +1,6 @@
 use std::pin::Pin;
 
-use formatting::{bold, colored, format_serror, muted, Color};
+use formatting::{Color, bold, colored, format_serror, muted};
 use komodo_client::{
   api::execute::{
     BatchExecutionResponse, BatchRunProcedure, RunProcedure,

bin/core/src/api/execute/repo.rs

@@ -1,6 +1,6 @@
 use std::{collections::HashSet, future::IntoFuture, time::Duration};
 
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use formatting::format_serror;
 use komodo_client::{
   api::{execute::*, write::RefreshRepoCache},
@@ -695,7 +695,9 @@ impl Resolve<ExecuteArgs> for CancelRepoBuild {
       )
       .await
       {
-        warn!("failed to set CancelRepoBuild Update status Complete after timeout | {e:#}")
+        warn!(
+          "failed to set CancelRepoBuild Update status Complete after timeout | {e:#}"
+        )
       }
     });
 

bin/core/src/api/execute/server_template.rs

@@ -1,4 +1,4 @@
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use formatting::format_serror;
 use komodo_client::{
   api::{execute::LaunchServer, write::CreateServer},

bin/core/src/api/execute/stack.rs

@@ -89,7 +89,7 @@ impl Resolve<ExecuteArgs> for DeployStack {
 
     if !self.services.is_empty() {
       update.logs.push(Log::simple(
-        &format!("Service/s",),
+        "Service/s",
         format!(
           "Execution requested for Stack service/s {}",
           self.services.join(", ")
@@ -392,14 +392,15 @@ pub async fn pull_stack_inner(
   mut update: Option<&mut Update>,
 ) -> anyhow::Result<ComposePullResponse> {
   if let Some(update) = update.as_mut() {
-    if !services.is_empty() {}
-    update.logs.push(Log::simple(
-      &format!("Service/s"),
-      format!(
-        "Execution requested for Stack service/s {}",
-        services.join(", ")
-      ),
-    ))
+    if !services.is_empty() {
+      update.logs.push(Log::simple(
+        "Service/s",
+        format!(
+          "Execution requested for Stack service/s {}",
+          services.join(", ")
+        ),
+      ))
+    }
   }
 
   let git_token = crate::helpers::git_token(

bin/core/src/api/execute/sync.rs

@@ -1,11 +1,11 @@
 use std::{collections::HashMap, str::FromStr};
 
-use anyhow::{anyhow, Context};
-use formatting::{colored, format_serror, Color};
+use anyhow::{Context, anyhow};
+use formatting::{Color, colored, format_serror};
 use komodo_client::{
   api::{execute::RunSync, write::RefreshResourceSyncPending},
   entities::{
-    self,
+    self, ResourceTargetVariant,
     action::Action,
     alerter::Alerter,
     build::Build,
@@ -21,7 +21,6 @@ use komodo_client::{
     sync::ResourceSync,
     update::{Log, Update},
     user::sync_user,
-    ResourceTargetVariant,
   },
 };
 use mongo_indexed::doc;
@@ -34,12 +33,12 @@ use crate::{
   resource,
   state::{action_states, db_client},
   sync::{
-    deploy::{
-      build_deploy_cache, deploy_from_cache, SyncDeployParams,
-    },
-    execute::{get_updates_for_execution, ExecuteResourceSync},
-    remote::RemoteResources,
     AllResourcesById, ResourceSyncTrait,
+    deploy::{
+      SyncDeployParams, build_deploy_cache, deploy_from_cache,
+    },
+    execute::{ExecuteResourceSync, get_updates_for_execution},
+    remote::RemoteResources,
   },
 };
 
@@ -207,7 +206,7 @@ impl Resolve<ExecuteArgs> for RunSync {
 
     let delete = sync.config.managed || sync.config.delete;
 
-    let (servers_to_create, servers_to_update, servers_to_delete) =
+    let server_deltas = if sync.config.include_resources {
       get_updates_for_execution::<Server>(
         resources.servers,
         delete,
@@ -217,22 +216,11 @@ impl Resolve<ExecuteArgs> for RunSync {
         &id_to_tags,
         &sync.config.match_tags,
       )
-      .await?;
-    let (
-      deployments_to_create,
-      deployments_to_update,
-      deployments_to_delete,
-    ) = get_updates_for_execution::<Deployment>(
-      resources.deployments,
-      delete,
-      &all_resources,
-      match_resource_type,
-      match_resources.as_deref(),
-      &id_to_tags,
-      &sync.config.match_tags,
-    )
-    .await?;
-    let (stacks_to_create, stacks_to_update, stacks_to_delete) =
+      .await?
+    } else {
+      Default::default()
+    };
+    let stack_deltas = if sync.config.include_resources {
       get_updates_for_execution::<Stack>(
         resources.stacks,
         delete,
@@ -242,8 +230,25 @@ impl Resolve<ExecuteArgs> for RunSync {
         &id_to_tags,
         &sync.config.match_tags,
       )
-      .await?;
-    let (builds_to_create, builds_to_update, builds_to_delete) =
+      .await?
+    } else {
+      Default::default()
+    };
+    let deployment_deltas = if sync.config.include_resources {
+      get_updates_for_execution::<Deployment>(
+        resources.deployments,
+        delete,
+        &all_resources,
+        match_resource_type,
+        match_resources.as_deref(),
+        &id_to_tags,
+        &sync.config.match_tags,
+      )
+      .await?
+    } else {
+      Default::default()
+    };
+    let build_deltas = if sync.config.include_resources {
       get_updates_for_execution::<Build>(
         resources.builds,
         delete,
@@ -253,8 +258,11 @@ impl Resolve<ExecuteArgs> for RunSync {
         &id_to_tags,
         &sync.config.match_tags,
       )
-      .await?;
-    let (repos_to_create, repos_to_update, repos_to_delete) =
+      .await?
+    } else {
+      Default::default()
+    };
+    let repo_deltas = if sync.config.include_resources {
       get_updates_for_execution::<Repo>(
         resources.repos,
         delete,
@@ -264,22 +272,25 @@ impl Resolve<ExecuteArgs> for RunSync {
         &id_to_tags,
         &sync.config.match_tags,
       )
-      .await?;
-    let (
-      procedures_to_create,
-      procedures_to_update,
-      procedures_to_delete,
-    ) = get_updates_for_execution::<Procedure>(
-      resources.procedures,
-      delete,
-      &all_resources,
-      match_resource_type,
-      match_resources.as_deref(),
-      &id_to_tags,
-      &sync.config.match_tags,
-    )
-    .await?;
-    let (actions_to_create, actions_to_update, actions_to_delete) =
+      .await?
+    } else {
+      Default::default()
+    };
+    let procedure_deltas = if sync.config.include_resources {
+      get_updates_for_execution::<Procedure>(
+        resources.procedures,
+        delete,
+        &all_resources,
+        match_resource_type,
+        match_resources.as_deref(),
+        &id_to_tags,
+        &sync.config.match_tags,
+      )
+      .await?
+    } else {
+      Default::default()
+    };
+    let action_deltas = if sync.config.include_resources {
       get_updates_for_execution::<Action>(
         resources.actions,
         delete,
@@ -289,8 +300,11 @@ impl Resolve<ExecuteArgs> for RunSync {
         &id_to_tags,
         &sync.config.match_tags,
       )
-      .await?;
-    let (builders_to_create, builders_to_update, builders_to_delete) =
+      .await?
+    } else {
+      Default::default()
+    };
+    let builder_deltas = if sync.config.include_resources {
       get_updates_for_execution::<Builder>(
         resources.builders,
         delete,
@@ -300,8 +314,11 @@ impl Resolve<ExecuteArgs> for RunSync {
         &id_to_tags,
         &sync.config.match_tags,
       )
-      .await?;
-    let (alerters_to_create, alerters_to_update, alerters_to_delete) =
+      .await?
+    } else {
+      Default::default()
+    };
+    let alerter_deltas = if sync.config.include_resources {
       get_updates_for_execution::<Alerter>(
         resources.alerters,
         delete,
@@ -311,35 +328,38 @@ impl Resolve<ExecuteArgs> for RunSync {
         &id_to_tags,
         &sync.config.match_tags,
       )
-      .await?;
-    let (
-      server_templates_to_create,
-      server_templates_to_update,
-      server_templates_to_delete,
-    ) = get_updates_for_execution::<ServerTemplate>(
-      resources.server_templates,
-      delete,
-      &all_resources,
-      match_resource_type,
-      match_resources.as_deref(),
-      &id_to_tags,
-      &sync.config.match_tags,
-    )
-    .await?;
-    let (
-      resource_syncs_to_create,
-      resource_syncs_to_update,
-      resource_syncs_to_delete,
-    ) = get_updates_for_execution::<entities::sync::ResourceSync>(
-      resources.resource_syncs,
-      delete,
-      &all_resources,
-      match_resource_type,
-      match_resources.as_deref(),
-      &id_to_tags,
-      &sync.config.match_tags,
-    )
-    .await?;
+      .await?
+    } else {
+      Default::default()
+    };
+    let server_template_deltas = if sync.config.include_resources {
+      get_updates_for_execution::<ServerTemplate>(
+        resources.server_templates,
+        delete,
+        &all_resources,
+        match_resource_type,
+        match_resources.as_deref(),
+        &id_to_tags,
+        &sync.config.match_tags,
+      )
+      .await?
+    } else {
+      Default::default()
+    };
+    let resource_sync_deltas = if sync.config.include_resources {
+      get_updates_for_execution::<entities::sync::ResourceSync>(
+        resources.resource_syncs,
+        delete,
+        &all_resources,
+        match_resource_type,
+        match_resources.as_deref(),
+        &id_to_tags,
+        &sync.config.match_tags,
+      )
+      .await?
+    } else {
+      Default::default()
+    };
 
     let (
       variables_to_create,
@@ -347,12 +367,11 @@ impl Resolve<ExecuteArgs> for RunSync {
       variables_to_delete,
     ) = if match_resource_type.is_none()
       && match_resources.is_none()
-      && sync.config.match_tags.is_empty()
+      && sync.config.include_variables
     {
       crate::sync::variables::get_updates_for_execution(
         resources.variables,
-        // Delete doesn't work with variables when match tags are set
-        sync.config.match_tags.is_empty() && delete,
+        delete,
       )
       .await?
     } else {
@@ -364,12 +383,11 @@ impl Resolve<ExecuteArgs> for RunSync {
       user_groups_to_delete,
     ) = if match_resource_type.is_none()
       && match_resources.is_none()
-      && sync.config.match_tags.is_empty()
+      && sync.config.include_user_groups
     {
       crate::sync::user_groups::get_updates_for_execution(
         resources.user_groups,
-        // Delete doesn't work with user groups when match tags are set
-        sync.config.match_tags.is_empty() && delete,
+        delete,
         &all_resources,
       )
       .await?
@@ -378,39 +396,17 @@ impl Resolve<ExecuteArgs> for RunSync {
     };
 
     if deploy_cache.is_empty()
-      && resource_syncs_to_create.is_empty()
-      && resource_syncs_to_update.is_empty()
-      && resource_syncs_to_delete.is_empty()
-      && server_templates_to_create.is_empty()
-      && server_templates_to_update.is_empty()
-      && server_templates_to_delete.is_empty()
-      && servers_to_create.is_empty()
-      && servers_to_update.is_empty()
-      && servers_to_delete.is_empty()
-      && deployments_to_create.is_empty()
-      && deployments_to_update.is_empty()
-      && deployments_to_delete.is_empty()
-      && stacks_to_create.is_empty()
-      && stacks_to_update.is_empty()
-      && stacks_to_delete.is_empty()
-      && builds_to_create.is_empty()
-      && builds_to_update.is_empty()
-      && builds_to_delete.is_empty()
-      && builders_to_create.is_empty()
-      && builders_to_update.is_empty()
-      && builders_to_delete.is_empty()
-      && alerters_to_create.is_empty()
-      && alerters_to_update.is_empty()
-      && alerters_to_delete.is_empty()
-      && repos_to_create.is_empty()
-      && repos_to_update.is_empty()
-      && repos_to_delete.is_empty()
-      && procedures_to_create.is_empty()
-      && procedures_to_update.is_empty()
-      && procedures_to_delete.is_empty()
-      && actions_to_create.is_empty()
-      && actions_to_update.is_empty()
-      && actions_to_delete.is_empty()
+      && resource_sync_deltas.no_changes()
+      && server_template_deltas.no_changes()
+      && server_deltas.no_changes()
+      && deployment_deltas.no_changes()
+      && stack_deltas.no_changes()
+      && build_deltas.no_changes()
+      && builder_deltas.no_changes()
+      && alerter_deltas.no_changes()
+      && repo_deltas.no_changes()
+      && procedure_deltas.no_changes()
+      && action_deltas.no_changes()
       && user_groups_to_create.is_empty()
       && user_groups_to_update.is_empty()
       && user_groups_to_delete.is_empty()
@@ -453,111 +449,57 @@ impl Resolve<ExecuteArgs> for RunSync {
     );
     maybe_extend(
       &mut update.logs,
-      ResourceSync::execute_sync_updates(
-        resource_syncs_to_create,
-        resource_syncs_to_update,
-        resource_syncs_to_delete,
-      )
-      .await,
+      ResourceSync::execute_sync_updates(resource_sync_deltas).await,
     );
     maybe_extend(
       &mut update.logs,
-      ServerTemplate::execute_sync_updates(
-        server_templates_to_create,
-        server_templates_to_update,
-        server_templates_to_delete,
-      )
-      .await,
+      ServerTemplate::execute_sync_updates(server_template_deltas)
+        .await,
     );
     maybe_extend(
       &mut update.logs,
-      Server::execute_sync_updates(
-        servers_to_create,
-        servers_to_update,
-        servers_to_delete,
-      )
-      .await,
+      Server::execute_sync_updates(server_deltas).await,
     );
     maybe_extend(
       &mut update.logs,
-      Alerter::execute_sync_updates(
-        alerters_to_create,
-        alerters_to_update,
-        alerters_to_delete,
-      )
-      .await,
+      Alerter::execute_sync_updates(alerter_deltas).await,
     );
     maybe_extend(
       &mut update.logs,
-      Action::execute_sync_updates(
-        actions_to_create,
-        actions_to_update,
-        actions_to_delete,
-      )
-      .await,
+      Action::execute_sync_updates(action_deltas).await,
     );
 
     // Dependent on server
     maybe_extend(
       &mut update.logs,
-      Builder::execute_sync_updates(
-        builders_to_create,
-        builders_to_update,
-        builders_to_delete,
-      )
-      .await,
+      Builder::execute_sync_updates(builder_deltas).await,
     );
     maybe_extend(
       &mut update.logs,
-      Repo::execute_sync_updates(
-        repos_to_create,
-        repos_to_update,
-        repos_to_delete,
-      )
-      .await,
+      Repo::execute_sync_updates(repo_deltas).await,
     );
 
     // Dependant on builder
     maybe_extend(
       &mut update.logs,
-      Build::execute_sync_updates(
-        builds_to_create,
-        builds_to_update,
-        builds_to_delete,
-      )
-      .await,
+      Build::execute_sync_updates(build_deltas).await,
     );
 
     // Dependant on server / build
     maybe_extend(
       &mut update.logs,
-      Deployment::execute_sync_updates(
-        deployments_to_create,
-        deployments_to_update,
-        deployments_to_delete,
-      )
-      .await,
+      Deployment::execute_sync_updates(deployment_deltas).await,
     );
     // stack only depends on server, but maybe will depend on build later.
     maybe_extend(
       &mut update.logs,
-      Stack::execute_sync_updates(
-        stacks_to_create,
-        stacks_to_update,
-        stacks_to_delete,
-      )
-      .await,
+      Stack::execute_sync_updates(stack_deltas).await,
     );
 
     // Dependant on everything
     maybe_extend(
       &mut update.logs,
-      Procedure::execute_sync_updates(
-        procedures_to_create,
-        procedures_to_update,
-        procedures_to_delete,
-      )
-      .await,
+      Procedure::execute_sync_updates(procedure_deltas).await,
     );
 
     // Execute the deploy cache

bin/core/src/api/read/build.rs

@@ -6,11 +6,11 @@ use futures::TryStreamExt;
 use komodo_client::{
   api::read::*,
   entities::{
+    Operation,
     build::{Build, BuildActionState, BuildListItem, BuildState},
     config::core::CoreConfig,
     permission::PermissionLevel,
     update::UpdateStatus,
-    Operation,
   },
 };
 use mungos::{

bin/core/src/api/read/deployment.rs

@@ -1,6 +1,6 @@
 use std::{cmp, collections::HashSet};
 
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use komodo_client::{
   api::read::*,
   entities::{
@@ -51,12 +51,20 @@ impl Resolve<ReadArgs> for ListDeployments {
     } else {
       get_all_tags(None).await?
     };
-    Ok(
-      resource::list_for_user::<Deployment>(
-        self.query, user, &all_tags,
-      )
-      .await?,
+    let only_update_available = self.query.specific.update_available;
+    let deployments = resource::list_for_user::<Deployment>(
+      self.query, user, &all_tags,
     )
+    .await?;
+    let deployments = if only_update_available {
+      deployments
+        .into_iter()
+        .filter(|deployment| deployment.info.update_available)
+        .collect()
+    } else {
+      deployments
+    };
+    Ok(deployments)
   }
 }
 

bin/core/src/api/read/mod.rs

@@ -1,10 +1,11 @@
 use std::{collections::HashSet, sync::OnceLock, time::Instant};
 
-use anyhow::{anyhow, Context};
-use axum::{middleware, routing::post, Extension, Router};
+use anyhow::{Context, anyhow};
+use axum::{Extension, Router, middleware, routing::post};
 use komodo_client::{
   api::read::*,
   entities::{
+    ResourceTarget,
     build::Build,
     builder::{Builder, BuilderConfig},
     config::{DockerRegistry, GitProvider},
@@ -12,7 +13,6 @@ use komodo_client::{
     server::Server,
     sync::ResourceSync,
     user::User,
-    ResourceTarget,
   },
 };
 use resolver_api::Resolve;
@@ -315,7 +315,7 @@ impl Resolve<ReadArgs> for ListSecrets {
         _ => {
           return Err(
             anyhow!("target must be `Server` or `Builder`").into(),
-          )
+          );
         }
       };
       if let Some(id) = server_id {
@@ -373,7 +373,7 @@ impl Resolve<ReadArgs> for ListGitProvidersFromConfig {
         _ => {
           return Err(
             anyhow!("target must be `Server` or `Builder`").into(),
-          )
+          );
         }
       }
     }
@@ -473,7 +473,7 @@ impl Resolve<ReadArgs> for ListDockerRegistriesFromConfig {
         _ => {
           return Err(
             anyhow!("target must be `Server` or `Builder`").into(),
-          )
+          );
         }
       }
     }

bin/core/src/api/read/permission.rs

@@ -1,4 +1,4 @@
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use komodo_client::{
   api::read::{
     GetPermissionLevel, GetPermissionLevelResponse, ListPermissions,

bin/core/src/api/read/provider.rs

@@ -1,6 +1,6 @@
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use komodo_client::api::read::*;
-use mongo_indexed::{doc, Document};
+use mongo_indexed::{Document, doc};
 use mungos::{
   by_id::find_one_by_id, find::find_collect,
   mongodb::options::FindOptions,

bin/core/src/api/read/server.rs

@@ -4,13 +4,14 @@ use std::{
   sync::{Arc, OnceLock},
 };
 
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use async_timing_util::{
-  get_timelength_in_ms, unix_timestamp_ms, FIFTEEN_SECONDS_MS,
+  FIFTEEN_SECONDS_MS, get_timelength_in_ms, unix_timestamp_ms,
 };
 use komodo_client::{
   api::read::*,
   entities::{
+    ResourceTarget,
     deployment::Deployment,
     docker::{
       container::{Container, ContainerListItem},
@@ -25,7 +26,6 @@ use komodo_client::{
     stack::{Stack, StackServiceNames},
     stats::{SystemInformation, SystemProcess},
     update::Log,
-    ResourceTarget,
   },
 };
 use mungos::{

bin/core/src/api/read/stack.rs

@@ -194,10 +194,25 @@ impl Resolve<ReadArgs> for ListStacks {
     } else {
       get_all_tags(None).await?
     };
-    Ok(
+    let only_update_available = self.query.specific.update_available;
+    let stacks =
       resource::list_for_user::<Stack>(self.query, user, &all_tags)
-        .await?,
-    )
+        .await?;
+    let stacks = if only_update_available {
+      stacks
+        .into_iter()
+        .filter(|stack| {
+          stack
+            .info
+            .services
+            .iter()
+            .any(|service| service.update_available)
+        })
+        .collect()
+    } else {
+      stacks
+    };
+    Ok(stacks)
   }
 }
 

bin/core/src/api/read/sync.rs

@@ -138,7 +138,9 @@ impl Resolve<ReadArgs> for GetResourceSyncsSummary {
         .syncing
       {
         res.syncing += 1;
+        continue;
       }
+      res.ok += 1;
     }
 
     Ok(res)

bin/core/src/api/read/toml.rs

@@ -6,11 +6,12 @@ use komodo_client::{
     ListUserGroups,
   },
   entities::{
-    action::Action, alerter::Alerter, build::Build, builder::Builder,
-    deployment::Deployment, permission::PermissionLevel,
-    procedure::Procedure, repo::Repo, resource::ResourceQuery,
-    server::Server, server_template::ServerTemplate, stack::Stack,
-    sync::ResourceSync, toml::ResourcesToml, ResourceTarget,
+    ResourceTarget, action::Action, alerter::Alerter, build::Build,
+    builder::Builder, deployment::Deployment,
+    permission::PermissionLevel, procedure::Procedure, repo::Repo,
+    resource::ResourceQuery, server::Server,
+    server_template::ServerTemplate, stack::Stack,
+    sync::ResourceSync, toml::ResourcesToml, user::User,
   },
 };
 use mungos::find::find_collect;
@@ -23,156 +24,168 @@ use crate::{
   resource,
   state::db_client,
   sync::{
-    toml::{convert_resource, ToToml, TOML_PRETTY_OPTIONS},
-    user_groups::convert_user_groups,
     AllResourcesById,
+    toml::{TOML_PRETTY_OPTIONS, ToToml, convert_resource},
+    user_groups::convert_user_groups,
   },
 };
 
 use super::ReadArgs;
 
+async fn get_all_targets(
+  tags: &[String],
+  user: &User,
+) -> anyhow::Result<Vec<ResourceTarget>> {
+  let mut targets = Vec::<ResourceTarget>::new();
+  let all_tags = if tags.is_empty() {
+    vec![]
+  } else {
+    get_all_tags(None).await?
+  };
+  targets.extend(
+    resource::list_for_user::<Alerter>(
+      ResourceQuery::builder().tags(tags).build(),
+      user,
+      &all_tags,
+    )
+    .await?
+    .into_iter()
+    .map(|resource| ResourceTarget::Alerter(resource.id)),
+  );
+  targets.extend(
+    resource::list_for_user::<Builder>(
+      ResourceQuery::builder().tags(tags).build(),
+      user,
+      &all_tags,
+    )
+    .await?
+    .into_iter()
+    .map(|resource| ResourceTarget::Builder(resource.id)),
+  );
+  targets.extend(
+    resource::list_for_user::<Server>(
+      ResourceQuery::builder().tags(tags).build(),
+      user,
+      &all_tags,
+    )
+    .await?
+    .into_iter()
+    .map(|resource| ResourceTarget::Server(resource.id)),
+  );
+  targets.extend(
+    resource::list_for_user::<Stack>(
+      ResourceQuery::builder().tags(tags).build(),
+      user,
+      &all_tags,
+    )
+    .await?
+    .into_iter()
+    .map(|resource| ResourceTarget::Stack(resource.id)),
+  );
+  targets.extend(
+    resource::list_for_user::<Deployment>(
+      ResourceQuery::builder().tags(tags).build(),
+      user,
+      &all_tags,
+    )
+    .await?
+    .into_iter()
+    .map(|resource| ResourceTarget::Deployment(resource.id)),
+  );
+  targets.extend(
+    resource::list_for_user::<Build>(
+      ResourceQuery::builder().tags(tags).build(),
+      user,
+      &all_tags,
+    )
+    .await?
+    .into_iter()
+    .map(|resource| ResourceTarget::Build(resource.id)),
+  );
+  targets.extend(
+    resource::list_for_user::<Repo>(
+      ResourceQuery::builder().tags(tags).build(),
+      user,
+      &all_tags,
+    )
+    .await?
+    .into_iter()
+    .map(|resource| ResourceTarget::Repo(resource.id)),
+  );
+  targets.extend(
+    resource::list_for_user::<Procedure>(
+      ResourceQuery::builder().tags(tags).build(),
+      user,
+      &all_tags,
+    )
+    .await?
+    .into_iter()
+    .map(|resource| ResourceTarget::Procedure(resource.id)),
+  );
+  targets.extend(
+    resource::list_for_user::<Action>(
+      ResourceQuery::builder().tags(tags).build(),
+      user,
+      &all_tags,
+    )
+    .await?
+    .into_iter()
+    .map(|resource| ResourceTarget::Action(resource.id)),
+  );
+  targets.extend(
+    resource::list_for_user::<ServerTemplate>(
+      ResourceQuery::builder().tags(tags).build(),
+      user,
+      &all_tags,
+    )
+    .await?
+    .into_iter()
+    .map(|resource| ResourceTarget::ServerTemplate(resource.id)),
+  );
+  targets.extend(
+    resource::list_full_for_user::<ResourceSync>(
+      ResourceQuery::builder().tags(tags).build(),
+      user,
+      &all_tags,
+    )
+    .await?
+    .into_iter()
+    // These will already be filtered by [ExportResourcesToToml]
+    .map(|resource| ResourceTarget::ResourceSync(resource.id)),
+  );
+  Ok(targets)
+}
+
 impl Resolve<ReadArgs> for ExportAllResourcesToToml {
   async fn resolve(
     self,
     args: &ReadArgs,
   ) -> serror::Result<ExportAllResourcesToTomlResponse> {
-    let mut targets = Vec::<ResourceTarget>::new();
-
-    let all_tags = if self.tags.is_empty() {
-      vec![]
+    let targets = if self.include_resources {
+      get_all_targets(&self.tags, &args.user).await?
     } else {
-      get_all_tags(None).await?
+      Vec::new()
     };
 
-    let ReadArgs { user } = args;
-
-    targets.extend(
-      resource::list_for_user::<Alerter>(
-        ResourceQuery::builder().tags(self.tags.clone()).build(),
-        user,
-        &all_tags,
-      )
-      .await?
-      .into_iter()
-      .map(|resource| ResourceTarget::Alerter(resource.id)),
-    );
-    targets.extend(
-      resource::list_for_user::<Builder>(
-        ResourceQuery::builder().tags(self.tags.clone()).build(),
-        user,
-        &all_tags,
-      )
-      .await?
-      .into_iter()
-      .map(|resource| ResourceTarget::Builder(resource.id)),
-    );
-    targets.extend(
-      resource::list_for_user::<Server>(
-        ResourceQuery::builder().tags(self.tags.clone()).build(),
-        user,
-        &all_tags,
-      )
-      .await?
-      .into_iter()
-      .map(|resource| ResourceTarget::Server(resource.id)),
-    );
-    targets.extend(
-      resource::list_for_user::<Stack>(
-        ResourceQuery::builder().tags(self.tags.clone()).build(),
-        user,
-        &all_tags,
-      )
-      .await?
-      .into_iter()
-      .map(|resource| ResourceTarget::Stack(resource.id)),
-    );
-    targets.extend(
-      resource::list_for_user::<Deployment>(
-        ResourceQuery::builder().tags(self.tags.clone()).build(),
-        user,
-        &all_tags,
-      )
-      .await?
-      .into_iter()
-      .map(|resource| ResourceTarget::Deployment(resource.id)),
-    );
-    targets.extend(
-      resource::list_for_user::<Build>(
-        ResourceQuery::builder().tags(self.tags.clone()).build(),
-        user,
-        &all_tags,
-      )
-      .await?
-      .into_iter()
-      .map(|resource| ResourceTarget::Build(resource.id)),
-    );
-    targets.extend(
-      resource::list_for_user::<Repo>(
-        ResourceQuery::builder().tags(self.tags.clone()).build(),
-        user,
-        &all_tags,
-      )
-      .await?
-      .into_iter()
-      .map(|resource| ResourceTarget::Repo(resource.id)),
-    );
-    targets.extend(
-      resource::list_for_user::<Procedure>(
-        ResourceQuery::builder().tags(self.tags.clone()).build(),
-        user,
-        &all_tags,
-      )
-      .await?
-      .into_iter()
-      .map(|resource| ResourceTarget::Procedure(resource.id)),
-    );
-    targets.extend(
-      resource::list_for_user::<Action>(
-        ResourceQuery::builder().tags(self.tags.clone()).build(),
-        user,
-        &all_tags,
-      )
-      .await?
-      .into_iter()
-      .map(|resource| ResourceTarget::Action(resource.id)),
-    );
-    targets.extend(
-      resource::list_for_user::<ServerTemplate>(
-        ResourceQuery::builder().tags(self.tags.clone()).build(),
-        user,
-        &all_tags,
-      )
-      .await?
-      .into_iter()
-      .map(|resource| ResourceTarget::ServerTemplate(resource.id)),
-    );
-    targets.extend(
-      resource::list_full_for_user::<ResourceSync>(
-        ResourceQuery::builder().tags(self.tags.clone()).build(),
-        user,
-        &all_tags,
-      )
-      .await?
-      .into_iter()
-      // These will already be filtered by [ExportResourcesToToml]
-      .map(|resource| ResourceTarget::ResourceSync(resource.id)),
-    );
-
-    let user_groups = if user.admin && self.tags.is_empty() {
-      find_collect(&db_client().user_groups, None, None)
-        .await
-        .context("failed to query db for user groups")?
-        .into_iter()
-        .map(|user_group| user_group.id)
-        .collect()
+    let user_groups = if self.include_user_groups {
+      if args.user.admin {
+        find_collect(&db_client().user_groups, None, None)
+          .await
+          .context("failed to query db for user groups")?
+          .into_iter()
+          .map(|user_group| user_group.id)
+          .collect()
+      } else {
+        get_user_user_group_ids(&args.user.id).await?
+      }
     } else {
-      get_user_user_group_ids(&user.id).await?
+      Vec::new()
     };
 
     ExportResourcesToToml {
       targets,
       user_groups,
-      include_variables: self.tags.is_empty(),
+      include_variables: self.include_variables,
     }
     .resolve(args)
     .await

bin/core/src/api/read/update.rs

@@ -1,9 +1,10 @@
 use std::collections::HashMap;
 
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use komodo_client::{
   api::read::{GetUpdate, ListUpdates, ListUpdatesResponse},
   entities::{
+    ResourceTarget,
     action::Action,
     alerter::Alerter,
     build::Build,
@@ -18,7 +19,6 @@ use komodo_client::{
     sync::ResourceSync,
     update::{Update, UpdateListItem},
     user::User,
-    ResourceTarget,
   },
 };
 use mungos::{
@@ -242,7 +242,7 @@ impl Resolve<ReadArgs> for GetUpdate {
       ResourceTarget::System(_) => {
         return Err(
           anyhow!("user must be admin to view system updates").into(),
-        )
+        );
       }
       ResourceTarget::Server(id) => {
         resource::get_check_permissions::<Server>(

bin/core/src/api/read/user.rs

@@ -1,4 +1,4 @@
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use komodo_client::{
   api::read::{
     FindUser, FindUserResponse, GetUsername, GetUsernameResponse,
@@ -6,7 +6,7 @@ use komodo_client::{
     ListApiKeysForServiceUserResponse, ListApiKeysResponse,
     ListUsers, ListUsersResponse,
   },
-  entities::user::{admin_service_user, UserConfig},
+  entities::user::{UserConfig, admin_service_user},
 };
 use mungos::{
   by_id::find_one_by_id,

bin/core/src/api/read/user_group.rs

@@ -5,7 +5,7 @@ use komodo_client::api::read::*;
 use mungos::{
   find::find_collect,
   mongodb::{
-    bson::{doc, oid::ObjectId, Document},
+    bson::{Document, doc, oid::ObjectId},
     options::FindOptions,
   },
 };

bin/core/src/api/user.rs

@@ -1,7 +1,7 @@
 use std::{collections::VecDeque, time::Instant};
 
-use anyhow::{anyhow, Context};
-use axum::{middleware, routing::post, Extension, Json, Router};
+use anyhow::{Context, anyhow};
+use axum::{Extension, Json, Router, middleware, routing::post};
 use derive_variants::EnumVariants;
 use komodo_client::{
   api::user::*,

bin/core/src/api/write/build.rs

@@ -1,13 +1,13 @@
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use git::GitRes;
 use komodo_client::{
   api::write::*,
   entities::{
+    CloneArgs, NoData,
     build::{Build, BuildInfo, PartialBuildConfig},
     config::core::CoreConfig,
     permission::PermissionLevel,
     update::Update,
-    CloneArgs, NoData,
   },
 };
 use mongo_indexed::doc;

bin/core/src/api/write/builder.rs

@@ -72,9 +72,6 @@ impl Resolve<WriteArgs> for RenameBuilder {
     self,
     WriteArgs { user }: &WriteArgs,
   ) -> serror::Result<Update> {
-    Ok(
-      resource::rename::<Builder>(&self.id, &self.name, user)
-        .await?,
-    )
+    Ok(resource::rename::<Builder>(&self.id, &self.name, user).await?)
   }
 }

bin/core/src/api/write/deployment.rs

@@ -1,7 +1,8 @@
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use komodo_client::{
   api::write::*,
   entities::{
+    Operation,
     deployment::{
       Deployment, DeploymentImage, DeploymentState,
       PartialDeploymentConfig, RestartMode,
@@ -12,7 +13,6 @@ use komodo_client::{
     server::{Server, ServerState},
     to_komodo_name,
     update::Update,
-    Operation,
   },
 };
 use mungos::{by_id::update_one_by_id, mongodb::bson::doc};
@@ -58,12 +58,8 @@ impl Resolve<WriteArgs> for CopyDeployment {
       )
       .await?;
     Ok(
-      resource::create::<Deployment>(
-        &self.name,
-        config.into(),
-        user,
-      )
-      .await?,
+      resource::create::<Deployment>(&self.name, config.into(), user)
+        .await?,
     )
   }
 }

bin/core/src/api/write/description.rs

@@ -2,10 +2,10 @@ use anyhow::anyhow;
 use komodo_client::{
   api::write::{UpdateDescription, UpdateDescriptionResponse},
   entities::{
-    action::Action, alerter::Alerter, build::Build, builder::Builder,
-    deployment::Deployment, procedure::Procedure, repo::Repo,
-    server::Server, server_template::ServerTemplate, stack::Stack,
-    sync::ResourceSync, ResourceTarget,
+    ResourceTarget, action::Action, alerter::Alerter, build::Build,
+    builder::Builder, deployment::Deployment, procedure::Procedure,
+    repo::Repo, server::Server, server_template::ServerTemplate,
+    stack::Stack, sync::ResourceSync,
   },
 };
 use resolver_api::Resolve;
@@ -27,7 +27,7 @@ impl Resolve<WriteArgs> for UpdateDescription {
             "cannot update description of System resource target"
           )
           .into(),
-        )
+        );
       }
       ResourceTarget::Server(id) => {
         resource::update_description::<Server>(

bin/core/src/api/write/mod.rs

@@ -1,7 +1,7 @@
 use std::time::Instant;
 
 use anyhow::Context;
-use axum::{middleware, routing::post, Extension, Router};
+use axum::{Extension, Router, middleware, routing::post};
 use derive_variants::{EnumVariants, ExtractVariant};
 use komodo_client::{api::write::*, entities::user::User};
 use resolver_api::Resolve;
@@ -172,6 +172,7 @@ pub enum WriteRequest {
   CreateTag(CreateTag),
   DeleteTag(DeleteTag),
   RenameTag(RenameTag),
+  UpdateTagColor(UpdateTagColor),
   UpdateTagsOnResource(UpdateTagsOnResource),
 
   // ==== VARIABLE ====

bin/core/src/api/write/permissions.rs

@@ -1,17 +1,17 @@
 use std::str::FromStr;
 
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use komodo_client::{
   api::write::*,
   entities::{
-    permission::{UserTarget, UserTargetVariant},
     ResourceTarget, ResourceTargetVariant,
+    permission::{UserTarget, UserTargetVariant},
   },
 };
 use mungos::{
   by_id::{find_one_by_id, update_one_by_id},
   mongodb::{
-    bson::{doc, oid::ObjectId, Document},
+    bson::{Document, doc, oid::ObjectId},
     options::UpdateOptions,
   },
 };

bin/core/src/api/write/provider.rs

@@ -1,9 +1,9 @@
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use komodo_client::{
   api::write::*,
   entities::{
-    provider::{DockerRegistryAccount, GitProviderAccount},
     Operation, ResourceTarget,
+    provider::{DockerRegistryAccount, GitProviderAccount},
   },
 };
 use mungos::{

bin/core/src/api/write/repo.rs

@@ -1,9 +1,10 @@
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use formatting::format_serror;
 use git::GitRes;
 use komodo_client::{
   api::write::*,
   entities::{
+    CloneArgs, NoData, Operation,
     config::core::CoreConfig,
     komodo_timestamp,
     permission::PermissionLevel,
@@ -11,7 +12,6 @@ use komodo_client::{
     server::Server,
     to_komodo_name,
     update::{Log, Update},
-    CloneArgs, NoData, Operation,
   },
 };
 use mongo_indexed::doc;

bin/core/src/api/write/server.rs

@@ -2,10 +2,10 @@ use formatting::format_serror;
 use komodo_client::{
   api::write::*,
   entities::{
+    Operation,
     permission::PermissionLevel,
     server::Server,
     update::{Update, UpdateStatus},
-    Operation,
   },
 };
 use periphery_client::api;

bin/core/src/api/write/service_user.rs

@@ -1,6 +1,6 @@
 use std::str::FromStr;
 
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use komodo_client::{
   api::{user::CreateApiKey, write::*},
   entities::{

bin/core/src/api/write/stack.rs

@@ -1,15 +1,15 @@
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use formatting::format_serror;
 use komodo_client::{
   api::write::*,
   entities::{
+    FileContents, NoData, Operation,
     config::core::CoreConfig,
     permission::PermissionLevel,
     server::ServerState,
     stack::{PartialStackConfig, Stack, StackInfo},
     update::Update,
     user::stack_user,
-    FileContents, NoData, Operation,
   },
 };
 use mungos::mongodb::bson::{doc, to_document};
@@ -33,7 +33,7 @@ use crate::{
   resource,
   stack::{
     get_stack_and_server,
-    remote::{get_repo_compose_contents, RemoteComposeContents},
+    remote::{RemoteComposeContents, get_repo_compose_contents},
     services::extract_services_into_res,
   },
   state::{db_client, github_client},

bin/core/src/api/write/sync.rs

@@ -1,11 +1,11 @@
 use std::{collections::HashMap, path::PathBuf};
 
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use formatting::format_serror;
 use komodo_client::{
   api::{read::ExportAllResourcesToToml, write::*},
   entities::{
-    self,
+    self, CloneArgs, NoData, Operation, ResourceTarget,
     action::Action,
     alert::{Alert, AlertData, SeverityLevel},
     alerter::Alerter,
@@ -23,11 +23,11 @@ use komodo_client::{
     stack::Stack,
     sync::{
       PartialResourceSyncConfig, ResourceSync, ResourceSyncInfo,
+      SyncDeployUpdate,
     },
     to_komodo_name,
     update::{Log, Update},
     user::sync_user,
-    CloneArgs, NoData, Operation, ResourceTarget,
   },
 };
 use mungos::{
@@ -52,8 +52,8 @@ use crate::{
   resource,
   state::{db_client, github_client},
   sync::{
-    deploy::SyncDeployParams, remote::RemoteResources,
-    view::push_updates_for_view, AllResourcesById,
+    AllResourcesById, deploy::SyncDeployParams,
+    remote::RemoteResources, view::push_updates_for_view,
   },
 };
 
@@ -165,7 +165,9 @@ async fn write_sync_file_contents_on_host(
 
   if let Err(e) =
     fs::write(&full_path, &contents).await.with_context(|| {
-      format!("Failed to write resource file contents to {full_path:?}")
+      format!(
+        "Failed to write resource file contents to {full_path:?}"
+      )
     })
   {
     update.push_error_log("Write File", format_serror(&e.into()));
@@ -261,7 +263,9 @@ async fn write_sync_file_contents_git(
 
   if let Err(e) =
     fs::write(&full_path, &contents).await.with_context(|| {
-      format!("Failed to write resource file contents to {full_path:?}")
+      format!(
+        "Failed to write resource file contents to {full_path:?}"
+      )
     })
   {
     update.push_error_log("Write File", format_serror(&e.into()));
@@ -385,7 +389,10 @@ impl Resolve<WriteArgs> for CommitSync {
       };
 
     let res = ExportAllResourcesToToml {
+      include_resources: sync.config.include_resources,
       tags: sync.config.match_tags.clone(),
+      include_variables: sync.config.include_variables,
+      include_user_groups: sync.config.include_user_groups,
     }
     .resolve(&ReadArgs {
       user: sync_user().to_owned(),
@@ -545,162 +552,168 @@ impl Resolve<WriteArgs> for RefreshResourceSyncPending {
       }
 
       let resources = resources?;
-
-      let id_to_tags = get_id_to_tags(None).await?;
+      let delete = sync.config.managed || sync.config.delete;
       let all_resources = AllResourcesById::load().await?;
 
-      let deployments_by_name = all_resources
-        .deployments
-        .values()
-        .map(|deployment| {
-          (deployment.name.clone(), deployment.clone())
-        })
-        .collect::<HashMap<_, _>>();
-      let stacks_by_name = all_resources
-        .stacks
-        .values()
-        .map(|stack| (stack.name.clone(), stack.clone()))
-        .collect::<HashMap<_, _>>();
+      let (resource_updates, deploy_updates) =
+        if sync.config.include_resources {
+          let id_to_tags = get_id_to_tags(None).await?;
 
-      let deploy_updates =
-        crate::sync::deploy::get_updates_for_view(SyncDeployParams {
-          deployments: &resources.deployments,
-          deployment_map: &deployments_by_name,
-          stacks: &resources.stacks,
-          stack_map: &stacks_by_name,
-          all_resources: &all_resources,
-        })
-        .await;
+          let deployments_by_name = all_resources
+            .deployments
+            .values()
+            .map(|deployment| {
+              (deployment.name.clone(), deployment.clone())
+            })
+            .collect::<HashMap<_, _>>();
+          let stacks_by_name = all_resources
+            .stacks
+            .values()
+            .map(|stack| (stack.name.clone(), stack.clone()))
+            .collect::<HashMap<_, _>>();
 
-      let delete = sync.config.managed || sync.config.delete;
+          let deploy_updates =
+            crate::sync::deploy::get_updates_for_view(
+              SyncDeployParams {
+                deployments: &resources.deployments,
+                deployment_map: &deployments_by_name,
+                stacks: &resources.stacks,
+                stack_map: &stacks_by_name,
+                all_resources: &all_resources,
+              },
+            )
+            .await;
 
-      let mut diffs = Vec::new();
+          let mut diffs = Vec::new();
 
-      {
-        push_updates_for_view::<Server>(
-          resources.servers,
-          delete,
-          &all_resources,
-          None,
-          None,
-          &id_to_tags,
-          &sync.config.match_tags,
-          &mut diffs,
-        )
-        .await?;
-        push_updates_for_view::<Stack>(
-          resources.stacks,
-          delete,
-          &all_resources,
-          None,
-          None,
-          &id_to_tags,
-          &sync.config.match_tags,
-          &mut diffs,
-        )
-        .await?;
-        push_updates_for_view::<Deployment>(
-          resources.deployments,
-          delete,
-          &all_resources,
-          None,
-          None,
-          &id_to_tags,
-          &sync.config.match_tags,
-          &mut diffs,
-        )
-        .await?;
-        push_updates_for_view::<Build>(
-          resources.builds,
-          delete,
-          &all_resources,
-          None,
-          None,
-          &id_to_tags,
-          &sync.config.match_tags,
-          &mut diffs,
-        )
-        .await?;
-        push_updates_for_view::<Repo>(
-          resources.repos,
-          delete,
-          &all_resources,
-          None,
-          None,
-          &id_to_tags,
-          &sync.config.match_tags,
-          &mut diffs,
-        )
-        .await?;
-        push_updates_for_view::<Procedure>(
-          resources.procedures,
-          delete,
-          &all_resources,
-          None,
-          None,
-          &id_to_tags,
-          &sync.config.match_tags,
-          &mut diffs,
-        )
-        .await?;
-        push_updates_for_view::<Action>(
-          resources.actions,
-          delete,
-          &all_resources,
-          None,
-          None,
-          &id_to_tags,
-          &sync.config.match_tags,
-          &mut diffs,
-        )
-        .await?;
-        push_updates_for_view::<Builder>(
-          resources.builders,
-          delete,
-          &all_resources,
-          None,
-          None,
-          &id_to_tags,
-          &sync.config.match_tags,
-          &mut diffs,
-        )
-        .await?;
-        push_updates_for_view::<Alerter>(
-          resources.alerters,
-          delete,
-          &all_resources,
-          None,
-          None,
-          &id_to_tags,
-          &sync.config.match_tags,
-          &mut diffs,
-        )
-        .await?;
-        push_updates_for_view::<ServerTemplate>(
-          resources.server_templates,
-          delete,
-          &all_resources,
-          None,
-          None,
-          &id_to_tags,
-          &sync.config.match_tags,
-          &mut diffs,
-        )
-        .await?;
-        push_updates_for_view::<ResourceSync>(
-          resources.resource_syncs,
-          delete,
-          &all_resources,
-          None,
-          None,
-          &id_to_tags,
-          &sync.config.match_tags,
-          &mut diffs,
-        )
-        .await?;
-      }
+          push_updates_for_view::<Server>(
+            resources.servers,
+            delete,
+            &all_resources,
+            None,
+            None,
+            &id_to_tags,
+            &sync.config.match_tags,
+            &mut diffs,
+          )
+          .await?;
+          push_updates_for_view::<Stack>(
+            resources.stacks,
+            delete,
+            &all_resources,
+            None,
+            None,
+            &id_to_tags,
+            &sync.config.match_tags,
+            &mut diffs,
+          )
+          .await?;
+          push_updates_for_view::<Deployment>(
+            resources.deployments,
+            delete,
+            &all_resources,
+            None,
+            None,
+            &id_to_tags,
+            &sync.config.match_tags,
+            &mut diffs,
+          )
+          .await?;
+          push_updates_for_view::<Build>(
+            resources.builds,
+            delete,
+            &all_resources,
+            None,
+            None,
+            &id_to_tags,
+            &sync.config.match_tags,
+            &mut diffs,
+          )
+          .await?;
+          push_updates_for_view::<Repo>(
+            resources.repos,
+            delete,
+            &all_resources,
+            None,
+            None,
+            &id_to_tags,
+            &sync.config.match_tags,
+            &mut diffs,
+          )
+          .await?;
+          push_updates_for_view::<Procedure>(
+            resources.procedures,
+            delete,
+            &all_resources,
+            None,
+            None,
+            &id_to_tags,
+            &sync.config.match_tags,
+            &mut diffs,
+          )
+          .await?;
+          push_updates_for_view::<Action>(
+            resources.actions,
+            delete,
+            &all_resources,
+            None,
+            None,
+            &id_to_tags,
+            &sync.config.match_tags,
+            &mut diffs,
+          )
+          .await?;
+          push_updates_for_view::<Builder>(
+            resources.builders,
+            delete,
+            &all_resources,
+            None,
+            None,
+            &id_to_tags,
+            &sync.config.match_tags,
+            &mut diffs,
+          )
+          .await?;
+          push_updates_for_view::<Alerter>(
+            resources.alerters,
+            delete,
+            &all_resources,
+            None,
+            None,
+            &id_to_tags,
+            &sync.config.match_tags,
+            &mut diffs,
+          )
+          .await?;
+          push_updates_for_view::<ServerTemplate>(
+            resources.server_templates,
+            delete,
+            &all_resources,
+            None,
+            None,
+            &id_to_tags,
+            &sync.config.match_tags,
+            &mut diffs,
+          )
+          .await?;
+          push_updates_for_view::<ResourceSync>(
+            resources.resource_syncs,
+            delete,
+            &all_resources,
+            None,
+            None,
+            &id_to_tags,
+            &sync.config.match_tags,
+            &mut diffs,
+          )
+          .await?;
 
-      let variable_updates = if sync.config.match_tags.is_empty() {
+          (diffs, deploy_updates)
+        } else {
+          (Vec::new(), SyncDeployUpdate::default())
+        };
+
+      let variable_updates = if sync.config.include_variables {
         crate::sync::variables::get_updates_for_view(
           &resources.variables,
           delete,
@@ -710,7 +723,7 @@ impl Resolve<WriteArgs> for RefreshResourceSyncPending {
         Default::default()
       };
 
-      let user_group_updates = if sync.config.match_tags.is_empty() {
+      let user_group_updates = if sync.config.include_user_groups {
         crate::sync::user_groups::get_updates_for_view(
           resources.user_groups,
           delete,
@@ -722,7 +735,7 @@ impl Resolve<WriteArgs> for RefreshResourceSyncPending {
       };
 
       anyhow::Ok((
-        diffs,
+        resource_updates,
         deploy_updates,
         variable_updates,
         user_group_updates,

bin/core/src/api/write/tag.rs

@@ -1,17 +1,26 @@
 use std::str::FromStr;
 
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use komodo_client::{
   api::write::{
-    CreateTag, DeleteTag, RenameTag, UpdateTagsOnResource,
-    UpdateTagsOnResourceResponse,
+    CreateTag, DeleteTag, RenameTag, UpdateTagColor,
+    UpdateTagsOnResource, UpdateTagsOnResourceResponse,
   },
   entities::{
-    action::Action, alerter::Alerter, build::Build, builder::Builder,
-    deployment::Deployment, permission::PermissionLevel,
-    procedure::Procedure, repo::Repo, server::Server,
-    server_template::ServerTemplate, stack::Stack,
-    sync::ResourceSync, tag::Tag, ResourceTarget,
+    ResourceTarget,
+    action::Action,
+    alerter::Alerter,
+    build::Build,
+    builder::Builder,
+    deployment::Deployment,
+    permission::PermissionLevel,
+    procedure::Procedure,
+    repo::Repo,
+    server::Server,
+    server_template::ServerTemplate,
+    stack::Stack,
+    sync::ResourceSync,
+    tag::{Tag, TagColor},
   },
 };
 use mungos::{
@@ -41,6 +50,7 @@ impl Resolve<WriteArgs> for CreateTag {
     let mut tag = Tag {
       id: Default::default(),
       name: self.name,
+      color: TagColor::Slate,
       owner: user.id.clone(),
     };
 
@@ -83,6 +93,27 @@ impl Resolve<WriteArgs> for RenameTag {
   }
 }
 
+impl Resolve<WriteArgs> for UpdateTagColor {
+  #[instrument(name = "UpdateTagColor", skip(user))]
+  async fn resolve(
+    self,
+    WriteArgs { user }: &WriteArgs,
+  ) -> serror::Result<Tag> {
+    let tag = get_tag_check_owner(&self.tag, user).await?;
+
+    update_one_by_id(
+      &db_client().tags,
+      &tag.id,
+      doc! { "$set": { "color": self.color.as_ref() } },
+      None,
+    )
+    .await
+    .context("failed to rename tag on db")?;
+
+    Ok(get_tag(&self.tag).await?)
+  }
+}
+
 impl Resolve<WriteArgs> for DeleteTag {
   #[instrument(name = "DeleteTag", skip(user))]
   async fn resolve(
@@ -118,7 +149,7 @@ impl Resolve<WriteArgs> for UpdateTagsOnResource {
     let WriteArgs { user } = args;
     match self.target {
       ResourceTarget::System(_) => {
-        return Err(anyhow!("Invalid target type: System").into())
+        return Err(anyhow!("Invalid target type: System").into());
       }
       ResourceTarget::Build(id) => {
         resource::get_check_permissions::<Build>(

bin/core/src/api/write/user.rs

@@ -1,13 +1,13 @@
 use std::str::FromStr;
 
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use komodo_client::{
   api::write::{
     DeleteUser, DeleteUserResponse, UpdateUserPassword,
     UpdateUserPasswordResponse, UpdateUserUsername,
     UpdateUserUsernameResponse,
   },
-  entities::{user::UserConfig, NoData},
+  entities::{NoData, user::UserConfig},
 };
 use mungos::mongodb::bson::{doc, oid::ObjectId};
 use resolver_api::Resolve;

bin/core/src/api/write/user_group.rs

@@ -1,6 +1,6 @@
 use std::{collections::HashMap, str::FromStr};
 
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use komodo_client::{
   api::write::{
     AddUserToUserGroup, CreateUserGroup, DeleteUserGroup,

bin/core/src/api/write/variable.rs

@@ -1,7 +1,7 @@
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use komodo_client::{
   api::write::*,
-  entities::{variable::Variable, Operation, ResourceTarget},
+  entities::{Operation, ResourceTarget, variable::Variable},
 };
 use mungos::mongodb::bson::doc;
 use resolver_api::Resolve;

bin/core/src/auth/github/client.rs

@@ -1,11 +1,11 @@
 use std::sync::OnceLock;
 
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use komodo_client::entities::config::core::{
   CoreConfig, OauthCredentials,
 };
 use reqwest::StatusCode;
-use serde::{de::DeserializeOwned, Deserialize, Serialize};
+use serde::{Deserialize, Serialize, de::DeserializeOwned};
 use tokio::sync::Mutex;
 
 use crate::{
@@ -47,15 +47,21 @@ impl GithubOauthClient {
       return None;
     }
     if host.is_empty() {
-      warn!("github oauth is enabled, but 'config.host' is not configured");
+      warn!(
+        "github oauth is enabled, but 'config.host' is not configured"
+      );
       return None;
     }
     if id.is_empty() {
-      warn!("github oauth is enabled, but 'config.github_oauth.id' is not configured");
+      warn!(
+        "github oauth is enabled, but 'config.github_oauth.id' is not configured"
+      );
       return None;
     }
     if secret.is_empty() {
-      warn!("github oauth is enabled, but 'config.github_oauth.secret' is not configured");
+      warn!(
+        "github oauth is enabled, but 'config.github_oauth.secret' is not configured"
+      );
       return None;
     }
     GithubOauthClient {

bin/core/src/auth/github/mod.rs

@@ -1,6 +1,6 @@
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use axum::{
-  extract::Query, response::Redirect, routing::get, Router,
+  Router, extract::Query, response::Redirect, routing::get,
 };
 use komodo_client::entities::{
   komodo_timestamp,
@@ -72,7 +72,7 @@ async fn callback(
     .context("failed at find user query from database")?;
   let jwt = match user {
     Some(user) => jwt_client()
-      .generate(user.id)
+      .encode(user.id)
       .context("failed to generate jwt")?,
     None => {
       let ts = komodo_timestamp();
@@ -109,7 +109,7 @@ async fn callback(
         .context("inserted_id is not ObjectId")?
         .to_string();
       jwt_client()
-        .generate(user_id)
+        .encode(user_id)
         .context("failed to generate jwt")?
     }
   };

bin/core/src/auth/google/client.rs

@@ -1,13 +1,12 @@
 use std::sync::OnceLock;
 
-use anyhow::{anyhow, Context};
-use jwt::Token;
+use anyhow::{Context, anyhow};
+use jsonwebtoken::{DecodingKey, Validation, decode};
 use komodo_client::entities::config::core::{
   CoreConfig, OauthCredentials,
 };
 use reqwest::StatusCode;
-use serde::{de::DeserializeOwned, Deserialize};
-use serde_json::Value;
+use serde::{Deserialize, de::DeserializeOwned};
 use tokio::sync::Mutex;
 
 use crate::{
@@ -49,15 +48,21 @@ impl GoogleOauthClient {
       return None;
     }
     if host.is_empty() {
-      warn!("google oauth is enabled, but 'config.host' is not configured");
+      warn!(
+        "google oauth is enabled, but 'config.host' is not configured"
+      );
       return None;
     }
     if id.is_empty() {
-      warn!("google oauth is enabled, but 'config.google_oauth.id' is not configured");
+      warn!(
+        "google oauth is enabled, but 'config.google_oauth.id' is not configured"
+      );
       return None;
     }
     if secret.is_empty() {
-      warn!("google oauth is enabled, but 'config.google_oauth.secret' is not configured");
+      warn!(
+        "google oauth is enabled, but 'config.google_oauth.secret' is not configured"
+      );
       return None;
     }
     let scopes = urlencoding::encode(
@@ -139,10 +144,16 @@ impl GoogleOauthClient {
     &self,
     id_token: &str,
   ) -> anyhow::Result<GoogleUser> {
-    let t: Token<Value, GoogleUser, jwt::Unverified> =
-      Token::parse_unverified(id_token)
-        .context("failed to parse id_token")?;
-    Ok(t.claims().to_owned())
+    let mut v = Validation::new(Default::default());
+    v.insecure_disable_signature_validation();
+    v.validate_aud = false;
+    let res = decode::<GoogleUser>(
+      id_token,
+      &DecodingKey::from_secret(b""),
+      &v,
+    )
+    .context("failed to decode google id token")?;
+    Ok(res.claims)
   }
 
   #[instrument(level = "debug", skip(self))]

bin/core/src/auth/google/mod.rs

@@ -1,7 +1,7 @@
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use async_timing_util::unix_timestamp_ms;
 use axum::{
-  extract::Query, response::Redirect, routing::get, Router,
+  Router, extract::Query, response::Redirect, routing::get,
 };
 use komodo_client::entities::user::{User, UserConfig};
 use mongo_indexed::Document;
@@ -81,7 +81,7 @@ async fn callback(
     .context("failed at find user query from mongo")?;
   let jwt = match user {
     Some(user) => jwt_client()
-      .generate(user.id)
+      .encode(user.id)
       .context("failed to generate jwt")?,
     None => {
       let ts = unix_timestamp_ms() as i64;
@@ -124,7 +124,7 @@ async fn callback(
         .context("inserted_id is not ObjectId")?
         .to_string();
       jwt_client()
-        .generate(user_id)
+        .encode(user_id)
         .context("failed to generate jwt")?
     }
   };

bin/core/src/auth/jwt.rs

@@ -1,15 +1,15 @@
 use std::collections::HashMap;
 
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use async_timing_util::{
-  get_timelength_in_ms, unix_timestamp_ms, Timelength,
+  Timelength, get_timelength_in_ms, unix_timestamp_ms,
+};
+use jsonwebtoken::{
+  DecodingKey, EncodingKey, Header, Validation, decode, encode,
 };
-use hmac::{Hmac, Mac};
-use jwt::SignWithKey;
 use komodo_client::entities::config::core::CoreConfig;
 use mungos::mongodb::bson::doc;
 use serde::{Deserialize, Serialize};
-use sha2::Sha256;
 use tokio::sync::Mutex;
 
 use crate::helpers::random_string;
@@ -24,7 +24,10 @@ pub struct JwtClaims {
 }
 
 pub struct JwtClient {
-  pub key: Hmac<Sha256>,
+  header: Header,
+  validation: Validation,
+  encoding_key: EncodingKey,
+  decoding_key: DecodingKey,
   ttl_ms: u128,
   exchange_tokens: ExchangeTokenMap,
 }
@@ -36,10 +39,11 @@ impl JwtClient {
     } else {
       config.jwt_secret.clone()
     };
-    let key = Hmac::new_from_slice(secret.as_bytes())
-      .context("failed at taking HmacSha256 of jwt secret")?;
     Ok(JwtClient {
-      key,
+      header: Header::default(),
+      validation: Validation::new(Default::default()),
+      encoding_key: EncodingKey::from_secret(secret.as_bytes()),
+      decoding_key: DecodingKey::from_secret(secret.as_bytes()),
       ttl_ms: get_timelength_in_ms(
         config.jwt_ttl.to_string().parse()?,
       ),
@@ -47,7 +51,7 @@ impl JwtClient {
     })
   }
 
-  pub fn generate(&self, user_id: String) -> anyhow::Result<String> {
+  pub fn encode(&self, user_id: String) -> anyhow::Result<String> {
     let iat = unix_timestamp_ms();
     let exp = iat + self.ttl_ms;
     let claims = JwtClaims {
@@ -55,10 +59,14 @@ impl JwtClient {
       iat,
       exp,
     };
-    let jwt = claims
-      .sign_with_key(&self.key)
-      .context("failed at signing claim")?;
-    Ok(jwt)
+    encode(&self.header, &claims, &self.encoding_key)
+      .context("failed at signing claim")
+  }
+
+  pub fn decode(&self, jwt: &str) -> anyhow::Result<JwtClaims> {
+    decode::<JwtClaims>(jwt, &self.decoding_key, &self.validation)
+      .map(|res| res.claims)
+      .context("failed to decode token claims")
   }
 
   #[instrument(level = "debug", skip_all)]

bin/core/src/auth/local.rs

@@ -1,6 +1,6 @@
 use std::str::FromStr;
 
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use async_timing_util::unix_timestamp_ms;
 use komodo_client::{
   api::auth::{
@@ -85,7 +85,7 @@ impl Resolve<AuthArgs> for CreateLocalUser {
       .to_string();
 
     let jwt = jwt_client()
-      .generate(user_id)
+      .encode(user_id)
       .context("failed to generate jwt for user")?;
 
     Ok(CreateLocalUserResponse { jwt })
@@ -131,7 +131,7 @@ impl Resolve<AuthArgs> for LoginLocalUser {
     }
 
     let jwt = jwt_client()
-      .generate(user.id)
+      .encode(user.id)
       .context("failed at generating jwt for user")?;
 
     Ok(LoginLocalUserResponse { jwt })

bin/core/src/auth/mod.rs

@@ -1,5 +1,4 @@
-use ::jwt::VerifyWithKey;
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use async_timing_util::unix_timestamp_ms;
 use axum::{
   extract::Request, http::HeaderMap, middleware::Next,
@@ -71,7 +70,9 @@ pub async fn get_user_id_from_headers(
     }
     _ => {
       // AUTH FAIL
-      Err(anyhow!("must attach either AUTHORIZATION header with jwt OR pass X-API-KEY and X-API-SECRET"))
+      Err(anyhow!(
+        "must attach either AUTHORIZATION header with jwt OR pass X-API-KEY and X-API-SECRET"
+      ))
     }
   }
 }
@@ -93,9 +94,7 @@ pub async fn authenticate_check_enabled(
 pub async fn auth_jwt_get_user_id(
   jwt: &str,
 ) -> anyhow::Result<String> {
-  let claims: JwtClaims = jwt
-    .verify_with_key(&jwt_client().key)
-    .context("failed to verify claims")?;
+  let claims: JwtClaims = jwt_client().decode(jwt)?;
   if claims.exp > unix_timestamp_ms() {
     Ok(claims.id)
   } else {

bin/core/src/auth/oidc/client.rs

@@ -1,10 +1,11 @@
-use std::sync::OnceLock;
+use std::{sync::OnceLock, time::Duration};
 
 use anyhow::Context;
+use arc_swap::ArcSwapOption;
 use openidconnect::{
-  core::*, Client, ClientId, ClientSecret, EmptyAdditionalClaims,
+  Client, ClientId, ClientSecret, EmptyAdditionalClaims,
   EndpointMaybeSet, EndpointNotSet, EndpointSet, IssuerUrl,
-  RedirectUrl, StandardErrorResponse,
+  RedirectUrl, StandardErrorResponse, core::*,
 };
 
 use crate::config::core_config;
@@ -29,59 +30,65 @@ type OidcClient = Client<
   EndpointMaybeSet,
 >;
 
-static DEFAULT_OIDC_CLIENT: OnceLock<Option<OidcClient>> =
-  OnceLock::new();
-
-pub fn default_oidc_client() -> Option<&'static OidcClient> {
-  DEFAULT_OIDC_CLIENT
-    .get()
-    .expect("OIDC client get before init")
-    .as_ref()
+pub fn oidc_client() -> &'static ArcSwapOption<OidcClient> {
+  static OIDC_CLIENT: OnceLock<ArcSwapOption<OidcClient>> =
+    OnceLock::new();
+  OIDC_CLIENT.get_or_init(Default::default)
 }
 
-pub async fn init_default_oidc_client() {
+/// The OIDC client must be reinitialized to
+/// pick up the latest provider JWKs. This
+/// function spawns a management thread to do this
+/// on a loop.
+pub async fn spawn_oidc_client_management() {
   let config = core_config();
   if !config.oidc_enabled
     || config.oidc_provider.is_empty()
     || config.oidc_client_id.is_empty()
-    || config.oidc_client_secret.is_empty()
   {
-    DEFAULT_OIDC_CLIENT
-      .set(None)
-      .expect("Default OIDC client initialized twice");
     return;
   }
-  async {
-    // Use OpenID Connect Discovery to fetch the provider metadata.
-    let provider_metadata = CoreProviderMetadata::discover_async(
-      IssuerUrl::new(config.oidc_provider.clone())?,
-      super::reqwest_client(),
-    )
+  reset_oidc_client()
     .await
-    .context(
-      "Failed to get OIDC /.well-known/openid-configuration",
-    )?;
+    .context("Failed to initialize OIDC client.")
+    .unwrap();
+  tokio::spawn(async move {
+    loop {
+      tokio::time::sleep(Duration::from_secs(60)).await;
+      if let Err(e) = reset_oidc_client().await {
+        warn!("Failed to reinitialize OIDC client | {e:#}");
+      }
+    }
+  });
+}
 
-    // Create an OpenID Connect client by specifying the client ID, client secret, authorization URL
-    // and token URL.
-    let client = CoreClient::from_provider_metadata(
-      provider_metadata,
-      ClientId::new(config.oidc_client_id.to_string()),
-      Some(ClientSecret::new(config.oidc_client_secret.to_string())),
-    )
-    // Set the URL the user will be redirected to after the authorization process.
-    .set_redirect_uri(RedirectUrl::new(format!(
-      "{}/auth/oidc/callback",
-      core_config().host
-    ))?);
+async fn reset_oidc_client() -> anyhow::Result<()> {
+  let config = core_config();
+  // Use OpenID Connect Discovery to fetch the provider metadata.
+  let provider_metadata = CoreProviderMetadata::discover_async(
+    IssuerUrl::new(config.oidc_provider.clone())?,
+    super::reqwest_client(),
+  )
+  .await
+  .context("Failed to get OIDC /.well-known/openid-configuration")?;
+
+  let client = CoreClient::from_provider_metadata(
+    provider_metadata,
+    ClientId::new(config.oidc_client_id.to_string()),
+    // The secret may be empty / ommitted if auth provider supports PKCE
+    if config.oidc_client_secret.is_empty() {
+      None
+    } else {
+      Some(ClientSecret::new(config.oidc_client_secret.to_string()))
+    },
+  )
+  // Set the URL the user will be redirected to after the authorization process.
+  .set_redirect_uri(RedirectUrl::new(format!(
+    "{}/auth/oidc/callback",
+    core_config().host
+  ))?);
 
-    DEFAULT_OIDC_CLIENT
-      .set(Some(client))
-      .expect("Default OIDC client initialized twice");
+  oidc_client().store(Some(client.into()));
 
-    anyhow::Ok(())
-  }
-  .await
-  .context("Failed to init default OIDC client")
-  .unwrap();
+  Ok(())
 }

bin/core/src/auth/oidc/mod.rs

@@ -1,19 +1,20 @@
 use std::sync::OnceLock;
 
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use axum::{
-  extract::Query, response::Redirect, routing::get, Router,
+  Router, extract::Query, response::Redirect, routing::get,
 };
-use client::default_oidc_client;
+use client::oidc_client;
 use dashmap::DashMap;
 use komodo_client::entities::{
   komodo_timestamp,
   user::{User, UserConfig},
 };
-use mungos::mongodb::bson::{doc, Document};
+use mungos::mongodb::bson::{Document, doc};
 use openidconnect::{
-  core::CoreAuthenticationFlow, AuthorizationCode, CsrfToken, Nonce,
-  Scope, TokenResponse,
+  AccessTokenHash, AuthorizationCode, CsrfToken, Nonce,
+  OAuth2TokenResponse, PkceCodeChallenge, PkceCodeVerifier, Scope,
+  TokenResponse, core::CoreAuthenticationFlow,
 };
 use reqwest::StatusCode;
 use serde::Deserialize;
@@ -43,10 +44,13 @@ fn reqwest_client() -> &'static reqwest::Client {
 const CSRF_VALID_FOR_MS: i64 = 120_000; // 2 minutes for user to log in.
 
 type RedirectUrl = Option<String>;
-type CsrfMap = DashMap<String, (Nonce, RedirectUrl, i64)>;
-fn csrf_verifier_tokens() -> &'static CsrfMap {
-  static CSRF: OnceLock<CsrfMap> = OnceLock::new();
-  CSRF.get_or_init(Default::default)
+/// Maps the csrf secrets to other information added in the "login" method (before auth provider redirect).
+/// This information is retrieved in the "callback" method (after auth provider redirect).
+type VerifierMap =
+  DashMap<String, (PkceCodeVerifier, Nonce, RedirectUrl, i64)>;
+fn verifier_tokens() -> &'static VerifierMap {
+  static VERIFIERS: OnceLock<VerifierMap> = OnceLock::new();
+  VERIFIERS.get_or_init(Default::default)
 }
 
 pub fn router() -> Router {
@@ -69,8 +73,12 @@ pub fn router() -> Router {
 async fn login(
   Query(RedirectQuery { redirect }): Query<RedirectQuery>,
 ) -> anyhow::Result<Redirect> {
+  let client = oidc_client().load();
   let client =
-    default_oidc_client().context("OIDC Client not configured")?;
+    client.as_ref().context("OIDC Client not configured")?;
+
+  let (pkce_challenge, pkce_verifier) =
+    PkceCodeChallenge::new_random_sha256();
 
   // Generate the authorization URL.
   let (auth_url, csrf_token, nonce) = client
@@ -79,14 +87,20 @@ async fn login(
       CsrfToken::new_random,
       Nonce::new_random,
     )
+    .set_pkce_challenge(pkce_challenge)
     .add_scope(Scope::new("openid".to_string()))
     .add_scope(Scope::new("email".to_string()))
     .url();
 
   // Data inserted here will be matched on callback side for csrf protection.
-  csrf_verifier_tokens().insert(
+  verifier_tokens().insert(
     csrf_token.secret().clone(),
-    (nonce, redirect, komodo_timestamp() + CSRF_VALID_FOR_MS),
+    (
+      pkce_verifier,
+      nonce,
+      redirect,
+      komodo_timestamp() + CSRF_VALID_FOR_MS,
+    ),
   );
 
   let config = core_config();
@@ -121,8 +135,9 @@ struct CallbackQuery {
 async fn callback(
   Query(query): Query<CallbackQuery>,
 ) -> anyhow::Result<Redirect> {
+  let client = oidc_client().load();
   let client =
-    default_oidc_client().context("OIDC Client not configured")?;
+    client.as_ref().context("OIDC Client not configured")?;
 
   if let Some(e) = query.error {
     return Err(anyhow!("Provider returned error: {e}"));
@@ -133,9 +148,10 @@ async fn callback(
     query.state.context("Provider did not return state")?,
   );
 
-  let (_, (nonce, redirect, valid_until)) = csrf_verifier_tokens()
-    .remove(state.secret())
-    .context("CSRF Token invalid")?;
+  let (_, (pkce_verifier, nonce, redirect, valid_until)) =
+    verifier_tokens()
+      .remove(state.secret())
+      .context("CSRF token invalid")?;
 
   if komodo_timestamp() > valid_until {
     return Err(anyhow!(
@@ -146,6 +162,7 @@ async fn callback(
   let token_response = client
     .exchange_code(AuthorizationCode::new(code))
     .context("Failed to get Oauth token at exchange code")?
+    .set_pkce_verifier(pkce_verifier)
     .request_async(reqwest_client())
     .await
     .context("Failed to get Oauth token")?;
@@ -169,7 +186,21 @@ async fn callback(
 
   let claims = id_token
     .claims(&verifier, &nonce)
-    .context("Failed to verify token claims")?;
+    .context("Failed to verify token claims. This issue may be temporary (60 seconds max).")?;
+
+  // Verify the access token hash to ensure that the access token hasn't been substituted for
+  // another user's.
+  if let Some(expected_access_token_hash) = claims.access_token_hash()
+  {
+    let actual_access_token_hash = AccessTokenHash::from_token(
+      token_response.access_token(),
+      id_token.signing_alg()?,
+      id_token.signing_key(&verifier)?,
+    )?;
+    if actual_access_token_hash != *expected_access_token_hash {
+      return Err(anyhow!("Invalid access token"));
+    }
+  }
 
   let user_id = claims.subject().as_str();
 
@@ -185,7 +216,7 @@ async fn callback(
 
   let jwt = match user {
     Some(user) => jwt_client()
-      .generate(user.id)
+      .encode(user.id)
       .context("failed to generate jwt")?,
     None => {
       let ts = komodo_timestamp();
@@ -241,7 +272,7 @@ async fn callback(
         .context("inserted_id is not ObjectId")?
         .to_string();
       jwt_client()
-        .generate(user_id)
+        .encode(user_id)
         .context("failed to generate jwt")?
     }
   };

bin/core/src/cloud/aws/ec2.rs

@@ -1,22 +1,22 @@
 use std::{str::FromStr, time::Duration};
 
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use aws_config::{BehaviorVersion, Region};
 use aws_sdk_ec2::{
+  Client,
   types::{
     BlockDeviceMapping, EbsBlockDevice,
     InstanceNetworkInterfaceSpecification, InstanceStateChange,
     InstanceStateName, InstanceStatus, InstanceType, ResourceType,
     Tag, TagSpecification, VolumeType,
   },
-  Client,
 };
 use base64::Engine;
 use komodo_client::entities::{
+  ResourceTarget,
   alert::{Alert, AlertData, SeverityLevel},
   komodo_timestamp,
   server_template::aws::AwsServerTemplateConfig,
-  ResourceTarget,
 };
 
 use crate::{alert::send_alerts, config::core_config};
@@ -29,20 +29,40 @@ pub struct Ec2Instance {
   pub ip: String,
 }
 
+/// Provides credentials in the core config file to the AWS client
+#[derive(Debug)]
+struct CredentialsFromConfig;
+
+impl aws_credential_types::provider::ProvideCredentials
+  for CredentialsFromConfig
+{
+  fn provide_credentials<'a>(
+    &'a self,
+  ) -> aws_credential_types::provider::future::ProvideCredentials<'a>
+  where
+    Self: 'a,
+  {
+    aws_credential_types::provider::future::ProvideCredentials::new(
+      async {
+        let config = core_config();
+        Ok(aws_credential_types::Credentials::new(
+          &config.aws.access_key_id,
+          &config.aws.secret_access_key,
+          None,
+          None,
+          "komodo-config",
+        ))
+      },
+    )
+  }
+}
+
 #[instrument]
 async fn create_ec2_client(region: String) -> Client {
-  // There may be a better way to pass these keys to client
-  std::env::set_var(
-    "AWS_ACCESS_KEY_ID",
-    &core_config().aws.access_key_id,
-  );
-  std::env::set_var(
-    "AWS_SECRET_ACCESS_KEY",
-    &core_config().aws.secret_access_key,
-  );
   let region = Region::new(region);
-  let config = aws_config::defaults(BehaviorVersion::v2024_03_28())
+  let config = aws_config::defaults(BehaviorVersion::latest())
     .region(region)
+    .credentials_provider(CredentialsFromConfig)
     .load()
     .await;
   Client::new(&config)

bin/core/src/cloud/hetzner/client.rs

@@ -1,7 +1,7 @@
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use axum::http::{HeaderName, HeaderValue};
 use reqwest::{RequestBuilder, StatusCode};
-use serde::{de::DeserializeOwned, Serialize};
+use serde::{Serialize, de::DeserializeOwned};
 
 use super::{
   common::{

bin/core/src/cloud/hetzner/mod.rs

@@ -3,7 +3,7 @@ use std::{
   time::Duration,
 };
 
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use futures::future::join_all;
 use komodo_client::entities::server_template::hetzner::{
   HetznerDatacenter, HetznerServerTemplateConfig, HetznerServerType,

bin/core/src/db.rs

@@ -89,7 +89,9 @@ impl DbClient {
         client = client.address(address);
       }
       _ => {
-        error!("config.mongo not configured correctly. must pass either config.mongo.uri, or config.mongo.address + config.mongo.username? + config.mongo.password?");
+        error!(
+          "config.mongo not configured correctly. must pass either config.mongo.uri, or config.mongo.address + config.mongo.username? + config.mongo.password?"
+        );
         std::process::exit(1)
       }
     }

bin/core/src/helpers/builder.rs

@@ -1,27 +1,27 @@
 use std::time::Duration;
 
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use formatting::muted;
 use komodo_client::entities::{
+  Version,
   builder::{AwsBuilderConfig, Builder, BuilderConfig},
   komodo_timestamp,
   server::Server,
   server_template::aws::AwsServerTemplateConfig,
   update::{Log, Update},
-  Version,
 };
 use periphery_client::{
-  api::{self, GetVersionResponse},
   PeripheryClient,
+  api::{self, GetVersionResponse},
 };
 
 use crate::{
   cloud::{
-    aws::ec2::{
-      launch_ec2_instance, terminate_ec2_instance_with_retry,
-      Ec2Instance,
-    },
     BuildCleanupData,
+    aws::ec2::{
+      Ec2Instance, launch_ec2_instance,
+      terminate_ec2_instance_with_retry,
+    },
   },
   config::core_config,
   helpers::update::update_update,

bin/core/src/helpers/cache.rs

@@ -9,9 +9,9 @@ pub struct Cache<K: PartialEq + Eq + Hash, T: Clone + Default> {
 }
 
 impl<
-    K: PartialEq + Eq + Hash + std::fmt::Debug + Clone,
-    T: Clone + Default,
-  > Cache<K, T>
+  K: PartialEq + Eq + Hash + std::fmt::Debug + Clone,
+  T: Clone + Default,
+> Cache<K, T>
 {
   #[instrument(level = "debug", skip(self))]
   pub async fn get(&self, key: &K) -> Option<T> {
@@ -70,9 +70,9 @@ impl<
 }
 
 impl<
-    K: PartialEq + Eq + Hash + std::fmt::Debug + Clone,
-    T: Clone + Default + Busy,
-  > Cache<K, T>
+  K: PartialEq + Eq + Hash + std::fmt::Debug + Clone,
+  T: Clone + Default + Busy,
+> Cache<K, T>
 {
   #[instrument(level = "debug", skip(self))]
   pub async fn busy(&self, id: &K) -> bool {

bin/core/src/helpers/channel.rs

@@ -1,11 +1,11 @@
 use std::sync::OnceLock;
 
 use komodo_client::entities::update::{Update, UpdateListItem};
-use tokio::sync::{broadcast, Mutex};
+use tokio::sync::{Mutex, broadcast};
 
 /// A channel sending (build_id, update_id)
-pub fn build_cancel_channel(
-) -> &'static BroadcastChannel<(String, Update)> {
+pub fn build_cancel_channel()
+-> &'static BroadcastChannel<(String, Update)> {
   static BUILD_CANCEL_CHANNEL: OnceLock<
     BroadcastChannel<(String, Update)>,
   > = OnceLock::new();
@@ -13,8 +13,8 @@ pub fn build_cancel_channel(
 }
 
 /// A channel sending (repo_id, update_id)
-pub fn repo_cancel_channel(
-) -> &'static BroadcastChannel<(String, Update)> {
+pub fn repo_cancel_channel()
+-> &'static BroadcastChannel<(String, Update)> {
   static REPO_CANCEL_CHANNEL: OnceLock<
     BroadcastChannel<(String, Update)>,
   > = OnceLock::new();

bin/core/src/helpers/interpolate.rs

@@ -1,7 +1,7 @@
 use std::collections::HashSet;
 
 use anyhow::Context;
-use komodo_client::entities::{update::Update, SystemCommand};
+use komodo_client::entities::{SystemCommand, update::Update};
 
 use super::query::VariablesAndSecrets;
 

bin/core/src/helpers/mod.rs

@@ -1,24 +1,24 @@
 use std::{str::FromStr, time::Duration};
 
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use futures::future::join_all;
 use komodo_client::{
   api::write::{CreateBuilder, CreateServer},
   entities::{
+    ResourceTarget,
     builder::{PartialBuilderConfig, PartialServerBuilderConfig},
     komodo_timestamp,
     permission::{Permission, PermissionLevel, UserTarget},
     server::{PartialServerConfig, Server},
     sync::ResourceSync,
     update::Log,
-    user::{system_user, User},
-    ResourceTarget,
+    user::{User, system_user},
   },
 };
 use mongo_indexed::Document;
 use mungos::{
   find::find_collect,
-  mongodb::bson::{doc, oid::ObjectId, to_document, Bson},
+  mongodb::bson::{Bson, doc, oid::ObjectId, to_document},
 };
 use periphery_client::PeripheryClient;
 use rand::Rng;
@@ -208,7 +208,9 @@ pub async fn startup_cleanup() {
 async fn startup_in_progress_update_cleanup() {
   let log = Log::error(
     "Komodo shutdown",
-    String::from("Komodo shutdown during execution. If this is a build, the builder may not have been terminated.")
+    String::from(
+      "Komodo shutdown during execution. If this is a build, the builder may not have been terminated.",
+    ),
   );
   // This static log won't fail to serialize, unwrap ok.
   let log = to_document(&log).unwrap();
@@ -319,7 +321,10 @@ pub async fn ensure_first_server_and_builder() {
     {
       Ok(server) => server,
       Err(e) => {
-        error!("Failed to initialize 'first_server'. Failed to CreateServer. {:#}", e.error);
+        error!(
+          "Failed to initialize 'first_server'. Failed to CreateServer. {:#}",
+          e.error
+        );
         return;
       }
     }
@@ -342,6 +347,9 @@ pub async fn ensure_first_server_and_builder() {
   })
   .await
   {
-    error!("Failed to initialize 'first_builder' | Failed to CreateBuilder | {:#}", e.error);
+    error!(
+      "Failed to initialize 'first_builder' | Failed to CreateBuilder | {:#}",
+      e.error
+    );
   }
-}
+}

bin/core/src/helpers/procedure.rs

@@ -1,7 +1,7 @@
 use std::time::{Duration, Instant};
 
-use anyhow::{anyhow, Context};
-use formatting::{bold, colored, format_serror, muted, Color};
+use anyhow::{Context, anyhow};
+use formatting::{Color, bold, colored, format_serror, muted};
 use futures::future::join_all;
 use komodo_client::{
   api::execute::*,
@@ -25,7 +25,7 @@ use crate::{
     execute::{ExecuteArgs, ExecuteRequest},
     write::WriteArgs,
   },
-  resource::{list_full_for_user_using_pattern, KomodoResource},
+  resource::{KomodoResource, list_full_for_user_using_pattern},
   state::db_client,
 };
 
@@ -206,7 +206,7 @@ async fn execute_stage(
   join_all(futures)
     .await
     .into_iter()
-    .collect::<anyhow::Result<_>>()?;
+    .collect::<anyhow::Result<Vec<_>>>()?;
   Ok(())
 }
 

bin/core/src/helpers/prune.rs

@@ -1,6 +1,6 @@
 use anyhow::Context;
 use async_timing_util::{
-  unix_timestamp_ms, wait_until_timelength, Timelength, ONE_DAY_MS,
+  ONE_DAY_MS, Timelength, unix_timestamp_ms, wait_until_timelength,
 };
 use futures::future::join_all;
 use mungos::{find::find_collect, mongodb::bson::doc};

bin/core/src/helpers/query.rs

@@ -1,7 +1,8 @@
 use std::{collections::HashMap, str::FromStr};
 
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use komodo_client::entities::{
+  Operation, ResourceTarget, ResourceTargetVariant,
   action::Action,
   alerter::Alerter,
   build::Build,
@@ -17,15 +18,14 @@ use komodo_client::entities::{
   sync::ResourceSync,
   tag::Tag,
   update::Update,
-  user::{admin_service_user, User},
+  user::{User, admin_service_user},
   user_group::UserGroup,
   variable::Variable,
-  Operation, ResourceTarget, ResourceTargetVariant,
 };
 use mungos::{
   find::find_collect,
   mongodb::{
-    bson::{doc, oid::ObjectId, Document},
+    bson::{Document, doc, oid::ObjectId},
     options::FindOneOptions,
   },
 };
@@ -359,8 +359,8 @@ pub struct VariablesAndSecrets {
   pub secrets: HashMap<String, String>,
 }
 
-pub async fn get_variables_and_secrets(
-) -> anyhow::Result<VariablesAndSecrets> {
+pub async fn get_variables_and_secrets()
+-> anyhow::Result<VariablesAndSecrets> {
   let variables = find_collect(&db_client().variables, None, None)
     .await
     .context("failed to get all variables from db")?;

bin/core/src/helpers/update.rs

@@ -1,5 +1,6 @@
 use anyhow::Context;
 use komodo_client::entities::{
+  Operation, ResourceTarget,
   action::Action,
   alerter::Alerter,
   build::Build,
@@ -13,7 +14,6 @@ use komodo_client::entities::{
   sync::ResourceSync,
   update::{Update, UpdateListItem},
   user::User,
-  Operation, ResourceTarget,
 };
 use mungos::{
   by_id::{find_one_by_id, update_one_by_id},
@@ -263,7 +263,7 @@ pub async fn init_execution_update(
       ),
     ),
     ExecuteRequest::BatchDeploy(_data) => {
-      return Ok(Default::default())
+      return Ok(Default::default());
     }
     ExecuteRequest::PullDeployment(data) => (
       Operation::PullDeployment,
@@ -308,7 +308,7 @@ pub async fn init_execution_update(
       ),
     ),
     ExecuteRequest::BatchDestroyDeployment(_data) => {
-      return Ok(Default::default())
+      return Ok(Default::default());
     }
 
     // Build
@@ -319,7 +319,7 @@ pub async fn init_execution_update(
       ),
     ),
     ExecuteRequest::BatchRunBuild(_data) => {
-      return Ok(Default::default())
+      return Ok(Default::default());
     }
     ExecuteRequest::CancelBuild(data) => (
       Operation::CancelBuild,
@@ -336,7 +336,7 @@ pub async fn init_execution_update(
       ),
     ),
     ExecuteRequest::BatchCloneRepo(_data) => {
-      return Ok(Default::default())
+      return Ok(Default::default());
     }
     ExecuteRequest::PullRepo(data) => (
       Operation::PullRepo,
@@ -345,7 +345,7 @@ pub async fn init_execution_update(
       ),
     ),
     ExecuteRequest::BatchPullRepo(_data) => {
-      return Ok(Default::default())
+      return Ok(Default::default());
     }
     ExecuteRequest::BuildRepo(data) => (
       Operation::BuildRepo,
@@ -354,7 +354,7 @@ pub async fn init_execution_update(
       ),
     ),
     ExecuteRequest::BatchBuildRepo(_data) => {
-      return Ok(Default::default())
+      return Ok(Default::default());
     }
     ExecuteRequest::CancelRepoBuild(data) => (
       Operation::CancelRepoBuild,
@@ -371,7 +371,7 @@ pub async fn init_execution_update(
       ),
     ),
     ExecuteRequest::BatchRunProcedure(_) => {
-      return Ok(Default::default())
+      return Ok(Default::default());
     }
 
     // Action
@@ -382,7 +382,7 @@ pub async fn init_execution_update(
       ),
     ),
     ExecuteRequest::BatchRunAction(_) => {
-      return Ok(Default::default())
+      return Ok(Default::default());
     }
 
     // Server template
@@ -415,7 +415,7 @@ pub async fn init_execution_update(
       ),
     ),
     ExecuteRequest::BatchDeployStack(_data) => {
-      return Ok(Default::default())
+      return Ok(Default::default());
     }
     ExecuteRequest::DeployStackIfChanged(data) => (
       Operation::DeployStack,
@@ -424,7 +424,7 @@ pub async fn init_execution_update(
       ),
     ),
     ExecuteRequest::BatchDeployStackIfChanged(_data) => {
-      return Ok(Default::default())
+      return Ok(Default::default());
     }
     ExecuteRequest::StartStack(data) => (
       if !data.services.is_empty() {
@@ -497,7 +497,7 @@ pub async fn init_execution_update(
       ),
     ),
     ExecuteRequest::BatchDestroyStack(_data) => {
-      return Ok(Default::default())
+      return Ok(Default::default());
     }
 
     // Alerter

bin/core/src/listener/integrations/github.rs

@@ -1,4 +1,4 @@
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use axum::http::HeaderMap;
 use hex::ToHex;
 use hmac::{Hmac, Mac};

bin/core/src/listener/integrations/gitlab.rs

@@ -1,4 +1,4 @@
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use serde::Deserialize;
 
 use crate::{

bin/core/src/listener/mod.rs

@@ -1,6 +1,6 @@
 use std::sync::Arc;
 
-use axum::{http::HeaderMap, Router};
+use axum::{Router, http::HeaderMap};
 use komodo_client::entities::resource::Resource;
 use tokio::sync::Mutex;
 

bin/core/src/listener/resources.rs

@@ -22,7 +22,7 @@ use crate::{
   helpers::update::init_execution_update,
 };
 
-use super::{ListenerLockCache, ANY_BRANCH};
+use super::{ANY_BRANCH, ListenerLockCache};
 
 // =======
 //  BUILD

bin/core/src/listener/router.rs

@@ -1,4 +1,4 @@
-use axum::{extract::Path, http::HeaderMap, routing::post, Router};
+use axum::{Router, extract::Path, http::HeaderMap, routing::post};
 use komodo_client::entities::{
   action::Action, build::Build, procedure::Procedure, repo::Repo,
   resource::Resource, stack::Stack, sync::ResourceSync,
@@ -11,13 +11,13 @@ use tracing::Instrument;
 use crate::resource::KomodoResource;
 
 use super::{
+  CustomSecret, VerifyBranch, VerifySecret,
   resources::{
+    RepoWebhookOption, StackWebhookOption, SyncWebhookOption,
     handle_action_webhook, handle_build_webhook,
     handle_procedure_webhook, handle_repo_webhook,
-    handle_stack_webhook, handle_sync_webhook, RepoWebhookOption,
-    StackWebhookOption, SyncWebhookOption,
+    handle_stack_webhook, handle_sync_webhook,
   },
-  CustomSecret, VerifyBranch, VerifySecret,
 };
 
 #[derive(Deserialize)]

bin/core/src/main.rs

@@ -40,8 +40,8 @@ async fn app() -> anyhow::Result<()> {
   tokio::join!(
     // Init db_client check to crash on db init failure
     state::init_db_client(),
-    // Init default OIDC client (defined in config / env vars / compose secret file)
-    auth::oidc::client::init_default_oidc_client()
+    // Manage OIDC client (defined in config / env vars / compose secret file)
+    auth::oidc::client::spawn_oidc_client_management()
   );
   tokio::join!(
     // Maybe initialize first server

bin/core/src/monitor/alert/deployment.rs

@@ -1,9 +1,9 @@
 use std::collections::HashMap;
 
 use komodo_client::entities::{
+  ResourceTarget,
   alert::{Alert, AlertData, SeverityLevel},
   deployment::{Deployment, DeploymentState},
-  ResourceTarget,
 };
 
 use crate::{

bin/core/src/monitor/alert/mod.rs

@@ -30,8 +30,8 @@ pub async fn check_alerts(ts: i64) {
 }
 
 #[instrument(level = "debug")]
-async fn get_all_servers_map(
-) -> anyhow::Result<(HashMap<String, Server>, HashMap<String, String>)>
+async fn get_all_servers_map()
+-> anyhow::Result<(HashMap<String, Server>, HashMap<String, String>)>
 {
   let servers = resource::list_full_for_user::<Server>(
     ResourceQuery::default(),

bin/core/src/monitor/alert/server.rs

@@ -3,10 +3,10 @@ use std::{collections::HashMap, path::PathBuf, str::FromStr};
 use anyhow::Context;
 use derive_variants::ExtractVariant;
 use komodo_client::entities::{
+  ResourceTarget,
   alert::{Alert, AlertData, AlertDataVariant, SeverityLevel},
   komodo_timestamp, optional_string,
   server::{Server, ServerState},
-  ResourceTarget,
 };
 use mongo_indexed::Indexed;
 use mungos::{
@@ -85,7 +85,9 @@ pub async fn alert_servers(
             id, name, region, ..
           } => (id, name, region),
           data => {
-            error!("got incorrect alert data in ServerStatus handler. got {data:?}");
+            error!(
+              "got incorrect alert data in ServerStatus handler. got {data:?}"
+            );
             continue;
           }
         };
@@ -530,8 +532,8 @@ async fn resolve_alerts(alerts: &[(Alert, SendAlerts)]) {
 }
 
 #[instrument(level = "debug")]
-async fn get_open_alerts(
-) -> anyhow::Result<(OpenAlertMap, OpenDiskAlertMap)> {
+async fn get_open_alerts()
+-> anyhow::Result<(OpenAlertMap, OpenDiskAlertMap)> {
   let alerts = find_collect(
     &db_client().alerts,
     doc! { "resolved": false },

bin/core/src/monitor/alert/stack.rs

@@ -1,9 +1,9 @@
 use std::collections::HashMap;
 
 use komodo_client::entities::{
+  ResourceTarget,
   alert::{Alert, AlertData, SeverityLevel},
   stack::{Stack, StackState},
-  ResourceTarget,
 };
 
 use crate::{

bin/core/src/monitor/lists.rs

@@ -6,8 +6,8 @@ use komodo_client::entities::{
   stack::ComposeProject,
 };
 use periphery_client::{
-  api::{GetDockerLists, GetDockerListsResponse},
   PeripheryClient,
+  api::{GetDockerLists, GetDockerListsResponse},
 };
 
 pub async fn get_docker_lists(

bin/core/src/monitor/mod.rs

@@ -7,8 +7,7 @@ use komodo_client::entities::{
     container::ContainerListItem, image::ImageListItem,
     network::NetworkListItem, volume::VolumeListItem,
   },
-  komodo_timestamp,
-  optional_string,
+  komodo_timestamp, optional_string,
   server::{Server, ServerHealth, ServerState},
   stack::{ComposeProject, StackService, StackState},
   stats::SystemStats,

bin/core/src/monitor/record.rs

@@ -1,5 +1,5 @@
 use komodo_client::entities::stats::{
-  sum_disk_usage, SystemStatsRecord, TotalDiskUsage,
+  SystemStatsRecord, TotalDiskUsage, sum_disk_usage,
 };
 
 use crate::state::{db_client, server_status_cache};

bin/core/src/monitor/resources.rs

@@ -7,6 +7,7 @@ use anyhow::Context;
 use komodo_client::{
   api::execute::{Deploy, DeployStack},
   entities::{
+    ResourceTarget,
     alert::{Alert, AlertData, SeverityLevel},
     build::Build,
     deployment::{Deployment, DeploymentImage, DeploymentState},
@@ -17,7 +18,6 @@ use komodo_client::{
     komodo_timestamp,
     stack::{Stack, StackService, StackServiceNames, StackState},
     user::auto_redeploy_user,
-    ResourceTarget,
   },
 };
 
@@ -222,8 +222,8 @@ pub async fn update_deployment_cache(
 }
 
 /// (StackId, Service)
-fn stack_alert_sent_cache(
-) -> &'static Mutex<HashSet<(String, String)>> {
+fn stack_alert_sent_cache()
+-> &'static Mutex<HashSet<(String, String)>> {
   static CACHE: OnceLock<Mutex<HashSet<(String, String)>>> =
     OnceLock::new();
   CACHE.get_or_init(Default::default)

bin/core/src/resource/action.rs

@@ -2,6 +2,7 @@ use std::time::Duration;
 
 use anyhow::Context;
 use komodo_client::entities::{
+  Operation, ResourceTargetVariant,
   action::{
     Action, ActionConfig, ActionConfigDiff, ActionInfo,
     ActionListItem, ActionListItemInfo, ActionQuerySpecifics,
@@ -10,11 +11,10 @@ use komodo_client::entities::{
   resource::Resource,
   update::Update,
   user::User,
-  Operation, ResourceTargetVariant,
 };
 use mungos::{
   find::find_collect,
-  mongodb::{bson::doc, options::FindOneOptions, Collection},
+  mongodb::{Collection, bson::doc, options::FindOneOptions},
 };
 
 use crate::state::{action_state_cache, action_states, db_client};

bin/core/src/resource/alerter.rs

@@ -1,5 +1,6 @@
 use derive_variants::ExtractVariant;
 use komodo_client::entities::{
+  Operation, ResourceTargetVariant,
   alerter::{
     Alerter, AlerterConfig, AlerterConfigDiff, AlerterListItem,
     AlerterListItemInfo, AlerterQuerySpecifics, PartialAlerterConfig,
@@ -7,7 +8,6 @@ use komodo_client::entities::{
   resource::Resource,
   update::Update,
   user::User,
-  Operation, ResourceTargetVariant,
 };
 use mungos::mongodb::Collection;
 

bin/core/src/resource/build.rs

@@ -2,6 +2,7 @@ use std::time::Duration;
 
 use anyhow::Context;
 use komodo_client::entities::{
+  Operation, ResourceTargetVariant,
   build::{
     Build, BuildConfig, BuildConfigDiff, BuildInfo, BuildListItem,
     BuildListItemInfo, BuildQuerySpecifics, BuildState,
@@ -13,11 +14,10 @@ use komodo_client::entities::{
   resource::Resource,
   update::Update,
   user::User,
-  Operation, ResourceTargetVariant,
 };
 use mungos::{
   find::find_collect,
-  mongodb::{bson::doc, options::FindOptions, Collection},
+  mongodb::{Collection, bson::doc, options::FindOptions},
 };
 
 use crate::{

bin/core/src/resource/builder.rs

@@ -1,5 +1,6 @@
 use anyhow::Context;
 use komodo_client::entities::{
+  MergePartial, Operation, ResourceTargetVariant,
   builder::{
     Builder, BuilderConfig, BuilderConfigDiff, BuilderConfigVariant,
     BuilderListItem, BuilderListItemInfo, BuilderQuerySpecifics,
@@ -10,11 +11,10 @@ use komodo_client::entities::{
   server::Server,
   update::Update,
   user::User,
-  MergePartial, Operation, ResourceTargetVariant,
 };
 use mungos::mongodb::{
-  bson::{doc, to_document, Document},
   Collection,
+  bson::{Document, doc, to_document},
 };
 
 use crate::state::db_client;

bin/core/src/resource/deployment.rs

@@ -1,12 +1,13 @@
 use anyhow::Context;
 use formatting::format_serror;
 use komodo_client::entities::{
+  Operation, ResourceTargetVariant,
   build::Build,
   deployment::{
-    conversions_from_str, Deployment, DeploymentConfig,
-    DeploymentConfigDiff, DeploymentImage, DeploymentListItem,
-    DeploymentListItemInfo, DeploymentQuerySpecifics,
-    DeploymentState, PartialDeploymentConfig,
+    Deployment, DeploymentConfig, DeploymentConfigDiff,
+    DeploymentImage, DeploymentListItem, DeploymentListItemInfo,
+    DeploymentQuerySpecifics, DeploymentState,
+    PartialDeploymentConfig, conversions_from_str,
   },
   environment_vars_from_str,
   permission::PermissionLevel,
@@ -14,7 +15,6 @@ use komodo_client::entities::{
   server::Server,
   update::Update,
   user::User,
-  Operation, ResourceTargetVariant,
 };
 use mungos::mongodb::Collection;
 use periphery_client::api::container::RemoveContainer;

bin/core/src/resource/mod.rs

@@ -3,20 +3,20 @@ use std::{
   str::FromStr,
 };
 
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use formatting::format_serror;
-use futures::{future::join_all, FutureExt};
+use futures::{FutureExt, future::join_all};
 use komodo_client::{
   api::{read::ExportResourcesToToml, write::CreateTag},
   entities::{
+    Operation, ResourceTarget, ResourceTargetVariant,
     komodo_timestamp,
     permission::PermissionLevel,
     resource::{AddFilters, Resource, ResourceQuery},
     tag::Tag,
     to_komodo_name,
     update::Update,
-    user::{system_user, User},
-    Operation, ResourceTarget, ResourceTargetVariant,
+    user::{User, system_user},
   },
   parsers::parse_string_list,
 };
@@ -24,14 +24,14 @@ use mungos::{
   by_id::{delete_one_by_id, update_one_by_id},
   find::find_collect,
   mongodb::{
-    bson::{doc, oid::ObjectId, to_document, Document},
-    options::FindOptions,
     Collection,
+    bson::{Document, doc, oid::ObjectId, to_document},
+    options::FindOptions,
   },
 };
 use partial_derive2::{Diff, FieldDiff, MaybeNone, PartialDiff};
 use resolver_api::Resolve;
-use serde::{de::DeserializeOwned, Serialize};
+use serde::{Serialize, de::DeserializeOwned};
 
 use crate::{
   api::{read::ReadArgs, write::WriteArgs},
@@ -971,7 +971,9 @@ where
     })
     .await
   {
-    warn!("failed to delete_many permissions matching target {target:?} | {e:#}");
+    warn!(
+      "failed to delete_many permissions matching target {target:?} | {e:#}"
+    );
   }
 }
 

bin/core/src/resource/procedure.rs

@@ -1,9 +1,10 @@
 use std::time::Duration;
 
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use komodo_client::{
   api::execute::Execution,
   entities::{
+    Operation, ResourceTargetVariant,
     action::Action,
     alerter::Alerter,
     build::Build,
@@ -21,12 +22,11 @@ use komodo_client::{
     sync::ResourceSync,
     update::Update,
     user::User,
-    Operation, ResourceTargetVariant,
   },
 };
 use mungos::{
   find::find_collect,
-  mongodb::{bson::doc, options::FindOneOptions, Collection},
+  mongodb::{Collection, bson::doc, options::FindOneOptions},
 };
 
 use crate::{
@@ -173,7 +173,7 @@ async fn validate_config(
             Some(id) if procedure.id == id => {
               return Err(anyhow!(
                 "Cannot have self-referential procedure"
-              ))
+              ));
             }
             _ => {}
           }

bin/core/src/resource/refresh.rs

@@ -1,6 +1,6 @@
 use std::time::Duration;
 
-use async_timing_util::{get_timelength_in_ms, Timelength};
+use async_timing_util::{Timelength, get_timelength_in_ms};
 use komodo_client::{
   api::write::{
     RefreshBuildCache, RefreshRepoCache, RefreshResourceSyncPending,
@@ -95,7 +95,9 @@ async fn refresh_deployments() {
         if let Err(e) =
           pull_deployment_inner(deployment, server).await
         {
-          warn!("Failed to pull latest image for Deployment {name} | {e:#}");
+          warn!(
+            "Failed to pull latest image for Deployment {name} | {e:#}"
+          );
         }
       }
     }
@@ -151,15 +153,17 @@ async fn refresh_repos() {
 }
 
 async fn refresh_syncs() {
-  let Ok(syncs) =
-    find_collect(&db_client().resource_syncs, None, None)
-      .await
-      .inspect_err(|e| {
-        warn!(
+  let Ok(syncs) = find_collect(
+    &db_client().resource_syncs,
+    None,
+    None,
+  )
+  .await
+  .inspect_err(|e| {
+    warn!(
       "failed to get resource syncs from db in refresh task | {e:#}"
     )
-      })
-  else {
+  }) else {
     return;
   };
   for sync in syncs {

bin/core/src/resource/repo.rs

@@ -3,6 +3,7 @@ use std::time::Duration;
 use anyhow::Context;
 use formatting::format_serror;
 use komodo_client::entities::{
+  Operation, ResourceTargetVariant,
   builder::Builder,
   permission::PermissionLevel,
   repo::{
@@ -14,11 +15,10 @@ use komodo_client::entities::{
   to_komodo_name,
   update::Update,
   user::User,
-  Operation, ResourceTargetVariant,
 };
 use mungos::{
   find::find_collect,
-  mongodb::{bson::doc, options::FindOneOptions, Collection},
+  mongodb::{Collection, bson::doc, options::FindOneOptions},
 };
 use periphery_client::api::git::DeleteRepo;
 

bin/core/src/resource/server.rs

@@ -1,6 +1,6 @@
 use anyhow::Context;
 use komodo_client::entities::{
-  komodo_timestamp,
+  Operation, ResourceTargetVariant, komodo_timestamp,
   resource::Resource,
   server::{
     PartialServerConfig, Server, ServerConfig, ServerConfigDiff,
@@ -8,9 +8,8 @@ use komodo_client::entities::{
   },
   update::Update,
   user::User,
-  Operation, ResourceTargetVariant,
 };
-use mungos::mongodb::{bson::doc, Collection};
+use mungos::mongodb::{Collection, bson::doc};
 
 use crate::{
   config::core_config,

bin/core/src/resource/server_template.rs

@@ -1,4 +1,5 @@
 use komodo_client::entities::{
+  MergePartial, Operation, ResourceTargetVariant,
   resource::Resource,
   server_template::{
     PartialServerTemplateConfig, ServerTemplate,
@@ -8,11 +9,10 @@ use komodo_client::entities::{
   },
   update::Update,
   user::User,
-  MergePartial, Operation, ResourceTargetVariant,
 };
 use mungos::mongodb::{
-  bson::{to_document, Document},
   Collection,
+  bson::{Document, to_document},
 };
 
 use crate::state::db_client;

bin/core/src/resource/stack.rs

@@ -3,6 +3,7 @@ use formatting::format_serror;
 use komodo_client::{
   api::write::RefreshStackCache,
   entities::{
+    Operation, ResourceTargetVariant,
     permission::PermissionLevel,
     resource::Resource,
     server::Server,
@@ -12,8 +13,7 @@ use komodo_client::{
       StackQuerySpecifics, StackServiceWithUpdate, StackState,
     },
     update::Update,
-    user::{stack_user, User},
-    Operation, ResourceTargetVariant,
+    user::{User, stack_user},
   },
 };
 use mungos::mongodb::Collection;

bin/core/src/resource/sync.rs

@@ -3,7 +3,7 @@ use formatting::format_serror;
 use komodo_client::{
   api::write::RefreshResourceSyncPending,
   entities::{
-    komodo_timestamp,
+    Operation, ResourceTargetVariant, komodo_timestamp,
     resource::Resource,
     sync::{
       PartialResourceSyncConfig, ResourceSync, ResourceSyncConfig,
@@ -12,8 +12,7 @@ use komodo_client::{
       ResourceSyncState,
     },
     update::Update,
-    user::{sync_user, User},
-    Operation, ResourceTargetVariant,
+    user::{User, sync_user},
   },
 };
 use mongo_indexed::doc;

bin/core/src/stack/execute.rs

@@ -7,7 +7,7 @@ use komodo_client::{
     user::User,
   },
 };
-use periphery_client::{api::compose::*, PeripheryClient};
+use periphery_client::{PeripheryClient, api::compose::*};
 
 use crate::{
   helpers::{periphery_client, update::update_update},
@@ -76,6 +76,14 @@ pub async fn execute_compose<T: ExecuteCompose>(
   Ok(update)
 }
 
+fn service_args(services: &[String]) -> String {
+  if !services.is_empty() {
+    format!(" {}", services.join(" "))
+  } else {
+    String::new()
+  }
+}
+
 impl ExecuteCompose for StartStack {
   type Extras = ();
   async fn execute(
@@ -84,11 +92,7 @@ impl ExecuteCompose for StartStack {
     services: Vec<String>,
     _: Self::Extras,
   ) -> anyhow::Result<Log> {
-    let service_args = if !services.is_empty() {
-      services.join(" ")
-    } else {
-      String::new()
-    };
+    let service_args = service_args(&services);
     periphery
       .request(ComposeExecution {
         project: stack.project_name(false),
@@ -106,11 +110,7 @@ impl ExecuteCompose for RestartStack {
     services: Vec<String>,
     _: Self::Extras,
   ) -> anyhow::Result<Log> {
-    let service_args = if !services.is_empty() {
-      services.join(" ")
-    } else {
-      String::new()
-    };
+    let service_args = service_args(&services);
     periphery
       .request(ComposeExecution {
         project: stack.project_name(false),
@@ -128,11 +128,7 @@ impl ExecuteCompose for PauseStack {
     services: Vec<String>,
     _: Self::Extras,
   ) -> anyhow::Result<Log> {
-    let service_args = if !services.is_empty() {
-      services.join(" ")
-    } else {
-      String::new()
-    };
+    let service_args = service_args(&services);
     periphery
       .request(ComposeExecution {
         project: stack.project_name(false),
@@ -150,11 +146,7 @@ impl ExecuteCompose for UnpauseStack {
     services: Vec<String>,
     _: Self::Extras,
   ) -> anyhow::Result<Log> {
-    let service_args = if !services.is_empty() {
-      services.join(" ")
-    } else {
-      String::new()
-    };
+    let service_args = service_args(&services);
     periphery
       .request(ComposeExecution {
         project: stack.project_name(false),
@@ -172,11 +164,7 @@ impl ExecuteCompose for StopStack {
     services: Vec<String>,
     timeout: Self::Extras,
   ) -> anyhow::Result<Log> {
-    let service_args = if !services.is_empty() {
-      services.join(" ")
-    } else {
-      String::new()
-    };
+    let service_args = service_args(&services);
     let maybe_timeout = maybe_timeout(timeout);
     periphery
       .request(ComposeExecution {
@@ -195,11 +183,7 @@ impl ExecuteCompose for DestroyStack {
     services: Vec<String>,
     (timeout, remove_orphans): Self::Extras,
   ) -> anyhow::Result<Log> {
-    let service_args = if !services.is_empty() {
-      services.join(" ")
-    } else {
-      String::new()
-    };
+    let service_args = service_args(&services);
     let maybe_timeout = maybe_timeout(timeout);
     let maybe_remove_orphans = if remove_orphans {
       " --remove-orphans"

bin/core/src/stack/mod.rs

@@ -1,4 +1,4 @@
-use anyhow::{anyhow, Context};
+use anyhow::{Context, anyhow};
 use komodo_client::entities::{
   permission::PermissionLevel,
   server::{Server, ServerState},

bin/core/src/stack/remote.rs

@@ -3,7 +3,7 @@ use std::{fs, path::PathBuf};
 use anyhow::Context;
 use formatting::format_serror;
 use komodo_client::entities::{
-  stack::Stack, update::Log, CloneArgs, FileContents,
+  CloneArgs, FileContents, stack::Stack, update::Log,
 };
 
 use crate::{config::core_config, helpers::git_token};