Files
komodo/bin/core/src/api/execute/stack.rs
Maxwell Becker db1cf786ac 1.17.0 (#248)
* resolver v3

add new ec2 instance types

clean up testing config

document the libraries a bit

clean up main

update sysinfo and otel

update client resolver 3.0

resolver v3 prog

clean up gitignore

implement periphery resolver v3

clean up

core read api v3

more prog

execute api

missing apis

compiling

1.16.13

work on more granular traits

prog on crud

* fmt

* format

* resource2 not really a benefit

* axum to 0.8

* bump aws deps

* just make it 1.17.0

* clean up cors

* the komodo env file should be highest priority over additional files

* add entities / message for test alerter

* test alert implementation

* rust 1.84.0

* axum update :param to {param} syntax

* fix last axum updates

* Add test alerter button

* higher quality / colored icons

* komodo-logo

* simplify network stats

* rename Test Alerter button

* escape incoming sync backslashes (BREAKING)

* clean up rust client websocket subscription

* finish oidc comment

* show update available stack table

* update available deployment table

* feature: use the repo path instead of name in GetLatestCommit (#282)

* Update repo path handling in commit fetching

- Changed `name` to `path` for repository identification.
- Updated cache update function to use the new path field.
- Improved error message for non-directory repo paths.

* feat: use optional name and path in GetLatestCommit

* review: don't use optional for name

* review: use helper

* review: remove redundant to_string()

* 1.17.0-dev

* feature: add post_deploy command (#288)

* feature: add post_deploy command

* review: do not run post_deploy if deploy failed

* feature: interpolate secrets in custom alerter (#289)

* feature: interpolate secrets in custom alerter

* fix rust warning

* review: sanitize errors

* review: sanitize error message

* Remove .git from remote_url (#299)

Remove .git from remote_url

Co-authored-by: Deon Marshall <dmarshall@ccp.com.au>

* mbecker20 -> moghtech

* remove example from cargo toml workspace

* dev-1

* fix login screen logo

* more legible favicon

* fix new compose images

* docs new organization

* typescript subscribe_to_update_websocket

* add donate button docsite

* add config save button in desktop sidebar navigator

* add save button to config bottom

* feature: allow docker image text to overflow in table (#301)

* feature: allow docker image text to overflow in table

* review: use break-words

* wip: revert line break in css file

* feature: update devcontainer node release

* improve First Login docs

* FIx PullStack re #302 and record docker compose config on stack deploy

* requery alerts more often

* improve update indicator style and also put on home screen

* Add all services stack log

* 1.17.0-dev-2

* fix api name chnage

* choose which stack services to include in logs

* feature: improve tables quick actions on mobile (#312)

* feature: improve tables quick actions on mobile

* review: fix gap4

* review: use flex-wrap

* improve pull to git init on existing folder without .git

* Fix unclear ComposePull log re #244

* use komodo_client.subscribe_to_update_websocket, and click indicator to reconnect

* dev-3

* ServerTemplate description

* improve WriteComposeContentsToHost instrument fields

* give server stat charts labels

* filters wrap

* show provider usernames from config file

* Stack: Fix git repo new compose file initialization

* init sync file new repo

* set branch on git init folder

* ResourceSync: pending view toggle between "Execute" vs "Commit" sync direction

* Improve resource sync Execute / Pending view selector

* standardize running commands with interpolation / output sanitizations

* fix all clippy lints

* fix rand

* lock certain users username / password, prevent demo creds from being changed.

* revert to login screen whenever the call to check login fails

* ResourceSync state resolution refinement

* make sure parent directories exist whenever writing files

* don't prune images if server not enabled

* update most deps

* update openidconnect dependency, and use reqwest rustls-tls-native-roots

* dev-4

* resource sync only add escaping on toml between the """

* Stacks executions take list of services -- Auto update only redeploys services with update

* auto update all service deploy option

* dev-5 fix the stack service executions

* clean up service_args

* rust 1.85

* store sync edits on localstorage

* stack edits on localstorage and show last deployed config

* add yarn install to runfile

* Fix actions when core on https

* add update_available query parameter to filter for only stacks /deployments with available update

* rust 2024 and fmt

* rename test.compose.yaml to dev.compose.yaml, and update runfile

* update .devcontainer / dev docs for updated runfile

* use png in topbar logo, svg quality sometimes bad

* OIDC: Support PKCE auth (secret optional)

* update docs on OIDC and client secret

* cycle the oidc client on interval to ensure up to date JWKs

* add KOMODO_LOCK_LOGIN_CREDENTIALS_FOR in config doc

* update deps

* resource sync toggle resource / variable / user group inclusion independantly

* use jsonwebtoken

* improve variable value table overflow

* colored tags

* fix sync summary count ok

* default new tag colors to grey

* soften tag opacity a bit

* Update config.tsx (#358)

* isolate stacks / deployments with pending updates

* update some deps

* use Tooltip component instead of HoverCard for mobile compatibility

* batch Build builds

* link to typescript client in the intro

* add link to main docs from client docs

* doc tweaks

* use moghtech/komodo-core and moghtech/komodo-periphery as images

* remove unnecessary explicit network

* periphery.compose.yaml

* clean up periphery compose

* add link to config

* update periphery container compose config

* rust 1.85.1

* update sync docs

* 1.17.0

---------

Co-authored-by: unsync <1211591+unsync@users.noreply.github.com>
Co-authored-by: Deon Marshall <dmarshall@ccp.com.au>
Co-authored-by: komodo <komodo@komo.do>
Co-authored-by: wlatic <jamesoh@gmail.com>
2025-03-23 16:47:06 -07:00

647 lines
17 KiB
Rust

use std::collections::HashSet;
use anyhow::Context;
use formatting::format_serror;
use komodo_client::{
api::{execute::*, write::RefreshStackCache},
entities::{
permission::PermissionLevel,
server::Server,
stack::{Stack, StackInfo},
update::{Log, Update},
},
};
use mungos::mongodb::bson::{doc, to_document};
use periphery_client::api::compose::*;
use resolver_api::Resolve;
use crate::{
api::write::WriteArgs,
helpers::{
interpolate::{
add_interp_update_log,
interpolate_variables_secrets_into_extra_args,
interpolate_variables_secrets_into_string,
interpolate_variables_secrets_into_system_command,
},
periphery_client,
query::get_variables_and_secrets,
update::{add_update_without_send, update_update},
},
monitor::update_cache_for_server,
resource,
stack::{execute::execute_compose, get_stack_and_server},
state::{action_states, db_client},
};
use super::{ExecuteArgs, ExecuteRequest};
impl super::BatchExecute for BatchDeployStack {
type Resource = Stack;
fn single_request(stack: String) -> ExecuteRequest {
ExecuteRequest::DeployStack(DeployStack {
stack,
services: Vec::new(),
stop_time: None,
})
}
}
impl Resolve<ExecuteArgs> for BatchDeployStack {
#[instrument(name = "BatchDeployStack", skip(user), fields(user_id = user.id))]
async fn resolve(
self,
ExecuteArgs { user, .. }: &ExecuteArgs,
) -> serror::Result<BatchExecutionResponse> {
Ok(
super::batch_execute::<BatchDeployStack>(&self.pattern, user)
.await?,
)
}
}
impl Resolve<ExecuteArgs> for DeployStack {
#[instrument(name = "DeployStack", skip(user, update), fields(user_id = user.id, update_id = update.id))]
async fn resolve(
self,
ExecuteArgs { user, update }: &ExecuteArgs,
) -> serror::Result<Update> {
let (mut stack, server) = get_stack_and_server(
&self.stack,
user,
PermissionLevel::Execute,
true,
)
.await?;
// get the action state for the stack (or insert default).
let action_state =
action_states().stack.get_or_insert_default(&stack.id).await;
// Will check to ensure stack not already busy before updating, and return Err if so.
// The returned guard will set the action state back to default when dropped.
let _action_guard =
action_state.update(|state| state.deploying = true)?;
let mut update = update.clone();
update_update(update.clone()).await?;
if !self.services.is_empty() {
update.logs.push(Log::simple(
"Service/s",
format!(
"Execution requested for Stack service/s {}",
self.services.join(", ")
),
))
}
let git_token = crate::helpers::git_token(
&stack.config.git_provider,
&stack.config.git_account,
|https| stack.config.git_https = https,
).await.with_context(
|| format!("Failed to get git token in call to db. Stopping run. | {} | {}", stack.config.git_provider, stack.config.git_account),
)?;
let registry_token = crate::helpers::registry_token(
&stack.config.registry_provider,
&stack.config.registry_account,
).await.with_context(
|| format!("Failed to get registry token in call to db. Stopping run. | {} | {}", stack.config.registry_provider, stack.config.registry_account),
)?;
// interpolate variables / secrets, returning the sanitizing replacers to send to
// periphery so it may sanitize the final command for safe logging (avoids exposing secret values)
let secret_replacers = if !stack.config.skip_secret_interp {
let vars_and_secrets = get_variables_and_secrets().await?;
let mut global_replacers = HashSet::new();
let mut secret_replacers = HashSet::new();
interpolate_variables_secrets_into_string(
&vars_and_secrets,
&mut stack.config.file_contents,
&mut global_replacers,
&mut secret_replacers,
)?;
interpolate_variables_secrets_into_string(
&vars_and_secrets,
&mut stack.config.environment,
&mut global_replacers,
&mut secret_replacers,
)?;
interpolate_variables_secrets_into_extra_args(
&vars_and_secrets,
&mut stack.config.extra_args,
&mut global_replacers,
&mut secret_replacers,
)?;
interpolate_variables_secrets_into_extra_args(
&vars_and_secrets,
&mut stack.config.build_extra_args,
&mut global_replacers,
&mut secret_replacers,
)?;
interpolate_variables_secrets_into_system_command(
&vars_and_secrets,
&mut stack.config.pre_deploy,
&mut global_replacers,
&mut secret_replacers,
)?;
interpolate_variables_secrets_into_system_command(
&vars_and_secrets,
&mut stack.config.post_deploy,
&mut global_replacers,
&mut secret_replacers,
)?;
add_interp_update_log(
&mut update,
&global_replacers,
&secret_replacers,
);
secret_replacers
} else {
Default::default()
};
let ComposeUpResponse {
logs,
deployed,
services,
file_contents,
missing_files,
remote_errors,
compose_config,
commit_hash,
commit_message,
} = periphery_client(&server)?
.request(ComposeUp {
stack: stack.clone(),
services: self.services,
git_token,
registry_token,
replacers: secret_replacers.into_iter().collect(),
})
.await?;
update.logs.extend(logs);
let update_info = async {
let latest_services = if services.is_empty() {
// maybe better to do something else here for services.
stack.info.latest_services.clone()
} else {
services
};
// This ensures to get the latest project name,
// as it may have changed since the last deploy.
let project_name = stack.project_name(true);
let (
deployed_services,
deployed_contents,
deployed_config,
deployed_hash,
deployed_message,
) = if deployed {
(
Some(latest_services.clone()),
Some(file_contents.clone()),
compose_config,
commit_hash.clone(),
commit_message.clone(),
)
} else {
(
stack.info.deployed_services,
stack.info.deployed_contents,
stack.info.deployed_config,
stack.info.deployed_hash,
stack.info.deployed_message,
)
};
let info = StackInfo {
missing_files,
deployed_project_name: project_name.into(),
deployed_services,
deployed_contents,
deployed_config,
deployed_hash,
deployed_message,
latest_services,
remote_contents: stack
.config
.file_contents
.is_empty()
.then_some(file_contents),
remote_errors: stack
.config
.file_contents
.is_empty()
.then_some(remote_errors),
latest_hash: commit_hash,
latest_message: commit_message,
};
let info = to_document(&info)
.context("failed to serialize stack info to bson")?;
db_client()
.stacks
.update_one(
doc! { "name": &stack.name },
doc! { "$set": { "info": info } },
)
.await
.context("failed to update stack info on db")?;
anyhow::Ok(())
};
// This will be weird with single service deploys. Come back to it.
if let Err(e) = update_info.await {
update.push_error_log(
"refresh stack info",
format_serror(
&e.context("failed to refresh stack info on db").into(),
),
)
}
// Ensure cached stack state up to date by updating server cache
update_cache_for_server(&server).await;
update.finalize();
update_update(update.clone()).await?;
Ok(update)
}
}
impl super::BatchExecute for BatchDeployStackIfChanged {
type Resource = Stack;
fn single_request(stack: String) -> ExecuteRequest {
ExecuteRequest::DeployStackIfChanged(DeployStackIfChanged {
stack,
stop_time: None,
})
}
}
impl Resolve<ExecuteArgs> for BatchDeployStackIfChanged {
#[instrument(name = "BatchDeployStackIfChanged", skip(user), fields(user_id = user.id))]
async fn resolve(
self,
ExecuteArgs { user, .. }: &ExecuteArgs,
) -> serror::Result<BatchExecutionResponse> {
Ok(
super::batch_execute::<BatchDeployStackIfChanged>(
&self.pattern,
user,
)
.await?,
)
}
}
impl Resolve<ExecuteArgs> for DeployStackIfChanged {
#[instrument(name = "DeployStackIfChanged", skip(user, update), fields(user_id = user.id))]
async fn resolve(
self,
ExecuteArgs { user, update }: &ExecuteArgs,
) -> serror::Result<Update> {
let stack = resource::get_check_permissions::<Stack>(
&self.stack,
user,
PermissionLevel::Execute,
)
.await?;
RefreshStackCache {
stack: stack.id.clone(),
}
.resolve(&WriteArgs { user: user.clone() })
.await?;
let stack = resource::get::<Stack>(&stack.id).await?;
let changed = match (
&stack.info.deployed_contents,
&stack.info.remote_contents,
) {
(Some(deployed_contents), Some(latest_contents)) => {
let changed = || {
for latest in latest_contents {
let Some(deployed) = deployed_contents
.iter()
.find(|c| c.path == latest.path)
else {
return true;
};
if latest.contents != deployed.contents {
return true;
}
}
false
};
changed()
}
(None, _) => true,
_ => false,
};
let mut update = update.clone();
if !changed {
update.push_simple_log(
"Diff compose files",
String::from("Deploy cancelled after no changes detected."),
);
update.finalize();
return Ok(update);
}
// Don't actually send it here, let the handler send it after it can set action state.
// This is usually done in crate::helpers::update::init_execution_update.
update.id = add_update_without_send(&update).await?;
DeployStack {
stack: stack.name,
services: Vec::new(),
stop_time: self.stop_time,
}
.resolve(&ExecuteArgs {
user: user.clone(),
update,
})
.await
}
}
pub async fn pull_stack_inner(
mut stack: Stack,
services: Vec<String>,
server: &Server,
mut update: Option<&mut Update>,
) -> anyhow::Result<ComposePullResponse> {
if let Some(update) = update.as_mut() {
if !services.is_empty() {
update.logs.push(Log::simple(
"Service/s",
format!(
"Execution requested for Stack service/s {}",
services.join(", ")
),
))
}
}
let git_token = crate::helpers::git_token(
&stack.config.git_provider,
&stack.config.git_account,
|https| stack.config.git_https = https,
).await.with_context(
|| format!("Failed to get git token in call to db. Stopping run. | {} | {}", stack.config.git_provider, stack.config.git_account),
)?;
let registry_token = crate::helpers::registry_token(
&stack.config.registry_provider,
&stack.config.registry_account,
).await.with_context(
|| format!("Failed to get registry token in call to db. Stopping run. | {} | {}", stack.config.registry_provider, stack.config.registry_account),
)?;
// interpolate variables / secrets
if !stack.config.skip_secret_interp {
let vars_and_secrets = get_variables_and_secrets().await?;
let mut global_replacers = HashSet::new();
let mut secret_replacers = HashSet::new();
interpolate_variables_secrets_into_string(
&vars_and_secrets,
&mut stack.config.file_contents,
&mut global_replacers,
&mut secret_replacers,
)?;
interpolate_variables_secrets_into_string(
&vars_and_secrets,
&mut stack.config.environment,
&mut global_replacers,
&mut secret_replacers,
)?;
if let Some(update) = update {
add_interp_update_log(
update,
&global_replacers,
&secret_replacers,
);
}
};
let res = periphery_client(server)?
.request(ComposePull {
stack,
services,
git_token,
registry_token,
})
.await?;
// Ensure cached stack state up to date by updating server cache
update_cache_for_server(server).await;
Ok(res)
}
impl Resolve<ExecuteArgs> for PullStack {
#[instrument(name = "PullStack", skip(user, update), fields(user_id = user.id))]
async fn resolve(
self,
ExecuteArgs { user, update }: &ExecuteArgs,
) -> serror::Result<Update> {
let (stack, server) = get_stack_and_server(
&self.stack,
user,
PermissionLevel::Execute,
true,
)
.await?;
// get the action state for the stack (or insert default).
let action_state =
action_states().stack.get_or_insert_default(&stack.id).await;
// Will check to ensure stack not already busy before updating, and return Err if so.
// The returned guard will set the action state back to default when dropped.
let _action_guard =
action_state.update(|state| state.pulling = true)?;
let mut update = update.clone();
update_update(update.clone()).await?;
let res = pull_stack_inner(
stack,
self.services,
&server,
Some(&mut update),
)
.await?;
update.logs.extend(res.logs);
update.finalize();
update_update(update.clone()).await?;
Ok(update)
}
}
impl Resolve<ExecuteArgs> for StartStack {
#[instrument(name = "StartStack", skip(user, update), fields(user_id = user.id))]
async fn resolve(
self,
ExecuteArgs { user, update }: &ExecuteArgs,
) -> serror::Result<Update> {
execute_compose::<StartStack>(
&self.stack,
self.services,
user,
|state| state.starting = true,
update.clone(),
(),
)
.await
.map_err(Into::into)
}
}
impl Resolve<ExecuteArgs> for RestartStack {
#[instrument(name = "RestartStack", skip(user, update), fields(user_id = user.id))]
async fn resolve(
self,
ExecuteArgs { user, update }: &ExecuteArgs,
) -> serror::Result<Update> {
execute_compose::<RestartStack>(
&self.stack,
self.services,
user,
|state| {
state.restarting = true;
},
update.clone(),
(),
)
.await
.map_err(Into::into)
}
}
impl Resolve<ExecuteArgs> for PauseStack {
#[instrument(name = "PauseStack", skip(user, update), fields(user_id = user.id, update_id = update.id))]
async fn resolve(
self,
ExecuteArgs { user, update }: &ExecuteArgs,
) -> serror::Result<Update> {
execute_compose::<PauseStack>(
&self.stack,
self.services,
user,
|state| state.pausing = true,
update.clone(),
(),
)
.await
.map_err(Into::into)
}
}
impl Resolve<ExecuteArgs> for UnpauseStack {
#[instrument(name = "UnpauseStack", skip(user, update), fields(user_id = user.id, update_id = update.id))]
async fn resolve(
self,
ExecuteArgs { user, update }: &ExecuteArgs,
) -> serror::Result<Update> {
execute_compose::<UnpauseStack>(
&self.stack,
self.services,
user,
|state| state.unpausing = true,
update.clone(),
(),
)
.await
.map_err(Into::into)
}
}
impl Resolve<ExecuteArgs> for StopStack {
#[instrument(name = "StopStack", skip(user, update), fields(user_id = user.id, update_id = update.id))]
async fn resolve(
self,
ExecuteArgs { user, update }: &ExecuteArgs,
) -> serror::Result<Update> {
execute_compose::<StopStack>(
&self.stack,
self.services,
user,
|state| state.stopping = true,
update.clone(),
self.stop_time,
)
.await
.map_err(Into::into)
}
}
impl super::BatchExecute for BatchDestroyStack {
type Resource = Stack;
fn single_request(stack: String) -> ExecuteRequest {
ExecuteRequest::DestroyStack(DestroyStack {
stack,
services: Vec::new(),
remove_orphans: false,
stop_time: None,
})
}
}
impl Resolve<ExecuteArgs> for BatchDestroyStack {
#[instrument(name = "BatchDestroyStack", skip(user), fields(user_id = user.id))]
async fn resolve(
self,
ExecuteArgs { user, .. }: &ExecuteArgs,
) -> serror::Result<BatchExecutionResponse> {
super::batch_execute::<BatchDestroyStack>(&self.pattern, user)
.await
.map_err(Into::into)
}
}
impl Resolve<ExecuteArgs> for DestroyStack {
#[instrument(name = "DestroyStack", skip(user, update), fields(user_id = user.id, update_id = update.id))]
async fn resolve(
self,
ExecuteArgs { user, update }: &ExecuteArgs,
) -> serror::Result<Update> {
execute_compose::<DestroyStack>(
&self.stack,
self.services,
user,
|state| state.destroying = true,
update.clone(),
(self.stop_time, self.remove_orphans),
)
.await
.map_err(Into::into)
}
}