Files
komodo/bin/core/src/api/read/toml.rs
Maxwell Becker 2fa9d9ecce 1.19.0 (#722)
* start 1.18.5

* prevent empty additional permission check (ie for new resources)

* dev-2

* bump rust to 1.88

* tweaks

* repo based stack commit happens from core repo cache rather than on server to simplify

* clippy auto fix

* clippy lints periphery

* clippy fix komodo_client

* dev-3

* emphasize ferret version pinning

* bump svi with PR fix

* dev-4

* webhook disabled early return

* Fix missing alert types for whitelist

* add "ScheduleRun"

* fix status cache not cleaning on resource delete

* dev-5

* forgot to pipe through poll in previous refactor

* refetch given in ms

* fix configure build extra args

* reorder resource sync config

* Implement ability to run actions at startup (#664)

* Implement ability to run actions at startup

* run post-startup actions after server is listening

* startup use action query

* fmt

* Fix Google Login enabled message (#668)

- it was showing "Github Login" instead of "Google Login"

* Allow CIDR ranges in Allowed IPs (#666)

* Allow CIDR ranges in Allowed IPs

* Catch mixed IPv4/IPv6 mappings that are probably intended to match

* forgiving vec

* dev-6

* forgiving vec log. allowed ips docs

* server stats UI: move current disk breakdown above charts

* searchable container stats, toggle collaple container / disk sections

* Add Clear repo cache method

* fix execute usage docs

* Komodo managed env-file should take precedence in all cases (ie come last in env file list)

* tag include unused flag for future use

* combine users page search

* util backup / restore

* refactor backup/restore duplication

* cleanup restore

* core image include util binary

* dev-7

* back to LinesCodec

* dev-8

* clean up

* clean up logs

* rename to komodo-util

* dev-9

* enable_fance_toml

* dev-10 enable fancy toml

* add user agent to oidc requests (#701)

Co-authored-by: eleith <online-github@eleith.com>

* fmt

* use database library

* clippy lint

* consolidate and standardize cli

* dev-11

* dev-12 implement backup using cli

* dev-13 logs

* command variant fields need to be #[arg]

* tweak cli

* gen client

* fix terminal reconnect issue

* rename cli to `km`

* tweaks for the cli logs

* wait for enter on --yes empty println

* fix --yes

* dev-15

* bump deps

* update croner to latest, use static parser

* dev-16

* cli execute polls updates until complete before logging

* remove repo cache mount

* cli nice

* /backup -> /backups

* dev-17 config loading preserves CONFIG_PATHS precedence

* update dockerfile default docker cli config keywords

* dev-18

* support .kmignore

* add ignores log

* Implement automatic backup pruning, default 14 backups before prune

* db copy / restore uses idempotent upsert

* cli update variable - "km set var VAR value"

* improve cli initial logs

* time the executions

* implement update for most resources

* dev 20

* add update page

* dev 21 support cli update link

* dev-22 test the deploy

* dev-23 use indexmap

* install-cli.py

* Frontend mobile fixes (#714)

* Allow ResourcePageHeader items to wrap

* Allow CardHeader items to wrap

* Increase z-index of sticky TableHeader, fixes #690

* Remove fixed widths from ActionButton, let them flex more to fit more layouts

* Make Section scroll overflow

* Remove grid class from Tabs, seems to prevent them from overflowing at small sizes

* deploy 1.18.5-dev-24

* auto version increment and deploy

* cli: profiles support aliases and merge on top of Default (root) config

* fix page set titles

* rust 1.89 and improve config logs

* skip serializing for proper merge

* fix clippy lints re 1.89

* remove layouts overflow-x-scroll

* deploy 1.18.5-dev-25

* 1.89 docker images not ready yet

* km cfg -a (print all profiles)

* include commit variables

* skip serializing profiles when empty

* skip serialize default db / log configs

* km cfg --debug print mode

* correct defaults for CLI and only can pass restore folder from cli arg

* some more skip serialization

* db restore / copy index optional

* add runfile command aliases

* remove second schedule updating loop, can causes some schedules to be missed

* deploy 1.18.5-dev-26

* add log when target db indexing disabled

* cli: user password reset, update user super admin

* Add manual network interface configuration for multi-NIC Docker environments (#719)

* Add iproute2 to debian-debs

* feat: Add manual network interface configuration for multi-NIC support

Complete implementation of manual interface configuration:
- Add internet_interface config option
- Implement manual gateway routing
- Add NET_ADMIN capability requirement
- Clean up codebase changes

* fix: Update internet interface handling for multi-NIC support

* refactor: Enhance error messages and logging in networking module

* refactor: Simplify interface argument handling and improve logging in network configuration and cleanup

* refactor(network): simplify startup integration and improve error handling

- Move config access and error handling into network::configure_internet_gateway()
- Simplify startup.rs to single function call without parameters
- Remove redundant check_network_privileges() function
- Improve error handling by checking actual command output instead of pre-validation
- Better separation of concerns between startup and network modules

Addresses feedback from PR discussion:
https://github.com/moghtech/komodo/pull/719#discussion_r2261542921

* fix(config): update default internet interface setting
Addresses feedback from PR discussion:
https://github.com/moghtech/komodo/pull/719#discussion_r2261552279

* fix(config): remove custom default for internet interface in CoreConfig

* move mod.rs -> network.rs
Addresses feedback from PR discussion:
https://github.com/moghtech/komodo/pull/719#discussion_r2261558332

* add internet interface example

* docs(build-images): document multi-platform builds with Docker Buildx (#721)

* docs(build-images): add multi-platform buildx guide to builders.md

* docs(build-images): add multi-platform buildx guide and clarify platform selection in Komodo UI Extra Args field

* move to 1.19.0

* core support reading from multiple config files

* config support yaml

* deploy 1.19.0-dev-1

* deploy 1.19.0-dev-2

* add default komodo cli config

* better config merge with base

* no need to panic if empty config paths

* improve km --help

* prog on cli docs

* tweak cli docs

* tweak doc

* split the runfile commands

* update docsite deps

* km ps initial

* km ls

* list resource apis

* km con inspect

* deploy 1.19.0-dev-3

* fix: need serde default

* dev-4 fix container parsing issue

* tweak

* use include-based file finding for much faster discovery

* just move to standard config dir .config/komodo/komodo.cli.*

* update fe w/ new contianer info minimal serialization

* add links to table names

* deploy 1.19.0-dev-5

* links in tables

* backend for Action arguments

* deploy 1.19.0-dev-6

* deploy 1.19.0-dev-7

* deploy 1.19.0-dev-8

* no space at front of KeyValue default args

* webhook branch / body optional

* The incoming arguments

* deploy 1.19.0-dev-9

* con -> cn

* add config -> cf alias

* .kmignore

* .peripheryinclude

* outdated

* optional links, configurable table format

* table_format -> table_borders

* get types

* include docsite in yarn install

* update runnables command in docs

* tweak

* improve km ls only show important stuff

* Add BackupCoreDatabase

* deploy 1.19.0-dev-10

* backup command needs "--yes"

* deploy 1.19.0-dev-11

* update rustc 1.89.0

* cli tweak

* try chef

* Fix chef (after dependencies)

* try other compile command

* fix

* fix comment

* cleanup stats page

* ensure database backup procedure

* UI allow configure Backup Core Database in Procedures

* procedure description

* deploy 1.19.0-dev-12

* deploy 1.19.0-dev-13

* GlobalAutoUpdate

* deploy 1.19.0-dev-14

* default tags and global auto update procedure

* deploy 1.19.0-dev-15

* trim the default procedure descriptions

* deploy 1.19.0-dev-16

* in "system" theme, also poll for updates to the theme based on time.

* Add next run to Action / Procedure column

* km ls support filter by templates

* fix procedure toml serialization when params = {}

* deploy 1.19.0-dev-17

* KOMODO_INIT_ADMIN_USERNAME

* KOMODO_FIRST_SERVER_NAME

* add server.config.external_address for use with links

* deploy 1.19.0-dev-18

* improve auto prune

* fix system theme auto update

* deploy 1.19.0-dev-19

* rename auth/CreateLocalUser -> SignUpLocalUser. Add write/CreateLocalUser for in-ui initialization.

* deploy 1.19.0-dev-20

* UI can handle multiple active logins

* deploy 1.19.0-dev-21

* fix

* add logout function

* fix oauth redirect

* fix multi user exchange token function

* default external address

* just Add

* style account switcher

* backup and restore docs

* rework docsite file / sidebar structure, start auto update docs

* auto update docs

* tweak

* fix doc links

* only pull / update running stacks / deployments images

* deploy 1.19.0-dev-22

* deploy 1.19.0-dev-23

* fix #737

* community docs

* add BackupCoreDatabase link to docs

* update ferret v2 update guide using komodo-cli

* fix data table headers overlapping topbar

* don't alert when deploying

* CommitSync returns Update

* deploy 1.19.0-dev-24

* trim the decoded branch

* action uses file contents deserializer

* deploy 1.19.0-dev-25

* remove Toml from action args format

* clarify External Address purpose

* Fix podman compatibility in `get_container_stats` (#739)

* Add podman compability for querying stats

Podman and docker stats differ in results in significant ways but this filter change they will output the same stats

* syntax fix

* feat(dashboard): display CPU, memory, and disk usage on server cards (#729)

* feat: mini-stats-card: Expose Server CPU , Memory, Disk Usage to Dashboard View

* comment: resolved

* Feat: fix overflow card , DRY stats-mini, add unreachable mini stats

* lint: fix

* deploy 1.19.0-dev-26

* 1.19.0

* linux, macos container install

* cli main config

---------

Co-authored-by: Brian Bradley <brian.bradley.p@gmail.com>
Co-authored-by: Daniel <daniel.barabasa@gmail.com>
Co-authored-by: eleith <eleith@users.noreply.github.com>
Co-authored-by: eleith <online-github@eleith.com>
Co-authored-by: Sam Edwards <sam@samedwards.ca>
Co-authored-by: Marcel Pfennig <82059270+MP-Tool@users.noreply.github.com>
Co-authored-by: itsmesid <693151+arevindh@users.noreply.github.com>
Co-authored-by: mbecker20 <max@mogh.tech>
Co-authored-by: Rhyn <Rhyn@users.noreply.github.com>
Co-authored-by: Anh Nguyen <tuananh131001@gmail.com>
2025-08-17 17:25:45 -07:00

517 lines
13 KiB
Rust

use anyhow::Context;
use database::mungos::find::find_collect;
use komodo_client::{
api::read::{
ExportAllResourcesToToml, ExportAllResourcesToTomlResponse,
ExportResourcesToToml, ExportResourcesToTomlResponse,
ListUserGroups,
},
entities::{
ResourceTarget, action::Action, alerter::Alerter, build::Build,
builder::Builder, deployment::Deployment,
permission::PermissionLevel, procedure::Procedure, repo::Repo,
resource::ResourceQuery, server::Server, stack::Stack,
sync::ResourceSync, toml::ResourcesToml, user::User,
},
};
use resolver_api::Resolve;
use crate::{
helpers::query::{
get_all_tags, get_id_to_tags, get_user_user_group_ids,
},
permission::get_check_permissions,
resource,
state::db_client,
sync::{
toml::{ToToml, convert_resource},
user_groups::{convert_user_groups, user_group_to_toml},
variables::variable_to_toml,
},
};
use super::ReadArgs;
async fn get_all_targets(
tags: &[String],
user: &User,
) -> anyhow::Result<Vec<ResourceTarget>> {
let mut targets = Vec::<ResourceTarget>::new();
let all_tags = if tags.is_empty() {
vec![]
} else {
get_all_tags(None).await?
};
targets.extend(
resource::list_full_for_user::<Alerter>(
ResourceQuery::builder().tags(tags).build(),
user,
PermissionLevel::Read.into(),
&all_tags,
)
.await?
.into_iter()
.map(|resource| ResourceTarget::Alerter(resource.id)),
);
targets.extend(
resource::list_full_for_user::<Builder>(
ResourceQuery::builder().tags(tags).build(),
user,
PermissionLevel::Read.into(),
&all_tags,
)
.await?
.into_iter()
.map(|resource| ResourceTarget::Builder(resource.id)),
);
targets.extend(
resource::list_full_for_user::<Server>(
ResourceQuery::builder().tags(tags).build(),
user,
PermissionLevel::Read.into(),
&all_tags,
)
.await?
.into_iter()
.map(|resource| ResourceTarget::Server(resource.id)),
);
targets.extend(
resource::list_full_for_user::<Stack>(
ResourceQuery::builder().tags(tags).build(),
user,
PermissionLevel::Read.into(),
&all_tags,
)
.await?
.into_iter()
.map(|resource| ResourceTarget::Stack(resource.id)),
);
targets.extend(
resource::list_full_for_user::<Deployment>(
ResourceQuery::builder().tags(tags).build(),
user,
PermissionLevel::Read.into(),
&all_tags,
)
.await?
.into_iter()
.map(|resource| ResourceTarget::Deployment(resource.id)),
);
targets.extend(
resource::list_full_for_user::<Build>(
ResourceQuery::builder().tags(tags).build(),
user,
PermissionLevel::Read.into(),
&all_tags,
)
.await?
.into_iter()
.map(|resource| ResourceTarget::Build(resource.id)),
);
targets.extend(
resource::list_full_for_user::<Repo>(
ResourceQuery::builder().tags(tags).build(),
user,
PermissionLevel::Read.into(),
&all_tags,
)
.await?
.into_iter()
.map(|resource| ResourceTarget::Repo(resource.id)),
);
targets.extend(
resource::list_full_for_user::<Procedure>(
ResourceQuery::builder().tags(tags).build(),
user,
PermissionLevel::Read.into(),
&all_tags,
)
.await?
.into_iter()
.map(|resource| ResourceTarget::Procedure(resource.id)),
);
targets.extend(
resource::list_full_for_user::<Action>(
ResourceQuery::builder().tags(tags).build(),
user,
PermissionLevel::Read.into(),
&all_tags,
)
.await?
.into_iter()
.map(|resource| ResourceTarget::Action(resource.id)),
);
targets.extend(
resource::list_full_for_user::<ResourceSync>(
ResourceQuery::builder().tags(tags).build(),
user,
PermissionLevel::Read.into(),
&all_tags,
)
.await?
.into_iter()
// These will already be filtered by [ExportResourcesToToml]
.map(|resource| ResourceTarget::ResourceSync(resource.id)),
);
Ok(targets)
}
impl Resolve<ReadArgs> for ExportAllResourcesToToml {
async fn resolve(
self,
args: &ReadArgs,
) -> serror::Result<ExportAllResourcesToTomlResponse> {
let targets = if self.include_resources {
get_all_targets(&self.tags, &args.user).await?
} else {
Vec::new()
};
let user_groups = if self.include_user_groups {
if args.user.admin {
find_collect(&db_client().user_groups, None, None)
.await
.context("failed to query db for user groups")?
.into_iter()
.map(|user_group| user_group.id)
.collect()
} else {
get_user_user_group_ids(&args.user.id).await?
}
} else {
Vec::new()
};
ExportResourcesToToml {
targets,
user_groups,
include_variables: self.include_variables,
}
.resolve(args)
.await
}
}
impl Resolve<ReadArgs> for ExportResourcesToToml {
async fn resolve(
self,
args: &ReadArgs,
) -> serror::Result<ExportResourcesToTomlResponse> {
let ExportResourcesToToml {
targets,
user_groups,
include_variables,
} = self;
let mut res = ResourcesToml::default();
let id_to_tags = get_id_to_tags(None).await?;
let ReadArgs { user } = args;
for target in targets {
match target {
ResourceTarget::Alerter(id) => {
let mut alerter = get_check_permissions::<Alerter>(
&id,
user,
PermissionLevel::Read.into(),
)
.await?;
Alerter::replace_ids(&mut alerter);
res.alerters.push(convert_resource::<Alerter>(
alerter,
false,
vec![],
&id_to_tags,
))
}
ResourceTarget::ResourceSync(id) => {
let mut sync = get_check_permissions::<ResourceSync>(
&id,
user,
PermissionLevel::Read.into(),
)
.await?;
if sync.config.file_contents.is_empty()
&& (sync.config.files_on_host
|| !sync.config.repo.is_empty()
|| !sync.config.linked_repo.is_empty())
{
ResourceSync::replace_ids(&mut sync);
res.resource_syncs.push(convert_resource::<ResourceSync>(
sync,
false,
vec![],
&id_to_tags,
))
}
}
ResourceTarget::Server(id) => {
let mut server = get_check_permissions::<Server>(
&id,
user,
PermissionLevel::Read.into(),
)
.await?;
Server::replace_ids(&mut server);
res.servers.push(convert_resource::<Server>(
server,
false,
vec![],
&id_to_tags,
))
}
ResourceTarget::Builder(id) => {
let mut builder = get_check_permissions::<Builder>(
&id,
user,
PermissionLevel::Read.into(),
)
.await?;
Builder::replace_ids(&mut builder);
res.builders.push(convert_resource::<Builder>(
builder,
false,
vec![],
&id_to_tags,
))
}
ResourceTarget::Build(id) => {
let mut build = get_check_permissions::<Build>(
&id,
user,
PermissionLevel::Read.into(),
)
.await?;
Build::replace_ids(&mut build);
res.builds.push(convert_resource::<Build>(
build,
false,
vec![],
&id_to_tags,
))
}
ResourceTarget::Deployment(id) => {
let mut deployment = get_check_permissions::<Deployment>(
&id,
user,
PermissionLevel::Read.into(),
)
.await?;
Deployment::replace_ids(&mut deployment);
res.deployments.push(convert_resource::<Deployment>(
deployment,
false,
vec![],
&id_to_tags,
))
}
ResourceTarget::Repo(id) => {
let mut repo = get_check_permissions::<Repo>(
&id,
user,
PermissionLevel::Read.into(),
)
.await?;
Repo::replace_ids(&mut repo);
res.repos.push(convert_resource::<Repo>(
repo,
false,
vec![],
&id_to_tags,
))
}
ResourceTarget::Stack(id) => {
let mut stack = get_check_permissions::<Stack>(
&id,
user,
PermissionLevel::Read.into(),
)
.await?;
Stack::replace_ids(&mut stack);
res.stacks.push(convert_resource::<Stack>(
stack,
false,
vec![],
&id_to_tags,
))
}
ResourceTarget::Procedure(id) => {
let mut procedure = get_check_permissions::<Procedure>(
&id,
user,
PermissionLevel::Read.into(),
)
.await?;
Procedure::replace_ids(&mut procedure);
res.procedures.push(convert_resource::<Procedure>(
procedure,
false,
vec![],
&id_to_tags,
));
}
ResourceTarget::Action(id) => {
let mut action = get_check_permissions::<Action>(
&id,
user,
PermissionLevel::Read.into(),
)
.await?;
Action::replace_ids(&mut action);
res.actions.push(convert_resource::<Action>(
action,
false,
vec![],
&id_to_tags,
));
}
ResourceTarget::System(_) => continue,
};
}
add_user_groups(user_groups, &mut res, args)
.await
.context("failed to add user groups")?;
if include_variables {
res.variables =
find_collect(&db_client().variables, None, None)
.await
.context("failed to get variables from db")?
.into_iter()
.map(|mut variable| {
if !user.admin && variable.is_secret {
variable.value = "#".repeat(variable.value.len())
}
variable
})
.collect();
}
let toml = serialize_resources_toml(res)
.context("failed to serialize resources to toml")?;
Ok(ExportResourcesToTomlResponse { toml })
}
}
async fn add_user_groups(
user_groups: Vec<String>,
res: &mut ResourcesToml,
args: &ReadArgs,
) -> anyhow::Result<()> {
let user_groups = ListUserGroups {}
.resolve(args)
.await
.map_err(|e| e.error)?
.into_iter()
.filter(|ug| {
user_groups.contains(&ug.name) || user_groups.contains(&ug.id)
});
let mut ug = Vec::with_capacity(user_groups.size_hint().0);
convert_user_groups(user_groups, &mut ug).await?;
res.user_groups = ug.into_iter().map(|ug| ug.1).collect();
Ok(())
}
fn serialize_resources_toml(
resources: ResourcesToml,
) -> anyhow::Result<String> {
let mut toml = String::new();
for server in resources.servers {
if !toml.is_empty() {
toml.push_str("\n\n##\n\n");
}
toml.push_str("[[server]]\n");
Server::push_to_toml_string(server, &mut toml)?;
}
for stack in resources.stacks {
if !toml.is_empty() {
toml.push_str("\n\n##\n\n");
}
toml.push_str("[[stack]]\n");
Stack::push_to_toml_string(stack, &mut toml)?;
}
for deployment in resources.deployments {
if !toml.is_empty() {
toml.push_str("\n\n##\n\n");
}
toml.push_str("[[deployment]]\n");
Deployment::push_to_toml_string(deployment, &mut toml)?;
}
for build in resources.builds {
if !toml.is_empty() {
toml.push_str("\n\n##\n\n");
}
toml.push_str("[[build]]\n");
Build::push_to_toml_string(build, &mut toml)?;
}
for repo in resources.repos {
if !toml.is_empty() {
toml.push_str("\n\n##\n\n");
}
toml.push_str("[[repo]]\n");
Repo::push_to_toml_string(repo, &mut toml)?;
}
for procedure in resources.procedures {
if !toml.is_empty() {
toml.push_str("\n\n##\n\n");
}
toml.push_str("[[procedure]]\n");
Procedure::push_to_toml_string(procedure, &mut toml)?;
}
for action in resources.actions {
if !toml.is_empty() {
toml.push_str("\n\n##\n\n");
}
toml.push_str("[[action]]\n");
Action::push_to_toml_string(action, &mut toml)?;
}
for alerter in resources.alerters {
if !toml.is_empty() {
toml.push_str("\n\n##\n\n");
}
toml.push_str("[[alerter]]\n");
Alerter::push_to_toml_string(alerter, &mut toml)?;
}
for builder in resources.builders {
if !toml.is_empty() {
toml.push_str("\n\n##\n\n");
}
toml.push_str("[[builder]]\n");
Builder::push_to_toml_string(builder, &mut toml)?;
}
for resource_sync in resources.resource_syncs {
if !toml.is_empty() {
toml.push_str("\n\n##\n\n");
}
toml.push_str("[[resource_sync]]\n");
ResourceSync::push_to_toml_string(resource_sync, &mut toml)?;
}
for variable in &resources.variables {
if !toml.is_empty() {
toml.push_str("\n\n##\n\n");
}
toml.push_str(&variable_to_toml(variable)?);
}
for user_group in resources.user_groups {
if !toml.is_empty() {
toml.push_str("\n\n##\n\n");
}
toml.push_str(&user_group_to_toml(user_group)?);
}
Ok(toml)
}