Compare commits

..
13 Commits
Author SHA1 Message Date
118ae9b92c 1.18.4 (#604)
* update easy deps

* update otel deps

* implement template in types + update resource meta

* ts types

* dev-2

* dev-3 default template query is include

* Toggle resource is template in resource header

* dev-4 support CopyServer

* gen ts

* style template selector in New Resource menu

* fix new menu show 0

* add template market in omni search bar

* fix some dynamic import behavior

* template badge on dashboard

* dev-5

* standardize interpolation methods with nice api

* core use new interpolation methods

* refactor git usage

* dev-6 refactor interpolation / git methods

* fix pull stack passed replacers

*  new types

* remove redundant interpolation for build secret args

* clean up periphery docker client

* dev-7 include ports in container summary, see if they actually come through

* show container ports in container table

* refresh processes without tasks (more efficient)

* dev-8 keep container stats cache, include with ContainerListItem

* gen types

* display more container ports

* dev-9 fix repo clone when repo doesn't exist initially

* Add ports display to more spots

* fix function name

* add Periphery full container stats api, may be used later

* server container stats list

* dev-10

* 1.18.4 release

* Use reset instead of invalidate to fix GetUser spam on token expiry (#618)

---------

Co-authored-by: Jacky Fong <hello@huzky.dev>
2025-06-24 16:32:39 -07:00
LukeandGitHub 2205a81e79 Update webhooks.md (#611) 2025-06-20 11:56:05 -07:00
mbecker20 e2280f38df fix: allow Build / Repo add Attach permission 2025-06-16 00:40:24 -07:00
Maxwell BeckerandGitHub 545196d7eb 1.18.3 (#603)
* start 1.18.3 branch

* git::pull will fetch before checkout

* dev-2

* 1.18.3 quick release
2025-06-15 23:45:50 -07:00
23f8ecc1d9 1.18.2 (#591)
* feat: add maintenance window management to suppress alerts during planned activities (#550)

* feat: add scheduled maintenance windows to server configuration

- Add maintenance window configuration to server entities
- Implement maintenance window UI components with data table layout
- Add maintenance tab to server interface
- Suppress alerts during maintenance windows

* chore: enhance maintenance windows with types and permission improvements

- Add chrono dependency to Rust client core for time handling
- Add comprehensive TypeScript types for maintenance windows (MaintenanceWindow, MaintenanceScheduleType, MaintenanceTime, DayOfWeek)
- Improve maintenance config component to use usePermissions hook for better permission handling
- Update package dependencies

* feat: restore alert buffer system to prevent noise

* fix yarn fe

* fix the merge with new alerting changes

* move alert buffer handle out of loop

* nit

* fix server version changes

* unneeded buffer clear

---------

Co-authored-by: mbecker20 <becker.maxh@gmail.com>

* set version 1.18.2

* failed OIDC provider init doesn't cause panic, just  error log

* OIDC: use userinfo endpoint to get preffered username for user.

* add profile to scopes and account for username already taken

* search through server docker lists

* move maintenance stuff

* refactor maintenance schedules to have more toml compatible structure

* daily schedule type use struct

* add timezone to core info response

* frontend can build with new maintenance types

* Action monaco expose KomodoClient to init another client

* flatten out the nested enum

* update maintenance schedule types

* dev-3

* implement maintenance windows on alerters

* dev-4

* add IanaTimezone enum

* typeshare timezone enum

* maintenance modes almost done on servers AND alerters

* maintenance schedules working

* remove mention of migrator

* Procedure / Action schedule timezone selector

* improve timezone selector to display configure core TZ

* dev-5

* refetch core version

* add version to server list item info

* add periphery version in server table

* dev-6

* capitalize Unknown server status in cache

* handle unknown version case

* set server table sizes

* default resource_poll_interval 1-hr

* ensure parent folder exists before cloning

* document Build Attach permission

* git actions return absolute path

* stack linked repos

* resource toml replace linked_repo id with name

* validate incoming linked repo

* add linked repo to stack list item info

* stack list item info resolved linked repo information

* configure linked repo stack

* to repo links

* dev-7

* sync: replace linked repo with name for execute compare

* obscure provider tokens in table view

* clean up stack write w/ refactor

* Resource Sync / Build start support Repo attach

* add stack clone path config

* Builds + syncs can link to repos

* dev-9

* update ts

* fix linked repo not included in resource sync list item info

* add linked repo UI for builds / syncs

* fix commit linked repo sync

* include linked repo syncs

* correct Sync / Build config mode

* dev-12 fix resource sync inclusion w/ linked_repo

* remove unneed sync commit todo!()

* fix other config.repo.is_empty issues

* replace ids in all to toml exports

* Ensure git pull before commit for linear history, add to update logs

* fix fe for linked repo cases

* consolidate linked repo config component

* fix resource sync commit behavior

* dev 17

* Build uses Pull or Clone api to setup build source

* capitalize Clone Repo stage

* mount PullOrCloneRepo

* dev-19

* Expand supported container names and also avoid unnecessary name formatting

* dev-20

* add periphery /terminal/execute/container api

* periphery client execute_container_exec method

* implement execute container, deployment, stack exec

* gen types

* execute container exec method

* clean up client / fix fe

* enumerate exec ts methods for each resource type

* fix and gen ts client

* fix FE use connect_exec

* add url log when terminal ws fail to connect

* ts client server allow terminal.js

* FE preload terminal.js / .d.ts

* dev-23 fix stack terminal fail to connect when not explicitly setting container name

* update docs on attach perms

* 1.18.2

---------

Co-authored-by: Samuel Cardoso <R3D2@users.noreply.github.com>
2025-06-15 16:42:36 -07:00
4d401d7f20 1.18.1 (#566)
* 1.18.1

* improve stack header / all resource links

* disable build config selector

* clean up deployment header

* update build header

* builder header

* update repo header

* start adding repo links from api

* implement list item repo link

* clean up fe

* gen client

* repo links across the board

* include state tracking buffer, so alerts are only triggered by consecutive out of bounds conditions

* add runnables-cli link in runfile

* improve frontend first load time through some code splitting

* add services count to stack header

* fix repo on pull

* Add dedicated Deploying state to Deployments and Stacks

* move predeploy script before compose config (#584)

* Periphery / core version mismatch check / red text

* move builders / alerts out of sidebar, into settings

* remove force push

* list schedules api

* dev-1

* actually dev-3

* fix action

* filter none procedures

* fix schedule api

* dev-5

* basic schedules page

* prog on schedule page

* simplify schedule

* use name to sort target

* add resource tags to schedule

* Schedule page working

* dev-6

* remove schedule table type column

* reorder schedule table

* force confirm  dialogs for delete, even if disabled in config

* 1.18.1

---------

Co-authored-by: undaunt <31376520+undaunt@users.noreply.github.com>
2025-06-06 23:08:51 -07:00
mbecker20 4165e25332 further clarify ferretdb setup for existing users 2025-06-01 13:50:03 -04:00
Maxwell BeckerandGitHub 4cc0817b0f Update copy-database.md 2025-05-30 15:08:19 -07:00
mbecker20 51cf1e2b05 clarify mongo / ferret in docs 2025-05-30 17:14:42 -04:00
mbecker20 5309c70929 update runfile 2025-05-30 17:01:15 -04:00
mbecker20 1278c62859 update specific permission in docs 2025-05-30 16:58:28 -04:00
mbecker20 6d6acdbc0b fix permissions list 2025-05-30 16:49:27 -04:00
mbecker20 d22000331e remove logging driver from compose example 2025-05-30 16:14:21 -04:00
253 changed files with 14301 additions and 8257 deletions
Generated
+185 -208
View File
@@ -165,9 +165,9 @@ checksum = "ace50bade8e6234aa140d9a2f552bbee1db4d353f69b8217bc503490fc1a9f26"
[[package]]
name = "aws-config"
version = "1.6.3"
version = "1.8.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "02a18fd934af6ae7ca52410d4548b98eb895aab0f1ea417d168d85db1434a141"
checksum = "455e9fb7743c6f6267eb2830ccc08686fbb3d13c9a689369562fd4d4ef9ea462"
dependencies = [
"aws-credential-types",
"aws-runtime",
@@ -230,9 +230,9 @@ dependencies = [
[[package]]
name = "aws-runtime"
version = "1.5.7"
version = "1.5.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6c4063282c69991e57faab9e5cb21ae557e59f5b0fb285c196335243df8dc25c"
checksum = "4f6c68419d8ba16d9a7463671593c54f81ba58cab466e9b759418da606dcc2e2"
dependencies = [
"aws-credential-types",
"aws-sigv4",
@@ -254,9 +254,9 @@ dependencies = [
[[package]]
name = "aws-sdk-ec2"
version = "1.134.0"
version = "1.139.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a9a84e95f739e79d157409fa00e41008dabd181022193dabfabc68ddccbd6055"
checksum = "3b92e3d00f89108bc36102fcb4f23fb0b59ee12a6dc62156c27ce40e42337127"
dependencies = [
"aws-credential-types",
"aws-runtime",
@@ -277,9 +277,9 @@ dependencies = [
[[package]]
name = "aws-sdk-sso"
version = "1.70.0"
version = "1.73.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "83447efb7179d8e2ad2afb15ceb9c113debbc2ecdf109150e338e2e28b86190b"
checksum = "b2ac1674cba7872061a29baaf02209fefe499ff034dfd91bd4cc59e4d7741489"
dependencies = [
"aws-credential-types",
"aws-runtime",
@@ -299,9 +299,9 @@ dependencies = [
[[package]]
name = "aws-sdk-ssooidc"
version = "1.71.0"
version = "1.74.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c5f9bfbbda5e2b9fe330de098f14558ee8b38346408efe9f2e9cee82dc1636a4"
checksum = "3a6a22f077f5fd3e3c0270d4e1a110346cddf6769e9433eb9e6daceb4ca3b149"
dependencies = [
"aws-credential-types",
"aws-runtime",
@@ -321,9 +321,9 @@ dependencies = [
[[package]]
name = "aws-sdk-sts"
version = "1.71.0"
version = "1.74.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e17b984a66491ec08b4f4097af8911251db79296b3e4a763060b45805746264f"
checksum = "19d440e1d368759bd10df0dbdddbfff6473d7cd73e9d9ef2363dc9995ac2d711"
dependencies = [
"aws-credential-types",
"aws-runtime",
@@ -344,9 +344,9 @@ dependencies = [
[[package]]
name = "aws-sigv4"
version = "1.3.1"
version = "1.3.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3503af839bd8751d0bdc5a46b9cac93a003a353e635b0c12cf2376b5b53e41ea"
checksum = "ddfb9021f581b71870a17eac25b52335b82211cdc092e02b6876b2bcefa61666"
dependencies = [
"aws-credential-types",
"aws-smithy-http",
@@ -419,15 +419,15 @@ dependencies = [
"rustls-native-certs 0.8.1",
"rustls-pki-types",
"tokio",
"tower 0.5.2",
"tower",
"tracing",
]
[[package]]
name = "aws-smithy-json"
version = "0.61.3"
version = "0.61.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "92144e45819cae7dc62af23eac5a038a58aa544432d2102609654376a900bd07"
checksum = "a16e040799d29c17412943bdbf488fd75db04112d0c0d4b9290bacf5ae0014b9"
dependencies = [
"aws-smithy-types",
]
@@ -477,9 +477,9 @@ dependencies = [
[[package]]
name = "aws-smithy-runtime-api"
version = "1.8.0"
version = "1.8.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a1e5d9e3a80a18afa109391fb5ad09c3daf887b516c6fd805a157c6ea7994a57"
checksum = "bd8531b6d8882fd8f48f82a9754e682e29dd44cff27154af51fa3eb730f59efb"
dependencies = [
"aws-smithy-async",
"aws-smithy-types",
@@ -494,9 +494,9 @@ dependencies = [
[[package]]
name = "aws-smithy-types"
version = "1.3.1"
version = "1.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "40076bd09fadbc12d5e026ae080d0930defa606856186e31d83ccc6a255eeaf3"
checksum = "d498595448e43de7f4296b7b7a18a8a02c61ec9349128c80a368f7c3b4ab11a8"
dependencies = [
"base64-simd",
"bytes",
@@ -520,9 +520,9 @@ dependencies = [
[[package]]
name = "aws-smithy-xml"
version = "0.60.9"
version = "0.60.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ab0b0166827aa700d3dc519f72f8b3a91c35d0b8d042dc5d643a91e6f80648fc"
checksum = "3db87b96cb1b16c024980f133968d52882ca0daaee3a086c6decc500f6c99728"
dependencies = [
"xmlparser",
]
@@ -572,8 +572,8 @@ dependencies = [
"sha1",
"sync_wrapper",
"tokio",
"tokio-tungstenite",
"tower 0.5.2",
"tokio-tungstenite 0.26.2",
"tower",
"tower-layer",
"tower-service",
"tracing",
@@ -617,7 +617,7 @@ dependencies = [
"pin-project-lite",
"rustversion",
"serde",
"tower 0.5.2",
"tower",
"tower-layer",
"tower-service",
]
@@ -791,9 +791,9 @@ dependencies = [
[[package]]
name = "bollard"
version = "0.19.0"
version = "0.19.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "af706e9dc793491dd382c99c22fde6e9934433d4cc0d6a4b34eb2cdc57a5c917"
checksum = "899ca34eb6924d6ec2a77c6f7f5c7339e60fd68235eaf91edd5a15f12958bb06"
dependencies = [
"base64 0.22.1",
"bollard-stubs",
@@ -824,9 +824,9 @@ dependencies = [
[[package]]
name = "bollard-stubs"
version = "1.48.2-rc.28.0.4"
version = "1.48.3-rc.28.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "79cdf0fccd5341b38ae0be74b74410bdd5eceeea8876dc149a13edfe57e3b259"
checksum = "64ea257e555d16a2c01e5593f40b73865cdf12efbceda33c6d14a2d8d1490368"
dependencies = [
"serde",
"serde_json",
@@ -890,7 +890,7 @@ dependencies = [
[[package]]
name = "cache"
version = "1.18.0"
version = "1.18.4"
dependencies = [
"anyhow",
"tokio",
@@ -993,9 +993,9 @@ dependencies = [
[[package]]
name = "clap"
version = "4.5.38"
version = "4.5.40"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ed93b9805f8ba930df42c2590f05453d5ec36cbb85d018868a5b24d31f6ac000"
checksum = "40b6887a1d8685cebccf115538db5c0efe625ccac9696ad45c409d96566e910f"
dependencies = [
"clap_builder",
"clap_derive",
@@ -1003,9 +1003,9 @@ dependencies = [
[[package]]
name = "clap_builder"
version = "4.5.38"
version = "4.5.40"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "379026ff283facf611b0ea629334361c4211d1b12ee01024eec1591133b04120"
checksum = "e0c66c08ce9f0c698cbce5c0279d0bb6ac936d8674174fe48f736533b964f59e"
dependencies = [
"anstream",
"anstyle",
@@ -1015,9 +1015,9 @@ dependencies = [
[[package]]
name = "clap_derive"
version = "4.5.32"
version = "4.5.40"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "09176aae279615badda0765c0c0b3f6ed53f4709118af73cf4655d85d1530cd7"
checksum = "d2c7947ae4cc3d851207c1adb5b5e260ff0cca11446b1d6d1423788e442257ce"
dependencies = [
"heck",
"proc-macro2",
@@ -1057,10 +1057,8 @@ dependencies = [
[[package]]
name = "command"
version = "1.18.0"
version = "1.18.4"
dependencies = [
"anyhow",
"formatting",
"komodo_client",
"run_command",
"svi",
@@ -1539,9 +1537,19 @@ dependencies = [
"syn 2.0.100",
]
[[package]]
name = "environment"
version = "1.18.4"
dependencies = [
"anyhow",
"formatting",
"komodo_client",
"tokio",
]
[[package]]
name = "environment_file"
version = "1.18.0"
version = "1.18.4"
dependencies = [
"thiserror 2.0.12",
]
@@ -1621,7 +1629,7 @@ dependencies = [
[[package]]
name = "formatting"
version = "1.18.0"
version = "1.18.4"
dependencies = [
"serror",
]
@@ -1783,7 +1791,7 @@ checksum = "07e28edb80900c19c28f1072f2e8aeca7fa06b23cd4169cefe1af5aa3260783f"
[[package]]
name = "git"
version = "1.18.0"
version = "1.18.4"
dependencies = [
"anyhow",
"cache",
@@ -1791,7 +1799,6 @@ dependencies = [
"formatting",
"komodo_client",
"run_command",
"svi",
"tokio",
"tracing",
]
@@ -2181,22 +2188,28 @@ dependencies = [
[[package]]
name = "hyper-util"
version = "0.1.11"
version = "0.1.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "497bbc33a26fdd4af9ed9c70d63f61cf56a938375fbb32df34db9b1cd6d643f2"
checksum = "dc2fdfdbff08affe55bb779f33b053aa1fe5dd5b54c257343c17edfa55711bdb"
dependencies = [
"base64 0.22.1",
"bytes",
"futures-channel",
"futures-core",
"futures-util",
"http 1.3.1",
"http-body 1.0.1",
"hyper 1.6.0",
"ipnet",
"libc",
"percent-encoding",
"pin-project-lite",
"socket2",
"system-configuration",
"tokio",
"tower-service",
"tracing",
"windows-registry",
]
[[package]]
@@ -2426,6 +2439,15 @@ dependencies = [
"web-sys",
]
[[package]]
name = "interpolate"
version = "1.18.4"
dependencies = [
"anyhow",
"komodo_client",
"svi",
]
[[package]]
name = "ipconfig"
version = "0.3.2"
@@ -2444,6 +2466,16 @@ version = "2.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "469fb0b9cefa57e3ef31275ee7cacb78f2fdca44e4765491884a2b119d4eb130"
[[package]]
name = "iri-string"
version = "0.7.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dbc5ebe9c3a1a7a5127f920a418f7585e9e758e911d0466ed004f393b0e380b2"
dependencies = [
"memchr",
"serde",
]
[[package]]
name = "is_terminal_polyfill"
version = "1.70.1"
@@ -2520,7 +2552,7 @@ dependencies = [
[[package]]
name = "komodo_cli"
version = "1.18.0"
version = "1.18.4"
dependencies = [
"anyhow",
"clap",
@@ -2536,7 +2568,7 @@ dependencies = [
[[package]]
name = "komodo_client"
version = "1.18.0"
version = "1.18.4"
dependencies = [
"anyhow",
"async_timing_util",
@@ -2559,7 +2591,7 @@ dependencies = [
"strum 0.27.1",
"thiserror 2.0.12",
"tokio",
"tokio-tungstenite",
"tokio-tungstenite 0.27.0",
"tokio-util",
"tracing",
"typeshare",
@@ -2568,7 +2600,7 @@ dependencies = [
[[package]]
name = "komodo_core"
version = "1.18.0"
version = "1.18.4"
dependencies = [
"anyhow",
"arc-swap",
@@ -2598,6 +2630,7 @@ dependencies = [
"hex",
"hmac",
"indexmap 2.9.0",
"interpolate",
"jsonwebtoken",
"komodo_client",
"logger",
@@ -2623,7 +2656,7 @@ dependencies = [
"slack_client_rs",
"svi",
"tokio",
"tokio-tungstenite",
"tokio-tungstenite 0.27.0",
"tokio-util",
"toml",
"toml_pretty",
@@ -2637,9 +2670,10 @@ dependencies = [
[[package]]
name = "komodo_periphery"
version = "1.18.0"
version = "1.18.4"
dependencies = [
"anyhow",
"arc-swap",
"async_timing_util",
"axum",
"axum-server",
@@ -2650,11 +2684,13 @@ dependencies = [
"command",
"derive_variants",
"dotenvy",
"environment",
"environment_file",
"envy",
"formatting",
"futures",
"git",
"interpolate",
"komodo_client",
"logger",
"merge_config_files",
@@ -2670,7 +2706,6 @@ dependencies = [
"serde_json",
"serde_yaml",
"serror",
"svi",
"sysinfo",
"tokio",
"tokio-stream",
@@ -2681,7 +2716,7 @@ dependencies = [
[[package]]
name = "komodo_util"
version = "1.18.0"
version = "1.18.4"
dependencies = [
"anyhow",
"dotenvy",
@@ -2770,7 +2805,7 @@ dependencies = [
[[package]]
name = "logger"
version = "1.18.0"
version = "1.18.4"
dependencies = [
"anyhow",
"komodo_client",
@@ -3263,9 +3298,9 @@ checksum = "d05e27ee213611ffe7d6348b942e8f942b37114c00cc03cec254295a4a17852e"
[[package]]
name = "opentelemetry"
version = "0.29.1"
version = "0.30.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9e87237e2775f74896f9ad219d26a2081751187eb7c9f5c58dde20a23b95d16c"
checksum = "aaf416e4cb72756655126f7dd7bb0af49c674f4c1b9903e80c009e0c37e552e6"
dependencies = [
"futures-core",
"futures-sink",
@@ -3277,25 +3312,23 @@ dependencies = [
[[package]]
name = "opentelemetry-http"
version = "0.29.0"
version = "0.30.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "46d7ab32b827b5b495bd90fa95a6cb65ccc293555dcc3199ae2937d2d237c8ed"
checksum = "50f6639e842a97dbea8886e3439710ae463120091e2e064518ba8e716e6ac36d"
dependencies = [
"async-trait",
"bytes",
"http 1.3.1",
"opentelemetry",
"reqwest",
"tracing",
]
[[package]]
name = "opentelemetry-otlp"
version = "0.29.0"
version = "0.30.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d899720fe06916ccba71c01d04ecd77312734e2de3467fd30d9d580c8ce85656"
checksum = "dbee664a43e07615731afc539ca60c6d9f1a9425e25ca09c57bc36c87c55852b"
dependencies = [
"futures-core",
"http 1.3.1",
"opentelemetry",
"opentelemetry-http",
@@ -3310,9 +3343,9 @@ dependencies = [
[[package]]
name = "opentelemetry-proto"
version = "0.29.0"
version = "0.30.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8c40da242381435e18570d5b9d50aca2a4f4f4d8e146231adb4e7768023309b3"
checksum = "2e046fd7660710fe5a05e8748e70d9058dc15c94ba914e7c4faa7c728f0e8ddc"
dependencies = [
"opentelemetry",
"opentelemetry_sdk",
@@ -3322,20 +3355,19 @@ dependencies = [
[[package]]
name = "opentelemetry-semantic-conventions"
version = "0.29.0"
version = "0.30.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "84b29a9f89f1a954936d5aa92f19b2feec3c8f3971d3e96206640db7f9706ae3"
checksum = "83d059a296a47436748557a353c5e6c5705b9470ef6c95cfc52c21a8814ddac2"
[[package]]
name = "opentelemetry_sdk"
version = "0.29.0"
version = "0.30.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "afdefb21d1d47394abc1ba6c57363ab141be19e27cc70d0e422b7f303e4d290b"
checksum = "11f644aa9e5e31d11896e024305d7e3c98a88884d9f8919dbf37a9991bc47a4b"
dependencies = [
"futures-channel",
"futures-executor",
"futures-util",
"glob",
"opentelemetry",
"percent-encoding",
"rand 0.9.1",
@@ -3343,7 +3375,6 @@ dependencies = [
"thiserror 2.0.12",
"tokio",
"tokio-stream",
"tracing",
]
[[package]]
@@ -3525,7 +3556,7 @@ checksum = "e3148f5046208a5d56bcfc03053e3ca6334e51da8dfb19b6cdc8b306fae3283e"
[[package]]
name = "periphery_client"
version = "1.18.0"
version = "1.18.4"
dependencies = [
"anyhow",
"komodo_client",
@@ -3537,7 +3568,7 @@ dependencies = [
"serde_qs",
"serror",
"tokio",
"tokio-tungstenite",
"tokio-tungstenite 0.27.0",
"tracing",
]
@@ -3908,9 +3939,9 @@ checksum = "2b15c43186be67a4fd63bee50d0303afffcef381492ebe2c5d87f324e1b8815c"
[[package]]
name = "reqwest"
version = "0.12.15"
version = "0.12.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d19c46a6fdd48bc4dab94b6103fccc55d34c67cc0ad04653aad4ea2a07cd7bbb"
checksum = "eabf4c97d9130e2bf606614eb937e86edac8292eaa6f422f995d7e8de1eb1813"
dependencies = [
"base64 0.22.1",
"bytes",
@@ -3925,36 +3956,32 @@ dependencies = [
"hyper 1.6.0",
"hyper-rustls 0.27.5",
"hyper-util",
"ipnet",
"js-sys",
"log",
"mime",
"mime_guess",
"once_cell",
"percent-encoding",
"pin-project-lite",
"quinn",
"rustls 0.23.27",
"rustls-native-certs 0.8.1",
"rustls-pemfile 2.2.0",
"rustls-pki-types",
"serde",
"serde_json",
"serde_urlencoded",
"sync_wrapper",
"system-configuration",
"tokio",
"tokio-rustls 0.26.2",
"tokio-util",
"tower 0.5.2",
"tower",
"tower-http",
"tower-service",
"url",
"wasm-bindgen",
"wasm-bindgen-futures",
"wasm-streams",
"web-sys",
"webpki-roots 0.26.8",
"windows-registry",
"webpki-roots 1.0.0",
]
[[package]]
@@ -4053,7 +4080,7 @@ dependencies = [
[[package]]
name = "response"
version = "1.18.0"
version = "1.18.4"
dependencies = [
"anyhow",
"axum",
@@ -4502,9 +4529,9 @@ dependencies = [
[[package]]
name = "serde_spanned"
version = "0.6.8"
version = "0.6.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "87607cb1398ed59d48732e575a4c28a7a8ebf2454b964fe3f224f2afc07909e1"
checksum = "bf41e0cfaf7226dca15e8197172c295a782857fcb97fad1808a166870dee75a3"
dependencies = [
"serde",
]
@@ -4816,11 +4843,11 @@ checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292"
[[package]]
name = "svi"
version = "1.0.1"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ecfc625e6292e18db65d1a122cb1ea014dd909622ab0154f068a3990413ff017"
checksum = "e8d4d1d576ce8f4b01ac90fd0d9f4c95bf5cebbc9c08e89499b152bef770cd98"
dependencies = [
"thiserror 1.0.69",
"thiserror 2.0.12",
]
[[package]]
@@ -4867,9 +4894,9 @@ dependencies = [
[[package]]
name = "sysinfo"
version = "0.35.1"
version = "0.35.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "79251336d17c72d9762b8b54be4befe38d2db56fbbc0241396d70f173c39d47a"
checksum = "3c3ffa3e4ff2b324a57f7aeb3c349656c7b127c3c189520251a648102a92496e"
dependencies = [
"libc",
"memchr",
@@ -5093,6 +5120,18 @@ name = "tokio-tungstenite"
version = "0.26.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7a9daff607c6d2bf6c16fd681ccb7eecc83e4e2cdc1ca067ffaadfca5de7f084"
dependencies = [
"futures-util",
"log",
"tokio",
"tungstenite 0.26.2",
]
[[package]]
name = "tokio-tungstenite"
version = "0.27.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "489a59b6730eda1b0171fcfda8b121f4bee2b35cba8645ca35c5f7ba3eb736c1"
dependencies = [
"futures-util",
"log",
@@ -5101,7 +5140,7 @@ dependencies = [
"rustls-pki-types",
"tokio",
"tokio-rustls 0.26.2",
"tungstenite",
"tungstenite 0.27.0",
]
[[package]]
@@ -5120,9 +5159,9 @@ dependencies = [
[[package]]
name = "toml"
version = "0.8.22"
version = "0.8.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "05ae329d1f08c4d17a59bed7ff5b5a769d062e64a62d34a3261b219e62cd5aae"
checksum = "dc1beb996b9d83529a9e75c17a1686767d148d70663143c7854d8b4a09ced362"
dependencies = [
"serde",
"serde_spanned",
@@ -5132,18 +5171,18 @@ dependencies = [
[[package]]
name = "toml_datetime"
version = "0.6.9"
version = "0.6.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3da5db5a963e24bc68be8b17b6fa82814bb22ee8660f192bb182771d498f09a3"
checksum = "22cddaf88f4fbc13c51aebbf5f8eceb5c7c5a9da2ac40a13519eb5b0a0e8f11c"
dependencies = [
"serde",
]
[[package]]
name = "toml_edit"
version = "0.22.26"
version = "0.22.27"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "310068873db2c5b3e7659d2cc35d21855dbafa50d1ce336397c666e3cb08137e"
checksum = "41fe8c660ae4257887cf66394862d21dbca4a6ddd26f04a3560410406a2f819a"
dependencies = [
"indexmap 2.9.0",
"serde",
@@ -5167,15 +5206,15 @@ dependencies = [
[[package]]
name = "toml_write"
version = "0.1.1"
version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bfb942dfe1d8e29a7ee7fcbde5bd2b9a25fb89aa70caea2eba3bee836ff41076"
checksum = "5d99f8c9a7727884afe522e9bd5edbfc91a3312b36a77b5fb8926e4c31a41801"
[[package]]
name = "tonic"
version = "0.12.3"
version = "0.13.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "877c5b330756d856ffcc4553ab34a5684481ade925ecc54bcd1bf02b1d0d4d52"
checksum = "7e581ba15a835f4d9ea06c55ab1bd4dce26fc53752c69a04aac00703bfb49ba9"
dependencies = [
"async-trait",
"base64 0.22.1",
@@ -5190,31 +5229,10 @@ dependencies = [
"pin-project",
"prost",
"rustls-native-certs 0.8.1",
"rustls-pemfile 2.2.0",
"tokio",
"tokio-rustls 0.26.2",
"tokio-stream",
"tower 0.4.13",
"tower-layer",
"tower-service",
"tracing",
]
[[package]]
name = "tower"
version = "0.4.13"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b8fa9be0de6cf49e536ce1851f987bd21a43b771b09473c3549a6c853db37c1c"
dependencies = [
"futures-core",
"futures-util",
"indexmap 1.9.3",
"pin-project",
"pin-project-lite",
"rand 0.8.5",
"slab",
"tokio",
"tokio-util",
"tower",
"tower-layer",
"tower-service",
"tracing",
@@ -5228,9 +5246,12 @@ checksum = "d039ad9159c98b70ecfd540b2573b97f7f52c3e8d9f8ad57a24b916a536975f9"
dependencies = [
"futures-core",
"futures-util",
"indexmap 2.9.0",
"pin-project-lite",
"slab",
"sync_wrapper",
"tokio",
"tokio-util",
"tower-layer",
"tower-service",
"tracing",
@@ -5238,24 +5259,27 @@ dependencies = [
[[package]]
name = "tower-http"
version = "0.6.4"
version = "0.6.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0fdb0c213ca27a9f57ab69ddb290fd80d970922355b83ae380b395d3986b8a2e"
checksum = "adc82fd73de2a9722ac5da747f12383d2bfdb93591ee6c58486e0097890f05f2"
dependencies = [
"bitflags 2.9.0",
"bytes",
"futures-core",
"futures-util",
"http 1.3.1",
"http-body 1.0.1",
"http-body-util",
"http-range-header",
"httpdate",
"iri-string",
"mime",
"mime_guess",
"percent-encoding",
"pin-project-lite",
"tokio",
"tokio-util",
"tower",
"tower-layer",
"tower-service",
"tracing",
@@ -5319,9 +5343,9 @@ dependencies = [
[[package]]
name = "tracing-opentelemetry"
version = "0.30.0"
version = "0.31.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fd8e764bd6f5813fd8bebc3117875190c5b0415be8f7f8059bffb6ecd979c444"
checksum = "ddcf5959f39507d0d04d6413119c04f33b623f4f951ebcbdddddfad2d0623a9c"
dependencies = [
"js-sys",
"once_cell",
@@ -5373,6 +5397,23 @@ name = "tungstenite"
version = "0.26.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4793cb5e56680ecbb1d843515b23b6de9a75eb04b66643e256a396d43be33c13"
dependencies = [
"bytes",
"data-encoding",
"http 1.3.1",
"httparse",
"log",
"rand 0.9.1",
"sha1",
"thiserror 2.0.12",
"utf-8",
]
[[package]]
name = "tungstenite"
version = "0.27.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "eadc29d668c91fcc564941132e17b28a7ceb2f3ebf0b9dae3e03fd7a6748eb0d"
dependencies = [
"bytes",
"data-encoding",
@@ -5708,6 +5749,15 @@ dependencies = [
"rustls-pki-types",
]
[[package]]
name = "webpki-roots"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2853738d1cc4f2da3a225c18ec6c3721abb31961096e9dbf5ab35fa88b19cfdb"
dependencies = [
"rustls-pki-types",
]
[[package]]
name = "which"
version = "4.4.2"
@@ -5789,7 +5839,7 @@ dependencies = [
"windows-interface",
"windows-link",
"windows-result",
"windows-strings 0.4.0",
"windows-strings",
]
[[package]]
@@ -5842,13 +5892,13 @@ dependencies = [
[[package]]
name = "windows-registry"
version = "0.4.0"
version = "0.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4286ad90ddb45071efd1a66dfa43eb02dd0dfbae1545ad6cc3c51cf34d7e8ba3"
checksum = "ad1da3e436dc7653dfdf3da67332e22bff09bb0e28b0239e1624499c7830842e"
dependencies = [
"windows-link",
"windows-result",
"windows-strings 0.3.1",
"windows-targets 0.53.0",
"windows-strings",
]
[[package]]
@@ -5860,15 +5910,6 @@ dependencies = [
"windows-link",
]
[[package]]
name = "windows-strings"
version = "0.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "87fa48cc5d406560701792be122a10132491cff9d0aeb23583cc2dcafc847319"
dependencies = [
"windows-link",
]
[[package]]
name = "windows-strings"
version = "0.4.0"
@@ -5929,29 +5970,13 @@ dependencies = [
"windows_aarch64_gnullvm 0.52.6",
"windows_aarch64_msvc 0.52.6",
"windows_i686_gnu 0.52.6",
"windows_i686_gnullvm 0.52.6",
"windows_i686_gnullvm",
"windows_i686_msvc 0.52.6",
"windows_x86_64_gnu 0.52.6",
"windows_x86_64_gnullvm 0.52.6",
"windows_x86_64_msvc 0.52.6",
]
[[package]]
name = "windows-targets"
version = "0.53.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b1e4c7e8ceaaf9cb7d7507c974735728ab453b67ef8f18febdd7c11fe59dca8b"
dependencies = [
"windows_aarch64_gnullvm 0.53.0",
"windows_aarch64_msvc 0.53.0",
"windows_i686_gnu 0.53.0",
"windows_i686_gnullvm 0.53.0",
"windows_i686_msvc 0.53.0",
"windows_x86_64_gnu 0.53.0",
"windows_x86_64_gnullvm 0.53.0",
"windows_x86_64_msvc 0.53.0",
]
[[package]]
name = "windows_aarch64_gnullvm"
version = "0.48.5"
@@ -5964,12 +5989,6 @@ version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3"
[[package]]
name = "windows_aarch64_gnullvm"
version = "0.53.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "86b8d5f90ddd19cb4a147a5fa63ca848db3df085e25fee3cc10b39b6eebae764"
[[package]]
name = "windows_aarch64_msvc"
version = "0.48.5"
@@ -5982,12 +6001,6 @@ version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469"
[[package]]
name = "windows_aarch64_msvc"
version = "0.53.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c7651a1f62a11b8cbd5e0d42526e55f2c99886c77e007179efff86c2b137e66c"
[[package]]
name = "windows_i686_gnu"
version = "0.48.5"
@@ -6000,24 +6013,12 @@ version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b"
[[package]]
name = "windows_i686_gnu"
version = "0.53.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c1dc67659d35f387f5f6c479dc4e28f1d4bb90ddd1a5d3da2e5d97b42d6272c3"
[[package]]
name = "windows_i686_gnullvm"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66"
[[package]]
name = "windows_i686_gnullvm"
version = "0.53.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9ce6ccbdedbf6d6354471319e781c0dfef054c81fbc7cf83f338a4296c0cae11"
[[package]]
name = "windows_i686_msvc"
version = "0.48.5"
@@ -6030,12 +6031,6 @@ version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66"
[[package]]
name = "windows_i686_msvc"
version = "0.53.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "581fee95406bb13382d2f65cd4a908ca7b1e4c2f1917f143ba16efe98a589b5d"
[[package]]
name = "windows_x86_64_gnu"
version = "0.48.5"
@@ -6048,12 +6043,6 @@ version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78"
[[package]]
name = "windows_x86_64_gnu"
version = "0.53.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2e55b5ac9ea33f2fc1716d1742db15574fd6fc8dadc51caab1c16a3d3b4190ba"
[[package]]
name = "windows_x86_64_gnullvm"
version = "0.48.5"
@@ -6066,12 +6055,6 @@ version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d"
[[package]]
name = "windows_x86_64_gnullvm"
version = "0.53.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0a6e035dd0599267ce1ee132e51c27dd29437f63325753051e71dd9e42406c57"
[[package]]
name = "windows_x86_64_msvc"
version = "0.48.5"
@@ -6084,17 +6067,11 @@ version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec"
[[package]]
name = "windows_x86_64_msvc"
version = "0.53.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "271414315aff87387382ec3d271b52d7ae78726f5d44ac98b4f4030c91880486"
[[package]]
name = "winnow"
version = "0.7.7"
version = "0.7.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6cb8234a863ea0e8cd7284fcdd4f145233eb00fee02bbdd9861aec44e6477bc5"
checksum = "74c7b26e3480b707944fc872477815d29a8e429d2f93a1ce000f5fa84a15cbcd"
dependencies = [
"memchr",
]
+17 -15
View File
@@ -8,7 +8,7 @@ members = [
]
[workspace.package]
version = "1.18.0"
version = "1.18.4"
edition = "2024"
authors = ["mbecker20 <becker.maxh@gmail.com>"]
license = "GPL-3.0-or-later"
@@ -20,6 +20,8 @@ homepage = "https://komo.do"
komodo_client = { path = "client/core/rs" }
periphery_client = { path = "client/periphery/rs" }
environment_file = { path = "lib/environment_file" }
environment = { path = "lib/environment" }
interpolate = { path = "lib/interpolate" }
formatting = { path = "lib/formatting" }
response = { path = "lib/response" }
command = { path = "lib/command" }
@@ -41,10 +43,10 @@ mongo_indexed = "2.0.1"
resolver_api = "3.0.0"
toml_pretty = "1.1.2"
mungos = "3.2.0"
svi = "1.0.1"
svi = "1.1.0"
# ASYNC
reqwest = { version = "0.12.15", default-features = false, features = ["json", "stream", "rustls-tls-native-roots"] }
reqwest = { version = "0.12.20", default-features = false, features = ["json", "stream", "rustls-tls-native-roots"] }
tokio = { version = "1.45.1", features = ["full"] }
tokio-util = { version = "0.7.15", features = ["io", "codec"] }
tokio-stream = { version = "0.1.17", features = ["sync"] }
@@ -54,7 +56,7 @@ futures-util = "0.3.31"
arc-swap = "1.7.1"
# SERVER
tokio-tungstenite = { version = "0.26.2", features = ["rustls-tls-native-roots"] }
tokio-tungstenite = { version = "0.27.0", features = ["rustls-tls-native-roots"] }
axum-extra = { version = "0.10.1", features = ["typed-header"] }
tower-http = { version = "0.6.4", features = ["fs", "cors"] }
axum-server = { version = "0.7.2", features = ["tls-rustls"] }
@@ -67,23 +69,23 @@ strum = { version = "0.27.1", features = ["derive"] }
serde_json = "1.0.140"
serde_yaml = "0.9.34"
serde_qs = "0.15.0"
toml = "0.8.22"
toml = "0.8.23"
# ERROR
anyhow = "1.0.98"
thiserror = "2.0.12"
# LOGGING
opentelemetry-otlp = { version = "0.29.0", features = ["tls-roots", "reqwest-rustls"] }
opentelemetry_sdk = { version = "0.29.0", features = ["rt-tokio"] }
opentelemetry-otlp = { version = "0.30.0", features = ["tls-roots", "reqwest-rustls"] }
opentelemetry_sdk = { version = "0.30.0", features = ["rt-tokio"] }
tracing-subscriber = { version = "0.3.19", features = ["json"] }
opentelemetry-semantic-conventions = "0.29.0"
tracing-opentelemetry = "0.30.0"
opentelemetry = "0.29.1"
opentelemetry-semantic-conventions = "0.30.0"
tracing-opentelemetry = "0.31.0"
opentelemetry = "0.30.0"
tracing = "0.1.41"
# CONFIG
clap = { version = "4.5.38", features = ["derive"] }
clap = { version = "4.5.40", features = ["derive"] }
dotenvy = "0.15.7"
envy = "0.4.2"
@@ -103,12 +105,12 @@ hex = "0.4.3"
# SYSTEM
portable-pty = "0.9.0"
bollard = "0.19.0"
sysinfo = "0.35.1"
bollard = "0.19.1"
sysinfo = "0.35.2"
# CLOUD
aws-config = "1.6.3"
aws-sdk-ec2 = "1.134.0"
aws-config = "1.8.0"
aws-sdk-ec2 = "1.139.0"
aws-credential-types = "1.2.3"
## CRON
+50 -50
View File
@@ -12,7 +12,7 @@ use crate::{
};
pub enum ExecutionResult {
Single(Update),
Single(Box<Update>),
Batch(BatchExecutionResponse),
}
@@ -227,7 +227,7 @@ pub async fn run(execution: Execution) -> anyhow::Result<()> {
Execution::RunAction(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::BatchRunAction(request) => komodo_client()
.execute(request)
.await
@@ -235,7 +235,7 @@ pub async fn run(execution: Execution) -> anyhow::Result<()> {
Execution::RunProcedure(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::BatchRunProcedure(request) => komodo_client()
.execute(request)
.await
@@ -243,7 +243,7 @@ pub async fn run(execution: Execution) -> anyhow::Result<()> {
Execution::RunBuild(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::BatchRunBuild(request) => komodo_client()
.execute(request)
.await
@@ -251,11 +251,11 @@ pub async fn run(execution: Execution) -> anyhow::Result<()> {
Execution::CancelBuild(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::Deploy(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::BatchDeploy(request) => komodo_client()
.execute(request)
.await
@@ -263,31 +263,31 @@ pub async fn run(execution: Execution) -> anyhow::Result<()> {
Execution::PullDeployment(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::StartDeployment(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::RestartDeployment(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::PauseDeployment(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::UnpauseDeployment(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::StopDeployment(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::DestroyDeployment(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::BatchDestroyDeployment(request) => komodo_client()
.execute(request)
.await
@@ -295,7 +295,7 @@ pub async fn run(execution: Execution) -> anyhow::Result<()> {
Execution::CloneRepo(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::BatchCloneRepo(request) => komodo_client()
.execute(request)
.await
@@ -303,7 +303,7 @@ pub async fn run(execution: Execution) -> anyhow::Result<()> {
Execution::PullRepo(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::BatchPullRepo(request) => komodo_client()
.execute(request)
.await
@@ -311,7 +311,7 @@ pub async fn run(execution: Execution) -> anyhow::Result<()> {
Execution::BuildRepo(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::BatchBuildRepo(request) => komodo_client()
.execute(request)
.await
@@ -319,103 +319,103 @@ pub async fn run(execution: Execution) -> anyhow::Result<()> {
Execution::CancelRepoBuild(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::StartContainer(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::RestartContainer(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::PauseContainer(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::UnpauseContainer(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::StopContainer(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::DestroyContainer(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::StartAllContainers(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::RestartAllContainers(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::PauseAllContainers(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::UnpauseAllContainers(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::StopAllContainers(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::PruneContainers(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::DeleteNetwork(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::PruneNetworks(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::DeleteImage(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::PruneImages(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::DeleteVolume(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::PruneVolumes(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::PruneDockerBuilders(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::PruneBuildx(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::PruneSystem(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::RunSync(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::CommitSync(request) => komodo_client()
.write(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::DeployStack(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::BatchDeployStack(request) => komodo_client()
.execute(request)
.await
@@ -423,7 +423,7 @@ pub async fn run(execution: Execution) -> anyhow::Result<()> {
Execution::DeployStackIfChanged(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::BatchDeployStackIfChanged(request) => komodo_client()
.execute(request)
.await
@@ -431,7 +431,7 @@ pub async fn run(execution: Execution) -> anyhow::Result<()> {
Execution::PullStack(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::BatchPullStack(request) => komodo_client()
.execute(request)
.await
@@ -439,27 +439,27 @@ pub async fn run(execution: Execution) -> anyhow::Result<()> {
Execution::StartStack(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::RestartStack(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::PauseStack(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::UnpauseStack(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::StopStack(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::DestroyStack(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::BatchDestroyStack(request) => komodo_client()
.execute(request)
.await
@@ -467,7 +467,7 @@ pub async fn run(execution: Execution) -> anyhow::Result<()> {
Execution::TestAlerter(request) => komodo_client()
.execute(request)
.await
.map(ExecutionResult::Single),
.map(|u| ExecutionResult::Single(u.into())),
Execution::Sleep(request) => {
let duration =
Duration::from_millis(request.duration_ms as u64);
+1
View File
@@ -18,6 +18,7 @@ path = "src/main.rs"
komodo_client = { workspace = true, features = ["mongo"] }
periphery_client.workspace = true
environment_file.workspace = true
interpolate.workspace = true
formatting.workspace = true
response.workspace = true
command.workspace = true
+10 -12
View File
@@ -210,24 +210,22 @@ pub async fn send_alert(
AlertData::None {} => Default::default(),
};
if !content.is_empty() {
let vars_and_secrets = get_variables_and_secrets().await?;
let mut global_replacers = HashSet::new();
let mut secret_replacers = HashSet::new();
let VariablesAndSecrets { variables, secrets } =
get_variables_and_secrets().await?;
let mut url_interpolated = url.to_string();
// interpolate variables and secrets into the url
interpolate_variables_secrets_into_string(
&vars_and_secrets,
&mut url_interpolated,
&mut global_replacers,
&mut secret_replacers,
)?;
let mut interpolator =
Interpolator::new(Some(&variables), &secrets);
interpolator.interpolate_string(&mut url_interpolated)?;
send_message(&url_interpolated, &content)
.await
.map_err(|e| {
let replacers =
secret_replacers.into_iter().collect::<Vec<_>>();
let replacers = interpolator
.secret_replacers
.into_iter()
.collect::<Vec<_>>();
let sanitized_error =
svi::replace_in_string(&format!("{e:?}"), &replacers);
anyhow::Error::msg(format!(
+22 -14
View File
@@ -2,19 +2,22 @@ use ::slack::types::Block;
use anyhow::{Context, anyhow};
use derive_variants::ExtractVariant;
use futures::future::join_all;
use interpolate::Interpolator;
use komodo_client::entities::{
ResourceTargetVariant,
alert::{Alert, AlertData, AlertDataVariant, SeverityLevel},
alerter::*,
deployment::DeploymentState,
komodo_timestamp,
stack::StackState,
};
use mungos::{find::find_collect, mongodb::bson::doc};
use std::collections::HashSet;
use tracing::Instrument;
use crate::helpers::interpolate::interpolate_variables_secrets_into_string;
use crate::helpers::query::get_variables_and_secrets;
use crate::helpers::{
maintenance::is_in_maintenance, query::VariablesAndSecrets,
};
use crate::{config::core_config, state::db_client};
mod discord;
@@ -80,6 +83,13 @@ pub async fn send_alert_to_alerter(
return Ok(());
}
if is_in_maintenance(
&alerter.config.maintenance_windows,
komodo_timestamp(),
) {
return Ok(());
}
let alert_type = alert.data.extract_variant();
// In the test case, we don't want the filters inside this
@@ -156,18 +166,14 @@ async fn send_custom_alert(
url: &str,
alert: &Alert,
) -> anyhow::Result<()> {
let vars_and_secrets = get_variables_and_secrets().await?;
let mut global_replacers = HashSet::new();
let mut secret_replacers = HashSet::new();
let VariablesAndSecrets { variables, secrets } =
get_variables_and_secrets().await?;
let mut url_interpolated = url.to_string();
// interpolate variables and secrets into the url
interpolate_variables_secrets_into_string(
&vars_and_secrets,
&mut url_interpolated,
&mut global_replacers,
&mut secret_replacers,
)?;
let mut interpolator =
Interpolator::new(Some(&variables), &secrets);
interpolator.interpolate_string(&mut url_interpolated)?;
let res = reqwest::Client::new()
.post(url_interpolated)
@@ -175,8 +181,10 @@ async fn send_custom_alert(
.send()
.await
.map_err(|e| {
let replacers =
secret_replacers.into_iter().collect::<Vec<_>>();
let replacers = interpolator
.secret_replacers
.into_iter()
.collect::<Vec<_>>();
let sanitized_error =
svi::replace_in_string(&format!("{e:?}"), &replacers);
anyhow::Error::msg(format!(
+10 -12
View File
@@ -432,23 +432,21 @@ pub async fn send_alert(
AlertData::None {} => Default::default(),
};
if !text.is_empty() {
let vars_and_secrets = get_variables_and_secrets().await?;
let mut global_replacers = HashSet::new();
let mut secret_replacers = HashSet::new();
let VariablesAndSecrets { variables, secrets } =
get_variables_and_secrets().await?;
let mut url_interpolated = url.to_string();
// interpolate variables and secrets into the url
interpolate_variables_secrets_into_string(
&vars_and_secrets,
&mut url_interpolated,
&mut global_replacers,
&mut secret_replacers,
)?;
let mut interpolator =
Interpolator::new(Some(&variables), &secrets);
interpolator.interpolate_string(&mut url_interpolated)?;
let slack = ::slack::Client::new(url_interpolated);
slack.send_message(text, blocks).await.map_err(|e| {
let replacers =
secret_replacers.into_iter().collect::<Vec<_>>();
let replacers = interpolator
.secret_replacers
.into_iter()
.collect::<Vec<_>>();
let sanitized_error =
svi::replace_in_string(&format!("{e:?}"), &replacers);
anyhow::Error::msg(format!(
+4 -10
View File
@@ -16,7 +16,7 @@ use crate::{
get_user_id_from_headers,
github::{self, client::github_oauth_client},
google::{self, client::google_oauth_client},
oidc,
oidc::{self, client::oidc_client},
},
config::core_config,
helpers::query::get_user,
@@ -114,15 +114,9 @@ fn login_options_reponse() -> &'static GetLoginOptionsResponse {
let config = core_config();
GetLoginOptionsResponse {
local: config.local_auth,
github: config.github_oauth.enabled
&& !config.github_oauth.id.is_empty()
&& !config.github_oauth.secret.is_empty(),
google: config.google_oauth.enabled
&& !config.google_oauth.id.is_empty()
&& !config.google_oauth.secret.is_empty(),
oidc: config.oidc_enabled
&& !config.oidc_provider.is_empty()
&& !config.oidc_client_id.is_empty(),
github: github_oauth_client().is_some(),
google: google_oauth_client().is_some(),
oidc: oidc_client().load().is_some(),
registration_disabled: config.disable_user_registration,
}
})
+14 -23
View File
@@ -7,6 +7,7 @@ use std::{
use anyhow::Context;
use command::run_komodo_command;
use interpolate::Interpolator;
use komodo_client::{
api::{
execute::{BatchExecutionResponse, BatchRunAction, RunAction},
@@ -31,11 +32,7 @@ use crate::{
api::{execute::ExecuteRequest, user::UserArgs},
config::core_config,
helpers::{
interpolate::{
add_interp_update_log,
interpolate_variables_secrets_into_string,
},
query::get_variables_and_secrets,
query::{VariablesAndSecrets, get_variables_and_secrets},
random_string,
update::update_update,
},
@@ -221,28 +218,22 @@ async fn interpolate(
key: String,
secret: String,
) -> serror::Result<HashSet<(String, String)>> {
let mut vars_and_secrets = get_variables_and_secrets().await?;
let VariablesAndSecrets {
variables,
mut secrets,
} = get_variables_and_secrets().await?;
vars_and_secrets
.secrets
.insert(String::from("ACTION_API_KEY"), key);
vars_and_secrets
.secrets
.insert(String::from("ACTION_API_SECRET"), secret);
secrets.insert(String::from("ACTION_API_KEY"), key);
secrets.insert(String::from("ACTION_API_SECRET"), secret);
let mut global_replacers = HashSet::new();
let mut secret_replacers = HashSet::new();
let mut interpolator =
Interpolator::new(Some(&variables), &secrets);
interpolate_variables_secrets_into_string(
&vars_and_secrets,
contents,
&mut global_replacers,
&mut secret_replacers,
)?;
interpolator
.interpolate_string(contents)?
.push_logs(&mut update.logs);
add_interp_update_log(update, &global_replacers, &secret_replacers);
Ok(secret_replacers)
Ok(interpolator.secret_replacers)
}
fn full_contents(contents: &str, key: &str, secret: &str) -> String {
+50 -72
View File
@@ -1,8 +1,9 @@
use std::{collections::HashSet, future::IntoFuture, time::Duration};
use std::{future::IntoFuture, time::Duration};
use anyhow::{Context, anyhow};
use formatting::format_serror;
use futures::future::join_all;
use interpolate::Interpolator;
use komodo_client::{
api::execute::{
BatchExecutionResponse, BatchRunBuild, CancelBuild, Deploy,
@@ -16,6 +17,7 @@ use komodo_client::{
deployment::DeploymentState,
komodo_timestamp,
permission::PermissionLevel,
repo::Repo,
update::{Log, Update},
user::auto_redeploy_user,
},
@@ -35,16 +37,13 @@ use tokio_util::sync::CancellationToken;
use crate::{
alert::send_alerts,
helpers::{
build_git_token,
builder::{cleanup_builder_instance, get_builder_periphery},
channel::build_cancel_channel,
git_token,
interpolate::{
add_interp_update_log,
interpolate_variables_secrets_into_extra_args,
interpolate_variables_secrets_into_string,
interpolate_variables_secrets_into_system_command,
query::{
VariablesAndSecrets, get_deployment_state,
get_variables_and_secrets,
},
query::{get_deployment_state, get_variables_and_secrets},
registry_token,
update::{init_execution_update, update_update},
},
@@ -88,9 +87,23 @@ impl Resolve<ExecuteArgs> for RunBuild {
)
.await?;
let mut vars_and_secrets = get_variables_and_secrets().await?;
let mut repo = if !build.config.files_on_host
&& !build.config.linked_repo.is_empty()
{
crate::resource::get::<Repo>(&build.config.linked_repo)
.await?
.into()
} else {
None
};
let VariablesAndSecrets {
mut variables,
secrets,
} = get_variables_and_secrets().await?;
// Add the $VERSION to variables. Use with [[$VERSION]]
vars_and_secrets.variables.insert(
variables.insert(
String::from("$VERSION"),
build.config.version.to_string(),
);
@@ -117,15 +130,8 @@ impl Resolve<ExecuteArgs> for RunBuild {
update.version = build.config.version;
update_update(update.clone()).await?;
let git_token = git_token(
&build.config.git_provider,
&build.config.git_account,
|https| build.config.git_https = https,
)
.await
.with_context(
|| format!("Failed to get git token in call to db. This is a database error, not a token exisitence error. Stopping run. | {} | {}", build.config.git_provider, build.config.git_account),
)?;
let git_token =
build_git_token(&mut build, repo.as_mut()).await?;
let registry_token =
validate_account_extract_registry_token(&build).await?;
@@ -203,66 +209,36 @@ impl Resolve<ExecuteArgs> for RunBuild {
// INTERPOLATE VARIABLES
let secret_replacers = if !build.config.skip_secret_interp {
let mut global_replacers = HashSet::new();
let mut secret_replacers = HashSet::new();
let mut interpolator =
Interpolator::new(Some(&variables), &secrets);
interpolate_variables_secrets_into_system_command(
&vars_and_secrets,
&mut build.config.pre_build,
&mut global_replacers,
&mut secret_replacers,
)?;
interpolator.interpolate_build(&mut build)?;
interpolate_variables_secrets_into_string(
&vars_and_secrets,
&mut build.config.build_args,
&mut global_replacers,
&mut secret_replacers,
)?;
if let Some(repo) = repo.as_mut() {
interpolator.interpolate_repo(repo)?;
}
interpolate_variables_secrets_into_string(
&vars_and_secrets,
&mut build.config.secret_args,
&mut global_replacers,
&mut secret_replacers,
)?;
interpolator.push_logs(&mut update.logs);
interpolate_variables_secrets_into_string(
&vars_and_secrets,
&mut build.config.dockerfile,
&mut global_replacers,
&mut secret_replacers,
)?;
interpolate_variables_secrets_into_extra_args(
&vars_and_secrets,
&mut build.config.extra_args,
&mut global_replacers,
&mut secret_replacers,
)?;
add_interp_update_log(
&mut update,
&global_replacers,
&secret_replacers,
);
secret_replacers
interpolator.secret_replacers
} else {
Default::default()
};
let commit_message = if !build.config.files_on_host
&& !build.config.repo.is_empty()
&& (!build.config.repo.is_empty()
|| !build.config.linked_repo.is_empty())
{
// CLONE REPO
// PULL OR CLONE REPO
let res = tokio::select! {
res = periphery
.request(api::git::CloneRepo {
args: (&build).into(),
.request(api::git::PullOrCloneRepo {
args: repo.as_ref().map(Into::into).unwrap_or((&build).into()),
git_token,
environment: Default::default(),
env_file_path: Default::default(),
on_clone: None,
on_pull: None,
skip_secret_interp: Default::default(),
replacers: Default::default(),
}) => res,
@@ -279,16 +255,16 @@ impl Resolve<ExecuteArgs> for RunBuild {
let commit_message = match res {
Ok(res) => {
debug!("finished repo clone");
update.logs.extend(res.logs);
update.logs.extend(res.res.logs);
update.commit_hash =
res.commit_hash.unwrap_or_default().to_string();
res.commit_message.unwrap_or_default()
res.res.commit_hash.unwrap_or_default().to_string();
res.res.commit_message.unwrap_or_default()
}
Err(e) => {
warn!("failed build at clone repo | {e:#}");
warn!("Failed build at clone repo | {e:#}");
update.push_error_log(
"clone repo",
format_serror(&e.context("failed to clone repo").into()),
"Clone Repo",
format_serror(&e.context("Failed to clone repo").into()),
);
Default::default()
}
@@ -307,6 +283,7 @@ impl Resolve<ExecuteArgs> for RunBuild {
res = periphery
.request(api::build::Build {
build: build.clone(),
repo,
registry_token,
replacers: secret_replacers.into_iter().collect(),
// Push a commit hash tagged image
@@ -588,8 +565,9 @@ async fn handle_post_build_redeploy(build_id: &str) {
redeploy_deployments
.into_iter()
.map(|deployment| async move {
let state =
get_deployment_state(&deployment).await.unwrap_or_default();
let state = get_deployment_state(&deployment.id)
.await
.unwrap_or_default();
if state == DeploymentState::Running {
let req = super::ExecuteRequest::Deploy(Deploy {
deployment: deployment.id.clone(),
+11 -51
View File
@@ -1,8 +1,9 @@
use std::{collections::HashSet, sync::OnceLock};
use std::sync::OnceLock;
use anyhow::{Context, anyhow};
use cache::TimeoutCache;
use formatting::format_serror;
use interpolate::Interpolator;
use komodo_client::{
api::execute::*,
entities::{
@@ -23,13 +24,8 @@ use resolver_api::Resolve;
use crate::{
helpers::{
interpolate::{
add_interp_update_log,
interpolate_variables_secrets_into_extra_args,
interpolate_variables_secrets_into_string,
},
periphery_client,
query::get_variables_and_secrets,
query::{VariablesAndSecrets, get_variables_and_secrets},
registry_token,
update::update_update,
},
@@ -180,53 +176,17 @@ impl Resolve<ExecuteArgs> for Deploy {
// interpolate variables / secrets, returning the sanitizing replacers to send to
// periphery so it may sanitize the final command for safe logging (avoids exposing secret values)
let secret_replacers = if !deployment.config.skip_secret_interp {
let vars_and_secrets = get_variables_and_secrets().await?;
let VariablesAndSecrets { variables, secrets } =
get_variables_and_secrets().await?;
let mut global_replacers = HashSet::new();
let mut secret_replacers = HashSet::new();
let mut interpolator =
Interpolator::new(Some(&variables), &secrets);
interpolate_variables_secrets_into_string(
&vars_and_secrets,
&mut deployment.config.environment,
&mut global_replacers,
&mut secret_replacers,
)?;
interpolator
.interpolate_deployment(&mut deployment)?
.push_logs(&mut update.logs);
interpolate_variables_secrets_into_string(
&vars_and_secrets,
&mut deployment.config.ports,
&mut global_replacers,
&mut secret_replacers,
)?;
interpolate_variables_secrets_into_string(
&vars_and_secrets,
&mut deployment.config.volumes,
&mut global_replacers,
&mut secret_replacers,
)?;
interpolate_variables_secrets_into_extra_args(
&vars_and_secrets,
&mut deployment.config.extra_args,
&mut global_replacers,
&mut secret_replacers,
)?;
interpolate_variables_secrets_into_string(
&vars_and_secrets,
&mut deployment.config.command,
&mut global_replacers,
&mut secret_replacers,
)?;
add_interp_update_log(
&mut update,
&global_replacers,
&secret_replacers,
);
secret_replacers
interpolator.secret_replacers
} else {
Default::default()
};
+5 -2
View File
@@ -174,8 +174,11 @@ async fn handler(
Ok((TypedHeader(ContentType::json()), res))
}
#[typeshare(serialized_as = "Update")]
type BoxUpdate = Box<Update>;
pub enum ExecutionResult {
Single(Update),
Single(BoxUpdate),
/// The batch contents will be pre serialized here
Batch(String),
}
@@ -245,7 +248,7 @@ pub fn inner_handler(
}
});
Ok(ExecutionResult::Single(update))
Ok(ExecutionResult::Single(update.into()))
})
}
+31 -52
View File
@@ -2,6 +2,7 @@ use std::{collections::HashSet, future::IntoFuture, time::Duration};
use anyhow::{Context, anyhow};
use formatting::format_serror;
use interpolate::Interpolator;
use komodo_client::{
api::{execute::*, write::RefreshRepoCache},
entities::{
@@ -31,14 +32,8 @@ use crate::{
helpers::{
builder::{cleanup_builder_instance, get_builder_periphery},
channel::repo_cancel_channel,
git_token,
interpolate::{
add_interp_update_log,
interpolate_variables_secrets_into_string,
interpolate_variables_secrets_into_system_command,
},
periphery_client,
query::get_variables_and_secrets,
git_token, periphery_client,
query::{VariablesAndSecrets, get_variables_and_secrets},
update::update_update,
},
permission::get_check_permissions,
@@ -123,16 +118,18 @@ impl Resolve<ExecuteArgs> for CloneRepo {
git_token,
environment: repo.config.env_vars()?,
env_file_path: repo.config.env_file_path,
on_clone: repo.config.on_clone.into(),
on_pull: repo.config.on_pull.into(),
skip_secret_interp: repo.config.skip_secret_interp,
replacers: secret_replacers.into_iter().collect(),
})
.await
{
Ok(res) => res.logs,
Ok(res) => res.res.logs,
Err(e) => {
vec![Log::error(
"clone repo",
format_serror(&e.context("failed to clone repo").into()),
"Clone Repo",
format_serror(&e.context("Failed to clone repo").into()),
)]
}
};
@@ -156,14 +153,14 @@ impl Resolve<ExecuteArgs> for CloneRepo {
);
};
handle_server_update_return(update).await
handle_repo_update_return(update).await
}
}
impl super::BatchExecute for BatchPullRepo {
type Resource = Repo;
fn single_request(repo: String) -> ExecuteRequest {
ExecuteRequest::CloneRepo(CloneRepo { repo })
ExecuteRequest::PullRepo(PullRepo { repo })
}
}
@@ -236,14 +233,15 @@ impl Resolve<ExecuteArgs> for PullRepo {
git_token,
environment: repo.config.env_vars()?,
env_file_path: repo.config.env_file_path,
on_pull: repo.config.on_pull.into(),
skip_secret_interp: repo.config.skip_secret_interp,
replacers: secret_replacers.into_iter().collect(),
})
.await
{
Ok(res) => {
update.commit_hash = res.commit_hash.unwrap_or_default();
res.logs
update.commit_hash = res.res.commit_hash.unwrap_or_default();
res.res.logs
}
Err(e) => {
vec![Log::error(
@@ -273,12 +271,12 @@ impl Resolve<ExecuteArgs> for PullRepo {
);
};
handle_server_update_return(update).await
handle_repo_update_return(update).await
}
}
#[instrument(skip_all, fields(update_id = update.id))]
async fn handle_server_update_return(
async fn handle_repo_update_return(
update: Update,
) -> serror::Result<Update> {
// Need to manually update the update before cache refresh,
@@ -457,6 +455,8 @@ impl Resolve<ExecuteArgs> for BuildRepo {
git_token,
environment: repo.config.env_vars()?,
env_file_path: repo.config.env_file_path,
on_clone: repo.config.on_clone.into(),
on_pull: repo.config.on_pull.into(),
skip_secret_interp: repo.config.skip_secret_interp,
replacers: secret_replacers.into_iter().collect()
}) => res,
@@ -473,14 +473,15 @@ impl Resolve<ExecuteArgs> for BuildRepo {
let commit_message = match res {
Ok(res) => {
debug!("finished repo clone");
update.logs.extend(res.logs);
update.commit_hash = res.commit_hash.unwrap_or_default();
res.commit_message.unwrap_or_default()
update.logs.extend(res.res.logs);
update.commit_hash = res.res.commit_hash.unwrap_or_default();
res.res.commit_message.unwrap_or_default()
}
Err(e) => {
update.push_error_log(
"clone repo",
format_serror(&e.context("failed to clone repo").into()),
"Clone Repo",
format_serror(&e.context("Failed to clone repo").into()),
);
Default::default()
}
@@ -712,39 +713,17 @@ async fn interpolate(
update: &mut Update,
) -> anyhow::Result<HashSet<(String, String)>> {
if !repo.config.skip_secret_interp {
let vars_and_secrets = get_variables_and_secrets().await?;
let VariablesAndSecrets { variables, secrets } =
get_variables_and_secrets().await?;
let mut global_replacers = HashSet::new();
let mut secret_replacers = HashSet::new();
let mut interpolator =
Interpolator::new(Some(&variables), &secrets);
interpolate_variables_secrets_into_string(
&vars_and_secrets,
&mut repo.config.environment,
&mut global_replacers,
&mut secret_replacers,
)?;
interpolator
.interpolate_repo(repo)?
.push_logs(&mut update.logs);
interpolate_variables_secrets_into_system_command(
&vars_and_secrets,
&mut repo.config.on_clone,
&mut global_replacers,
&mut secret_replacers,
)?;
interpolate_variables_secrets_into_system_command(
&vars_and_secrets,
&mut repo.config.on_pull,
&mut global_replacers,
&mut secret_replacers,
)?;
add_interp_update_log(
update,
&global_replacers,
&secret_replacers,
);
Ok(secret_replacers)
Ok(interpolator.secret_replacers)
} else {
Ok(Default::default())
}
+60 -98
View File
@@ -1,11 +1,11 @@
use std::collections::HashSet;
use anyhow::Context;
use formatting::format_serror;
use interpolate::Interpolator;
use komodo_client::{
api::{execute::*, write::RefreshStackCache},
entities::{
permission::PermissionLevel,
repo::Repo,
server::Server,
stack::{Stack, StackInfo},
update::{Log, Update},
@@ -18,14 +18,9 @@ use resolver_api::Resolve;
use crate::{
api::write::WriteArgs,
helpers::{
interpolate::{
add_interp_update_log,
interpolate_variables_secrets_into_extra_args,
interpolate_variables_secrets_into_string,
interpolate_variables_secrets_into_system_command,
},
periphery_client,
query::get_variables_and_secrets,
query::{VariablesAndSecrets, get_variables_and_secrets},
stack_git_token,
update::{add_update_without_send, update_update},
},
monitor::update_cache_for_server,
@@ -75,6 +70,16 @@ impl Resolve<ExecuteArgs> for DeployStack {
)
.await?;
let mut repo = if !stack.config.files_on_host
&& !stack.config.linked_repo.is_empty()
{
crate::resource::get::<Repo>(&stack.config.linked_repo)
.await?
.into()
} else {
None
};
// get the action state for the stack (or insert default).
let action_state =
action_states().stack.get_or_insert_default(&stack.id).await;
@@ -98,13 +103,8 @@ impl Resolve<ExecuteArgs> for DeployStack {
))
}
let git_token = crate::helpers::git_token(
&stack.config.git_provider,
&stack.config.git_account,
|https| stack.config.git_https = https,
).await.with_context(
|| format!("Failed to get git token in call to db. Stopping run. | {} | {}", stack.config.git_provider, stack.config.git_account),
)?;
let git_token =
stack_git_token(&mut stack, repo.as_mut()).await?;
let registry_token = crate::helpers::registry_token(
&stack.config.registry_provider,
@@ -116,60 +116,21 @@ impl Resolve<ExecuteArgs> for DeployStack {
// interpolate variables / secrets, returning the sanitizing replacers to send to
// periphery so it may sanitize the final command for safe logging (avoids exposing secret values)
let secret_replacers = if !stack.config.skip_secret_interp {
let vars_and_secrets = get_variables_and_secrets().await?;
let VariablesAndSecrets { variables, secrets } =
get_variables_and_secrets().await?;
let mut global_replacers = HashSet::new();
let mut secret_replacers = HashSet::new();
let mut interpolator =
Interpolator::new(Some(&variables), &secrets);
interpolate_variables_secrets_into_string(
&vars_and_secrets,
&mut stack.config.file_contents,
&mut global_replacers,
&mut secret_replacers,
)?;
interpolator.interpolate_stack(&mut stack)?;
if let Some(repo) = repo.as_mut() {
if !repo.config.skip_secret_interp {
interpolator.interpolate_repo(repo)?;
}
}
interpolator.push_logs(&mut update.logs);
interpolate_variables_secrets_into_string(
&vars_and_secrets,
&mut stack.config.environment,
&mut global_replacers,
&mut secret_replacers,
)?;
interpolate_variables_secrets_into_extra_args(
&vars_and_secrets,
&mut stack.config.extra_args,
&mut global_replacers,
&mut secret_replacers,
)?;
interpolate_variables_secrets_into_extra_args(
&vars_and_secrets,
&mut stack.config.build_extra_args,
&mut global_replacers,
&mut secret_replacers,
)?;
interpolate_variables_secrets_into_system_command(
&vars_and_secrets,
&mut stack.config.pre_deploy,
&mut global_replacers,
&mut secret_replacers,
)?;
interpolate_variables_secrets_into_system_command(
&vars_and_secrets,
&mut stack.config.post_deploy,
&mut global_replacers,
&mut secret_replacers,
)?;
add_interp_update_log(
&mut update,
&global_replacers,
&secret_replacers,
);
secret_replacers
interpolator.secret_replacers
} else {
Default::default()
};
@@ -188,6 +149,7 @@ impl Resolve<ExecuteArgs> for DeployStack {
.request(ComposeUp {
stack: stack.clone(),
services: self.services,
repo,
git_token,
registry_token,
replacers: secret_replacers.into_iter().collect(),
@@ -413,6 +375,7 @@ pub async fn pull_stack_inner(
mut stack: Stack,
services: Vec<String>,
server: &Server,
mut repo: Option<Repo>,
mut update: Option<&mut Update>,
) -> anyhow::Result<ComposePullResponse> {
if let Some(update) = update.as_mut() {
@@ -427,13 +390,7 @@ pub async fn pull_stack_inner(
}
}
let git_token = crate::helpers::git_token(
&stack.config.git_provider,
&stack.config.git_account,
|https| stack.config.git_https = https,
).await.with_context(
|| format!("Failed to get git token in call to db. Stopping run. | {} | {}", stack.config.git_provider, stack.config.git_account),
)?;
let git_token = stack_git_token(&mut stack, repo.as_mut()).await?;
let registry_token = crate::helpers::registry_token(
&stack.config.registry_provider,
@@ -443,41 +400,35 @@ pub async fn pull_stack_inner(
)?;
// interpolate variables / secrets
if !stack.config.skip_secret_interp {
let vars_and_secrets = get_variables_and_secrets().await?;
let secret_replacers = if !stack.config.skip_secret_interp {
let VariablesAndSecrets { variables, secrets } =
get_variables_and_secrets().await?;
let mut global_replacers = HashSet::new();
let mut secret_replacers = HashSet::new();
let mut interpolator =
Interpolator::new(Some(&variables), &secrets);
interpolate_variables_secrets_into_string(
&vars_and_secrets,
&mut stack.config.file_contents,
&mut global_replacers,
&mut secret_replacers,
)?;
interpolate_variables_secrets_into_string(
&vars_and_secrets,
&mut stack.config.environment,
&mut global_replacers,
&mut secret_replacers,
)?;
if let Some(update) = update {
add_interp_update_log(
update,
&global_replacers,
&secret_replacers,
);
interpolator.interpolate_stack(&mut stack)?;
if let Some(repo) = repo.as_mut() {
if !repo.config.skip_secret_interp {
interpolator.interpolate_repo(repo)?;
}
}
if let Some(update) = update {
interpolator.push_logs(&mut update.logs);
}
interpolator.secret_replacers
} else {
Default::default()
};
let res = periphery_client(server)?
.request(ComposePull {
stack,
services,
repo,
git_token,
registry_token,
replacers: secret_replacers.into_iter().collect(),
})
.await?;
@@ -501,6 +452,16 @@ impl Resolve<ExecuteArgs> for PullStack {
)
.await?;
let repo = if !stack.config.files_on_host
&& !stack.config.linked_repo.is_empty()
{
crate::resource::get::<Repo>(&stack.config.linked_repo)
.await?
.into()
} else {
None
};
// get the action state for the stack (or insert default).
let action_state =
action_states().stack.get_or_insert_default(&stack.id).await;
@@ -517,6 +478,7 @@ impl Resolve<ExecuteArgs> for PullStack {
stack,
self.services,
&server,
repo,
Some(&mut update),
)
.await?;
+19 -17
View File
@@ -28,11 +28,14 @@ use resolver_api::Resolve;
use crate::{
api::write::WriteArgs,
helpers::{query::get_id_to_tags, update::update_update},
helpers::{
all_resources::AllResourcesById, query::get_id_to_tags,
update::update_update,
},
permission::get_check_permissions,
state::{action_states, db_client},
sync::{
AllResourcesById, ResourceSyncTrait,
ResourceSyncTrait,
deploy::{
SyncDeployParams, build_deploy_cache, deploy_from_cache,
},
@@ -61,6 +64,16 @@ impl Resolve<ExecuteArgs> for RunSync {
)
.await?;
let repo = if !sync.config.files_on_host
&& !sync.config.linked_repo.is_empty()
{
crate::resource::get::<Repo>(&sync.config.linked_repo)
.await?
.into()
} else {
None
};
// get the action state for the sync (or insert default).
let action_state = action_states()
.resource_sync
@@ -84,9 +97,10 @@ impl Resolve<ExecuteArgs> for RunSync {
message,
file_errors,
..
} = crate::sync::remote::get_remote_resources(&sync)
.await
.context("failed to get remote resources")?;
} =
crate::sync::remote::get_remote_resources(&sync, repo.as_ref())
.await
.context("failed to get remote resources")?;
update.logs.extend(logs);
update_update(update.clone()).await?;
@@ -197,7 +211,6 @@ impl Resolve<ExecuteArgs> for RunSync {
deployment_map: &deployments_by_name,
stacks: &resources.stacks,
stack_map: &stacks_by_name,
all_resources: &all_resources,
})
.await?;
@@ -207,7 +220,6 @@ impl Resolve<ExecuteArgs> for RunSync {
get_updates_for_execution::<Server>(
resources.servers,
delete,
&all_resources,
match_resource_type,
match_resources.as_deref(),
&id_to_tags,
@@ -221,7 +233,6 @@ impl Resolve<ExecuteArgs> for RunSync {
get_updates_for_execution::<Stack>(
resources.stacks,
delete,
&all_resources,
match_resource_type,
match_resources.as_deref(),
&id_to_tags,
@@ -235,7 +246,6 @@ impl Resolve<ExecuteArgs> for RunSync {
get_updates_for_execution::<Deployment>(
resources.deployments,
delete,
&all_resources,
match_resource_type,
match_resources.as_deref(),
&id_to_tags,
@@ -249,7 +259,6 @@ impl Resolve<ExecuteArgs> for RunSync {
get_updates_for_execution::<Build>(
resources.builds,
delete,
&all_resources,
match_resource_type,
match_resources.as_deref(),
&id_to_tags,
@@ -263,7 +272,6 @@ impl Resolve<ExecuteArgs> for RunSync {
get_updates_for_execution::<Repo>(
resources.repos,
delete,
&all_resources,
match_resource_type,
match_resources.as_deref(),
&id_to_tags,
@@ -277,7 +285,6 @@ impl Resolve<ExecuteArgs> for RunSync {
get_updates_for_execution::<Procedure>(
resources.procedures,
delete,
&all_resources,
match_resource_type,
match_resources.as_deref(),
&id_to_tags,
@@ -291,7 +298,6 @@ impl Resolve<ExecuteArgs> for RunSync {
get_updates_for_execution::<Action>(
resources.actions,
delete,
&all_resources,
match_resource_type,
match_resources.as_deref(),
&id_to_tags,
@@ -305,7 +311,6 @@ impl Resolve<ExecuteArgs> for RunSync {
get_updates_for_execution::<Builder>(
resources.builders,
delete,
&all_resources,
match_resource_type,
match_resources.as_deref(),
&id_to_tags,
@@ -319,7 +324,6 @@ impl Resolve<ExecuteArgs> for RunSync {
get_updates_for_execution::<Alerter>(
resources.alerters,
delete,
&all_resources,
match_resource_type,
match_resources.as_deref(),
&id_to_tags,
@@ -333,7 +337,6 @@ impl Resolve<ExecuteArgs> for RunSync {
get_updates_for_execution::<entities::sync::ResourceSync>(
resources.resource_syncs,
delete,
&all_resources,
match_resource_type,
match_resources.as_deref(),
&id_to_tags,
@@ -371,7 +374,6 @@ impl Resolve<ExecuteArgs> for RunSync {
crate::sync::user_groups::get_updates_for_execution(
resources.user_groups,
delete,
&all_resources,
)
.await?
} else {
+5
View File
@@ -43,6 +43,7 @@ mod permission;
mod procedure;
mod provider;
mod repo;
mod schedule;
mod server;
mod stack;
mod sync;
@@ -98,6 +99,9 @@ enum ReadRequest {
ListActions(ListActions),
ListFullActions(ListFullActions),
// ==== SCHEDULE ====
ListSchedules(ListSchedules),
// ==== SERVER ====
GetServersSummary(GetServersSummary),
GetServer(GetServer),
@@ -292,6 +296,7 @@ fn core_info() -> &'static GetCoreInfoResponse {
.iter()
.map(|i| i.namespace.to_string())
.collect(),
timezone: config.timezone.clone(),
}
})
}
+107
View File
@@ -0,0 +1,107 @@
use futures::future::join_all;
use komodo_client::{
api::read::*,
entities::{
ResourceTarget,
action::Action,
permission::PermissionLevel,
procedure::Procedure,
resource::{ResourceQuery, TemplatesQueryBehavior},
schedule::Schedule,
},
};
use resolver_api::Resolve;
use crate::{
helpers::query::{get_all_tags, get_last_run_at},
resource::list_full_for_user,
schedule::get_schedule_item_info,
};
use super::ReadArgs;
impl Resolve<ReadArgs> for ListSchedules {
async fn resolve(
self,
args: &ReadArgs,
) -> serror::Result<Vec<Schedule>> {
let all_tags = get_all_tags(None).await?;
let (actions, procedures) = tokio::try_join!(
list_full_for_user::<Action>(
ResourceQuery {
names: Default::default(),
templates: TemplatesQueryBehavior::Include,
tag_behavior: self.tag_behavior,
tags: self.tags.clone(),
specific: Default::default(),
},
&args.user,
PermissionLevel::Read.into(),
&all_tags,
),
list_full_for_user::<Procedure>(
ResourceQuery {
names: Default::default(),
templates: TemplatesQueryBehavior::Include,
tag_behavior: self.tag_behavior,
tags: self.tags.clone(),
specific: Default::default(),
},
&args.user,
PermissionLevel::Read.into(),
&all_tags,
)
)?;
let actions = actions.into_iter().map(async |action| {
let (next_scheduled_run, schedule_error) =
get_schedule_item_info(&ResourceTarget::Action(
action.id.clone(),
));
let last_run_at =
get_last_run_at::<Action>(&action.id).await.unwrap_or(None);
Schedule {
target: ResourceTarget::Action(action.id),
name: action.name,
enabled: action.config.schedule_enabled,
schedule_format: action.config.schedule_format,
schedule: action.config.schedule,
schedule_timezone: action.config.schedule_timezone,
tags: action.tags,
last_run_at,
next_scheduled_run,
schedule_error,
}
});
let procedures = procedures.into_iter().map(async |procedure| {
let (next_scheduled_run, schedule_error) =
get_schedule_item_info(&ResourceTarget::Procedure(
procedure.id.clone(),
));
let last_run_at = get_last_run_at::<Procedure>(&procedure.id)
.await
.unwrap_or(None);
Schedule {
target: ResourceTarget::Procedure(procedure.id),
name: procedure.name,
enabled: procedure.config.schedule_enabled,
schedule_format: procedure.config.schedule_format,
schedule: procedure.config.schedule,
schedule_timezone: procedure.config.schedule_timezone,
tags: procedure.tags,
last_run_at,
next_scheduled_run,
schedule_error,
}
});
let (actions, procedures) =
tokio::join!(join_all(actions), join_all(procedures));
Ok(
actions
.into_iter()
.chain(procedures)
.filter(|s| !s.schedule.is_empty())
.collect(),
)
}
}
+1 -1
View File
@@ -176,7 +176,7 @@ impl Resolve<ReadArgs> for InspectStackContainer {
.curr
.services;
let Some(name) = services
.into_iter()
.iter()
.find(|s| s.service == service)
.and_then(|s| s.container.as_ref().map(|c| c.name.clone()))
else {
+26 -25
View File
@@ -24,7 +24,6 @@ use crate::{
resource,
state::db_client,
sync::{
AllResourcesById,
toml::{ToToml, convert_resource},
user_groups::{convert_user_groups, user_group_to_toml},
variables::variable_to_toml,
@@ -44,7 +43,7 @@ async fn get_all_targets(
get_all_tags(None).await?
};
targets.extend(
resource::list_for_user::<Alerter>(
resource::list_full_for_user::<Alerter>(
ResourceQuery::builder().tags(tags).build(),
user,
PermissionLevel::Read.into(),
@@ -55,7 +54,7 @@ async fn get_all_targets(
.map(|resource| ResourceTarget::Alerter(resource.id)),
);
targets.extend(
resource::list_for_user::<Builder>(
resource::list_full_for_user::<Builder>(
ResourceQuery::builder().tags(tags).build(),
user,
PermissionLevel::Read.into(),
@@ -66,7 +65,7 @@ async fn get_all_targets(
.map(|resource| ResourceTarget::Builder(resource.id)),
);
targets.extend(
resource::list_for_user::<Server>(
resource::list_full_for_user::<Server>(
ResourceQuery::builder().tags(tags).build(),
user,
PermissionLevel::Read.into(),
@@ -77,7 +76,7 @@ async fn get_all_targets(
.map(|resource| ResourceTarget::Server(resource.id)),
);
targets.extend(
resource::list_for_user::<Stack>(
resource::list_full_for_user::<Stack>(
ResourceQuery::builder().tags(tags).build(),
user,
PermissionLevel::Read.into(),
@@ -88,7 +87,7 @@ async fn get_all_targets(
.map(|resource| ResourceTarget::Stack(resource.id)),
);
targets.extend(
resource::list_for_user::<Deployment>(
resource::list_full_for_user::<Deployment>(
ResourceQuery::builder().tags(tags).build(),
user,
PermissionLevel::Read.into(),
@@ -99,7 +98,7 @@ async fn get_all_targets(
.map(|resource| ResourceTarget::Deployment(resource.id)),
);
targets.extend(
resource::list_for_user::<Build>(
resource::list_full_for_user::<Build>(
ResourceQuery::builder().tags(tags).build(),
user,
PermissionLevel::Read.into(),
@@ -110,7 +109,7 @@ async fn get_all_targets(
.map(|resource| ResourceTarget::Build(resource.id)),
);
targets.extend(
resource::list_for_user::<Repo>(
resource::list_full_for_user::<Repo>(
ResourceQuery::builder().tags(tags).build(),
user,
PermissionLevel::Read.into(),
@@ -121,7 +120,7 @@ async fn get_all_targets(
.map(|resource| ResourceTarget::Repo(resource.id)),
);
targets.extend(
resource::list_for_user::<Procedure>(
resource::list_full_for_user::<Procedure>(
ResourceQuery::builder().tags(tags).build(),
user,
PermissionLevel::Read.into(),
@@ -132,7 +131,7 @@ async fn get_all_targets(
.map(|resource| ResourceTarget::Procedure(resource.id)),
);
targets.extend(
resource::list_for_user::<Action>(
resource::list_full_for_user::<Action>(
ResourceQuery::builder().tags(tags).build(),
user,
PermissionLevel::Read.into(),
@@ -204,18 +203,18 @@ impl Resolve<ReadArgs> for ExportResourcesToToml {
include_variables,
} = self;
let mut res = ResourcesToml::default();
let all = AllResourcesById::load().await?;
let id_to_tags = get_id_to_tags(None).await?;
let ReadArgs { user } = args;
for target in targets {
match target {
ResourceTarget::Alerter(id) => {
let alerter = get_check_permissions::<Alerter>(
let mut alerter = get_check_permissions::<Alerter>(
&id,
user,
PermissionLevel::Read.into(),
)
.await?;
Alerter::replace_ids(&mut alerter);
res.alerters.push(convert_resource::<Alerter>(
alerter,
false,
@@ -224,7 +223,7 @@ impl Resolve<ReadArgs> for ExportResourcesToToml {
))
}
ResourceTarget::ResourceSync(id) => {
let sync = get_check_permissions::<ResourceSync>(
let mut sync = get_check_permissions::<ResourceSync>(
&id,
user,
PermissionLevel::Read.into(),
@@ -232,8 +231,10 @@ impl Resolve<ReadArgs> for ExportResourcesToToml {
.await?;
if sync.config.file_contents.is_empty()
&& (sync.config.files_on_host
|| !sync.config.repo.is_empty())
|| !sync.config.repo.is_empty()
|| !sync.config.linked_repo.is_empty())
{
ResourceSync::replace_ids(&mut sync);
res.resource_syncs.push(convert_resource::<ResourceSync>(
sync,
false,
@@ -243,12 +244,13 @@ impl Resolve<ReadArgs> for ExportResourcesToToml {
}
}
ResourceTarget::Server(id) => {
let server = get_check_permissions::<Server>(
let mut server = get_check_permissions::<Server>(
&id,
user,
PermissionLevel::Read.into(),
)
.await?;
Server::replace_ids(&mut server);
res.servers.push(convert_resource::<Server>(
server,
false,
@@ -263,7 +265,7 @@ impl Resolve<ReadArgs> for ExportResourcesToToml {
PermissionLevel::Read.into(),
)
.await?;
Builder::replace_ids(&mut builder, &all);
Builder::replace_ids(&mut builder);
res.builders.push(convert_resource::<Builder>(
builder,
false,
@@ -278,7 +280,7 @@ impl Resolve<ReadArgs> for ExportResourcesToToml {
PermissionLevel::Read.into(),
)
.await?;
Build::replace_ids(&mut build, &all);
Build::replace_ids(&mut build);
res.builds.push(convert_resource::<Build>(
build,
false,
@@ -293,7 +295,7 @@ impl Resolve<ReadArgs> for ExportResourcesToToml {
PermissionLevel::Read.into(),
)
.await?;
Deployment::replace_ids(&mut deployment, &all);
Deployment::replace_ids(&mut deployment);
res.deployments.push(convert_resource::<Deployment>(
deployment,
false,
@@ -308,7 +310,7 @@ impl Resolve<ReadArgs> for ExportResourcesToToml {
PermissionLevel::Read.into(),
)
.await?;
Repo::replace_ids(&mut repo, &all);
Repo::replace_ids(&mut repo);
res.repos.push(convert_resource::<Repo>(
repo,
false,
@@ -323,7 +325,7 @@ impl Resolve<ReadArgs> for ExportResourcesToToml {
PermissionLevel::Read.into(),
)
.await?;
Stack::replace_ids(&mut stack, &all);
Stack::replace_ids(&mut stack);
res.stacks.push(convert_resource::<Stack>(
stack,
false,
@@ -338,7 +340,7 @@ impl Resolve<ReadArgs> for ExportResourcesToToml {
PermissionLevel::Read.into(),
)
.await?;
Procedure::replace_ids(&mut procedure, &all);
Procedure::replace_ids(&mut procedure);
res.procedures.push(convert_resource::<Procedure>(
procedure,
false,
@@ -353,7 +355,7 @@ impl Resolve<ReadArgs> for ExportResourcesToToml {
PermissionLevel::Read.into(),
)
.await?;
Action::replace_ids(&mut action, &all);
Action::replace_ids(&mut action);
res.actions.push(convert_resource::<Action>(
action,
false,
@@ -365,7 +367,7 @@ impl Resolve<ReadArgs> for ExportResourcesToToml {
};
}
add_user_groups(user_groups, &mut res, &all, args)
add_user_groups(user_groups, &mut res, args)
.await
.context("failed to add user groups")?;
@@ -394,7 +396,6 @@ impl Resolve<ReadArgs> for ExportResourcesToToml {
async fn add_user_groups(
user_groups: Vec<String>,
res: &mut ResourcesToml,
all: &AllResourcesById,
args: &ReadArgs,
) -> anyhow::Result<()> {
let user_groups = ListUserGroups {}
@@ -406,7 +407,7 @@ async fn add_user_groups(
user_groups.contains(&ug.name) || user_groups.contains(&ug.id)
});
let mut ug = Vec::with_capacity(user_groups.size_hint().0);
convert_user_groups(user_groups, all, &mut ug).await?;
convert_user_groups(user_groups, &mut ug).await?;
res.user_groups = ug.into_iter().map(|ug| ug.1).collect();
Ok(())
+232 -9
View File
@@ -1,9 +1,10 @@
use anyhow::Context;
use axum::{Extension, Router, middleware, routing::post};
use komodo_client::{
api::terminal::ExecuteTerminalBody,
api::terminal::*,
entities::{
permission::PermissionLevel, server::Server, user::User,
deployment::Deployment, permission::PermissionLevel,
server::Server, stack::Stack, user::User,
},
};
use serror::Json;
@@ -11,20 +12,28 @@ use uuid::Uuid;
use crate::{
auth::auth_request, helpers::periphery_client,
permission::get_check_permissions,
permission::get_check_permissions, resource::get,
state::stack_status_cache,
};
pub fn router() -> Router {
Router::new()
.route("/execute", post(execute))
.route("/execute", post(execute_terminal))
.route("/execute/container", post(execute_container_exec))
.route("/execute/deployment", post(execute_deployment_exec))
.route("/execute/stack", post(execute_stack_exec))
.layer(middleware::from_fn(auth_request))
}
async fn execute(
// =================
// ExecuteTerminal
// =================
async fn execute_terminal(
Extension(user): Extension<User>,
Json(request): Json<ExecuteTerminalBody>,
) -> serror::Result<axum::body::Body> {
execute_inner(Uuid::new_v4(), request, user).await
execute_terminal_inner(Uuid::new_v4(), request, user).await
}
#[instrument(
@@ -34,7 +43,7 @@ async fn execute(
user_id = user.id,
)
)]
async fn execute_inner(
async fn execute_terminal_inner(
req_id: Uuid,
ExecuteTerminalBody {
server,
@@ -43,7 +52,7 @@ async fn execute_inner(
}: ExecuteTerminalBody,
user: User,
) -> serror::Result<axum::body::Body> {
info!("/terminal request | user: {}", user.username);
info!("/terminal/execute request | user: {}", user.username);
let res = async {
let server = get_check_permissions::<Server>(
@@ -67,7 +76,221 @@ async fn execute_inner(
let stream = match res {
Ok(stream) => stream,
Err(e) => {
warn!("/terminal request {req_id} error: {e:#}");
warn!("/terminal/execute request {req_id} error: {e:#}");
return Err(e.into());
}
};
Ok(axum::body::Body::from_stream(stream.into_line_stream()))
}
// ======================
// ExecuteContainerExec
// ======================
async fn execute_container_exec(
Extension(user): Extension<User>,
Json(request): Json<ExecuteContainerExecBody>,
) -> serror::Result<axum::body::Body> {
execute_container_exec_inner(Uuid::new_v4(), request, user).await
}
#[instrument(
name = "ExecuteContainerExec",
skip(user),
fields(
user_id = user.id,
)
)]
async fn execute_container_exec_inner(
req_id: Uuid,
ExecuteContainerExecBody {
server,
container,
shell,
command,
}: ExecuteContainerExecBody,
user: User,
) -> serror::Result<axum::body::Body> {
info!(
"/terminal/execute/container request | user: {}",
user.username
);
let res = async {
let server = get_check_permissions::<Server>(
&server,
&user,
PermissionLevel::Read.terminal(),
)
.await?;
let periphery = periphery_client(&server)?;
let stream = periphery
.execute_container_exec(container, shell, command)
.await
.context(
"Failed to execute container exec command on periphery",
)?;
anyhow::Ok(stream)
}
.await;
let stream = match res {
Ok(stream) => stream,
Err(e) => {
warn!(
"/terminal/execute/container request {req_id} error: {e:#}"
);
return Err(e.into());
}
};
Ok(axum::body::Body::from_stream(stream.into_line_stream()))
}
// =======================
// ExecuteDeploymentExec
// =======================
async fn execute_deployment_exec(
Extension(user): Extension<User>,
Json(request): Json<ExecuteDeploymentExecBody>,
) -> serror::Result<axum::body::Body> {
execute_deployment_exec_inner(Uuid::new_v4(), request, user).await
}
#[instrument(
name = "ExecuteDeploymentExec",
skip(user),
fields(
user_id = user.id,
)
)]
async fn execute_deployment_exec_inner(
req_id: Uuid,
ExecuteDeploymentExecBody {
deployment,
shell,
command,
}: ExecuteDeploymentExecBody,
user: User,
) -> serror::Result<axum::body::Body> {
info!(
"/terminal/execute/deployment request | user: {}",
user.username
);
let res = async {
let deployment = get_check_permissions::<Deployment>(
&deployment,
&user,
PermissionLevel::Read.terminal(),
)
.await?;
let server = get::<Server>(&deployment.config.server_id).await?;
let periphery = periphery_client(&server)?;
let stream = periphery
.execute_container_exec(deployment.name, shell, command)
.await
.context(
"Failed to execute container exec command on periphery",
)?;
anyhow::Ok(stream)
}
.await;
let stream = match res {
Ok(stream) => stream,
Err(e) => {
warn!(
"/terminal/execute/deployment request {req_id} error: {e:#}"
);
return Err(e.into());
}
};
Ok(axum::body::Body::from_stream(stream.into_line_stream()))
}
// ==================
// ExecuteStackExec
// ==================
async fn execute_stack_exec(
Extension(user): Extension<User>,
Json(request): Json<ExecuteStackExecBody>,
) -> serror::Result<axum::body::Body> {
execute_stack_exec_inner(Uuid::new_v4(), request, user).await
}
#[instrument(
name = "ExecuteStackExec",
skip(user),
fields(
user_id = user.id,
)
)]
async fn execute_stack_exec_inner(
req_id: Uuid,
ExecuteStackExecBody {
stack,
service,
shell,
command,
}: ExecuteStackExecBody,
user: User,
) -> serror::Result<axum::body::Body> {
info!("/terminal/execute/stack request | user: {}", user.username);
let res = async {
let stack = get_check_permissions::<Stack>(
&stack,
&user,
PermissionLevel::Read.terminal(),
)
.await?;
let server = get::<Server>(&stack.config.server_id).await?;
let container = stack_status_cache()
.get(&stack.id)
.await
.context("could not get stack status")?
.curr
.services
.iter()
.find(|s| s.service == service)
.context("could not find service")?
.container
.as_ref()
.context("could not find service container")?
.name
.clone();
let periphery = periphery_client(&server)?;
let stream = periphery
.execute_container_exec(container, shell, command)
.await
.context(
"Failed to execute container exec command on periphery",
)?;
anyhow::Ok(stream)
}
.await;
let stream = match res {
Ok(stream) => stream,
Err(e) => {
warn!("/terminal/execute/stack request {req_id} error: {e:#}");
return Err(e.into());
}
};
+133 -75
View File
@@ -2,15 +2,16 @@ use std::{path::PathBuf, str::FromStr, time::Duration};
use anyhow::{Context, anyhow};
use formatting::format_serror;
use git::GitRes;
use komodo_client::{
api::write::*,
entities::{
CloneArgs, FileContents, NoData, Operation, all_logs_success,
FileContents, NoData, Operation, RepoExecutionArgs,
all_logs_success,
build::{Build, BuildInfo, PartialBuildConfig},
builder::{Builder, BuilderConfig},
config::core::CoreConfig,
permission::PermissionLevel,
repo::Repo,
server::ServerState,
update::Update,
},
@@ -114,7 +115,10 @@ impl Resolve<WriteArgs> for WriteBuildFileContents {
)
.await?;
if !build.config.files_on_host && build.config.repo.is_empty() {
if !build.config.files_on_host
&& build.config.repo.is_empty()
&& build.config.linked_repo.is_empty()
{
return Err(anyhow!(
"Build is not configured to use Files on Host or Git Repo, can't write dockerfile contents"
).into());
@@ -182,8 +186,18 @@ async fn write_dockerfile_contents_git(
) -> serror::Result<Update> {
let WriteBuildFileContents { build: _, contents } = req;
let mut clone_args: CloneArgs = (&build).into();
let mut clone_args: RepoExecutionArgs = if !build
.config
.files_on_host
&& !build.config.linked_repo.is_empty()
{
(&crate::resource::get::<Repo>(&build.config.linked_repo).await?)
.into()
} else {
(&build).into()
};
let root = clone_args.unique_path(&core_config().repo_directory)?;
clone_args.destination = Some(root.display().to_string());
let build_path = build
.config
@@ -206,19 +220,19 @@ async fn write_dockerfile_contents_git(
})?;
}
let access_token = if let Some(account) = &clone_args.account {
git_token(&clone_args.provider, account, |https| clone_args.https = https)
.await
.with_context(
|| format!("Failed to get git token in call to db. Stopping run. | {} | {account}", clone_args.provider),
)?
} else {
None
};
// Ensure the folder is initialized as git repo.
// This allows a new file to be committed on a branch that may not exist.
if !root.join(".git").exists() {
let access_token = if let Some(account) = &clone_args.account {
git_token(&clone_args.provider, account, |https| clone_args.https = https)
.await
.with_context(
|| format!("Failed to get git token in call to db. Stopping run. | {} | {account}", clone_args.provider),
)?
} else {
None
};
git::init_folder_as_repo(
&root,
&clone_args,
@@ -235,6 +249,30 @@ async fn write_dockerfile_contents_git(
}
}
// Pull latest changes to repo to ensure linear commit history
match git::pull_or_clone(
clone_args,
&core_config().repo_directory,
access_token,
)
.await
.context("Failed to pull latest changes before commit")
{
Ok((res, _)) => update.logs.extend(res.logs),
Err(e) => {
update.push_error_log("Pull Repo", format_serror(&e.into()));
update.finalize();
return Ok(update);
}
};
if !all_logs_success(&update.logs) {
update.finalize();
update.id = add_update(update.clone()).await?;
return Ok(update);
}
if let Err(e) =
fs::write(&full_path, &contents).await.with_context(|| {
format!("Failed to write dockerfile contents to {full_path:?}")
@@ -301,6 +339,16 @@ impl Resolve<WriteArgs> for RefreshBuildCache {
)
.await?;
let repo = if !build.config.files_on_host
&& !build.config.linked_repo.is_empty()
{
crate::resource::get::<Repo>(&build.config.linked_repo)
.await?
.into()
} else {
None
};
let (
remote_path,
remote_contents,
@@ -319,71 +367,20 @@ impl Resolve<WriteArgs> for RefreshBuildCache {
(None, None, Some(format_serror(&e.into())), None, None)
}
}
} else if !build.config.repo.is_empty() {
// ================
// REPO BASED BUILD
// ================
if build.config.git_provider.is_empty() {
} else if let Some(repo) = &repo {
let Some(res) = get_git_remote(&build, repo.into()).await?
else {
// Nothing to do here
return Ok(NoData {});
}
let config = core_config();
let mut clone_args: CloneArgs = (&build).into();
let repo_path =
clone_args.unique_path(&core_config().repo_directory)?;
clone_args.destination = Some(repo_path.display().to_string());
// Don't want to run these on core.
clone_args.on_clone = None;
clone_args.on_pull = None;
let access_token = if let Some(username) = &clone_args.account {
git_token(&clone_args.provider, username, |https| {
clone_args.https = https
})
.await
.with_context(
|| format!("Failed to get git token in call to db. Stopping run. | {} | {username}", clone_args.provider),
)?
} else {
None
};
let GitRes { hash, message, .. } = git::pull_or_clone(
clone_args,
&config.repo_directory,
access_token,
&[],
"",
None,
&[],
)
.await
.context("failed to clone build repo")?;
let relative_path = PathBuf::from_str(&build.config.build_path)
.context("Invalid build path")?
.join(&build.config.dockerfile_path);
let full_path = repo_path.join(&relative_path);
let (contents, error) = match fs::read_to_string(&full_path)
.await
.with_context(|| {
format!(
"Failed to read dockerfile contents at {full_path:?}"
)
}) {
Ok(contents) => (Some(contents), None),
Err(e) => (None, Some(format_serror(&e.into()))),
res
} else if !build.config.repo.is_empty() {
let Some(res) = get_git_remote(&build, (&build).into()).await?
else {
// Nothing to do here
return Ok(NoData {});
};
(
Some(relative_path.display().to_string()),
contents,
error,
hash,
message,
)
res
} else {
// =============
// UI BASED FILE
@@ -476,6 +473,67 @@ async fn get_on_host_dockerfile(
.await
}
async fn get_git_remote(
build: &Build,
mut clone_args: RepoExecutionArgs,
) -> anyhow::Result<
Option<(
Option<String>,
Option<String>,
Option<String>,
Option<String>,
Option<String>,
)>,
> {
if clone_args.provider.is_empty() {
// Nothing to do here
return Ok(None);
}
let config = core_config();
let repo_path = clone_args.unique_path(&config.repo_directory)?;
clone_args.destination = Some(repo_path.display().to_string());
let access_token = if let Some(username) = &clone_args.account {
git_token(&clone_args.provider, username, |https| {
clone_args.https = https
})
.await
.with_context(
|| format!("Failed to get git token in call to db. Stopping run. | {} | {username}", clone_args.provider),
)?
} else {
None
};
let (res, _) = git::pull_or_clone(
clone_args,
&config.repo_directory,
access_token,
)
.await
.context("failed to clone build repo")?;
let relative_path = PathBuf::from_str(&build.config.build_path)
.context("Invalid build path")?
.join(&build.config.dockerfile_path);
let full_path = repo_path.join(&relative_path);
let (contents, error) =
match fs::read_to_string(&full_path).await.with_context(|| {
format!("Failed to read dockerfile contents at {full_path:?}")
}) {
Ok(contents) => (Some(contents), None),
Err(e) => (None, Some(format_serror(&e.into()))),
};
Ok(Some((
Some(relative_path.display().to_string()),
contents,
error,
res.commit_hash,
res.commit_message,
)))
}
impl Resolve<WriteArgs> for CreateBuildWebhook {
#[instrument(name = "CreateBuildWebhook", skip(args))]
async fn resolve(
+4 -3
View File
@@ -11,7 +11,7 @@ use komodo_client::{
komodo_timestamp,
permission::PermissionLevel,
server::{Server, ServerState},
to_docker_compatible_name,
to_container_compatible_name,
update::Update,
},
};
@@ -207,9 +207,10 @@ impl Resolve<WriteArgs> for RenameDeployment {
let _action_guard =
action_state.update(|state| state.renaming = true)?;
let name = to_docker_compatible_name(&self.name);
let name = to_container_compatible_name(&self.name);
let container_state = get_deployment_state(&deployment).await?;
let container_state =
get_deployment_state(&deployment.id).await?;
if container_state == DeploymentState::Unknown {
return Err(
-114
View File
@@ -1,114 +0,0 @@
use anyhow::anyhow;
use komodo_client::{
api::write::{UpdateDescription, UpdateDescriptionResponse},
entities::{
ResourceTarget, action::Action, alerter::Alerter, build::Build,
builder::Builder, deployment::Deployment, procedure::Procedure,
repo::Repo, server::Server, stack::Stack, sync::ResourceSync,
},
};
use resolver_api::Resolve;
use crate::resource;
use super::WriteArgs;
impl Resolve<WriteArgs> for UpdateDescription {
#[instrument(name = "UpdateDescription", skip(user))]
async fn resolve(
self,
WriteArgs { user }: &WriteArgs,
) -> serror::Result<UpdateDescriptionResponse> {
match self.target {
ResourceTarget::System(_) => {
return Err(
anyhow!(
"cannot update description of System resource target"
)
.into(),
);
}
ResourceTarget::Server(id) => {
resource::update_description::<Server>(
&id,
&self.description,
user,
)
.await?;
}
ResourceTarget::Deployment(id) => {
resource::update_description::<Deployment>(
&id,
&self.description,
user,
)
.await?;
}
ResourceTarget::Build(id) => {
resource::update_description::<Build>(
&id,
&self.description,
user,
)
.await?;
}
ResourceTarget::Repo(id) => {
resource::update_description::<Repo>(
&id,
&self.description,
user,
)
.await?;
}
ResourceTarget::Builder(id) => {
resource::update_description::<Builder>(
&id,
&self.description,
user,
)
.await?;
}
ResourceTarget::Alerter(id) => {
resource::update_description::<Alerter>(
&id,
&self.description,
user,
)
.await?;
}
ResourceTarget::Procedure(id) => {
resource::update_description::<Procedure>(
&id,
&self.description,
user,
)
.await?;
}
ResourceTarget::Action(id) => {
resource::update_description::<Action>(
&id,
&self.description,
user,
)
.await?;
}
ResourceTarget::ResourceSync(id) => {
resource::update_description::<ResourceSync>(
&id,
&self.description,
user,
)
.await?;
}
ResourceTarget::Stack(id) => {
resource::update_description::<Stack>(
&id,
&self.description,
user,
)
.await?;
}
}
Ok(UpdateDescriptionResponse {})
}
}
+4 -4
View File
@@ -23,11 +23,11 @@ mod alerter;
mod build;
mod builder;
mod deployment;
mod description;
mod permissions;
mod procedure;
mod provider;
mod repo;
mod resource;
mod server;
mod service_user;
mod stack;
@@ -77,11 +77,12 @@ pub enum WriteRequest {
UpdatePermissionOnResourceType(UpdatePermissionOnResourceType),
UpdatePermissionOnTarget(UpdatePermissionOnTarget),
// ==== DESCRIPTION ====
UpdateDescription(UpdateDescription),
// ==== RESOURCE ====
UpdateResourceMeta(UpdateResourceMeta),
// ==== SERVER ====
CreateServer(CreateServer),
CopyServer(CopyServer),
DeleteServer(DeleteServer),
UpdateServer(UpdateServer),
RenameServer(RenameServer),
@@ -175,7 +176,6 @@ pub enum WriteRequest {
DeleteTag(DeleteTag),
RenameTag(RenameTag),
UpdateTagColor(UpdateTagColor),
UpdateTagsOnResource(UpdateTagsOnResource),
// ==== VARIABLE ====
CreateVariable(CreateVariable),
+5 -13
View File
@@ -1,10 +1,9 @@
use anyhow::{Context, anyhow};
use formatting::format_serror;
use git::GitRes;
use komodo_client::{
api::write::*,
entities::{
CloneArgs, NoData, Operation,
NoData, Operation, RepoExecutionArgs,
config::core::CoreConfig,
komodo_timestamp,
permission::PermissionLevel,
@@ -183,13 +182,10 @@ impl Resolve<WriteArgs> for RefreshRepoCache {
return Ok(NoData {});
}
let mut clone_args: CloneArgs = (&repo).into();
let mut clone_args: RepoExecutionArgs = (&repo).into();
let repo_path =
clone_args.unique_path(&core_config().repo_directory)?;
clone_args.destination = Some(repo_path.display().to_string());
// Don't want to run these on core.
clone_args.on_clone = None;
clone_args.on_pull = None;
let access_token = if let Some(username) = &clone_args.account {
git_token(&clone_args.provider, username, |https| {
@@ -203,14 +199,10 @@ impl Resolve<WriteArgs> for RefreshRepoCache {
None
};
let GitRes { hash, message, .. } = git::pull_or_clone(
let (res, _) = git::pull_or_clone(
clone_args,
&core_config().repo_directory,
access_token,
&[],
"",
None,
&[],
)
.await
.with_context(|| {
@@ -222,8 +214,8 @@ impl Resolve<WriteArgs> for RefreshRepoCache {
last_built_at: repo.info.last_built_at,
built_hash: repo.info.built_hash,
built_message: repo.info.built_message,
latest_hash: hash,
latest_message: message,
latest_hash: res.commit_hash,
latest_message: res.commit_message,
};
let info = to_document(&info)
+68
View File
@@ -0,0 +1,68 @@
use anyhow::anyhow;
use komodo_client::{
api::write::{UpdateResourceMeta, UpdateResourceMetaResponse},
entities::{
ResourceTarget, action::Action, alerter::Alerter, build::Build,
builder::Builder, deployment::Deployment, procedure::Procedure,
repo::Repo, server::Server, stack::Stack, sync::ResourceSync,
},
};
use resolver_api::Resolve;
use crate::resource::{self, ResourceMetaUpdate};
use super::WriteArgs;
impl Resolve<WriteArgs> for UpdateResourceMeta {
#[instrument(name = "UpdateResourceMeta", skip(args))]
async fn resolve(
self,
args: &WriteArgs,
) -> serror::Result<UpdateResourceMetaResponse> {
let meta = ResourceMetaUpdate {
description: self.description,
template: self.template,
tags: self.tags,
};
match self.target {
ResourceTarget::System(_) => {
return Err(
anyhow!("cannot update meta of System resource target")
.into(),
);
}
ResourceTarget::Server(id) => {
resource::update_meta::<Server>(&id, meta, args).await?;
}
ResourceTarget::Deployment(id) => {
resource::update_meta::<Deployment>(&id, meta, args).await?;
}
ResourceTarget::Build(id) => {
resource::update_meta::<Build>(&id, meta, args).await?;
}
ResourceTarget::Repo(id) => {
resource::update_meta::<Repo>(&id, meta, args).await?;
}
ResourceTarget::Builder(id) => {
resource::update_meta::<Builder>(&id, meta, args).await?;
}
ResourceTarget::Alerter(id) => {
resource::update_meta::<Alerter>(&id, meta, args).await?;
}
ResourceTarget::Procedure(id) => {
resource::update_meta::<Procedure>(&id, meta, args).await?;
}
ResourceTarget::Action(id) => {
resource::update_meta::<Action>(&id, meta, args).await?;
}
ResourceTarget::ResourceSync(id) => {
resource::update_meta::<ResourceSync>(&id, meta, args)
.await?;
}
ResourceTarget::Stack(id) => {
resource::update_meta::<Stack>(&id, meta, args).await?;
}
}
Ok(UpdateResourceMetaResponse {})
}
}
+19
View File
@@ -37,6 +37,25 @@ impl Resolve<WriteArgs> for CreateServer {
}
}
impl Resolve<WriteArgs> for CopyServer {
#[instrument(name = "CopyServer", skip(user))]
async fn resolve(
self,
WriteArgs { user }: &WriteArgs,
) -> serror::Result<Server> {
let Server { config, .. } = get_check_permissions::<Server>(
&self.id,
user,
PermissionLevel::Read.into(),
)
.await?;
Ok(
resource::create::<Server>(&self.name, config.into(), user)
.await?,
)
}
}
impl Resolve<WriteArgs> for DeleteServer {
#[instrument(name = "DeleteServer", skip(args))]
async fn resolve(self, args: &WriteArgs) -> serror::Result<Server> {
+41 -23
View File
@@ -6,6 +6,7 @@ use komodo_client::{
FileContents, NoData, Operation,
config::core::CoreConfig,
permission::PermissionLevel,
repo::Repo,
server::ServerState,
stack::{PartialStackConfig, Stack, StackInfo},
update::Update,
@@ -26,8 +27,9 @@ use crate::{
api::execute::pull_stack_inner,
config::core_config,
helpers::{
git_token, periphery_client,
periphery_client,
query::get_server_with_state,
stack_git_token,
update::{add_update, make_update},
},
permission::get_check_permissions,
@@ -120,9 +122,22 @@ impl Resolve<WriteArgs> for WriteStackFileContents {
)
.await?;
if !stack.config.files_on_host && stack.config.repo.is_empty() {
let mut repo = if !stack.config.files_on_host
&& !stack.config.linked_repo.is_empty()
{
crate::resource::get::<Repo>(&stack.config.linked_repo)
.await?
.into()
} else {
None
};
if !stack.config.files_on_host
&& stack.config.repo.is_empty()
&& stack.config.linked_repo.is_empty()
{
return Err(anyhow!(
"Stack is not configured to use Files on Host or Git Repo, can't write file contents"
"Stack is not configured to use Files on Host, Git Repo, or Linked Repo, can't write file contents"
).into());
}
@@ -155,25 +170,12 @@ impl Resolve<WriteArgs> for WriteStackFileContents {
}
};
} else {
let git_token = if !stack.config.git_account.is_empty() {
git_token(
&stack.config.git_provider,
&stack.config.git_account,
|https| stack.config.git_https = https,
)
.await
.with_context(|| {
format!(
"Failed to get git token. | {} | {}",
stack.config.git_account, stack.config.git_provider
)
})?
} else {
None
};
let git_token =
stack_git_token(&mut stack, repo.as_mut()).await?;
match periphery_client(&server)?
.request(WriteCommitComposeContents {
stack,
repo,
username: Some(user.username.clone()),
file_path,
contents,
@@ -236,8 +238,19 @@ impl Resolve<WriteArgs> for RefreshStackCache {
)
.await?;
let repo = if !stack.config.files_on_host
&& !stack.config.linked_repo.is_empty()
{
crate::resource::get::<Repo>(&stack.config.linked_repo)
.await?
.into()
} else {
None
};
let file_contents_empty = stack.config.file_contents.is_empty();
let repo_empty = stack.config.repo.is_empty();
let repo_empty =
stack.config.repo.is_empty() && repo.as_ref().is_none();
if !stack.config.files_on_host
&& file_contents_empty
@@ -320,8 +333,12 @@ impl Resolve<WriteArgs> for RefreshStackCache {
hash: latest_hash,
message: latest_message,
..
} = get_repo_compose_contents(&stack, Some(&mut missing_files))
.await?;
} = get_repo_compose_contents(
&stack,
repo.as_ref(),
Some(&mut missing_files),
)
.await?;
let project_name = stack.project_name(true);
@@ -402,7 +419,8 @@ impl Resolve<WriteArgs> for RefreshStackCache {
if state == ServerState::Ok {
let name = stack.name.clone();
if let Err(e) =
pull_stack_inner(stack, Vec::new(), &server, None).await
pull_stack_inner(stack, Vec::new(), &server, repo, None)
.await
{
warn!(
"Failed to pull latest images for Stack {name} | {e:#}",
+194 -77
View File
@@ -1,11 +1,14 @@
use std::{collections::HashMap, path::PathBuf};
use std::{
collections::HashMap,
path::{Path, PathBuf},
};
use anyhow::{Context, anyhow};
use formatting::format_serror;
use komodo_client::{
api::{read::ExportAllResourcesToToml, write::*},
entities::{
self, CloneArgs, NoData, Operation, ResourceTarget,
self, NoData, Operation, RepoExecutionArgs, ResourceTarget,
action::Action,
alert::{Alert, AlertData, SeverityLevel},
alerter::Alerter,
@@ -44,6 +47,7 @@ use crate::{
api::read::ReadArgs,
config::core_config,
helpers::{
all_resources::AllResourcesById,
git_token,
query::get_id_to_tags,
update::{add_update, make_update, update_update},
@@ -52,8 +56,8 @@ use crate::{
resource,
state::{db_client, github_client},
sync::{
AllResourcesById, deploy::SyncDeployParams,
remote::RemoteResources, view::push_updates_for_view,
deploy::SyncDeployParams, remote::RemoteResources,
view::push_updates_for_view,
},
};
@@ -142,7 +146,20 @@ impl Resolve<WriteArgs> for WriteSyncFileContents {
)
.await?;
if !sync.config.files_on_host && sync.config.repo.is_empty() {
let repo = if !sync.config.files_on_host
&& !sync.config.linked_repo.is_empty()
{
crate::resource::get::<Repo>(&sync.config.linked_repo)
.await?
.into()
} else {
None
};
if !sync.config.files_on_host
&& sync.config.repo.is_empty()
&& sync.config.linked_repo.is_empty()
{
return Err(
anyhow!(
"This method is only for 'files on host' or 'repo' based syncs."
@@ -159,7 +176,8 @@ impl Resolve<WriteArgs> for WriteSyncFileContents {
if sync.config.files_on_host {
write_sync_file_contents_on_host(self, args, sync, update).await
} else {
write_sync_file_contents_git(self, args, sync, update).await
write_sync_file_contents_git(self, args, sync, repo, update)
.await
}
}
}
@@ -237,6 +255,7 @@ async fn write_sync_file_contents_git(
req: WriteSyncFileContents,
args: &WriteArgs,
sync: ResourceSync,
repo: Option<Repo>,
mut update: Update,
) -> serror::Result<Update> {
let WriteSyncFileContents {
@@ -246,15 +265,34 @@ async fn write_sync_file_contents_git(
contents,
} = req;
let mut clone_args: CloneArgs = (&sync).into();
let mut clone_args: RepoExecutionArgs = if let Some(repo) = &repo {
repo.into()
} else {
(&sync).into()
};
let root = clone_args.unique_path(&core_config().repo_directory)?;
clone_args.destination = Some(root.display().to_string());
let access_token = if let Some(account) = &clone_args.account {
git_token(&clone_args.provider, account, |https| clone_args.https = https)
.await
.with_context(
|| format!("Failed to get git token in call to db. Stopping run. | {} | {account}", clone_args.provider),
)?
} else {
None
};
let file_path =
file_path.parse::<PathBuf>().context("Invalid file path")?;
let resource_path = resource_path
.parse::<PathBuf>()
.context("Invalid resource path")?;
let full_path = root.join(&resource_path).join(&file_path);
let full_path = root
.join(&resource_path)
.join(&file_path)
.components()
.collect::<PathBuf>();
if let Some(parent) = full_path.parent() {
fs::create_dir_all(parent).await.with_context(|| {
@@ -267,16 +305,6 @@ async fn write_sync_file_contents_git(
// Ensure the folder is initialized as git repo.
// This allows a new file to be committed on a branch that may not exist.
if !root.join(".git").exists() {
let access_token = if let Some(account) = &clone_args.account {
git_token(&clone_args.provider, account, |https| clone_args.https = https)
.await
.with_context(
|| format!("Failed to get git token in call to db. Stopping run. | {} | {account}", clone_args.provider),
)?
} else {
None
};
git::init_folder_as_repo(
&root,
&clone_args,
@@ -288,11 +316,33 @@ async fn write_sync_file_contents_git(
if !all_logs_success(&update.logs) {
update.finalize();
update.id = add_update(update.clone()).await?;
return Ok(update);
}
}
// Pull latest changes to repo to ensure linear commit history
match git::pull_or_clone(
clone_args,
&core_config().repo_directory,
access_token,
)
.await
.context("Failed to pull latest changes before commit")
{
Ok((res, _)) => update.logs.extend(res.logs),
Err(e) => {
update.push_error_log("Pull Repo", format_serror(&e.into()));
update.finalize();
return Ok(update);
}
};
if !all_logs_success(&update.logs) {
update.finalize();
update.id = add_update(update.clone()).await?;
return Ok(update);
}
if let Err(e) =
fs::write(&full_path, &contents).await.with_context(|| {
format!(
@@ -353,10 +403,21 @@ impl Resolve<WriteArgs> for CommitSync {
)
.await?;
let repo = if !sync.config.files_on_host
&& !sync.config.linked_repo.is_empty()
{
crate::resource::get::<Repo>(&sync.config.linked_repo)
.await?
.into()
} else {
None
};
let file_contents_empty = sync.config.file_contents_empty();
let fresh_sync = !sync.config.files_on_host
&& sync.config.repo.is_empty()
&& repo.is_none()
&& file_contents_empty;
if !sync.config.managed && !fresh_sync {
@@ -367,29 +428,31 @@ impl Resolve<WriteArgs> for CommitSync {
}
// Get this here so it can fail before update created.
let resource_path =
if sync.config.files_on_host || !sync.config.repo.is_empty() {
let resource_path = sync
.config
.resource_path
.first()
.context("Sync does not have resource path configured.")?
.parse::<PathBuf>()
.context("Invalid resource path")?;
let resource_path = if sync.config.files_on_host
|| !sync.config.repo.is_empty()
|| repo.is_some()
{
let resource_path = sync
.config
.resource_path
.first()
.context("Sync does not have resource path configured.")?
.parse::<PathBuf>()
.context("Invalid resource path")?;
if resource_path
.extension()
.context("Resource path missing '.toml' extension")?
!= "toml"
{
return Err(
anyhow!("Resource path missing '.toml' extension").into(),
);
}
Some(resource_path)
} else {
None
};
if resource_path
.extension()
.context("Resource path missing '.toml' extension")?
!= "toml"
{
return Err(
anyhow!("Resource path missing '.toml' extension").into(),
);
}
Some(resource_path)
} else {
None
};
let res = ExportAllResourcesToToml {
include_resources: sync.config.include_resources,
@@ -440,34 +503,43 @@ impl Resolve<WriteArgs> for CommitSync {
format!("File contents written to {file_path:?}"),
);
}
} else if let Some(repo) = &repo {
let Some(resource_path) = resource_path else {
// Resource path checked above for repo mode.
unreachable!()
};
let args: RepoExecutionArgs = repo.into();
if let Err(e) =
commit_git_sync(args, &resource_path, &res.toml, &mut update)
.await
{
update.push_error_log(
"Write resource file",
format_serror(&e.into()),
);
update.finalize();
add_update(update.clone()).await?;
return Ok(update);
}
} else if !sync.config.repo.is_empty() {
let Some(resource_path) = resource_path else {
// Resource path checked above for repo mode.
unreachable!()
};
// GIT REPO
let args: CloneArgs = (&sync).into();
let root = args.unique_path(&core_config().repo_directory)?;
match git::write_commit_file(
"Commit Sync",
&root,
&resource_path,
&res.toml,
&sync.config.branch,
)
.await
let args: RepoExecutionArgs = (&sync).into();
if let Err(e) =
commit_git_sync(args, &resource_path, &res.toml, &mut update)
.await
{
Ok(res) => update.logs.extend(res.logs),
Err(e) => {
update.push_error_log(
"Write resource file",
format_serror(&e.into()),
);
update.finalize();
add_update(update.clone()).await?;
return Ok(update);
}
update.push_error_log(
"Write resource file",
format_serror(&e.into()),
);
update.finalize();
add_update(update.clone()).await?;
return Ok(update);
}
// ===========
// UI DEFINED
} else if let Err(e) = db_client()
@@ -505,6 +577,49 @@ impl Resolve<WriteArgs> for CommitSync {
}
}
async fn commit_git_sync(
mut args: RepoExecutionArgs,
resource_path: &Path,
toml: &str,
update: &mut Update,
) -> anyhow::Result<()> {
let root = args.unique_path(&core_config().repo_directory)?;
args.destination = Some(root.display().to_string());
let access_token = if let Some(account) = &args.account {
git_token(&args.provider, account, |https| args.https = https)
.await
.with_context(
|| format!("Failed to get git token in call to db. Stopping run. | {} | {account}", args.provider),
)?
} else {
None
};
let (pull_res, _) = git::pull_or_clone(
args.clone(),
&core_config().repo_directory,
access_token,
)
.await?;
update.logs.extend(pull_res.logs);
if !all_logs_success(&update.logs) {
return Ok(());
}
let res = git::write_commit_file(
"Commit Sync",
&root,
resource_path,
toml,
&args.branch,
)
.await?;
update.logs.extend(res.logs);
Ok(())
}
impl Resolve<WriteArgs> for RefreshResourceSyncPending {
#[instrument(
name = "RefreshResourceSyncPending",
@@ -525,10 +640,21 @@ impl Resolve<WriteArgs> for RefreshResourceSyncPending {
)
.await?;
let repo = if !sync.config.files_on_host
&& !sync.config.linked_repo.is_empty()
{
crate::resource::get::<Repo>(&sync.config.linked_repo)
.await?
.into()
} else {
None
};
if !sync.config.managed
&& !sync.config.files_on_host
&& sync.config.file_contents.is_empty()
&& sync.config.repo.is_empty()
&& sync.config.linked_repo.is_empty()
{
// Sync not configured, nothing to refresh
return Ok(sync);
@@ -542,9 +668,12 @@ impl Resolve<WriteArgs> for RefreshResourceSyncPending {
hash,
message,
..
} = crate::sync::remote::get_remote_resources(&sync)
.await
.context("failed to get remote resources")?;
} = crate::sync::remote::get_remote_resources(
&sync,
repo.as_ref(),
)
.await
.context("failed to get remote resources")?;
sync.info.remote_contents = files;
sync.info.remote_errors = file_errors;
@@ -585,7 +714,6 @@ impl Resolve<WriteArgs> for RefreshResourceSyncPending {
deployment_map: &deployments_by_name,
stacks: &resources.stacks,
stack_map: &stacks_by_name,
all_resources: &all_resources,
},
)
.await;
@@ -595,7 +723,6 @@ impl Resolve<WriteArgs> for RefreshResourceSyncPending {
push_updates_for_view::<Server>(
resources.servers,
delete,
&all_resources,
None,
None,
&id_to_tags,
@@ -606,7 +733,6 @@ impl Resolve<WriteArgs> for RefreshResourceSyncPending {
push_updates_for_view::<Stack>(
resources.stacks,
delete,
&all_resources,
None,
None,
&id_to_tags,
@@ -617,7 +743,6 @@ impl Resolve<WriteArgs> for RefreshResourceSyncPending {
push_updates_for_view::<Deployment>(
resources.deployments,
delete,
&all_resources,
None,
None,
&id_to_tags,
@@ -628,7 +753,6 @@ impl Resolve<WriteArgs> for RefreshResourceSyncPending {
push_updates_for_view::<Build>(
resources.builds,
delete,
&all_resources,
None,
None,
&id_to_tags,
@@ -639,7 +763,6 @@ impl Resolve<WriteArgs> for RefreshResourceSyncPending {
push_updates_for_view::<Repo>(
resources.repos,
delete,
&all_resources,
None,
None,
&id_to_tags,
@@ -650,7 +773,6 @@ impl Resolve<WriteArgs> for RefreshResourceSyncPending {
push_updates_for_view::<Procedure>(
resources.procedures,
delete,
&all_resources,
None,
None,
&id_to_tags,
@@ -661,7 +783,6 @@ impl Resolve<WriteArgs> for RefreshResourceSyncPending {
push_updates_for_view::<Action>(
resources.actions,
delete,
&all_resources,
None,
None,
&id_to_tags,
@@ -672,7 +793,6 @@ impl Resolve<WriteArgs> for RefreshResourceSyncPending {
push_updates_for_view::<Builder>(
resources.builders,
delete,
&all_resources,
None,
None,
&id_to_tags,
@@ -683,7 +803,6 @@ impl Resolve<WriteArgs> for RefreshResourceSyncPending {
push_updates_for_view::<Alerter>(
resources.alerters,
delete,
&all_resources,
None,
None,
&id_to_tags,
@@ -694,7 +813,6 @@ impl Resolve<WriteArgs> for RefreshResourceSyncPending {
push_updates_for_view::<ResourceSync>(
resources.resource_syncs,
delete,
&all_resources,
None,
None,
&id_to_tags,
@@ -722,7 +840,6 @@ impl Resolve<WriteArgs> for RefreshResourceSyncPending {
crate::sync::user_groups::get_updates_for_view(
resources.user_groups,
delete,
&all_resources,
)
.await?
} else {
+5 -118
View File
@@ -2,18 +2,13 @@ use std::str::FromStr;
use anyhow::{Context, anyhow};
use komodo_client::{
api::write::{
CreateTag, DeleteTag, RenameTag, UpdateTagColor,
UpdateTagsOnResource, UpdateTagsOnResourceResponse,
},
api::write::{CreateTag, DeleteTag, RenameTag, UpdateTagColor},
entities::{
ResourceTarget,
action::Action,
alerter::Alerter,
build::Build,
builder::Builder,
deployment::Deployment,
permission::PermissionLevel,
procedure::Procedure,
repo::Repo,
server::Server,
@@ -30,7 +25,6 @@ use resolver_api::Resolve;
use crate::{
helpers::query::{get_tag, get_tag_check_owner},
permission::get_check_permissions,
resource,
state::db_client,
};
@@ -124,13 +118,15 @@ impl Resolve<WriteArgs> for DeleteTag {
tokio::try_join!(
resource::remove_tag_from_all::<Server>(&self.id),
resource::remove_tag_from_all::<Deployment>(&self.id),
resource::remove_tag_from_all::<Stack>(&self.id),
resource::remove_tag_from_all::<Deployment>(&self.id),
resource::remove_tag_from_all::<Build>(&self.id),
resource::remove_tag_from_all::<Repo>(&self.id),
resource::remove_tag_from_all::<Procedure>(&self.id),
resource::remove_tag_from_all::<Action>(&self.id),
resource::remove_tag_from_all::<ResourceSync>(&self.id),
resource::remove_tag_from_all::<Builder>(&self.id),
resource::remove_tag_from_all::<Alerter>(&self.id),
resource::remove_tag_from_all::<Procedure>(&self.id),
)?;
delete_one_by_id(&db_client().tags, &self.id, None).await?;
@@ -138,112 +134,3 @@ impl Resolve<WriteArgs> for DeleteTag {
Ok(tag)
}
}
impl Resolve<WriteArgs> for UpdateTagsOnResource {
#[instrument(name = "UpdateTagsOnResource", skip(args))]
async fn resolve(
self,
args: &WriteArgs,
) -> serror::Result<UpdateTagsOnResourceResponse> {
let WriteArgs { user } = args;
match self.target {
ResourceTarget::System(_) => {
return Err(anyhow!("Invalid target type: System").into());
}
ResourceTarget::Build(id) => {
get_check_permissions::<Build>(
&id,
user,
PermissionLevel::Write.into(),
)
.await?;
resource::update_tags::<Build>(&id, self.tags, args).await?;
}
ResourceTarget::Builder(id) => {
get_check_permissions::<Builder>(
&id,
user,
PermissionLevel::Write.into(),
)
.await?;
resource::update_tags::<Builder>(&id, self.tags, args).await?
}
ResourceTarget::Deployment(id) => {
get_check_permissions::<Deployment>(
&id,
user,
PermissionLevel::Write.into(),
)
.await?;
resource::update_tags::<Deployment>(&id, self.tags, args)
.await?
}
ResourceTarget::Server(id) => {
get_check_permissions::<Server>(
&id,
user,
PermissionLevel::Write.into(),
)
.await?;
resource::update_tags::<Server>(&id, self.tags, args).await?
}
ResourceTarget::Repo(id) => {
get_check_permissions::<Repo>(
&id,
user,
PermissionLevel::Write.into(),
)
.await?;
resource::update_tags::<Repo>(&id, self.tags, args).await?
}
ResourceTarget::Alerter(id) => {
get_check_permissions::<Alerter>(
&id,
user,
PermissionLevel::Write.into(),
)
.await?;
resource::update_tags::<Alerter>(&id, self.tags, args).await?
}
ResourceTarget::Procedure(id) => {
get_check_permissions::<Procedure>(
&id,
user,
PermissionLevel::Write.into(),
)
.await?;
resource::update_tags::<Procedure>(&id, self.tags, args)
.await?
}
ResourceTarget::Action(id) => {
get_check_permissions::<Action>(
&id,
user,
PermissionLevel::Write.into(),
)
.await?;
resource::update_tags::<Action>(&id, self.tags, args).await?
}
ResourceTarget::ResourceSync(id) => {
get_check_permissions::<ResourceSync>(
&id,
user,
PermissionLevel::Write.into(),
)
.await?;
resource::update_tags::<ResourceSync>(&id, self.tags, args)
.await?
}
ResourceTarget::Stack(id) => {
get_check_permissions::<Stack>(
&id,
user,
PermissionLevel::Write.into(),
)
.await?;
resource::update_tags::<Stack>(&id, self.tags, args).await?
}
};
Ok(UpdateTagsOnResourceResponse {})
}
}
+4 -1
View File
@@ -262,7 +262,10 @@ impl Resolve<WriteArgs> for SetEveryoneUserGroup {
Err(_) => doc! { "name": &self.user_group },
};
db.user_groups
.update_one(filter.clone(), doc! { "$set": { "everyone": self.everyone } })
.update_one(
filter.clone(),
doc! { "$set": { "everyone": self.everyone } },
)
.await
.context("failed to set everyone on user group")?;
let res = db
+16 -1
View File
@@ -14,6 +14,7 @@ use serror::AddStatusCode;
use crate::{
config::core_config,
helpers::random_string,
state::{db_client, jwt_client},
};
@@ -82,9 +83,23 @@ async fn callback(
if !no_users_exist && core_config.disable_user_registration {
return Err(anyhow!("User registration is disabled"));
}
let mut username = github_user.login;
// Modify username if it already exists
if db_client
.users
.find_one(doc! { "username": &username })
.await
.context("Failed to query users collection")?
.is_some()
{
username += "-";
username += &random_string(5);
};
let user = User {
id: Default::default(),
username: github_user.login,
username,
enabled: no_users_exist || core_config.enable_new_users,
admin: no_users_exist,
super_admin: no_users_exist,
+21 -7
View File
@@ -12,6 +12,7 @@ use serror::AddStatusCode;
use crate::{
config::core_config,
helpers::random_string,
state::{db_client, jwt_client},
};
@@ -91,15 +92,28 @@ async fn callback(
if !no_users_exist && core_config.disable_user_registration {
return Err(anyhow!("User registration is disabled"));
}
let mut username = google_user
.email
.split('@')
.collect::<Vec<&str>>()
.first()
.unwrap()
.to_string();
// Modify username if it already exists
if db_client
.users
.find_one(doc! { "username": &username })
.await
.context("Failed to query users collection")?
.is_some()
{
username += "-";
username += &random_string(5);
};
let user = User {
id: Default::default(),
username: google_user
.email
.split('@')
.collect::<Vec<&str>>()
.first()
.unwrap()
.to_string(),
username,
enabled: no_users_exist || core_config.enable_new_users,
admin: no_users_exist,
super_admin: no_users_exist,
+3 -4
View File
@@ -48,10 +48,9 @@ pub async fn spawn_oidc_client_management() {
{
return;
}
reset_oidc_client()
.await
.context("Failed to initialize OIDC client.")
.unwrap();
if let Err(e) = reset_oidc_client().await {
error!("Failed to initialize OIDC client | {e:#}");
}
tokio::spawn(async move {
loop {
tokio::time::sleep(Duration::from_secs(60)).await;
+40 -7
View File
@@ -12,9 +12,10 @@ use komodo_client::entities::{
};
use mungos::mongodb::bson::{Document, doc};
use openidconnect::{
AccessTokenHash, AuthorizationCode, CsrfToken, Nonce,
OAuth2TokenResponse, PkceCodeChallenge, PkceCodeVerifier, Scope,
TokenResponse, core::CoreAuthenticationFlow,
AccessTokenHash, AuthorizationCode, CsrfToken,
EmptyAdditionalClaims, Nonce, OAuth2TokenResponse,
PkceCodeChallenge, PkceCodeVerifier, Scope, TokenResponse,
core::{CoreAuthenticationFlow, CoreGenderClaim},
};
use reqwest::StatusCode;
use serde::Deserialize;
@@ -22,6 +23,7 @@ use serror::AddStatusCode;
use crate::{
config::core_config,
helpers::random_string,
state::{db_client, jwt_client},
};
@@ -89,6 +91,7 @@ async fn login(
)
.set_pkce_challenge(pkce_challenge)
.add_scope(Scope::new("openid".to_string()))
.add_scope(Scope::new("profile".to_string()))
.add_scope(Scope::new("email".to_string()))
.url();
@@ -137,7 +140,7 @@ async fn callback(
) -> anyhow::Result<Redirect> {
let client = oidc_client().load();
let client =
client.as_ref().context("OIDC Client not configured")?;
client.as_ref().context("OIDC Client not initialized successfully. Is the provider properly configured?")?;
if let Some(e) = query.error {
return Err(anyhow!("Provider returned error: {e}"));
@@ -159,11 +162,12 @@ async fn callback(
));
}
let reqwest_client = reqwest_client();
let token_response = client
.exchange_code(AuthorizationCode::new(code))
.context("Failed to get Oauth token at exchange code")?
.set_pkce_verifier(pkce_verifier)
.request_async(reqwest_client())
.request_async(reqwest_client)
.await
.context("Failed to get Oauth token")?;
@@ -226,12 +230,26 @@ async fn callback(
if !no_users_exist && core_config.disable_user_registration {
return Err(anyhow!("User registration is disabled"));
}
// Fetch user info
let user_info = client
.user_info(
token_response.access_token().clone(),
claims.subject().clone().into(),
)
.context("Invalid user info request")?
.request_async::<EmptyAdditionalClaims, _, CoreGenderClaim>(
reqwest_client,
)
.await
.context("Failed to fetch user info for new user")?;
// Will use preferred_username, then email, then user_id if it isn't available.
let username = claims
let mut username = user_info
.preferred_username()
.map(|username| username.to_string())
.unwrap_or_else(|| {
let email = claims
let email = user_info
.email()
.map(|email| email.as_str())
.unwrap_or(user_id);
@@ -245,6 +263,19 @@ async fn callback(
}
.to_string()
});
// Modify username if it already exists
if db_client
.users
.find_one(doc! { "username": &username })
.await
.context("Failed to query users collection")?
.is_some()
{
username += "-";
username += &random_string(5);
};
let user = User {
id: Default::default(),
username,
@@ -262,6 +293,7 @@ async fn callback(
user_id: user_id.to_string(),
},
};
let user_id = db_client
.users
.insert_one(user)
@@ -271,6 +303,7 @@ async fn callback(
.as_object_id()
.context("inserted_id is not ObjectId")?
.to_string();
jwt_client()
.encode(user_id)
.context("failed to generate jwt")?
+1
View File
@@ -135,6 +135,7 @@ pub fn core_config() -> &'static CoreConfig {
host: env.komodo_host.unwrap_or(config.host),
port: env.komodo_port.unwrap_or(config.port),
bind_ip: env.komodo_bind_ip.unwrap_or(config.bind_ip),
timezone: env.komodo_timezone.unwrap_or(config.timezone),
first_server: env.komodo_first_server.unwrap_or(config.first_server),
frontend_path: env.komodo_frontend_path.unwrap_or(config.frontend_path),
jwt_ttl: env
+73
View File
@@ -0,0 +1,73 @@
use std::collections::HashMap;
use komodo_client::entities::{
action::Action, alerter::Alerter, build::Build, builder::Builder,
deployment::Deployment, procedure::Procedure, repo::Repo,
server::Server, stack::Stack, sync::ResourceSync,
};
#[derive(Debug, Default)]
pub struct AllResourcesById {
pub servers: HashMap<String, Server>,
pub deployments: HashMap<String, Deployment>,
pub stacks: HashMap<String, Stack>,
pub builds: HashMap<String, Build>,
pub repos: HashMap<String, Repo>,
pub procedures: HashMap<String, Procedure>,
pub actions: HashMap<String, Action>,
pub builders: HashMap<String, Builder>,
pub alerters: HashMap<String, Alerter>,
pub syncs: HashMap<String, ResourceSync>,
}
impl AllResourcesById {
/// Use `match_tags` to filter resources by tag.
pub async fn load() -> anyhow::Result<Self> {
let map = HashMap::new();
let id_to_tags = &map;
let match_tags = &[];
Ok(Self {
servers: crate::resource::get_id_to_resource_map::<Server>(
id_to_tags, match_tags,
)
.await?,
deployments: crate::resource::get_id_to_resource_map::<
Deployment,
>(id_to_tags, match_tags)
.await?,
builds: crate::resource::get_id_to_resource_map::<Build>(
id_to_tags, match_tags,
)
.await?,
repos: crate::resource::get_id_to_resource_map::<Repo>(
id_to_tags, match_tags,
)
.await?,
procedures:
crate::resource::get_id_to_resource_map::<Procedure>(
id_to_tags, match_tags,
)
.await?,
actions: crate::resource::get_id_to_resource_map::<Action>(
id_to_tags, match_tags,
)
.await?,
builders: crate::resource::get_id_to_resource_map::<Builder>(
id_to_tags, match_tags,
)
.await?,
alerters: crate::resource::get_id_to_resource_map::<Alerter>(
id_to_tags, match_tags,
)
.await?,
syncs: crate::resource::get_id_to_resource_map::<ResourceSync>(
id_to_tags, match_tags,
)
.await?,
stacks: crate::resource::get_id_to_resource_map::<Stack>(
id_to_tags, match_tags,
)
.await?,
})
}
}
-164
View File
@@ -1,164 +0,0 @@
use std::collections::HashSet;
use anyhow::Context;
use komodo_client::entities::{SystemCommand, update::Update};
use super::query::VariablesAndSecrets;
pub fn interpolate_variables_secrets_into_extra_args(
VariablesAndSecrets { variables, secrets }: &VariablesAndSecrets,
extra_args: &mut Vec<String>,
global_replacers: &mut HashSet<(String, String)>,
secret_replacers: &mut HashSet<(String, String)>,
) -> anyhow::Result<()> {
for arg in extra_args {
if arg.is_empty() {
continue;
}
// first pass - global variables
let (res, more_replacers) = svi::interpolate_variables(
arg,
variables,
svi::Interpolator::DoubleBrackets,
false,
)
.with_context(|| {
format!(
"failed to interpolate global variables into extra arg '{arg}'",
)
})?;
global_replacers.extend(more_replacers);
// second pass - core secrets
let (res, more_replacers) = svi::interpolate_variables(
&res,
secrets,
svi::Interpolator::DoubleBrackets,
false,
)
.with_context(|| {
format!(
"failed to interpolate core secrets into extra arg '{arg}'",
)
})?;
secret_replacers.extend(more_replacers);
// set arg with the result
*arg = res;
}
Ok(())
}
pub fn interpolate_variables_secrets_into_string(
VariablesAndSecrets { variables, secrets }: &VariablesAndSecrets,
target: &mut String,
global_replacers: &mut HashSet<(String, String)>,
secret_replacers: &mut HashSet<(String, String)>,
) -> anyhow::Result<()> {
if target.is_empty() {
return Ok(());
}
// first pass - global variables
let (res, more_replacers) = svi::interpolate_variables(
target,
variables,
svi::Interpolator::DoubleBrackets,
false,
)
.context("Failed to interpolate core variables")?;
global_replacers.extend(more_replacers);
// second pass - core secrets
let (res, more_replacers) = svi::interpolate_variables(
&res,
secrets,
svi::Interpolator::DoubleBrackets,
false,
)
.context("Failed to interpolate core secrets")?;
secret_replacers.extend(more_replacers);
// set command with the result
*target = res;
Ok(())
}
pub fn interpolate_variables_secrets_into_system_command(
VariablesAndSecrets { variables, secrets }: &VariablesAndSecrets,
command: &mut SystemCommand,
global_replacers: &mut HashSet<(String, String)>,
secret_replacers: &mut HashSet<(String, String)>,
) -> anyhow::Result<()> {
if command.command.is_empty() {
return Ok(());
}
// first pass - global variables
let (res, more_replacers) = svi::interpolate_variables(
&command.command,
variables,
svi::Interpolator::DoubleBrackets,
false,
)
.with_context(|| {
format!(
"failed to interpolate global variables into command '{}'",
command.command
)
})?;
global_replacers.extend(more_replacers);
// second pass - core secrets
let (res, more_replacers) = svi::interpolate_variables(
&res,
secrets,
svi::Interpolator::DoubleBrackets,
false,
)
.with_context(|| {
format!(
"failed to interpolate core secrets into command '{}'",
command.command
)
})?;
secret_replacers.extend(more_replacers);
// set command with the result
command.command = res;
Ok(())
}
pub fn add_interp_update_log(
update: &mut Update,
global_replacers: &HashSet<(String, String)>,
secret_replacers: &HashSet<(String, String)>,
) {
// Show which variables were interpolated
if !global_replacers.is_empty() {
update.push_simple_log(
"interpolate global variables",
global_replacers
.iter()
.map(|(value, variable)| format!("<span class=\"text-muted-foreground\">{variable} =></span> {value}"))
.collect::<Vec<_>>()
.join("\n"),
);
}
// Only show names of interpolated secrets
if !secret_replacers.is_empty() {
update.push_simple_log(
"interpolate core secrets",
secret_replacers
.iter()
.map(|(_, variable)| format!("<span class=\"text-muted-foreground\">replaced:</span> {variable}"))
.collect::<Vec<_>>()
.join("\n"),
);
}
}
+114
View File
@@ -0,0 +1,114 @@
use std::str::FromStr;
use anyhow::Context;
use chrono::{Datelike, Local};
use komodo_client::entities::{
DayOfWeek, MaintenanceScheduleType, MaintenanceWindow,
};
use crate::config::core_config;
/// Check if a timestamp is currently in a maintenance window, given a list of windows.
pub fn is_in_maintenance(
windows: &[MaintenanceWindow],
timestamp: i64,
) -> bool {
windows
.iter()
.any(|window| is_maintenance_window_active(window, timestamp))
}
/// Check if the current timestamp falls within this maintenance window
pub fn is_maintenance_window_active(
window: &MaintenanceWindow,
timestamp: i64,
) -> bool {
if !window.enabled {
return false;
}
let dt = chrono::DateTime::from_timestamp(timestamp / 1000, 0)
.unwrap_or_else(chrono::Utc::now);
let (local_time, local_weekday, local_date) =
match (window.timezone.as_str(), core_config().timezone.as_str())
{
("", "") => {
let local_dt = dt.with_timezone(&Local);
(local_dt.time(), local_dt.weekday(), local_dt.date_naive())
}
("", timezone) | (timezone, _) => {
let tz: chrono_tz::Tz = match timezone
.parse()
.context("Failed to parse timezone")
{
Ok(tz) => tz,
Err(e) => {
warn!(
"Failed to parse maintenance window timezone: {e:#}"
);
return false;
}
};
let local_dt = dt.with_timezone(&tz);
(local_dt.time(), local_dt.weekday(), local_dt.date_naive())
}
};
match window.schedule_type {
MaintenanceScheduleType::Daily => {
is_time_in_window(window, local_time)
}
MaintenanceScheduleType::Weekly => {
let day_of_week =
DayOfWeek::from_str(&window.day_of_week).unwrap_or_default();
convert_day_of_week(local_weekday) == day_of_week
&& is_time_in_window(window, local_time)
}
MaintenanceScheduleType::OneTime => {
// Parse the date string and check if it matches current date
if let Ok(maintenance_date) =
chrono::NaiveDate::parse_from_str(&window.date, "%Y-%m-%d")
{
local_date == maintenance_date
&& is_time_in_window(window, local_time)
} else {
false
}
}
}
}
fn is_time_in_window(
window: &MaintenanceWindow,
current_time: chrono::NaiveTime,
) -> bool {
let start_time = chrono::NaiveTime::from_hms_opt(
window.hour as u32,
window.minute as u32,
0,
)
.unwrap_or(chrono::NaiveTime::from_hms_opt(0, 0, 0).unwrap());
let end_time = start_time
+ chrono::Duration::minutes(window.duration_minutes as i64);
// Handle case where maintenance window crosses midnight
if end_time < start_time {
current_time >= start_time || current_time <= end_time
} else {
current_time >= start_time && current_time <= end_time
}
}
fn convert_day_of_week(value: chrono::Weekday) -> DayOfWeek {
match value {
chrono::Weekday::Mon => DayOfWeek::Monday,
chrono::Weekday::Tue => DayOfWeek::Tuesday,
chrono::Weekday::Wed => DayOfWeek::Wednesday,
chrono::Weekday::Thu => DayOfWeek::Thursday,
chrono::Weekday::Fri => DayOfWeek::Friday,
chrono::Weekday::Sat => DayOfWeek::Saturday,
chrono::Weekday::Sun => DayOfWeek::Sunday,
}
}
+88 -2
View File
@@ -1,13 +1,16 @@
use std::time::Duration;
use std::{fmt::Write, time::Duration};
use anyhow::{Context, anyhow};
use indexmap::IndexSet;
use komodo_client::entities::{
ResourceTarget,
build::Build,
permission::{
Permission, PermissionLevel, SpecificPermission, UserTarget,
},
repo::Repo,
server::Server,
stack::Stack,
user::User,
};
use mongo_indexed::Document;
@@ -18,10 +21,11 @@ use rand::Rng;
use crate::{config::core_config, state::db_client};
pub mod action_state;
pub mod all_resources;
pub mod builder;
pub mod cache;
pub mod channel;
pub mod interpolate;
pub mod maintenance;
pub mod matcher;
pub mod procedure;
pub mod prune;
@@ -95,6 +99,70 @@ pub async fn git_token(
)
}
pub async fn stack_git_token(
stack: &mut Stack,
repo: Option<&mut Repo>,
) -> anyhow::Result<Option<String>> {
if let Some(repo) = repo {
return git_token(
&repo.config.git_provider,
&repo.config.git_account,
|https| repo.config.git_https = https,
)
.await
.with_context(|| {
format!(
"Failed to get git token. Stopping run. | {} | {}",
repo.config.git_provider, repo.config.git_account
)
});
}
git_token(
&stack.config.git_provider,
&stack.config.git_account,
|https| stack.config.git_https = https,
)
.await
.with_context(|| {
format!(
"Failed to get git token. Stopping run. | {} | {}",
stack.config.git_provider, stack.config.git_account
)
})
}
pub async fn build_git_token(
build: &mut Build,
repo: Option<&mut Repo>,
) -> anyhow::Result<Option<String>> {
if let Some(repo) = repo {
return git_token(
&repo.config.git_provider,
&repo.config.git_account,
|https| repo.config.git_https = https,
)
.await
.with_context(|| {
format!(
"Failed to get git token. Stopping run. | {} | {}",
repo.config.git_provider, repo.config.git_account
)
});
}
git_token(
&build.config.git_provider,
&build.config.git_account,
|https| build.config.git_https = https,
)
.await
.with_context(|| {
format!(
"Failed to get git token. Stopping run. | {} | {}",
build.config.git_provider, build.config.git_account
)
})
}
/// First checks db for token, then checks core config.
/// Only errors if db call errors.
pub async fn registry_token(
@@ -194,3 +262,21 @@ pub fn flatten_document(doc: Document) -> Document {
target
}
pub fn repo_link(
provider: &str,
repo: &str,
branch: &str,
https: bool,
) -> String {
let mut res = format!(
"http{}://{provider}/{repo}",
if https { "s" } else { "" }
);
// Each provider uses a different link format to get to branches.
// At least can support github for branch aware link.
if provider == "github.com" {
let _ = write!(&mut res, "/tree/{branch}");
}
res
}
+75 -6
View File
@@ -8,14 +8,14 @@ use anyhow::{Context, anyhow};
use async_timing_util::{ONE_MIN_MS, unix_timestamp_ms};
use komodo_client::entities::{
Operation, ResourceTarget, ResourceTargetVariant,
action::Action,
action::{Action, ActionState},
alerter::Alerter,
build::Build,
builder::Builder,
deployment::{Deployment, DeploymentState},
docker::container::{ContainerListItem, ContainerStateStatusEnum},
permission::{PermissionLevel, PermissionLevelAndSpecifics},
procedure::Procedure,
procedure::{Procedure, ProcedureState},
repo::Repo,
server::{Server, ServerState},
stack::{Stack, StackServiceNames, StackState},
@@ -40,9 +40,13 @@ use tokio::sync::Mutex;
use crate::{
config::core_config,
permission::get_user_permission_on_resource,
resource,
resource::{self, KomodoResource},
stack::compose_container_match_regex,
state::{db_client, deployment_status_cache, stack_status_cache},
state::{
action_state_cache, action_states, db_client,
deployment_status_cache, procedure_state_cache,
stack_status_cache,
},
};
use super::periphery_client;
@@ -88,10 +92,22 @@ pub async fn get_server_state(server: &Server) -> ServerState {
#[instrument(level = "debug")]
pub async fn get_deployment_state(
deployment: &Deployment,
id: &String,
) -> anyhow::Result<DeploymentState> {
if action_states()
.deployment
.get(id)
.await
.map(|s| s.get().map(|s| s.deploying))
.transpose()
.ok()
.flatten()
.unwrap_or_default()
{
return Ok(DeploymentState::Deploying);
}
let state = deployment_status_cache()
.get(&deployment.id)
.get(id)
.await
.unwrap_or_default()
.curr
@@ -424,3 +440,56 @@ pub async fn get_system_info(
};
Ok(res)
}
/// Get last time procedure / action was run using Update query.
/// Ignored whether run was successful.
pub async fn get_last_run_at<R: KomodoResource>(
id: &String,
) -> anyhow::Result<Option<i64>> {
let resource_type = R::resource_type();
let res = db_client()
.updates
.find_one(doc! {
"target.type": resource_type.as_ref(),
"target.id": id,
"operation": format!("Run{resource_type}"),
"status": "Complete"
})
.sort(doc! { "start_ts": -1 })
.await
.context("Failed to query updates collection for last run time")?
.map(|u| u.start_ts);
Ok(res)
}
pub async fn get_action_state(id: &String) -> ActionState {
if action_states()
.action
.get(id)
.await
.map(|s| s.get().map(|s| s.running))
.transpose()
.ok()
.flatten()
.unwrap_or_default()
{
return ActionState::Running;
}
action_state_cache().get(id).await.unwrap_or_default()
}
pub async fn get_procedure_state(id: &String) -> ProcedureState {
if action_states()
.procedure
.get(id)
.await
.map(|s| s.get().map(|s| s.running))
.transpose()
.ok()
.flatten()
.unwrap_or_default()
{
return ProcedureState::Running;
}
procedure_state_cache().get(id).await.unwrap_or_default()
}
+1
View File
@@ -65,6 +65,7 @@ async fn app() -> anyhow::Result<()> {
// Spawn background tasks
monitor::spawn_monitor_loop();
resource::spawn_resource_refresh_loop();
resource::spawn_all_resources_refresh_loop();
resource::spawn_build_state_refresh_loop();
resource::spawn_repo_state_refresh_loop();
resource::spawn_procedure_state_refresh_loop();
+198 -100
View File
@@ -1,4 +1,9 @@
use std::{collections::HashMap, path::PathBuf, str::FromStr};
use std::{
collections::HashMap,
path::PathBuf,
str::FromStr,
sync::{Mutex, OnceLock},
};
use anyhow::Context;
use derive_variants::ExtractVariant;
@@ -17,6 +22,7 @@ use mungos::{
use crate::{
alert::send_alerts,
helpers::maintenance::is_in_maintenance,
state::{db_client, server_status_cache},
};
@@ -25,6 +31,48 @@ type OpenAlertMap<T = AlertDataVariant> =
HashMap<ResourceTarget, HashMap<T, Alert>>;
type OpenDiskAlertMap = OpenAlertMap<PathBuf>;
/// Alert buffer to prevent immediate alerts on transient issues
struct AlertBuffer {
buffer: Mutex<HashMap<(String, AlertDataVariant), bool>>,
}
impl AlertBuffer {
fn new() -> Self {
Self {
buffer: Mutex::new(HashMap::new()),
}
}
/// Check if alert should be opened. Requires two consecutive calls to return true.
fn ready_to_open(
&self,
server_id: String,
variant: AlertDataVariant,
) -> bool {
let mut lock = self.buffer.lock().unwrap();
let ready = lock.entry((server_id, variant)).or_default();
if *ready {
*ready = false;
true
} else {
*ready = true;
false
}
}
/// Reset buffer state for a specific server/alert combination
fn reset(&self, server_id: String, variant: AlertDataVariant) {
let mut lock = self.buffer.lock().unwrap();
lock.remove(&(server_id, variant));
}
}
/// Global alert buffer instance
fn alert_buffer() -> &'static AlertBuffer {
static BUFFER: OnceLock<AlertBuffer> = OnceLock::new();
BUFFER.get_or_init(AlertBuffer::new)
}
#[instrument(level = "debug")]
pub async fn alert_servers(
ts: i64,
@@ -32,7 +80,8 @@ pub async fn alert_servers(
) {
let server_statuses = server_status_cache().get_list().await;
let (alerts, disk_alerts) = match get_open_alerts().await {
let (open_alerts, open_disk_alerts) = match get_open_alerts().await
{
Ok(alerts) => alerts,
Err(e) => {
error!("{e:#}");
@@ -44,12 +93,18 @@ pub async fn alert_servers(
let mut alerts_to_update = Vec::<(Alert, SendAlerts)>::new();
let mut alert_ids_to_close = Vec::<(Alert, SendAlerts)>::new();
let buffer = alert_buffer();
for server_status in server_statuses {
let Some(server) = servers.remove(&server_status.id) else {
continue;
};
let server_alerts =
alerts.get(&ResourceTarget::Server(server_status.id.clone()));
let server_alerts = open_alerts
.get(&ResourceTarget::Server(server_status.id.clone()));
// Check if server is in maintenance mode
let in_maintenance =
is_in_maintenance(&server.config.maintenance_windows, ts);
// ===================
// SERVER HEALTH
@@ -59,23 +114,30 @@ pub async fn alert_servers(
});
match (server_status.state, health_alert) {
(ServerState::NotOk, None) => {
// open unreachable alert
let alert = Alert {
id: Default::default(),
ts,
resolved: false,
resolved_ts: None,
level: SeverityLevel::Critical,
target: ResourceTarget::Server(server_status.id.clone()),
data: AlertData::ServerUnreachable {
id: server_status.id.clone(),
name: server.name.clone(),
region: optional_string(&server.config.region),
err: server_status.err.clone(),
},
};
alerts_to_open
.push((alert, server.config.send_unreachable_alerts))
// Only open unreachable alert if not in maintenance and buffer is ready
if !in_maintenance
&& buffer.ready_to_open(
server_status.id.clone(),
AlertDataVariant::ServerUnreachable,
)
{
let alert = Alert {
id: Default::default(),
ts,
resolved: false,
resolved_ts: None,
level: SeverityLevel::Critical,
target: ResourceTarget::Server(server_status.id.clone()),
data: AlertData::ServerUnreachable {
id: server_status.id.clone(),
name: server.name.clone(),
region: optional_string(&server.config.region),
err: server_status.err.clone(),
},
};
alerts_to_open
.push((alert, server.config.send_unreachable_alerts))
}
}
(ServerState::NotOk, Some(alert)) => {
// update alert err
@@ -109,7 +171,11 @@ pub async fn alert_servers(
server.config.send_unreachable_alerts,
));
}
_ => {}
(ServerState::Ok | ServerState::Disabled, None) => buffer
.reset(
server_status.id.clone(),
AlertDataVariant::ServerUnreachable,
),
}
let Some(health) = &server_status.health else {
@@ -126,34 +192,41 @@ pub async fn alert_servers(
match (health.cpu.level, cpu_alert, health.cpu.should_close_alert)
{
(SeverityLevel::Warning | SeverityLevel::Critical, None, _) => {
// open alert
let alert = Alert {
id: Default::default(),
ts,
resolved: false,
resolved_ts: None,
level: health.cpu.level,
target: ResourceTarget::Server(server_status.id.clone()),
data: AlertData::ServerCpu {
id: server_status.id.clone(),
name: server.name.clone(),
region: optional_string(&server.config.region),
percentage: server_status
.stats
.as_ref()
.map(|s| s.cpu_perc as f64)
.unwrap_or(0.0),
},
};
alerts_to_open.push((alert, server.config.send_cpu_alerts));
// Only open CPU alert if not in maintenance and buffer is ready
if !in_maintenance
&& buffer.ready_to_open(
server_status.id.clone(),
AlertDataVariant::ServerCpu,
)
{
let alert = Alert {
id: Default::default(),
ts,
resolved: false,
resolved_ts: None,
level: health.cpu.level,
target: ResourceTarget::Server(server_status.id.clone()),
data: AlertData::ServerCpu {
id: server_status.id.clone(),
name: server.name.clone(),
region: optional_string(&server.config.region),
percentage: server_status
.stats
.as_ref()
.map(|s| s.cpu_perc as f64)
.unwrap_or(0.0),
},
};
alerts_to_open.push((alert, server.config.send_cpu_alerts));
}
}
(
SeverityLevel::Warning | SeverityLevel::Critical,
Some(mut alert),
_,
) => {
// modify alert level only if it has increased
if alert.level < health.cpu.level {
// modify alert level only if it has increased and not in maintenance
if !in_maintenance && alert.level < health.cpu.level {
alert.level = health.cpu.level;
alert.data = AlertData::ServerCpu {
id: server_status.id.clone(),
@@ -184,7 +257,8 @@ pub async fn alert_servers(
alert_ids_to_close
.push((alert, server.config.send_cpu_alerts))
}
_ => {}
(SeverityLevel::Ok, _, _) => buffer
.reset(server_status.id.clone(), AlertDataVariant::ServerCpu),
}
// ===================
@@ -197,39 +271,46 @@ pub async fn alert_servers(
match (health.mem.level, mem_alert, health.mem.should_close_alert)
{
(SeverityLevel::Warning | SeverityLevel::Critical, None, _) => {
// open alert
let alert = Alert {
id: Default::default(),
ts,
resolved: false,
resolved_ts: None,
level: health.mem.level,
target: ResourceTarget::Server(server_status.id.clone()),
data: AlertData::ServerMem {
id: server_status.id.clone(),
name: server.name.clone(),
region: optional_string(&server.config.region),
total_gb: server_status
.stats
.as_ref()
.map(|s| s.mem_total_gb)
.unwrap_or(0.0),
used_gb: server_status
.stats
.as_ref()
.map(|s| s.mem_used_gb)
.unwrap_or(0.0),
},
};
alerts_to_open.push((alert, server.config.send_mem_alerts));
// Only open memory alert if not in maintenance and buffer is ready
if !in_maintenance
&& buffer.ready_to_open(
server_status.id.clone(),
AlertDataVariant::ServerMem,
)
{
let alert = Alert {
id: Default::default(),
ts,
resolved: false,
resolved_ts: None,
level: health.mem.level,
target: ResourceTarget::Server(server_status.id.clone()),
data: AlertData::ServerMem {
id: server_status.id.clone(),
name: server.name.clone(),
region: optional_string(&server.config.region),
total_gb: server_status
.stats
.as_ref()
.map(|s| s.mem_total_gb)
.unwrap_or(0.0),
used_gb: server_status
.stats
.as_ref()
.map(|s| s.mem_used_gb)
.unwrap_or(0.0),
},
};
alerts_to_open.push((alert, server.config.send_mem_alerts));
}
}
(
SeverityLevel::Warning | SeverityLevel::Critical,
Some(mut alert),
_,
) => {
// modify alert level only if it has increased
if alert.level < health.mem.level {
// modify alert level only if it has increased and not in maintenance
if !in_maintenance && alert.level < health.mem.level {
alert.level = health.mem.level;
alert.data = AlertData::ServerMem {
id: server_status.id.clone(),
@@ -270,14 +351,15 @@ pub async fn alert_servers(
alert_ids_to_close
.push((alert, server.config.send_mem_alerts))
}
_ => {}
(SeverityLevel::Ok, _, _) => buffer
.reset(server_status.id.clone(), AlertDataVariant::ServerMem),
}
// ===================
// SERVER DISK
// ===================
let server_disk_alerts = disk_alerts
let server_disk_alerts = open_disk_alerts
.get(&ResourceTarget::Server(server_status.id.clone()));
for (path, health) in &health.disks {
@@ -291,35 +373,48 @@ pub async fn alert_servers(
None,
_,
) => {
let disk = server_status.stats.as_ref().and_then(|stats| {
stats.disks.iter().find(|disk| disk.mount == *path)
});
let alert = Alert {
id: Default::default(),
ts,
resolved: false,
resolved_ts: None,
level: health.level,
target: ResourceTarget::Server(server_status.id.clone()),
data: AlertData::ServerDisk {
id: server_status.id.clone(),
name: server.name.clone(),
region: optional_string(&server.config.region),
path: path.to_owned(),
total_gb: disk.map(|d| d.total_gb).unwrap_or_default(),
used_gb: disk.map(|d| d.used_gb).unwrap_or_default(),
},
};
alerts_to_open
.push((alert, server.config.send_disk_alerts));
// Only open disk alert if not in maintenance and buffer is ready
if !in_maintenance
&& buffer.ready_to_open(
server_status.id.clone(),
AlertDataVariant::ServerDisk,
)
{
let disk =
server_status.stats.as_ref().and_then(|stats| {
stats.disks.iter().find(|disk| disk.mount == *path)
});
let alert = Alert {
id: Default::default(),
ts,
resolved: false,
resolved_ts: None,
level: health.level,
target: ResourceTarget::Server(
server_status.id.clone(),
),
data: AlertData::ServerDisk {
id: server_status.id.clone(),
name: server.name.clone(),
region: optional_string(&server.config.region),
path: path.to_owned(),
total_gb: disk
.map(|d| d.total_gb)
.unwrap_or_default(),
used_gb: disk.map(|d| d.used_gb).unwrap_or_default(),
},
};
alerts_to_open
.push((alert, server.config.send_disk_alerts));
}
}
(
SeverityLevel::Warning | SeverityLevel::Critical,
Some(mut alert),
_,
) => {
// modify alert level only if it has increased
if health.level < alert.level {
// modify alert level only if it has increased and not in maintenance
if !in_maintenance && health.level < alert.level {
let disk =
server_status.stats.as_ref().and_then(|stats| {
stats.disks.iter().find(|disk| disk.mount == *path)
@@ -354,7 +449,10 @@ pub async fn alert_servers(
alert_ids_to_close
.push((alert, server.config.send_disk_alerts))
}
_ => {}
(SeverityLevel::Ok, _, _) => buffer.reset(
server_status.id.clone(),
AlertDataVariant::ServerDisk,
),
}
}
@@ -372,14 +470,14 @@ pub async fn alert_servers(
}
tokio::join!(
open_alerts(&alerts_to_open),
open_new_alerts(&alerts_to_open),
update_alerts(&alerts_to_update),
resolve_alerts(&alert_ids_to_close),
);
}
#[instrument(level = "debug")]
async fn open_alerts(alerts: &[(Alert, SendAlerts)]) {
async fn open_new_alerts(alerts: &[(Alert, SendAlerts)]) {
if alerts.is_empty() {
return;
}
+6 -5
View File
@@ -145,8 +145,8 @@ pub async fn update_cache_for_server(server: &Server) {
// Handle server disabled
if !server.config.enabled {
insert_deployments_status_unknown(deployments).await;
insert_repos_status_unknown(repos).await;
insert_stacks_status_unknown(stacks).await;
insert_repos_status_unknown(repos).await;
insert_server_status(
server,
ServerState::Disabled,
@@ -170,12 +170,12 @@ pub async fn update_cache_for_server(server: &Server) {
Ok(version) => version.version,
Err(e) => {
insert_deployments_status_unknown(deployments).await;
insert_repos_status_unknown(repos).await;
insert_stacks_status_unknown(stacks).await;
insert_repos_status_unknown(repos).await;
insert_server_status(
server,
ServerState::NotOk,
String::from("unknown"),
String::from("Unknown"),
None,
(None, None, None, None, None),
Serror::from(&e),
@@ -190,8 +190,8 @@ pub async fn update_cache_for_server(server: &Server) {
Ok(stats) => Some(filter_volumes(server, stats)),
Err(e) => {
insert_deployments_status_unknown(deployments).await;
insert_repos_status_unknown(repos).await;
insert_stacks_status_unknown(stacks).await;
insert_repos_status_unknown(repos).await;
insert_server_status(
server,
ServerState::NotOk,
@@ -267,8 +267,9 @@ pub async fn update_cache_for_server(server: &Server) {
path: optional_string(&repo.config.path),
})
.await
.map(|r| (r.hash, r.message))
.ok()
.flatten()
.map(|c| (c.hash, c.message))
.unzip();
status_cache
.insert(
+12 -23
View File
@@ -2,11 +2,11 @@ use std::time::Duration;
use anyhow::Context;
use komodo_client::entities::{
Operation, ResourceTarget, ResourceTargetVariant,
NoData, Operation, ResourceTarget, ResourceTargetVariant,
action::{
Action, ActionConfig, ActionConfigDiff, ActionInfo,
ActionListItem, ActionListItemInfo, ActionQuerySpecifics,
ActionState, PartialActionConfig,
Action, ActionConfig, ActionConfigDiff, ActionListItem,
ActionListItemInfo, ActionQuerySpecifics, ActionState,
PartialActionConfig,
},
resource::Resource,
update::Update,
@@ -18,6 +18,7 @@ use mungos::{
};
use crate::{
helpers::query::{get_action_state, get_last_run_at},
schedule::{
cancel_schedule, get_schedule_item_info, update_schedule,
},
@@ -28,7 +29,7 @@ impl super::KomodoResource for Action {
type Config = ActionConfig;
type PartialConfig = PartialActionConfig;
type ConfigDiff = ActionConfigDiff;
type Info = ActionInfo;
type Info = NoData;
type ListItem = ActionListItem;
type QuerySpecifics = ActionQuerySpecifics;
@@ -48,18 +49,22 @@ impl super::KomodoResource for Action {
async fn to_list_item(
action: Resource<Self::Config, Self::Info>,
) -> Self::ListItem {
let state = get_action_state(&action.id).await;
let (state, last_run_at) = tokio::join!(
get_action_state(&action.id),
get_last_run_at::<Action>(&action.id)
);
let (next_scheduled_run, schedule_error) = get_schedule_item_info(
&ResourceTarget::Action(action.id.clone()),
);
ActionListItem {
name: action.name,
id: action.id,
template: action.template,
tags: action.tags,
resource_type: ResourceTargetVariant::Action,
info: ActionListItemInfo {
state,
last_run_at: action.info.last_run_at,
last_run_at: last_run_at.unwrap_or(None),
next_scheduled_run,
schedule_error,
},
@@ -181,22 +186,6 @@ pub async fn refresh_action_state_cache() {
});
}
async fn get_action_state(id: &String) -> ActionState {
if action_states()
.action
.get(id)
.await
.map(|s| s.get().map(|s| s.running))
.transpose()
.ok()
.flatten()
.unwrap_or_default()
{
return ActionState::Running;
}
action_state_cache().get(id).await.unwrap_or_default()
}
async fn get_action_state_from_db(id: &str) -> ActionState {
async {
let state = db_client()
+1
View File
@@ -40,6 +40,7 @@ impl super::KomodoResource for Alerter {
AlerterListItem {
name: alerter.name,
id: alerter.id,
template: alerter.template,
tags: alerter.tags,
resource_type: ResourceTargetVariant::Alerter,
info: AlerterListItemInfo {
+59 -5
View File
@@ -14,6 +14,7 @@ use komodo_client::{
builder::Builder,
environment_vars_from_str, optional_string,
permission::PermissionLevel,
repo::Repo,
resource::Resource,
to_docker_compatible_name,
update::Update,
@@ -29,8 +30,13 @@ use resolver_api::Resolve;
use crate::{
api::write::WriteArgs,
config::core_config,
helpers::{empty_or_only_spaces, query::get_latest_update},
state::{action_states, build_state_cache, db_client},
helpers::{
empty_or_only_spaces, query::get_latest_update, repo_link,
},
permission::get_check_permissions,
state::{
action_states, all_resources_cache, build_state_cache, db_client,
},
};
impl super::KomodoResource for Build {
@@ -62,9 +68,36 @@ impl super::KomodoResource for Build {
build: Resource<Self::Config, Self::Info>,
) -> Self::ListItem {
let state = get_build_state(&build.id).await;
let default_git = (
build.config.git_provider,
build.config.repo,
build.config.branch,
build.config.git_https,
);
let (git_provider, repo, branch, git_https) =
if build.config.linked_repo.is_empty() {
default_git
} else {
all_resources_cache()
.load()
.repos
.get(&build.config.linked_repo)
.map(|r| {
(
r.config.git_provider.clone(),
r.config.repo.clone(),
r.config.branch.clone(),
r.config.git_https,
)
})
.unwrap_or(default_git)
};
BuildListItem {
name: build.name,
id: build.id,
template: build.template,
tags: build.tags,
resource_type: ResourceTargetVariant::Build,
info: BuildListItemInfo {
@@ -72,9 +105,17 @@ impl super::KomodoResource for Build {
version: build.config.version,
builder_id: build.config.builder_id,
files_on_host: build.config.files_on_host,
git_provider: optional_string(build.config.git_provider),
repo: optional_string(build.config.repo),
branch: optional_string(build.config.branch),
dockerfile_contents: !build.config.dockerfile.is_empty(),
linked_repo: build.config.linked_repo,
repo_link: repo_link(
&git_provider,
&repo,
&branch,
git_https,
),
git_provider,
repo,
branch,
image_registry_domain: optional_string(
build.config.image_registry.domain,
),
@@ -226,6 +267,19 @@ async fn validate_config(
config.builder_id = Some(builder.id)
}
}
if let Some(linked_repo) = &config.linked_repo {
if !linked_repo.is_empty() {
let repo = get_check_permissions::<Repo>(
linked_repo,
user,
PermissionLevel::Read.attach(),
)
.await
.context("Cannot attach Repo to this Build")?;
// in case it comes in as name
config.linked_repo = Some(repo.id);
}
}
if let Some(build_args) = &config.build_args {
environment_vars_from_str(build_args)
.context("Invalid build_args")?;
+1
View File
@@ -64,6 +64,7 @@ impl super::KomodoResource for Builder {
BuilderListItem {
name: builder.name,
id: builder.id,
template: builder.template,
tags: builder.tags,
resource_type: ResourceTargetVariant::Builder,
info: BuilderListItemInfo {
+27 -15
View File
@@ -14,7 +14,7 @@ use komodo_client::entities::{
permission::{PermissionLevel, SpecificPermission},
resource::Resource,
server::Server,
to_docker_compatible_name,
to_container_compatible_name,
update::Update,
user::User,
};
@@ -50,7 +50,7 @@ impl super::KomodoResource for Deployment {
}
fn validated_name(name: &str) -> String {
to_docker_compatible_name(name)
to_container_compatible_name(name)
}
fn creator_specific_permissions() -> IndexSet<SpecificPermission> {
@@ -78,6 +78,20 @@ impl super::KomodoResource for Deployment {
deployment: Resource<Self::Config, Self::Info>,
) -> Self::ListItem {
let status = deployment_status_cache().get(&deployment.id).await;
let state = if action_states()
.deployment
.get(&deployment.id)
.await
.map(|s| s.get().map(|s| s.deploying))
.transpose()
.ok()
.flatten()
.unwrap_or_default()
{
DeploymentState::Deploying
} else {
status.as_ref().map(|s| s.curr.state).unwrap_or_default()
};
let (build_image, build_id) = match deployment.config.image {
DeploymentImage::Build { build_id, version } => {
let (build_name, build_id, build_version) =
@@ -114,13 +128,11 @@ impl super::KomodoResource for Deployment {
DeploymentListItem {
name: deployment.name,
id: deployment.id,
template: deployment.template,
tags: deployment.tags,
resource_type: ResourceTargetVariant::Deployment,
info: DeploymentListItemInfo {
state: status
.as_ref()
.map(|s| s.curr.state)
.unwrap_or_default(),
state,
status: status.as_ref().and_then(|s| {
s.curr.container.as_ref().and_then(|c| c.status.to_owned())
}),
@@ -217,9 +229,9 @@ impl super::KomodoResource for Deployment {
deployment: &Resource<Self::Config, Self::Info>,
update: &mut Update,
) -> anyhow::Result<()> {
let state = get_deployment_state(deployment)
let state = get_deployment_state(&deployment.id)
.await
.context("failed to get container state")?;
.context("Failed to get deployment state")?;
if matches!(
state,
DeploymentState::NotDeployed | DeploymentState::Unknown
@@ -235,7 +247,7 @@ impl super::KomodoResource for Deployment {
Ok(server) => server,
Err(e) => {
update.push_error_log(
"remove container",
"Remove Container",
format_serror(
&e.context(format!(
"failed to retrieve server at {} from db.",
@@ -250,8 +262,8 @@ impl super::KomodoResource for Deployment {
if !server.config.enabled {
// Don't need to
update.push_simple_log(
"remove container",
"skipping container removal, server is disabled.",
"Remove Container",
"Skipping container removal, server is disabled.",
);
return Ok(());
}
@@ -261,9 +273,9 @@ impl super::KomodoResource for Deployment {
// This case won't ever happen, as periphery_client only fallible if the server is disabled.
// Leaving it for completeness sake
update.push_error_log(
"remove container",
"Remove Container",
format_serror(
&e.context("failed to get periphery client").into(),
&e.context("Failed to get periphery client").into(),
),
);
return Ok(());
@@ -279,9 +291,9 @@ impl super::KomodoResource for Deployment {
{
Ok(log) => update.logs.push(log),
Err(e) => update.push_error_log(
"remove container",
"Remove Container",
format_serror(
&e.context("failed to remove container").into(),
&e.context("Failed to remove container").into(),
),
),
};
+60 -39
View File
@@ -69,7 +69,10 @@ pub use build::{
pub use procedure::{
refresh_procedure_state_cache, spawn_procedure_state_refresh_loop,
};
pub use refresh::spawn_resource_refresh_loop;
pub use refresh::{
refresh_all_resources_cache, spawn_all_resources_refresh_loop,
spawn_resource_refresh_loop,
};
pub use repo::{
refresh_repo_state_cache, spawn_repo_state_refresh_loop,
};
@@ -488,12 +491,13 @@ pub async fn create<T: KomodoResource>(
let resource = Resource::<T::Config, T::Info> {
id: Default::default(),
name,
updated_at: start_ts,
description: Default::default(),
template: Default::default(),
tags: Default::default(),
config: config.into(),
info: T::default_info().await?,
base_permission: PermissionLevel::None.into(),
updated_at: start_ts,
};
let resource_id = T::coll()
@@ -537,6 +541,8 @@ pub async fn create<T: KomodoResource>(
T::post_create(&resource, &mut update).await?;
refresh_all_resources_cache().await;
update.finalize();
add_update(update).await?;
@@ -632,8 +638,9 @@ pub async fn update<T: KomodoResource>(
T::post_update(&updated, &mut update).await?;
update.finalize();
refresh_all_resources_cache().await;
update.finalize();
add_update(update).await?;
Ok(updated)
@@ -659,53 +666,62 @@ fn resource_target<T: KomodoResource>(id: String) -> ResourceTarget {
}
}
pub async fn update_description<T: KomodoResource>(
pub struct ResourceMetaUpdate {
pub description: Option<String>,
pub template: Option<bool>,
pub tags: Option<Vec<String>>,
}
impl ResourceMetaUpdate {
pub fn is_none(&self) -> bool {
self.description.is_none()
&& self.template.is_none()
&& self.tags.is_none()
}
}
pub async fn update_meta<T: KomodoResource>(
id_or_name: &str,
description: &str,
user: &User,
meta: ResourceMetaUpdate,
args: &WriteArgs,
) -> anyhow::Result<()> {
get_check_permissions::<T>(
id_or_name,
user,
&args.user,
PermissionLevel::Write.into(),
)
.await?;
T::coll()
.update_one(
id_or_name_filter(id_or_name),
doc! { "$set": { "description": description } },
)
.await?;
Ok(())
}
pub async fn update_tags<T: KomodoResource>(
id_or_name: &str,
tags: Vec<String>,
args: &WriteArgs,
) -> anyhow::Result<()> {
let futures = tags.iter().map(|tag| async {
match get_tag(tag).await {
Ok(tag) => Ok(tag.id),
Err(_) => CreateTag {
name: tag.to_string(),
let mut set = Document::new();
if let Some(description) = meta.description {
set.insert("description", description);
}
if let Some(template) = meta.template {
set.insert("template", template);
}
if let Some(tags) = meta.tags {
// First normalize to tag ids only
let futures = tags.iter().map(|tag| async {
match get_tag(tag).await {
Ok(tag) => Ok(tag.id),
Err(_) => CreateTag {
name: tag.to_string(),
}
.resolve(args)
.await
.map(|tag| tag.id),
}
.resolve(args)
});
let tags = join_all(futures)
.await
.map(|tag| tag.id),
}
});
let tags = join_all(futures)
.await
.into_iter()
.flatten()
.collect::<Vec<_>>();
.into_iter()
.flatten()
.collect::<Vec<_>>();
set.insert("tags", tags);
}
T::coll()
.update_one(
id_or_name_filter(id_or_name),
doc! { "$set": { "tags": tags } },
)
.update_one(id_or_name_filter(id_or_name), doc! { "$set": set })
.await?;
refresh_all_resources_cache().await;
Ok(())
}
@@ -769,8 +785,11 @@ pub async fn rename<T: KomodoResource>(
),
);
refresh_all_resources_cache().await;
update.finalize();
update.id = add_update(update.clone()).await?;
Ok(update)
}
@@ -829,6 +848,8 @@ pub async fn delete<T: KomodoResource>(
update.push_error_log("post delete", format_serror(&e.into()));
}
refresh_all_resources_cache().await;
update.finalize();
add_update(update).await?;
+7 -17
View File
@@ -31,6 +31,7 @@ use mungos::{
use crate::{
config::core_config,
helpers::query::{get_last_run_at, get_procedure_state},
schedule::{
cancel_schedule, get_schedule_item_info, update_schedule,
},
@@ -61,18 +62,23 @@ impl super::KomodoResource for Procedure {
async fn to_list_item(
procedure: Resource<Self::Config, Self::Info>,
) -> Self::ListItem {
let state = get_procedure_state(&procedure.id).await;
let (state, last_run_at) = tokio::join!(
get_procedure_state(&procedure.id),
get_last_run_at::<Procedure>(&procedure.id)
);
let (next_scheduled_run, schedule_error) = get_schedule_item_info(
&ResourceTarget::Procedure(procedure.id.clone()),
);
ProcedureListItem {
name: procedure.name,
id: procedure.id,
template: procedure.template,
tags: procedure.tags,
resource_type: ResourceTargetVariant::Procedure,
info: ProcedureListItemInfo {
stages: procedure.config.stages.len() as i64,
state,
last_run_at: last_run_at.unwrap_or(None),
next_scheduled_run,
schedule_error,
},
@@ -754,22 +760,6 @@ pub async fn refresh_procedure_state_cache() {
});
}
async fn get_procedure_state(id: &String) -> ProcedureState {
if action_states()
.procedure
.get(id)
.await
.map(|s| s.get().map(|s| s.running))
.transpose()
.ok()
.flatten()
.unwrap_or_default()
{
return ProcedureState::Running;
}
procedure_state_cache().get(id).await.unwrap_or_default()
}
async fn get_procedure_state_from_db(id: &str) -> ProcedureState {
async {
let state = db_client()
+23 -4
View File
@@ -14,9 +14,31 @@ use resolver_api::Resolve;
use crate::{
api::{execute::pull_deployment_inner, write::WriteArgs},
config::core_config,
state::db_client,
helpers::all_resources::AllResourcesById,
state::{all_resources_cache, db_client},
};
pub fn spawn_all_resources_refresh_loop() {
tokio::spawn(async move {
let mut interval = tokio::time::interval(Duration::from_secs(15));
loop {
interval.tick().await;
refresh_all_resources_cache().await;
}
});
}
pub async fn refresh_all_resources_cache() {
let all = match AllResourcesById::load().await {
Ok(all) => all,
Err(e) => {
error!("Failed to load all resources by id cache | {e:#}");
return;
}
};
all_resources_cache().store(all.into());
}
pub fn spawn_resource_refresh_loop() {
let interval: Timelength = core_config()
.resource_poll_interval
@@ -167,9 +189,6 @@ async fn refresh_syncs() {
return;
};
for sync in syncs {
if sync.config.repo.is_empty() {
continue;
}
RefreshResourceSyncPending { sync: sync.id }
.resolve(
&WriteArgs { user: sync_user().clone() },
+8 -1
View File
@@ -24,7 +24,7 @@ use periphery_client::api::git::DeleteRepo;
use crate::{
config::core_config,
helpers::periphery_client,
helpers::{periphery_client, repo_link},
state::{
action_states, db_client, repo_state_cache, repo_status_cache,
},
@@ -66,6 +66,7 @@ impl super::KomodoResource for Repo {
RepoListItem {
name: repo.name,
id: repo.id,
template: repo.template,
tags: repo.tags,
resource_type: ResourceTargetVariant::Repo,
info: RepoListItemInfo {
@@ -73,6 +74,12 @@ impl super::KomodoResource for Repo {
builder_id: repo.config.builder_id,
last_pulled_at: repo.info.last_pulled_at,
last_built_at: repo.info.last_built_at,
repo_link: repo_link(
&repo.config.git_provider,
&repo.config.repo,
&repo.config.branch,
repo.config.git_https,
),
git_provider: repo.config.git_provider,
repo: repo.config.repo,
branch: repo.config.branch,
+5 -1
View File
@@ -65,10 +65,14 @@ impl super::KomodoResource for Server {
ServerListItem {
name: server.name,
id: server.id,
template: server.template,
tags: server.tags,
resource_type: ResourceTargetVariant::Server,
info: ServerListItemInfo {
state: status.map(|s| s.state).unwrap_or_default(),
state: status.as_ref().map(|s| s.state).unwrap_or_default(),
version: status
.map(|s| s.version.clone())
.unwrap_or(String::from("Unknown")),
region: server.config.region,
address: server.config.address,
send_unreachable_alerts: server
+69 -109
View File
@@ -6,6 +6,7 @@ use komodo_client::{
entities::{
Operation, ResourceTarget, ResourceTargetVariant,
permission::{PermissionLevel, SpecificPermission},
repo::Repo,
resource::Resource,
server::Server,
stack::{
@@ -25,10 +26,11 @@ use resolver_api::Resolve;
use crate::{
api::write::WriteArgs,
config::core_config,
helpers::{periphery_client, query::get_stack_state},
helpers::{periphery_client, query::get_stack_state, repo_link},
monitor::update_cache_for_server,
state::{
action_states, db_client, server_status_cache, stack_status_cache,
action_states, all_resources_cache, db_client,
server_status_cache, stack_status_cache,
},
};
@@ -79,8 +81,20 @@ impl super::KomodoResource for Stack {
stack: Resource<Self::Config, Self::Info>,
) -> Self::ListItem {
let status = stack_status_cache().get(&stack.id).await;
let state =
status.as_ref().map(|s| s.curr.state).unwrap_or_default();
let state = if action_states()
.stack
.get(&stack.id)
.await
.map(|s| s.get().map(|s| s.deploying))
.transpose()
.ok()
.flatten()
.unwrap_or_default()
{
StackState::Deploying
} else {
status.as_ref().map(|s| s.curr.state).unwrap_or_default()
};
let project_name = stack.project_name(false);
let services = status
.as_ref()
@@ -97,6 +111,31 @@ impl super::KomodoResource for Stack {
})
.unwrap_or_default();
let default_git = (
stack.config.git_provider,
stack.config.repo,
stack.config.branch,
stack.config.git_https,
);
let (git_provider, repo, branch, git_https) =
if stack.config.linked_repo.is_empty() {
default_git
} else {
all_resources_cache()
.load()
.repos
.get(&stack.config.linked_repo)
.map(|r| {
(
r.config.git_provider.clone(),
r.config.repo.clone(),
r.config.branch.clone(),
r.config.git_https,
)
})
.unwrap_or(default_git)
};
// This is only true if it is KNOWN to be true. so other cases are false.
let (project_missing, status) =
if stack.config.server_id.is_empty()
@@ -126,8 +165,9 @@ impl super::KomodoResource for Stack {
};
StackListItem {
id: stack.id,
name: stack.name,
id: stack.id,
template: stack.template,
tags: stack.tags,
resource_type: ResourceTargetVariant::Stack,
info: StackListItemInfo {
@@ -137,11 +177,18 @@ impl super::KomodoResource for Stack {
project_missing,
file_contents: !stack.config.file_contents.is_empty(),
server_id: stack.config.server_id,
linked_repo: stack.config.linked_repo,
missing_files: stack.info.missing_files,
files_on_host: stack.config.files_on_host,
git_provider: stack.config.git_provider,
repo: stack.config.repo,
branch: stack.config.branch,
repo_link: repo_link(
&git_provider,
&repo,
&branch,
git_https,
),
git_provider,
repo,
branch,
latest_hash: stack.info.latest_hash,
deployed_hash: stack.info.deployed_hash,
},
@@ -339,110 +386,23 @@ async fn validate_config(
PermissionLevel::Read.attach(),
)
.await
.context("Cannot attach stack to this Server")?;
.context("Cannot attach Stack to this Server")?;
// in case it comes in as name
config.server_id = Some(server.id);
}
}
if let Some(linked_repo) = &config.linked_repo {
if !linked_repo.is_empty() {
let repo = get_check_permissions::<Repo>(
linked_repo,
user,
PermissionLevel::Read.attach(),
)
.await
.context("Cannot attach Repo to this Stack")?;
// in case it comes in as name
config.linked_repo = Some(repo.id);
}
}
Ok(())
}
// pub fn spawn_resource_sync_state_refresh_loop() {
// tokio::spawn(async move {
// loop {
// refresh_resource_sync_state_cache().await;
// tokio::time::sleep(Duration::from_secs(60)).await;
// }
// });
// }
// pub async fn refresh_resource_sync_state_cache() {
// let _ = async {
// let resource_syncs =
// find_collect(&db_client().resource_syncs, None, None)
// .await
// .context("failed to get resource_syncs from db")?;
// let cache = resource_sync_state_cache();
// for resource_sync in resource_syncs {
// let state =
// get_resource_sync_state_from_db(&resource_sync.id).await;
// cache.insert(resource_sync.id, state).await;
// }
// anyhow::Ok(())
// }
// .await
// .inspect_err(|e| {
// error!("failed to refresh resource_sync state cache | {e:#}")
// });
// }
// async fn get_resource_sync_state(
// id: &String,
// data: &PendingSyncUpdatesData,
// ) -> StackState {
// if let Some(state) = action_states()
// .resource_sync
// .get(id)
// .await
// .and_then(|s| {
// s.get()
// .map(|s| {
// if s.syncing {
// Some(StackState::Syncing)
// } else {
// None
// }
// })
// .ok()
// })
// .flatten()
// {
// return state;
// }
// let data = match data {
// PendingSyncUpdatesData::Err(_) => return StackState::Failed,
// PendingSyncUpdatesData::Ok(data) => data,
// };
// if !data.no_updates() {
// return StackState::Pending;
// }
// resource_sync_state_cache()
// .get(id)
// .await
// .unwrap_or_default()
// }
// async fn get_resource_sync_state_from_db(id: &str) -> StackState {
// async {
// let state = db_client()
// .await
// .updates
// .find_one(doc! {
// "target.type": "Stack",
// "target.id": id,
// "operation": "RunSync"
// })
// .with_options(
// FindOneOptions::builder()
// .sort(doc! { "start_ts": -1 })
// .build(),
// )
// .await?
// .map(|u| {
// if u.success {
// StackState::Ok
// } else {
// StackState::Failed
// }
// })
// .unwrap_or(StackState::Ok);
// anyhow::Ok(state)
// }
// .await
// .inspect_err(|e| {
// warn!(
// "failed to get resource sync state from db for {id} | {e:#}"
// )
// })
// .unwrap_or(StackState::Unknown)
// }
+70 -11
View File
@@ -5,6 +5,8 @@ use komodo_client::{
entities::{
Operation, ResourceTarget, ResourceTargetVariant,
komodo_timestamp,
permission::PermissionLevel,
repo::Repo,
resource::Resource,
sync::{
PartialResourceSyncConfig, ResourceSync, ResourceSyncConfig,
@@ -22,7 +24,9 @@ use resolver_api::Resolve;
use crate::{
api::write::WriteArgs,
state::{action_states, db_client},
helpers::repo_link,
permission::get_check_permissions,
state::{action_states, all_resources_cache, db_client},
};
impl super::KomodoResource for ResourceSync {
@@ -52,18 +56,52 @@ impl super::KomodoResource for ResourceSync {
let state =
get_resource_sync_state(&resource_sync.id, &resource_sync.info)
.await;
let default_git = (
resource_sync.config.git_provider,
resource_sync.config.repo,
resource_sync.config.branch,
resource_sync.config.git_https,
);
let (git_provider, repo, branch, git_https) =
if resource_sync.config.linked_repo.is_empty() {
default_git
} else {
all_resources_cache()
.load()
.repos
.get(&resource_sync.config.linked_repo)
.map(|r| {
(
r.config.git_provider.clone(),
r.config.repo.clone(),
r.config.branch.clone(),
r.config.git_https,
)
})
.unwrap_or(default_git)
};
ResourceSyncListItem {
id: resource_sync.id,
name: resource_sync.name,
id: resource_sync.id,
template: resource_sync.template,
tags: resource_sync.tags,
resource_type: ResourceTargetVariant::ResourceSync,
info: ResourceSyncListItemInfo {
file_contents: !resource_sync.config.file_contents.is_empty(),
files_on_host: resource_sync.config.files_on_host,
managed: resource_sync.config.managed,
git_provider: resource_sync.config.git_provider,
repo: resource_sync.config.repo,
branch: resource_sync.config.branch,
linked_repo: resource_sync.config.linked_repo,
repo_link: repo_link(
&git_provider,
&repo,
&branch,
git_https,
),
git_provider,
repo,
branch,
last_sync_ts: resource_sync.info.last_sync_ts,
last_sync_hash: resource_sync.info.last_sync_hash,
last_sync_message: resource_sync.info.last_sync_message,
@@ -93,10 +131,10 @@ impl super::KomodoResource for ResourceSync {
}
async fn validate_create_config(
_config: &mut Self::PartialConfig,
_user: &User,
config: &mut Self::PartialConfig,
user: &User,
) -> anyhow::Result<()> {
Ok(())
validate_config(config, user).await
}
async fn post_create(
@@ -127,10 +165,10 @@ impl super::KomodoResource for ResourceSync {
async fn validate_update_config(
_id: &str,
_config: &mut Self::PartialConfig,
_user: &User,
config: &mut Self::PartialConfig,
user: &User,
) -> anyhow::Result<()> {
Ok(())
validate_config(config, user).await
}
async fn post_update(
@@ -178,6 +216,27 @@ impl super::KomodoResource for ResourceSync {
}
}
#[instrument(skip(user))]
async fn validate_config(
config: &mut PartialResourceSyncConfig,
user: &User,
) -> anyhow::Result<()> {
if let Some(linked_repo) = &config.linked_repo {
if !linked_repo.is_empty() {
let repo = get_check_permissions::<Repo>(
linked_repo,
user,
PermissionLevel::Read.attach(),
)
.await
.context("Cannot attach Repo to this Resource Sync")?;
// in case it comes in as name
config.linked_repo = Some(repo.id);
}
}
Ok(())
}
async fn get_resource_sync_state(
id: &String,
data: &ResourceSyncInfo,
+20 -17
View File
@@ -24,6 +24,7 @@ use resolver_api::Resolve;
use crate::{
alert::send_alerts,
api::execute::{ExecuteArgs, ExecuteRequest},
config::core_config,
helpers::update::init_execution_update,
state::db_client,
};
@@ -313,23 +314,25 @@ fn find_next_occurrence(
})?
}
};
let next = if schedule.timezone().is_empty() {
let tz_time = chrono::Local::now().with_timezone(&Local);
cron
.find_next_occurrence(&tz_time, false)
.context("Failed to find next run time")?
.timestamp_millis()
} else {
let tz: chrono_tz::Tz = schedule
.timezone()
.parse()
.context("Failed to parse schedule timezone")?;
let tz_time = chrono::Local::now().with_timezone(&tz);
cron
.find_next_occurrence(&tz_time, false)
.context("Failed to find next run time")?
.timestamp_millis()
};
let next =
match (schedule.timezone(), core_config().timezone.as_str()) {
("", "") => {
let tz_time = chrono::Local::now().with_timezone(&Local);
cron
.find_next_occurrence(&tz_time, false)
.context("Failed to find next run time")?
.timestamp_millis()
}
("", timezone) | (timezone, _) => {
let tz: chrono_tz::Tz =
timezone.parse().context("Failed to parse timezone")?;
let tz_time = chrono::Local::now().with_timezone(&tz);
cron
.find_next_occurrence(&tz_time, false)
.context("Failed to find next run time")?
.timestamp_millis()
}
};
Ok(next)
}
+12 -18
View File
@@ -3,7 +3,8 @@ use std::{fs, path::PathBuf};
use anyhow::Context;
use formatting::format_serror;
use komodo_client::entities::{
CloneArgs, FileContents, stack::Stack, update::Log,
FileContents, RepoExecutionArgs, repo::Repo, stack::Stack,
update::Log,
};
use crate::{config::core_config, helpers::git_token};
@@ -19,10 +20,12 @@ pub struct RemoteComposeContents {
/// Returns Result<(read paths, error paths, logs, short hash, commit message)>
pub async fn get_repo_compose_contents(
stack: &Stack,
repo: Option<&Repo>,
// Collect any files which are missing in the repo.
mut missing_files: Option<&mut Vec<String>>,
) -> anyhow::Result<RemoteComposeContents> {
let clone_args: CloneArgs = stack.into();
let clone_args: RepoExecutionArgs =
repo.map(Into::into).unwrap_or(stack.into());
let (repo_path, _logs, hash, message) =
ensure_remote_repo(clone_args)
.await
@@ -68,7 +71,7 @@ pub async fn get_repo_compose_contents(
/// Returns (destination, logs, hash, message)
pub async fn ensure_remote_repo(
mut clone_args: CloneArgs,
mut clone_args: RepoExecutionArgs,
) -> anyhow::Result<(PathBuf, Vec<Log>, Option<String>, Option<String>)>
{
let config = core_config();
@@ -88,20 +91,11 @@ pub async fn ensure_remote_repo(
let repo_path =
clone_args.unique_path(&core_config().repo_directory)?;
clone_args.destination = Some(repo_path.display().to_string());
// Don't want to run these on core.
clone_args.on_clone = None;
clone_args.on_pull = None;
git::pull_or_clone(
clone_args,
&config.repo_directory,
access_token,
&[],
"",
None,
&[],
)
.await
.context("Failed to clone stack repo")
.map(|res| (repo_path, res.logs, res.hash, res.message))
git::pull_or_clone(clone_args, &config.repo_directory, access_token)
.await
.context("Failed to clone stack repo")
.map(|(res, _)| {
(repo_path, res.logs, res.commit_hash, res.commit_message)
})
}
+11 -1
View File
@@ -4,6 +4,7 @@ use std::{
};
use anyhow::Context;
use arc_swap::ArcSwap;
use komodo_client::entities::{
action::ActionState,
build::BuildState,
@@ -21,7 +22,10 @@ use crate::{
auth::jwt::JwtClient,
config::core_config,
db::DbClient,
helpers::{action_state::ActionStates, cache::Cache},
helpers::{
action_state::ActionStates, all_resources::AllResourcesById,
cache::Cache,
},
monitor::{
CachedDeploymentStatus, CachedRepoStatus, CachedServerStatus,
CachedStackStatus, History,
@@ -196,3 +200,9 @@ pub fn action_state_cache() -> &'static ActionStateCache {
OnceLock::new();
ACTION_STATE_CACHE.get_or_init(Default::default)
}
pub fn all_resources_cache() -> &'static ArcSwap<AllResourcesById> {
static ALL_RESOURCES: OnceLock<ArcSwap<AllResourcesById>> =
OnceLock::new();
ALL_RESOURCES.get_or_init(Default::default)
}
+6 -19
View File
@@ -32,7 +32,7 @@ use crate::{
state::{deployment_status_cache, stack_status_cache},
};
use super::{AllResourcesById, ResourceSyncTrait};
use super::ResourceSyncTrait;
/// All entries in here are due to be deployed,
/// after the given dependencies,
@@ -48,7 +48,6 @@ pub struct SyncDeployParams<'a> {
pub stacks: &'a [ResourceToml<PartialStackConfig>],
// Names to stacks
pub stack_map: &'a HashMap<String, Stack>,
pub all_resources: &'a AllResourcesById,
}
pub async fn deploy_from_cache(
@@ -307,7 +306,6 @@ fn build_cache_for_deployment<'a>(
deployment_map,
stacks,
stack_map,
all_resources,
}: SyncDeployParams<'a>,
cache: &'a mut ToDeployCacheInner,
build_version_cache: &'a mut BuildVersionCache,
@@ -367,11 +365,8 @@ fn build_cache_for_deployment<'a>(
Deployment::validate_partial_config(&mut config);
let mut diff = Deployment::get_diff(
original.config.clone(),
config,
all_resources,
)?;
let mut diff =
Deployment::get_diff(original.config.clone(), config)?;
Deployment::validate_diff(&mut diff);
// Needs to only check config fields that affect docker run
@@ -486,7 +481,6 @@ fn build_cache_for_deployment<'a>(
deployment_map,
stacks,
stack_map,
all_resources,
},
cache,
build_version_cache,
@@ -502,7 +496,6 @@ fn build_cache_for_stack<'a>(
deployment_map,
stacks,
stack_map,
all_resources,
}: SyncDeployParams<'a>,
cache: &'a mut ToDeployCacheInner,
build_version_cache: &'a mut BuildVersionCache,
@@ -555,6 +548,7 @@ fn build_cache_for_stack<'a>(
// Here can diff the changes, to see if they merit a redeploy.
// See if any remote contents don't match deployed contents
#[allow(clippy::single_match)]
match (
&original.info.deployed_contents,
&original.info.remote_contents,
@@ -599,11 +593,8 @@ fn build_cache_for_stack<'a>(
Stack::validate_partial_config(&mut config);
let mut diff = Stack::get_diff(
original.config.clone(),
config,
all_resources,
)?;
let mut diff =
Stack::get_diff(original.config.clone(), config)?;
Stack::validate_diff(&mut diff);
// Needs to only check config fields that affect docker compose command
@@ -649,7 +640,6 @@ fn build_cache_for_stack<'a>(
deployment_map,
stacks,
stack_map,
all_resources,
},
cache,
build_version_cache,
@@ -666,7 +656,6 @@ async fn insert_target_using_after_list<'a>(
deployment_map,
stacks,
stack_map,
all_resources,
}: SyncDeployParams<'a>,
cache: &'a mut ToDeployCacheInner,
build_version_cache: &'a mut BuildVersionCache,
@@ -708,7 +697,6 @@ async fn insert_target_using_after_list<'a>(
deployment_map,
stacks,
stack_map,
all_resources,
},
cache,
build_version_cache,
@@ -755,7 +743,6 @@ async fn insert_target_using_after_list<'a>(
deployment_map,
stacks,
stack_map,
all_resources,
},
cache,
build_version_cache,
+35 -89
View File
@@ -2,22 +2,16 @@ use std::collections::HashMap;
use anyhow::Context;
use formatting::{Color, bold, colored, muted};
use komodo_client::{
api::write::{UpdateDescription, UpdateTagsOnResource},
entities::{
ResourceTargetVariant, tag::Tag, toml::ResourceToml, update::Log,
user::sync_user,
},
use komodo_client::entities::{
ResourceTargetVariant, tag::Tag, toml::ResourceToml, update::Log,
user::sync_user,
};
use mungos::find::find_collect;
use partial_derive2::MaybeNone;
use resolver_api::Resolve;
use crate::api::write::WriteArgs;
use crate::{api::write::WriteArgs, resource::ResourceMetaUpdate};
use super::{
AllResourcesById, ResourceSyncTrait, SyncDeltas, ToUpdateItem,
};
use super::{ResourceSyncTrait, SyncDeltas, ToUpdateItem};
/// Gets all the resources to update. For use in sync execution.
pub async fn get_updates_for_execution<
@@ -25,7 +19,6 @@ pub async fn get_updates_for_execution<
>(
resources: Vec<ResourceToml<Resource::PartialConfig>>,
delete: bool,
all_resources: &AllResourcesById,
match_resource_type: Option<ResourceTargetVariant>,
match_resources: Option<&[String]>,
id_to_tags: &HashMap<String, Tag>,
@@ -86,7 +79,6 @@ pub async fn get_updates_for_execution<
let mut diff = Resource::get_diff(
original.config.clone(),
resource.config,
all_resources,
)?;
Resource::validate_diff(&mut diff);
@@ -101,6 +93,7 @@ pub async fn get_updates_for_execution<
// or a change to tags / description
if diff.is_none()
&& resource.description == original.description
&& resource.template == original.template
&& resource.tags == original_tags
{
continue;
@@ -113,6 +106,7 @@ pub async fn get_updates_for_execution<
id: original.id.clone(),
update_description: resource.description
!= original.description,
update_template: resource.template != original.template,
update_tags: resource.tags != original_tags,
resource,
};
@@ -147,8 +141,6 @@ pub trait ExecuteResourceSync: ResourceSyncTrait {
for resource in to_create {
let name = resource.name.clone();
let tags = resource.tags.clone();
let description = resource.description.clone();
let id = match crate::resource::create::<Self>(
&resource.name,
resource.config,
@@ -168,18 +160,14 @@ pub trait ExecuteResourceSync: ResourceSyncTrait {
continue;
}
};
run_update_tags::<Self>(
run_update_meta::<Self>(
id.clone(),
&name,
tags,
&mut log,
&mut has_error,
)
.await;
run_update_description::<Self>(
id,
&name,
description,
ResourceMetaUpdate {
description: Some(resource.description),
template: Some(resource.template),
tags: Some(resource.tags),
},
&mut log,
&mut has_error,
)
@@ -197,30 +185,24 @@ pub trait ExecuteResourceSync: ResourceSyncTrait {
id,
resource,
update_description,
update_template,
update_tags,
} in to_update
{
// Update resource
let name = resource.name.clone();
let tags = resource.tags.clone();
let description = resource.description.clone();
if update_description {
run_update_description::<Self>(
let meta = ResourceMetaUpdate {
description: update_description
.then(|| resource.description.clone()),
template: update_template.then_some(resource.template),
tags: update_tags.then(|| resource.tags.clone()),
};
if !meta.is_none() {
run_update_meta::<Self>(
id.clone(),
&name,
description,
&mut log,
&mut has_error,
)
.await;
}
if update_tags {
run_update_tags::<Self>(
id.clone(),
&name,
tags,
meta,
&mut log,
&mut has_error,
)
@@ -290,21 +272,20 @@ pub trait ExecuteResourceSync: ResourceSyncTrait {
}
}
pub async fn run_update_tags<Resource: ResourceSyncTrait>(
pub async fn run_update_meta<Resource: ResourceSyncTrait>(
id: String,
name: &str,
tags: Vec<String>,
meta: ResourceMetaUpdate,
log: &mut String,
has_error: &mut bool,
) {
// Update tags
if let Err(e) = (UpdateTagsOnResource {
target: Resource::resource_target(id),
tags,
})
.resolve(&WriteArgs {
user: sync_user().to_owned(),
})
if let Err(e) = crate::resource::update_meta::<Resource>(
&id,
meta,
&WriteArgs {
user: sync_user().to_owned(),
},
)
.await
{
*has_error = true;
@@ -313,46 +294,11 @@ pub async fn run_update_tags<Resource: ResourceSyncTrait>(
colored("ERROR", Color::Red),
Resource::resource_type(),
bold(name),
e.error
e
))
} else {
log.push_str(&format!(
"\n{}: {} {} '{}' tags",
muted("INFO"),
colored("updated", Color::Blue),
Resource::resource_type(),
bold(name)
));
}
}
pub async fn run_update_description<Resource: ResourceSyncTrait>(
id: String,
name: &str,
description: String,
log: &mut String,
has_error: &mut bool,
) {
if let Err(e) = (UpdateDescription {
target: Resource::resource_target(id.clone()),
description,
})
.resolve(&WriteArgs {
user: sync_user().to_owned(),
})
.await
{
*has_error = true;
log.push_str(&format!(
"\n{}: failed to update description on {} '{}' | {:#}",
colored("ERROR", Color::Red),
Resource::resource_type(),
bold(name),
e.error
))
} else {
log.push_str(&format!(
"\n{}: {} {} '{}' description",
"\n{}: {} {} '{}' meta",
muted("INFO"),
colored("updated", Color::Blue),
Resource::resource_type(),
+1 -76
View File
@@ -3,16 +3,6 @@ use std::{collections::HashMap, str::FromStr};
use anyhow::anyhow;
use komodo_client::entities::{
ResourceTargetVariant,
action::Action,
alerter::Alerter,
build::Build,
builder::Builder,
deployment::Deployment,
procedure::Procedure,
repo::Repo,
server::Server,
stack::Stack,
sync::ResourceSync,
tag::Tag,
toml::{ResourceToml, ResourcesToml},
};
@@ -50,6 +40,7 @@ pub struct ToUpdateItem<T: Default> {
pub id: String,
pub resource: ResourceToml<T>,
pub update_description: bool,
pub update_template: bool,
pub update_tags: bool,
}
@@ -105,7 +96,6 @@ pub trait ResourceSyncTrait: ToToml + Sized {
fn get_diff(
original: Self::Config,
update: Self::PartialConfig,
resources: &AllResourcesById,
) -> anyhow::Result<Self::ConfigDiff>;
/// Apply any changes to computed config diff
@@ -155,71 +145,6 @@ pub fn include_resource_by_resource_type_and_name<
}
}
pub struct AllResourcesById {
pub servers: HashMap<String, Server>,
pub deployments: HashMap<String, Deployment>,
pub stacks: HashMap<String, Stack>,
pub builds: HashMap<String, Build>,
pub repos: HashMap<String, Repo>,
pub procedures: HashMap<String, Procedure>,
pub actions: HashMap<String, Action>,
pub builders: HashMap<String, Builder>,
pub alerters: HashMap<String, Alerter>,
pub syncs: HashMap<String, ResourceSync>,
}
impl AllResourcesById {
/// Use `match_tags` to filter resources by tag.
pub async fn load() -> anyhow::Result<Self> {
let map = HashMap::new();
let id_to_tags = &map;
let match_tags = &[];
Ok(Self {
servers: crate::resource::get_id_to_resource_map::<Server>(
id_to_tags, match_tags,
)
.await?,
deployments: crate::resource::get_id_to_resource_map::<
Deployment,
>(id_to_tags, match_tags)
.await?,
builds: crate::resource::get_id_to_resource_map::<Build>(
id_to_tags, match_tags,
)
.await?,
repos: crate::resource::get_id_to_resource_map::<Repo>(
id_to_tags, match_tags,
)
.await?,
procedures:
crate::resource::get_id_to_resource_map::<Procedure>(
id_to_tags, match_tags,
)
.await?,
actions: crate::resource::get_id_to_resource_map::<Action>(
id_to_tags, match_tags,
)
.await?,
builders: crate::resource::get_id_to_resource_map::<Builder>(
id_to_tags, match_tags,
)
.await?,
alerters: crate::resource::get_id_to_resource_map::<Alerter>(
id_to_tags, match_tags,
)
.await?,
syncs: crate::resource::get_id_to_resource_map::<ResourceSync>(
id_to_tags, match_tags,
)
.await?,
stacks: crate::resource::get_id_to_resource_map::<Stack>(
id_to_tags, match_tags,
)
.await?,
})
}
}
fn deserialize_resources_toml(
toml_str: &str,
) -> anyhow::Result<ResourcesToml> {
+85 -86
View File
@@ -1,7 +1,7 @@
use anyhow::{Context, anyhow};
use git::GitRes;
use anyhow::Context;
use komodo_client::entities::{
CloneArgs,
RepoExecutionArgs, RepoExecutionResponse,
repo::Repo,
sync::{ResourceSync, SyncFileContents},
to_path_compatible_name,
toml::ResourcesToml,
@@ -24,79 +24,49 @@ pub struct RemoteResources {
/// Use `match_tags` to filter resources by tag.
pub async fn get_remote_resources(
sync: &ResourceSync,
repo: Option<&Repo>,
) -> anyhow::Result<RemoteResources> {
if sync.config.files_on_host {
// =============
// FILES ON HOST
// =============
let root_path = core_config()
.sync_directory
.join(to_path_compatible_name(&sync.name));
let (mut logs, mut files, mut file_errors) =
(Vec::new(), Vec::new(), Vec::new());
let resources = super::file::read_resources(
&root_path,
&sync.config.resource_path,
&sync.config.match_tags,
&mut logs,
&mut files,
&mut file_errors,
);
return Ok(RemoteResources {
resources,
files,
file_errors,
logs,
hash: None,
message: None,
});
} else if sync.config.repo.is_empty() {
// ==========
// UI DEFINED
// ==========
let mut resources = ResourcesToml::default();
let resources = if !sync.config.file_contents.is_empty() {
super::deserialize_resources_toml(&sync.config.file_contents)
.context("failed to parse resource file contents")
.map(|more| {
extend_resources(
&mut resources,
more,
&sync.config.match_tags,
);
resources
})
} else {
Ok(resources)
};
return Ok(RemoteResources {
resources,
files: vec![SyncFileContents {
resource_path: String::new(),
path: "database file".to_string(),
contents: sync.config.file_contents.clone(),
}],
file_errors: vec![],
logs: vec![Log::simple(
"Read from database",
"Resources added from database file".to_string(),
)],
hash: None,
message: None,
});
get_files_on_host(sync).await
} else if let Some(repo) = repo {
get_repo(sync, repo.into()).await
} else if !sync.config.repo.is_empty() {
get_repo(sync, sync.into()).await
} else {
get_ui_defined(sync).await
}
}
// ===============
// REPO BASED SYNC
// ===============
if sync.config.repo.is_empty() {
return Err(anyhow!("No sync files configured"));
}
let mut clone_args: CloneArgs = sync.into();
async fn get_files_on_host(
sync: &ResourceSync,
) -> anyhow::Result<RemoteResources> {
let root_path = core_config()
.sync_directory
.join(to_path_compatible_name(&sync.name));
let (mut logs, mut files, mut file_errors) =
(Vec::new(), Vec::new(), Vec::new());
let resources = super::file::read_resources(
&root_path,
&sync.config.resource_path,
&sync.config.match_tags,
&mut logs,
&mut files,
&mut file_errors,
);
Ok(RemoteResources {
resources,
files,
file_errors,
logs,
hash: None,
message: None,
})
}
async fn get_repo(
sync: &ResourceSync,
mut clone_args: RepoExecutionArgs,
) -> anyhow::Result<RemoteResources> {
let access_token = if let Some(account) = &clone_args.account {
git_token(&clone_args.provider, account, |https| clone_args.https = https)
.await
@@ -110,23 +80,19 @@ pub async fn get_remote_resources(
let repo_path =
clone_args.unique_path(&core_config().repo_directory)?;
clone_args.destination = Some(repo_path.display().to_string());
// Don't want to run these on core.
clone_args.on_clone = None;
clone_args.on_pull = None;
let GitRes {
mut logs,
hash,
message,
..
} = git::pull_or_clone(
let (
RepoExecutionResponse {
mut logs,
commit_hash,
commit_message,
..
},
_,
) = git::pull_or_clone(
clone_args,
&core_config().repo_directory,
access_token,
&[],
"",
None,
&[],
)
.await
.with_context(|| {
@@ -152,7 +118,40 @@ pub async fn get_remote_resources(
files,
file_errors,
logs,
hash,
message,
hash: commit_hash,
message: commit_message,
})
}
async fn get_ui_defined(
sync: &ResourceSync,
) -> anyhow::Result<RemoteResources> {
let mut resources = ResourcesToml::default();
let resources =
super::deserialize_resources_toml(&sync.config.file_contents)
.context("failed to parse resource file contents")
.map(|more| {
extend_resources(
&mut resources,
more,
&sync.config.match_tags,
);
resources
});
Ok(RemoteResources {
resources,
files: vec![SyncFileContents {
resource_path: String::new(),
path: "database file".to_string(),
contents: sync.config.file_contents.clone(),
}],
file_errors: vec![],
logs: vec![Log::simple(
"Read from database",
"Resources added from database file".to_string(),
)],
hash: None,
message: None,
})
}
+55 -48
View File
@@ -24,16 +24,13 @@ use partial_derive2::{MaybeNone, PartialDiff};
use crate::{
api::write::WriteArgs,
resource::KomodoResource,
sync::{
ToUpdateItem,
execute::{run_update_description, run_update_tags},
},
resource::{KomodoResource, ResourceMetaUpdate},
state::all_resources_cache,
sync::{ToUpdateItem, execute::run_update_meta},
};
use super::{
AllResourcesById, ResourceSyncTrait, SyncDeltas,
execute::ExecuteResourceSync,
ResourceSyncTrait, SyncDeltas, execute::ExecuteResourceSync,
include_resource_by_resource_type_and_name,
include_resource_by_tags,
};
@@ -42,7 +39,6 @@ impl ResourceSyncTrait for Server {
fn get_diff(
original: Self::Config,
update: Self::PartialConfig,
_resources: &AllResourcesById,
) -> anyhow::Result<Self::ConfigDiff> {
Ok(original.partial_diff(update))
}
@@ -54,8 +50,8 @@ impl ResourceSyncTrait for Deployment {
fn get_diff(
mut original: Self::Config,
update: Self::PartialConfig,
resources: &AllResourcesById,
) -> anyhow::Result<Self::ConfigDiff> {
let resources = all_resources_cache().load();
// need to replace the server id with name
original.server_id = resources
.servers
@@ -87,14 +83,20 @@ impl ResourceSyncTrait for Stack {
fn get_diff(
mut original: Self::Config,
update: Self::PartialConfig,
resources: &AllResourcesById,
) -> anyhow::Result<Self::ConfigDiff> {
let resources = all_resources_cache().load();
// Need to replace server id with name
original.server_id = resources
.servers
.get(&original.server_id)
.map(|s| s.name.clone())
.unwrap_or_default();
// Replace linked repo with name
original.linked_repo = resources
.repos
.get(&original.linked_repo)
.map(|r| r.name.clone())
.unwrap_or_default();
Ok(original.partial_diff(update))
}
@@ -106,13 +108,18 @@ impl ResourceSyncTrait for Build {
fn get_diff(
mut original: Self::Config,
update: Self::PartialConfig,
resources: &AllResourcesById,
) -> anyhow::Result<Self::ConfigDiff> {
let resources = all_resources_cache().load();
original.builder_id = resources
.builders
.get(&original.builder_id)
.map(|b| b.name.clone())
.unwrap_or_default();
original.linked_repo = resources
.repos
.get(&original.linked_repo)
.map(|r| r.name.clone())
.unwrap_or_default();
Ok(original.partial_diff(update))
}
@@ -135,8 +142,8 @@ impl ResourceSyncTrait for Repo {
fn get_diff(
mut original: Self::Config,
update: Self::PartialConfig,
resources: &AllResourcesById,
) -> anyhow::Result<Self::ConfigDiff> {
let resources = all_resources_cache().load();
// Need to replace server id with name
original.server_id = resources
.servers
@@ -161,7 +168,6 @@ impl ResourceSyncTrait for Alerter {
fn get_diff(
original: Self::Config,
update: Self::PartialConfig,
_resources: &AllResourcesById,
) -> anyhow::Result<Self::ConfigDiff> {
Ok(original.partial_diff(update))
}
@@ -173,10 +179,10 @@ impl ResourceSyncTrait for Builder {
fn get_diff(
mut original: Self::Config,
update: Self::PartialConfig,
resources: &AllResourcesById,
) -> anyhow::Result<Self::ConfigDiff> {
// need to replace server builder id with name
if let BuilderConfig::Server(config) = &mut original {
let resources = all_resources_cache().load();
config.server_id = resources
.servers
.get(&config.server_id)
@@ -194,7 +200,6 @@ impl ResourceSyncTrait for Action {
fn get_diff(
original: Self::Config,
update: Self::PartialConfig,
_resources: &AllResourcesById,
) -> anyhow::Result<Self::ConfigDiff> {
Ok(original.partial_diff(update))
}
@@ -228,13 +233,14 @@ impl ResourceSyncTrait for ResourceSync {
if contents_empty
&& !config.files_on_host
&& config.repo.is_empty()
&& config.linked_repo.is_empty()
{
return false;
}
// The file contents MUST be empty
contents_empty &&
// The sync must be files on host mode OR git repo mode
(config.files_on_host || !config.repo.is_empty())
(config.files_on_host || !config.repo.is_empty() || !config.linked_repo.is_empty())
}
fn include_resource_partial(
@@ -267,20 +273,31 @@ impl ResourceSyncTrait for ResourceSync {
if contents_empty
&& !files_on_host
&& config.repo.as_ref().map(String::is_empty).unwrap_or(true)
&& config
.linked_repo
.as_ref()
.map(String::is_empty)
.unwrap_or(true)
{
return false;
}
// The file contents MUST be empty
contents_empty &&
// The sync must be files on host mode OR git repo mode
(files_on_host || !config.repo.as_deref().unwrap_or_default().is_empty())
(files_on_host || !config.repo.as_deref().unwrap_or_default().is_empty() || !config.linked_repo.as_deref().unwrap_or_default().is_empty())
}
fn get_diff(
original: Self::Config,
mut original: Self::Config,
update: Self::PartialConfig,
_resources: &AllResourcesById,
) -> anyhow::Result<Self::ConfigDiff> {
let resources = all_resources_cache().load();
original.linked_repo = resources
.repos
.get(&original.linked_repo)
.map(|r| r.name.clone())
.unwrap_or_default();
Ok(original.partial_diff(update))
}
}
@@ -291,8 +308,8 @@ impl ResourceSyncTrait for Procedure {
fn get_diff(
mut original: Self::Config,
update: Self::PartialConfig,
resources: &AllResourcesById,
) -> anyhow::Result<Self::ConfigDiff> {
let resources = all_resources_cache().load();
for stage in &mut original.stages {
for execution in &mut stage.executions {
match &mut execution.execution {
@@ -721,28 +738,24 @@ impl ExecuteResourceSync for Procedure {
id,
resource,
update_description,
update_template,
update_tags,
} in &to_update
{
// Update resource
let name = resource.name.clone();
let tags = resource.tags.clone();
let description = resource.description.clone();
if *update_description {
run_update_description::<Procedure>(
let meta = ResourceMetaUpdate {
description: update_description
.then(|| resource.description.clone()),
template: update_template.then(|| resource.template),
tags: update_tags.then(|| resource.tags.clone()),
};
if !meta.is_none() {
run_update_meta::<Procedure>(
id.clone(),
&name,
description,
&mut log,
&mut has_error,
)
.await;
}
if *update_tags {
run_update_tags::<Procedure>(
id.clone(),
&name,
tags,
meta,
&mut log,
&mut has_error,
)
@@ -784,8 +797,6 @@ impl ExecuteResourceSync for Procedure {
let mut to_pull = Vec::new();
for resource in &to_create {
let name = resource.name.clone();
let tags = resource.tags.clone();
let description = resource.description.clone();
let id = match crate::resource::create::<Procedure>(
&name,
resource.config.clone(),
@@ -807,18 +818,14 @@ impl ExecuteResourceSync for Procedure {
continue;
}
};
run_update_tags::<Procedure>(
run_update_meta::<Procedure>(
id.clone(),
&name,
tags,
&mut log,
&mut has_error,
)
.await;
run_update_description::<Procedure>(
id,
&name,
description,
ResourceMetaUpdate {
description: Some(resource.description.clone()),
template: Some(resource.template),
tags: Some(resource.tags.clone()),
},
&mut log,
&mut has_error,
)
+45 -37
View File
@@ -22,9 +22,7 @@ use komodo_client::{
};
use partial_derive2::{MaybeNone, PartialDiff};
use crate::resource::KomodoResource;
use super::AllResourcesById;
use crate::{resource::KomodoResource, state::all_resources_cache};
pub const TOML_PRETTY_OPTIONS: toml_pretty::Options =
toml_pretty::Options {
@@ -36,10 +34,7 @@ pub const TOML_PRETTY_OPTIONS: toml_pretty::Options =
pub trait ToToml: KomodoResource {
/// Replace linked ids (server_id, build_id, etc) with the resource name.
fn replace_ids(
_resource: &mut Resource<Self::Config, Self::Info>,
_all: &AllResourcesById,
) {
fn replace_ids(_resource: &mut Resource<Self::Config, Self::Info>) {
}
fn edit_config_object(
@@ -108,10 +103,9 @@ pub fn resource_push_to_toml<R: ToToml>(
deploy: bool,
after: Vec<String>,
toml: &mut String,
all: &AllResourcesById,
all_tags: &HashMap<String, Tag>,
) -> anyhow::Result<()> {
R::replace_ids(&mut resource, all);
R::replace_ids(&mut resource);
if !toml.is_empty() {
toml.push_str("\n\n##\n\n");
}
@@ -128,12 +122,11 @@ pub fn resource_to_toml<R: ToToml>(
resource: Resource<R::Config, R::Info>,
deploy: bool,
after: Vec<String>,
all: &AllResourcesById,
all_tags: &HashMap<String, Tag>,
) -> anyhow::Result<String> {
let mut toml = String::new();
resource_push_to_toml::<R>(
resource, deploy, after, &mut toml, all, all_tags,
resource, deploy, after, &mut toml, all_tags,
)?;
Ok(toml)
}
@@ -146,12 +139,13 @@ pub fn convert_resource<R: KomodoResource>(
) -> ResourceToml<R::PartialConfig> {
ResourceToml {
name: resource.name,
description: resource.description,
template: resource.template,
tags: resource
.tags
.iter()
.filter_map(|t| all_tags.get(t).map(|t| t.name.clone()))
.collect(),
description: resource.description,
deploy,
after,
// The config still needs to be minimized.
@@ -163,14 +157,24 @@ pub fn convert_resource<R: KomodoResource>(
// These have no linked resource ids to replace
impl ToToml for Alerter {}
impl ToToml for Server {}
impl ToToml for ResourceSync {}
impl ToToml for Action {}
impl ToToml for ResourceSync {
fn replace_ids(resource: &mut Resource<Self::Config, Self::Info>) {
let all = all_resources_cache().load();
resource.config.linked_repo.clone_from(
all
.repos
.get(&resource.config.linked_repo)
.map(|r| &r.name)
.unwrap_or(&String::new()),
);
}
}
impl ToToml for Stack {
fn replace_ids(
resource: &mut Resource<Self::Config, Self::Info>,
all: &AllResourcesById,
) {
fn replace_ids(resource: &mut Resource<Self::Config, Self::Info>) {
let all = all_resources_cache().load();
resource.config.server_id.clone_from(
all
.servers
@@ -178,6 +182,13 @@ impl ToToml for Stack {
.map(|s| &s.name)
.unwrap_or(&String::new()),
);
resource.config.linked_repo.clone_from(
all
.repos
.get(&resource.config.linked_repo)
.map(|r| &r.name)
.unwrap_or(&String::new()),
);
}
fn edit_config_object(
@@ -199,10 +210,8 @@ impl ToToml for Stack {
}
impl ToToml for Deployment {
fn replace_ids(
resource: &mut Resource<Self::Config, Self::Info>,
all: &AllResourcesById,
) {
fn replace_ids(resource: &mut Resource<Self::Config, Self::Info>) {
let all = all_resources_cache().load();
resource.config.server_id.clone_from(
all
.servers
@@ -263,10 +272,8 @@ impl ToToml for Deployment {
}
impl ToToml for Build {
fn replace_ids(
resource: &mut Resource<Self::Config, Self::Info>,
all: &AllResourcesById,
) {
fn replace_ids(resource: &mut Resource<Self::Config, Self::Info>) {
let all = all_resources_cache().load();
resource.config.builder_id.clone_from(
all
.builders
@@ -274,6 +281,13 @@ impl ToToml for Build {
.map(|s| &s.name)
.unwrap_or(&String::new()),
);
resource.config.linked_repo.clone_from(
all
.repos
.get(&resource.config.linked_repo)
.map(|r| &r.name)
.unwrap_or(&String::new()),
);
}
fn edit_config_object(
@@ -308,10 +322,8 @@ impl ToToml for Build {
}
impl ToToml for Repo {
fn replace_ids(
resource: &mut Resource<Self::Config, Self::Info>,
all: &AllResourcesById,
) {
fn replace_ids(resource: &mut Resource<Self::Config, Self::Info>) {
let all = all_resources_cache().load();
resource.config.server_id.clone_from(
all
.servers
@@ -349,11 +361,9 @@ impl ToToml for Repo {
}
impl ToToml for Builder {
fn replace_ids(
resource: &mut Resource<Self::Config, Self::Info>,
all: &AllResourcesById,
) {
fn replace_ids(resource: &mut Resource<Self::Config, Self::Info>) {
if let BuilderConfig::Server(config) = &mut resource.config {
let all = all_resources_cache().load();
config.server_id.clone_from(
all
.servers
@@ -382,10 +392,8 @@ impl ToToml for Builder {
}
impl ToToml for Procedure {
fn replace_ids(
resource: &mut Resource<Self::Config, Self::Info>,
all: &AllResourcesById,
) {
fn replace_ids(resource: &mut Resource<Self::Config, Self::Info>) {
let all = all_resources_cache().load();
for stage in &mut resource.config.stages {
for execution in &mut stage.executions {
match &mut execution.execution {
+27 -31
View File
@@ -34,10 +34,10 @@ use serde::Serialize;
use crate::{
api::{read::ReadArgs, write::WriteArgs},
helpers::matcher::Matcher,
state::db_client,
state::{all_resources_cache, db_client},
};
use super::{AllResourcesById, toml::TOML_PRETTY_OPTIONS};
use super::toml::TOML_PRETTY_OPTIONS;
/// Used to serialize user group
#[derive(Serialize)]
@@ -141,14 +141,12 @@ pub struct DeleteItem {
pub async fn get_updates_for_view(
user_groups: Vec<UserGroupToml>,
delete: bool,
all_resources: &AllResourcesById,
) -> anyhow::Result<Vec<DiffData>> {
let _curr = find_collect(&db_client().user_groups, None, None)
.await
.context("failed to query db for UserGroups")?;
let mut curr = Vec::with_capacity(_curr.capacity());
convert_user_groups(_curr.into_iter(), all_resources, &mut curr)
.await?;
convert_user_groups(_curr.into_iter(), &mut curr).await?;
let map = curr
.into_iter()
.map(|ug| (ug.1.name.clone(), ug))
@@ -175,17 +173,15 @@ pub async fn get_updates_for_view(
.permissions
.retain(|p| p.level > PermissionLevel::None);
user_group.permissions = expand_user_group_permissions(
user_group.permissions,
all_resources,
)
.await
.with_context(|| {
format!(
"failed to expand user group {} permissions",
user_group.name
)
})?;
user_group.permissions =
expand_user_group_permissions(user_group.permissions)
.await
.with_context(|| {
format!(
"failed to expand user group {} permissions",
user_group.name
)
})?;
let (_original_id, original) =
match map.get(&user_group.name).cloned() {
@@ -229,7 +225,6 @@ pub async fn get_updates_for_view(
pub async fn get_updates_for_execution(
user_groups: Vec<UserGroupToml>,
delete: bool,
all_resources: &AllResourcesById,
) -> anyhow::Result<(
Vec<UserGroupToml>,
Vec<UpdateItem>,
@@ -285,17 +280,15 @@ pub async fn get_updates_for_execution(
.permissions
.retain(|p| p.level > PermissionLevel::None);
user_group.permissions = expand_user_group_permissions(
user_group.permissions,
all_resources,
)
.await
.with_context(|| {
format!(
"Failed to expand user group {} permissions",
user_group.name
)
})?;
user_group.permissions =
expand_user_group_permissions(user_group.permissions)
.await
.with_context(|| {
format!(
"Failed to expand user group {} permissions",
user_group.name
)
})?;
let original = match map.get(&user_group.name).cloned() {
Some(original) => original,
@@ -313,6 +306,8 @@ pub async fn get_updates_for_execution(
})
.collect::<Vec<_>>();
let all_resources = all_resources_cache().load();
let mut original_permissions = (ListUserTargetPermissions {
user_target: UserTarget::UserGroup(original.id),
})
@@ -789,17 +784,17 @@ async fn run_update_permissions(
/// Expands any regex defined targets into the full list
async fn expand_user_group_permissions(
permissions: Vec<PermissionToml>,
all_resources: &AllResourcesById,
) -> anyhow::Result<Vec<PermissionToml>> {
let mut expanded =
Vec::<PermissionToml>::with_capacity(permissions.capacity());
let all_resources = all_resources_cache().load();
for permission in permissions {
let (variant, id) = permission.target.extract_variant_id();
if id.is_empty() {
continue;
}
let matcher = Matcher::new(&id)?;
let matcher = Matcher::new(id)?;
match variant {
ResourceTargetVariant::Build => {
let permissions = all_resources
@@ -1012,7 +1007,6 @@ fn specific_equal(
pub async fn convert_user_groups(
user_groups: impl Iterator<Item = UserGroup>,
all: &AllResourcesById,
res: &mut Vec<(String, UserGroupToml)>,
) -> anyhow::Result<()> {
let db = db_client();
@@ -1023,6 +1017,8 @@ pub async fn convert_user_groups(
.map(|user| (user.id, user.username))
.collect::<HashMap<_, _>>();
let all = all_resources_cache().load();
for mut user_group in user_groups {
user_group.all.retain(|_, p| {
p.level > PermissionLevel::None || !p.specific.is_empty()
+1 -5
View File
@@ -10,13 +10,12 @@ use komodo_client::entities::{
use mungos::find::find_collect;
use partial_derive2::MaybeNone;
use super::{AllResourcesById, ResourceSyncTrait};
use super::ResourceSyncTrait;
#[allow(clippy::too_many_arguments)]
pub async fn push_updates_for_view<Resource: ResourceSyncTrait>(
resources: Vec<ResourceToml<Resource::PartialConfig>>,
delete: bool,
all_resources: &AllResourcesById,
match_resource_type: Option<ResourceTargetVariant>,
match_resources: Option<&[String]>,
id_to_tags: &HashMap<String, Tag>,
@@ -68,7 +67,6 @@ pub async fn push_updates_for_view<Resource: ResourceSyncTrait>(
current_resource.clone(),
false,
vec![],
all_resources,
id_to_tags,
)?,
},
@@ -97,7 +95,6 @@ pub async fn push_updates_for_view<Resource: ResourceSyncTrait>(
let mut diff = Resource::get_diff(
current_resource.config.clone(),
proposed_resource.config,
all_resources,
)?;
Resource::validate_diff(&mut diff);
@@ -127,7 +124,6 @@ pub async fn push_updates_for_view<Resource: ResourceSyncTrait>(
current_resource.clone(),
proposed_resource.deploy,
proposed_resource.after,
all_resources,
id_to_tags,
)?,
proposed,
+2
View File
@@ -23,6 +23,8 @@ const ALLOWED_FILES: &[&str] = &[
"types.d.ts",
"responses.js",
"responses.d.ts",
"terminal.js",
"terminal.d.ts",
];
#[derive(Deserialize)]
+31 -9
View File
@@ -10,7 +10,10 @@ use komodo_client::{
},
};
use crate::{permission::get_check_permissions, resource::get};
use crate::{
permission::get_check_permissions, resource::get,
state::stack_status_cache,
};
#[instrument(name = "ConnectStackExec", skip(ws))]
pub async fn terminal(
@@ -58,16 +61,35 @@ pub async fn terminal(
}
};
let services = stack
.info
.deployed_services
.unwrap_or(stack.info.latest_services);
let Some(status) = stack_status_cache().get(&stack.id).await
else {
debug!("could not get stack status");
let _ = client_socket
.send(Message::text(String::from(
"ERROR: could not get stack status",
)))
.await;
let _ = client_socket.close().await;
return;
};
let container = match services
.into_iter()
.find(|s| s.service_name == service)
let container = match status
.curr
.services
.iter()
.find(|s| s.service == service)
.map(|s| s.container.as_ref())
{
Some(service) => service.container_name,
Some(Some(container)) => container.name.clone(),
Some(None) => {
let _ = client_socket
.send(Message::text(format!(
"ERROR: Service {service} container could not be found"
)))
.await;
let _ = client_socket.close().await;
return;
}
None => {
let _ = client_socket
.send(Message::text(format!(
+3 -1
View File
@@ -18,6 +18,8 @@ path = "src/main.rs"
komodo_client.workspace = true
periphery_client.workspace = true
environment_file.workspace = true
environment.workspace = true
interpolate.workspace = true
formatting.workspace = true
response.workspace = true
command.workspace = true
@@ -31,7 +33,6 @@ async_timing_util.workspace = true
derive_variants.workspace = true
resolver_api.workspace = true
run_command.workspace = true
svi.workspace = true
# external
pin-project-lite.workspace = true
tokio-stream.workspace = true
@@ -40,6 +41,7 @@ axum-server.workspace = true
serde_json.workspace = true
serde_yaml.workspace = true
tokio-util.workspace = true
arc-swap.workspace = true
futures.workspace = true
tracing.workspace = true
bollard.workspace = true
+75 -193
View File
@@ -1,23 +1,17 @@
use std::{
fmt::Write,
path::{Path, PathBuf},
};
use std::path::PathBuf;
use anyhow::{Context, anyhow};
use command::{
run_komodo_command, run_komodo_command_multiline,
run_komodo_command_with_interpolation,
run_komodo_command, run_komodo_command_with_sanitization,
};
use formatting::format_serror;
use komodo_client::{
entities::{
EnvironmentVar, Version,
build::{Build, BuildConfig},
environment_vars_from_str, get_image_name, optional_string,
to_docker_compatible_name, to_path_compatible_name,
update::Log,
},
parsers::QUOTE_PATTERN,
use interpolate::Interpolator;
use komodo_client::entities::{
EnvironmentVar, all_logs_success,
build::{Build, BuildConfig},
environment_vars_from_str, get_image_name, optional_string,
to_path_compatible_name,
update::Log,
};
use periphery_client::api::build::{
self, GetDockerfileContentsOnHost,
@@ -28,6 +22,9 @@ use resolver_api::Resolve;
use tokio::fs;
use crate::{
build::{
image_tags, parse_build_args, parse_secret_args, write_dockerfile,
},
config::periphery_config,
docker::docker_login,
helpers::{parse_extra_args, parse_labels},
@@ -123,18 +120,30 @@ impl Resolve<super::Args> for build::Build {
_: &super::Args,
) -> serror::Result<Vec<Log>> {
let build::Build {
build,
mut build,
repo: linked_repo,
registry_token,
additional_tags,
replacers: mut core_replacers,
mut replacers,
} = self;
let mut logs = Vec::new();
// Periphery side interpolation
let mut interpolator =
Interpolator::new(None, &periphery_config().secrets);
interpolator
.interpolate_build(&mut build)?
.push_logs(&mut logs);
replacers.extend(interpolator.secret_replacers);
let Build {
name,
config:
BuildConfig {
version,
image_tag,
skip_secret_interp,
build_path,
dockerfile_path,
build_args,
@@ -152,12 +161,14 @@ impl Resolve<super::Args> for build::Build {
..
} = &build;
if !*files_on_host && repo.is_empty() && dockerfile.is_empty() {
if !*files_on_host
&& repo.is_empty()
&& linked_repo.is_none()
&& dockerfile.is_empty()
{
return Err(anyhow!("Build must be files on host mode, have a repo attached, or have dockerfile contents set to build").into());
}
let mut logs = Vec::new();
// Maybe docker login
let should_push = match docker_login(
&image_registry.domain,
@@ -178,81 +189,59 @@ impl Resolve<super::Args> for build::Build {
}
};
let name = to_docker_compatible_name(name);
let build_path = if let Some(repo) = &linked_repo {
periphery_config()
.repo_dir()
.join(to_path_compatible_name(&repo.name))
.join(build_path)
} else {
periphery_config()
.build_dir()
.join(to_path_compatible_name(name))
.join(build_path)
}
.components()
.collect::<PathBuf>();
let build_path =
periphery_config().build_dir().join(&name).join(build_path);
let dockerfile_path = optional_string(dockerfile_path)
.unwrap_or("Dockerfile".to_owned());
// Write UI defined Dockerfile to host
if !*files_on_host && repo.is_empty() && !dockerfile.is_empty() {
let dockerfile = if *skip_secret_interp {
dockerfile.to_string()
} else {
let (dockerfile, replacers) = svi::interpolate_variables(
dockerfile,
&periphery_config().secrets,
svi::Interpolator::DoubleBrackets,
true,
).context("Failed to interpolate variables into UI defined dockerfile")?;
core_replacers.extend(replacers);
dockerfile
};
let full_path = build_path
.join(&dockerfile_path)
.components()
.collect::<PathBuf>();
// Ensure parent directory exists
if let Some(parent) = full_path.parent() {
if !parent.exists() {
tokio::fs::create_dir_all(parent)
.await
.with_context(|| format!("Failed to initialize dockerfile parent directory {parent:?}"))?;
}
if !*files_on_host
&& repo.is_empty()
&& linked_repo.is_none()
&& !dockerfile.is_empty()
{
write_dockerfile(
&build_path,
&dockerfile_path,
dockerfile,
&mut logs,
)
.await;
if !all_logs_success(&logs) {
return Ok(logs);
}
fs::write(&full_path, dockerfile).await.with_context(|| {
format!(
"Failed to write dockerfile contents to {full_path:?}"
)
})?;
logs.push(Log::simple(
"Write Dockerfile",
format!("Dockerfile contents written to {full_path:?}"),
));
};
// Pre Build
if !pre_build.is_none() {
let pre_build_path = build_path.join(&pre_build.path);
if let Some(log) = if !skip_secret_interp {
run_komodo_command_with_interpolation(
"Pre Build",
Some(pre_build_path.as_path()),
&pre_build.command,
true,
&periphery_config().secrets,
&core_replacers,
)
.await
} else {
run_komodo_command_multiline(
"Pre Build",
Some(pre_build_path.as_path()),
&pre_build.command,
)
.await
} {
if let Some(log) = run_komodo_command_with_sanitization(
"Pre Build",
pre_build_path.as_path(),
&pre_build.command,
true,
&replacers,
)
.await
{
let success = log.success;
logs.push(log);
if !success {
return Ok(logs);
}
};
}
}
// Get command parts
@@ -272,12 +261,8 @@ impl Resolve<super::Args> for build::Build {
let secret_args = environment_vars_from_str(secret_args)
.context("Invalid secret_args")?;
let command_secret_args = parse_secret_args(
&secret_args,
&build_path,
*skip_secret_interp,
)
.await?;
let command_secret_args =
parse_secret_args(&secret_args, &build_path).await?;
let labels = parse_labels(
&environment_vars_from_str(labels).context("Invalid labels")?,
@@ -293,125 +278,22 @@ impl Resolve<super::Args> for build::Build {
"docker{buildx} build{build_args}{command_secret_args}{extra_args}{labels}{image_tags}{maybe_push} -f {dockerfile_path} .",
);
if *skip_secret_interp {
let build_log = run_komodo_command(
"Docker Build",
build_path.as_ref(),
command,
)
.await;
logs.push(build_log);
} else if let Some(log) = run_komodo_command_with_interpolation(
if let Some(build_log) = run_komodo_command_with_sanitization(
"Docker Build",
build_path.as_ref(),
command,
false,
&periphery_config().secrets,
&core_replacers,
&replacers,
)
.await
{
logs.push(log)
}
logs.push(build_log);
};
Ok(logs)
}
}
fn image_tags(
image_name: &str,
custom_tag: &str,
version: &Version,
additional: &[String],
) -> String {
let Version { major, minor, .. } = version;
let custom_tag = if custom_tag.is_empty() {
String::new()
} else {
format!("-{custom_tag}")
};
let additional = additional
.iter()
.map(|tag| format!(" -t {image_name}:{tag}{custom_tag}"))
.collect::<Vec<_>>()
.join("");
format!(
" -t {image_name}:latest{custom_tag} -t {image_name}:{version}{custom_tag} -t {image_name}:{major}.{minor}{custom_tag} -t {image_name}:{major}{custom_tag}{additional}",
)
}
fn parse_build_args(build_args: &[EnvironmentVar]) -> String {
build_args
.iter()
.map(|p| {
if p.value.starts_with(QUOTE_PATTERN)
&& p.value.ends_with(QUOTE_PATTERN)
{
// If the value already wrapped in quotes, don't wrap it again
format!(" --build-arg {}={}", p.variable, p.value)
} else {
format!(" --build-arg {}=\"{}\"", p.variable, p.value)
}
})
.collect::<Vec<_>>()
.join("")
}
/// <https://docs.docker.com/build/building/secrets/#using-build-secrets>
async fn parse_secret_args(
secret_args: &[EnvironmentVar],
build_dir: &Path,
skip_secret_interp: bool,
) -> anyhow::Result<String> {
let periphery_config = periphery_config();
let mut res = String::new();
for EnvironmentVar { variable, value } in secret_args {
// Check edge cases
if variable.is_empty() {
return Err(anyhow!("secret variable cannot be empty string"));
} else if variable.contains('=') {
return Err(anyhow!(
"invalid variable {variable}. variable cannot contain '='"
));
}
// Interpolate in value
let value = if skip_secret_interp {
value.to_string()
} else {
svi::interpolate_variables(
value,
&periphery_config.secrets,
svi::Interpolator::DoubleBrackets,
true,
)
.context(
"Failed to interpolate periphery secrets into build secrets",
)?
.0
};
// Write the value to file to mount
let path = build_dir.join(variable);
tokio::fs::write(&path, value).await.with_context(|| {
format!(
"Failed to write build secret {variable} to {}",
path.display()
)
})?;
// Extend the command
write!(
&mut res,
" --secret id={variable},src={}",
path.display()
)
.with_context(|| {
format!(
"Failed to format build secret arguments for {variable}"
)
})?;
}
Ok(res)
}
//
impl Resolve<super::Args> for PruneBuilders {
+309 -72
View File
@@ -1,23 +1,34 @@
use std::{fmt::Write, path::PathBuf};
use anyhow::{Context, anyhow};
use command::run_komodo_command;
use command::{
run_komodo_command, run_komodo_command_with_sanitization,
};
use formatting::format_serror;
use git::{GitRes, write_commit_file};
use git::write_commit_file;
use interpolate::Interpolator;
use komodo_client::entities::{
FileContents, stack::ComposeProject, to_path_compatible_name,
FileContents, RepoExecutionResponse, all_logs_success,
stack::{
ComposeFile, ComposeProject, ComposeService,
ComposeServiceDeploy, StackServiceNames,
},
to_path_compatible_name,
update::Log,
};
use periphery_client::api::{compose::*, git::RepoActionResponse};
use periphery_client::api::compose::*;
use resolver_api::Resolve;
use serde::{Deserialize, Serialize};
use tokio::fs;
use crate::{
compose::{WriteStackRes, compose_up, docker_compose, write_stack},
compose::{
docker_compose, pull_or_clone_stack,
up::{maybe_login_registry, validate_files},
write::write_stack,
},
config::periphery_config,
docker::docker_login,
helpers::{log_grep, pull_or_clone_stack},
helpers::{log_grep, parse_extra_args},
};
impl Resolve<super::Args> for ListComposeProjects {
@@ -237,16 +248,18 @@ impl Resolve<super::Args> for WriteCommitComposeContents {
async fn resolve(
self,
_: &super::Args,
) -> serror::Result<RepoActionResponse> {
) -> serror::Result<RepoExecutionResponse> {
let WriteCommitComposeContents {
stack,
repo,
username,
file_path,
contents,
git_token,
} = self;
let root = pull_or_clone_stack(&stack, git_token).await?;
let root =
pull_or_clone_stack(&stack, repo.as_ref(), git_token).await?;
let file_path = stack
.config
@@ -261,37 +274,20 @@ impl Resolve<super::Args> for WriteCommitComposeContents {
"Write Compose File".to_string()
};
let GitRes {
logs,
hash,
message,
..
} = write_commit_file(
write_commit_file(
&msg,
&root,
&file_path,
&contents,
&stack.config.branch,
)
.await?;
Ok(RepoActionResponse {
logs,
commit_hash: hash,
commit_message: message,
env_file_path: None,
})
.await
.map_err(Into::into)
}
}
//
impl WriteStackRes for &mut ComposePullResponse {
fn logs(&mut self) -> &mut Vec<Log> {
&mut self.logs
}
}
impl Resolve<super::Args> for ComposePull {
#[instrument(
name = "ComposePull",
@@ -306,24 +302,42 @@ impl Resolve<super::Args> for ComposePull {
_: &super::Args,
) -> serror::Result<ComposePullResponse> {
let ComposePull {
stack,
mut stack,
repo,
services,
git_token,
registry_token,
mut replacers,
} = self;
let mut res = ComposePullResponse::default();
let (run_directory, env_file_path, _replacers) =
match write_stack(&stack, git_token, &mut res).await {
Ok(res) => res,
Err(e) => {
res.logs.push(Log::error(
"Write Stack",
format_serror(&e.into()),
));
return Ok(res);
}
};
let mut interpolator =
Interpolator::new(None, &periphery_config().secrets);
// Only interpolate Stack. Repo interpolation will be handled
// by the CloneRepo / PullOrCloneRepo call.
interpolator
.interpolate_stack(&mut stack)?
.push_logs(&mut res.logs);
replacers.extend(interpolator.secret_replacers);
let (run_directory, env_file_path) = match write_stack(
&stack,
repo.as_ref(),
git_token,
replacers.clone(),
&mut res,
)
.await
{
Ok(res) => res,
Err(e) => {
res
.logs
.push(Log::error("Write Stack", format_serror(&e.into())));
return Ok(res);
}
};
// Canonicalize the path to ensure it exists, and is the cleanest path to the run directory.
let run_directory = run_directory.canonicalize().context(
@@ -342,12 +356,18 @@ impl Resolve<super::Args> for ComposePull {
})
.collect::<Vec<_>>();
// Validate files
for (path, full_path) in &file_paths {
if !full_path.exists() {
return Err(anyhow!("Missing compose file at {path}").into());
}
}
maybe_login_registry(&stack, registry_token, &mut res.logs).await;
if !all_logs_success(&res.logs) {
return Ok(res);
}
let docker_compose = docker_compose();
let service_args = if services.is_empty() {
@@ -362,26 +382,6 @@ impl Resolve<super::Args> for ComposePull {
stack.config.file_paths.join(" -f ")
};
// Login to the registry to pull private images, if provider / account are set
if !stack.config.registry_provider.is_empty()
&& !stack.config.registry_account.is_empty()
{
docker_login(
&stack.config.registry_provider,
&stack.config.registry_account,
registry_token.as_deref(),
)
.await
.with_context(|| {
format!(
"domain: {} | account: {}",
stack.config.registry_provider,
stack.config.registry_account
)
})
.context("failed to login to image registry")?;
}
let env_file = env_file_path
.map(|path| format!(" --env-file {path}"))
.unwrap_or_default();
@@ -428,28 +428,265 @@ impl Resolve<super::Args> for ComposeUp {
_: &super::Args,
) -> serror::Result<ComposeUpResponse> {
let ComposeUp {
stack,
mut stack,
repo,
services,
git_token,
registry_token,
replacers,
mut replacers,
} = self;
let mut res = ComposeUpResponse::default();
if let Err(e) = compose_up(
stack,
services,
let mut interpolator =
Interpolator::new(None, &periphery_config().secrets);
// Only interpolate Stack. Repo interpolation will be handled
// by the CloneRepo / PullOrCloneRepo call.
interpolator
.interpolate_stack(&mut stack)?
.push_logs(&mut res.logs);
replacers.extend(interpolator.secret_replacers);
let (run_directory, env_file_path) = match write_stack(
&stack,
repo.as_ref(),
git_token,
registry_token,
replacers.clone(),
&mut res,
replacers,
)
.await
{
res.logs.push(Log::error(
"Compose Up - Failed",
format_serror(&e.into()),
));
Ok(res) => res,
Err(e) => {
res
.logs
.push(Log::error("Write Stack", format_serror(&e.into())));
return Ok(res);
}
};
// Canonicalize the path to ensure it exists, and is the cleanest path to the run directory.
let run_directory = run_directory.canonicalize().context(
"Failed to validate run directory on host after stack write (canonicalize error)",
)?;
validate_files(&stack, &run_directory, &mut res).await;
if !all_logs_success(&res.logs) {
return Ok(res);
}
maybe_login_registry(&stack, registry_token, &mut res.logs).await;
if !all_logs_success(&res.logs) {
return Ok(res);
}
// Pre deploy
if !stack.config.pre_deploy.is_none() {
let pre_deploy_path =
run_directory.join(&stack.config.pre_deploy.path);
if let Some(log) = run_komodo_command_with_sanitization(
"Pre Deploy",
pre_deploy_path.as_path(),
&stack.config.pre_deploy.command,
true,
&replacers,
)
.await
{
res.logs.push(log);
if !all_logs_success(&res.logs) {
return Ok(res);
}
};
}
let docker_compose = docker_compose();
let service_args = if services.is_empty() {
String::new()
} else {
format!(" {}", services.join(" "))
};
let file_args = if stack.config.file_paths.is_empty() {
String::from("compose.yaml")
} else {
stack.config.file_paths.join(" -f ")
};
// This will be the last project name, which is the one that needs to be destroyed.
// Might be different from the current project name, if user renames stack / changes to custom project name.
let last_project_name = stack.project_name(false);
let project_name = stack.project_name(true);
let env_file = env_file_path
.map(|path| format!(" --env-file {path}"))
.unwrap_or_default();
let additional_env_files = stack
.config
.additional_env_files
.iter()
.fold(String::new(), |mut output, file| {
let _ = write!(output, " --env-file {file}");
output
});
// Uses 'docker compose config' command to extract services (including image)
// after performing interpolation
{
let command = format!(
"{docker_compose} -p {project_name} -f {file_args}{additional_env_files}{env_file} config",
);
let Some(config_log) = run_komodo_command_with_sanitization(
"Compose Config",
run_directory.as_path(),
command,
false,
&replacers,
)
.await
else {
// Only reachable if command is empty,
// not the case since it is provided above.
unreachable!()
};
if !config_log.success {
res.logs.push(config_log);
return Ok(res);
}
let compose =
serde_yaml::from_str::<ComposeFile>(&config_log.stdout)
.context("Failed to parse compose contents")?;
// Record sanitized compose config output
res.compose_config = Some(config_log.stdout);
for (
service_name,
ComposeService {
container_name,
deploy,
image,
},
) in compose.services
{
let image = image.unwrap_or_default();
match deploy {
Some(ComposeServiceDeploy {
replicas: Some(replicas),
}) if replicas > 1 => {
for i in 1..1 + replicas {
res.services.push(StackServiceNames {
container_name: format!(
"{project_name}-{service_name}-{i}"
),
service_name: format!("{service_name}-{i}"),
image: image.clone(),
});
}
}
_ => {
res.services.push(StackServiceNames {
container_name: container_name.unwrap_or_else(|| {
format!("{project_name}-{service_name}")
}),
service_name,
image,
});
}
}
}
}
if stack.config.run_build {
let build_extra_args =
parse_extra_args(&stack.config.build_extra_args);
let command = format!(
"{docker_compose} -p {project_name} -f {file_args}{env_file}{additional_env_files} build{build_extra_args}{service_args}",
);
let Some(log) = run_komodo_command_with_sanitization(
"Compose Build",
run_directory.as_path(),
command,
false,
&replacers,
)
.await
else {
unreachable!()
};
res.logs.push(log);
if !all_logs_success(&res.logs) {
return Ok(res);
}
}
// Pull images before deploying
if stack.config.auto_pull {
// Pull images before destroying to minimize downtime.
// If this fails, do not continue.
let command = format!(
"{docker_compose} -p {project_name} -f {file_args}{env_file}{additional_env_files} pull{service_args}",
);
let log = run_komodo_command(
"Compose Pull",
run_directory.as_ref(),
command,
)
.await;
res.logs.push(log);
if !all_logs_success(&res.logs) {
return Ok(res);
}
}
if stack.config.destroy_before_deploy
// Also check if project name changed, which also requires taking down.
|| last_project_name != project_name
{
// Take down the existing containers.
// This one tries to use the previously deployed service name, to ensure the right stack is taken down.
crate::compose::down(&last_project_name, &services, &mut res)
.await
.context("failed to destroy existing containers")?;
}
// Run compose up
let extra_args = parse_extra_args(&stack.config.extra_args);
let command = format!(
"{docker_compose} -p {project_name} -f {file_args}{env_file}{additional_env_files} up -d{extra_args}{service_args}",
);
let Some(log) = run_komodo_command_with_sanitization(
"Compose Up",
run_directory.as_path(),
command,
false,
&replacers,
)
.await
else {
unreachable!()
};
res.deployed = log.success;
res.logs.push(log);
if res.deployed && !stack.config.post_deploy.is_none() {
let post_deploy_path =
run_directory.join(&stack.config.post_deploy.path);
if let Some(log) = run_komodo_command_with_sanitization(
"Post Deploy",
post_deploy_path.as_path(),
&stack.config.post_deploy.command,
true,
&replacers,
)
.await
{
res.logs.push(log);
};
}
Ok(res)
}
}
+28 -10
View File
@@ -1,16 +1,20 @@
use anyhow::{Context, anyhow};
use anyhow::Context;
use command::run_komodo_command;
use futures::future::join_all;
use komodo_client::entities::{
docker::container::{Container, ContainerListItem, ContainerStats},
to_docker_compatible_name,
docker::{
container::{Container, ContainerListItem, ContainerStats},
stats::FullContainerStats,
},
update::Log,
};
use periphery_client::api::container::*;
use resolver_api::Resolve;
use crate::{
docker::{container_stats, docker_client, stop_container_command},
docker::{
docker_client, stats::get_container_stats, stop_container_command,
},
helpers::log_grep,
};
@@ -81,22 +85,37 @@ impl Resolve<super::Args> for GetContainerStats {
self,
_: &super::Args,
) -> serror::Result<ContainerStats> {
let error = anyhow!("no stats matching {}", self.name);
let mut stats = container_stats(Some(self.name)).await?;
let stats = stats.pop().ok_or(error)?;
let mut stats = get_container_stats(Some(self.name)).await?;
let stats =
stats.pop().context("No stats found for container")?;
Ok(stats)
}
}
//
impl Resolve<super::Args> for GetFullContainerStats {
#[instrument(name = "GetFullContainerStats", level = "debug")]
async fn resolve(
self,
_: &super::Args,
) -> serror::Result<FullContainerStats> {
docker_client()
.full_container_stats(&self.name)
.await
.map_err(Into::into)
}
}
//
impl Resolve<super::Args> for GetContainerStatsList {
#[instrument(name = "GetContainerStatsList", level = "debug")]
async fn resolve(
self,
_: &super::Args,
) -> serror::Result<Vec<ContainerStats>> {
Ok(container_stats(None).await?)
Ok(get_container_stats(None).await?)
}
}
@@ -234,8 +253,7 @@ impl Resolve<super::Args> for RenameContainer {
curr_name,
new_name,
} = self;
let new = to_docker_compatible_name(&new_name);
let command = format!("docker rename {curr_name} {new}");
let command = format!("docker rename {curr_name} {new_name}");
Ok(run_komodo_command("Docker Rename", None, command).await)
}
}
+27 -26
View File
@@ -1,8 +1,7 @@
use anyhow::Context;
use command::{
run_komodo_command, run_komodo_command_with_interpolation,
};
use command::run_komodo_command_with_sanitization;
use formatting::format_serror;
use interpolate::Interpolator;
use komodo_client::{
entities::{
EnvironmentVar,
@@ -10,7 +9,7 @@ use komodo_client::{
Conversion, Deployment, DeploymentConfig, DeploymentImage,
RestartMode, conversions_from_str, extract_registry_domain,
},
environment_vars_from_str, to_docker_compatible_name,
environment_vars_from_str,
update::Log,
},
parsers::QUOTE_PATTERN,
@@ -36,12 +35,18 @@ impl Resolve<super::Args> for Deploy {
)]
async fn resolve(self, _: &super::Args) -> serror::Result<Log> {
let Deploy {
deployment,
mut deployment,
stop_signal,
stop_time,
registry_token,
replacers: core_replacers,
mut replacers,
} = self;
let mut interpolator =
Interpolator::new(None, &periphery_config().secrets);
interpolator.interpolate_deployment(&mut deployment)?;
replacers.extend(interpolator.secret_replacers);
let image = if let DeploymentImage::Image { image } =
&deployment.config.image
{
@@ -76,6 +81,7 @@ impl Resolve<super::Args> for Deploy {
let _ = pull_image(image).await;
debug!("image pulled");
let _ = (RemoveContainer {
name: deployment.name.clone(),
signal: stop_signal,
@@ -87,26 +93,22 @@ impl Resolve<super::Args> for Deploy {
let command = docker_run_command(&deployment, image)
.context("Unable to generate valid docker run command")?;
debug!("docker run command: {command}");
if deployment.config.skip_secret_interp {
Ok(run_komodo_command("Docker Run", None, command).await)
} else {
match run_komodo_command_with_interpolation(
"Docker Run",
None,
command,
false,
&periphery_config().secrets,
&core_replacers,
)
.await
{
Some(log) => Ok(log),
// The None case can not be reached, as the command is always non-empty
None => unreachable!(),
}
}
let Some(log) = run_komodo_command_with_sanitization(
"Docker Run",
None,
command,
false,
&replacers,
)
.await
else {
// The none case is only for empty command,
// this won't be the case given it is populated above.
unreachable!()
};
Ok(log)
}
}
@@ -129,7 +131,6 @@ fn docker_run_command(
}: &Deployment,
image: &str,
) -> anyhow::Result<String> {
let name = to_docker_compatible_name(name);
let ports = parse_conversions(
&conversions_from_str(ports).context("Invalid ports")?,
"-p",
+82 -138
View File
@@ -1,37 +1,38 @@
use anyhow::{Context, anyhow};
use axum::http::StatusCode;
use formatting::format_serror;
use git::GitRes;
use komodo_client::entities::{CloneArgs, LatestCommit, update::Log};
use komodo_client::entities::{
DefaultRepoFolder, LatestCommit, update::Log,
};
use periphery_client::api::git::{
CloneRepo, DeleteRepo, GetLatestCommit, PullOrCloneRepo, PullRepo,
RenameRepo, RepoActionResponse,
CloneRepo, DeleteRepo, GetLatestCommit,
PeripheryRepoExecutionResponse, PullOrCloneRepo, PullRepo,
RenameRepo,
};
use resolver_api::Resolve;
use serror::AddStatusCodeError;
use std::path::PathBuf;
use tokio::fs;
use crate::config::periphery_config;
use crate::{
config::periphery_config, git::handle_post_repo_execution,
};
impl Resolve<super::Args> for GetLatestCommit {
#[instrument(name = "GetLatestCommit", level = "debug")]
async fn resolve(
self,
_: &super::Args,
) -> serror::Result<LatestCommit> {
) -> serror::Result<Option<LatestCommit>> {
let repo_path = match self.path {
Some(p) => PathBuf::from(p),
None => periphery_config().repo_dir().join(self.name),
};
if !repo_path.is_dir() {
return Err(
anyhow!(
"Repo path {} is not directory. is it cloned?",
repo_path.display()
)
.into(),
);
// Make sure its a repo, or return null to avoid log spam
if !repo_path.is_dir() || !repo_path.join(".git").is_dir() {
return Ok(None);
}
Ok(git::get_commit_hash_info(&repo_path).await?)
Ok(Some(git::get_commit_hash_info(&repo_path).await?))
}
}
@@ -47,58 +48,33 @@ impl Resolve<super::Args> for CloneRepo {
async fn resolve(
self,
_: &super::Args,
) -> serror::Result<RepoActionResponse> {
) -> serror::Result<PeripheryRepoExecutionResponse> {
let CloneRepo {
args,
git_token,
environment,
env_file_path,
on_clone,
on_pull,
skip_secret_interp,
replacers,
} = self;
let CloneArgs {
provider, account, ..
} = &args;
let token = match (account, git_token) {
(None, _) => None,
(Some(_), Some(token)) => Some(token),
(Some(account), None) => Some(
crate::helpers::git_token(provider, account).map(ToString::to_string)
.with_context(
|| format!("Failed to get git token from periphery config | provider: {provider} | account: {account}")
)?,
),
};
let parent_dir = if args.is_build {
periphery_config().build_dir()
} else {
periphery_config().repo_dir()
};
git::clone(
args,
&parent_dir,
token,
&environment,
let token = crate::helpers::git_token(git_token, &args)?;
let root_repo_dir = default_folder(args.default_folder)?;
let res = git::clone(args, &root_repo_dir, token).await?;
handle_post_repo_execution(
res,
environment,
&env_file_path,
(!skip_secret_interp).then_some(&periphery_config().secrets),
&replacers,
on_clone,
on_pull,
skip_secret_interp,
replacers,
)
.await
.map(
|GitRes {
logs,
hash,
message,
env_file_path,
}| {
RepoActionResponse {
logs,
commit_hash: hash,
commit_message: message,
env_file_path,
}
},
)
.map_err(Into::into)
}
}
@@ -117,58 +93,32 @@ impl Resolve<super::Args> for PullRepo {
async fn resolve(
self,
_: &super::Args,
) -> serror::Result<RepoActionResponse> {
) -> serror::Result<PeripheryRepoExecutionResponse> {
let PullRepo {
args,
git_token,
environment,
env_file_path,
on_pull,
skip_secret_interp,
replacers,
} = self;
let CloneArgs {
provider, account, ..
} = &args;
let token = match (account, git_token) {
(None, _) => None,
(Some(_), Some(token)) => Some(token),
(Some(account), None) => Some(
crate::helpers::git_token(provider, account).map(ToString::to_string)
.with_context(
|| format!("Failed to get git token from periphery config | provider: {provider} | account: {account}")
)?,
),
};
let parent_dir = if args.is_build {
periphery_config().build_dir()
} else {
periphery_config().repo_dir()
};
git::pull(
args,
&parent_dir,
token,
&environment,
let token = crate::helpers::git_token(git_token, &args)?;
let parent_dir = default_folder(args.default_folder)?;
let res = git::pull(args, &parent_dir, token).await?;
handle_post_repo_execution(
res,
environment,
&env_file_path,
(!skip_secret_interp).then_some(&periphery_config().secrets),
&replacers,
None,
on_pull,
skip_secret_interp,
replacers,
)
.await
.map(
|GitRes {
logs,
hash,
message,
env_file_path,
}| {
RepoActionResponse {
logs,
commit_hash: hash,
commit_message: message,
env_file_path,
}
},
)
.map_err(Into::into)
}
}
@@ -187,58 +137,34 @@ impl Resolve<super::Args> for PullOrCloneRepo {
async fn resolve(
self,
_: &super::Args,
) -> serror::Result<RepoActionResponse> {
) -> serror::Result<PeripheryRepoExecutionResponse> {
let PullOrCloneRepo {
args,
git_token,
environment,
env_file_path,
on_clone,
on_pull,
skip_secret_interp,
replacers,
} = self;
let CloneArgs {
provider, account, ..
} = &args;
let token = match (account, git_token) {
(None, _) => None,
(Some(_), Some(token)) => Some(token),
(Some(account), None) => Some(
crate::helpers::git_token(provider, account).map(ToString::to_string)
.with_context(
|| format!("Failed to get git token from periphery config | provider: {provider} | account: {account}")
)?,
),
};
let parent_dir = if args.is_build {
periphery_config().build_dir()
} else {
periphery_config().repo_dir()
};
git::pull_or_clone(
args,
&parent_dir,
token,
&environment,
let token = crate::helpers::git_token(git_token, &args)?;
let parent_dir = default_folder(args.default_folder)?;
let (res, cloned) =
git::pull_or_clone(args, &parent_dir, token).await?;
handle_post_repo_execution(
res,
environment,
&env_file_path,
(!skip_secret_interp).then_some(&periphery_config().secrets),
&replacers,
cloned.then_some(on_clone).flatten(),
on_pull,
skip_secret_interp,
replacers,
)
.await
.map(
|GitRes {
logs,
hash,
message,
env_file_path,
}| {
RepoActionResponse {
logs,
commit_hash: hash,
commit_message: message,
env_file_path,
}
},
)
.map_err(Into::into)
}
}
@@ -291,3 +217,21 @@ impl Resolve<super::Args> for DeleteRepo {
Ok(log)
}
}
//
fn default_folder(
default_folder: DefaultRepoFolder,
) -> serror::Result<PathBuf> {
match default_folder {
DefaultRepoFolder::Stacks => Ok(periphery_config().stack_dir()),
DefaultRepoFolder::Builds => Ok(periphery_config().build_dir()),
DefaultRepoFolder::Repos => Ok(periphery_config().repo_dir()),
DefaultRepoFolder::NotApplicable => {
Err(
anyhow!("The clone args should not have a default_folder of NotApplicable using this method.")
.status_code(StatusCode::BAD_REQUEST)
)
}
}
}
+2
View File
@@ -63,6 +63,7 @@ pub enum PeripheryRequest {
// Repo (Write)
CloneRepo(CloneRepo),
PullRepo(PullRepo),
PullOrCloneRepo(PullOrCloneRepo),
RenameRepo(RenameRepo),
DeleteRepo(DeleteRepo),
@@ -91,6 +92,7 @@ pub enum PeripheryRequest {
GetContainerLogSearch(GetContainerLogSearch),
GetContainerStats(GetContainerStats),
GetContainerStatsList(GetContainerStatsList),
GetFullContainerStats(GetFullContainerStats),
// Container (Write)
Deploy(Deploy),
+4
View File
@@ -36,6 +36,10 @@ pub fn router() -> Router {
"/execute",
Router::new()
.route("/", post(super::terminal::execute_terminal))
.route(
"/container",
post(super::terminal::execute_container_exec),
)
.layer(middleware::from_fn(guard_request_by_passkey)),
),
)
+43 -100
View File
@@ -1,5 +1,3 @@
use std::{collections::HashMap, sync::OnceLock, task::Poll};
use anyhow::{Context, anyhow};
use axum::{
extract::{
@@ -10,16 +8,12 @@ use axum::{
response::Response,
};
use bytes::Bytes;
use futures::{SinkExt, Stream, StreamExt, TryStreamExt};
use futures::{SinkExt, StreamExt, TryStreamExt};
use komodo_client::{
api::write::TerminalRecreateMode,
entities::{
KOMODO_EXIT_CODE, NoData, komodo_timestamp, server::TerminalInfo,
},
entities::{KOMODO_EXIT_CODE, NoData, server::TerminalInfo},
};
use periphery_client::api::terminal::*;
use pin_project_lite::pin_project;
use rand::Rng;
use resolver_api::Resolve;
use serror::{AddStatusCodeError, Json};
use tokio_util::sync::CancellationToken;
@@ -81,53 +75,6 @@ impl Resolve<super::Args> for CreateTerminalAuthToken {
}
}
/// Tokens valid for 3 seconds
const TOKEN_VALID_FOR_MS: i64 = 3_000;
fn auth_tokens() -> &'static AuthTokens {
static AUTH_TOKENS: OnceLock<AuthTokens> = OnceLock::new();
AUTH_TOKENS.get_or_init(Default::default)
}
#[derive(Default)]
struct AuthTokens {
map: std::sync::Mutex<HashMap<String, i64>>,
}
impl AuthTokens {
pub fn create_auth_token(&self) -> String {
let mut lock = self.map.lock().unwrap();
// clear out any old tokens here (prevent unbounded growth)
let ts = komodo_timestamp();
lock.retain(|_, valid_until| *valid_until > ts);
let token: String = rand::rng()
.sample_iter(&rand::distr::Alphanumeric)
.take(30)
.map(char::from)
.collect();
lock.insert(token.clone(), ts + TOKEN_VALID_FOR_MS);
token
}
pub fn check_token(&self, token: String) -> serror::Result<()> {
let Some(valid_until) = self.map.lock().unwrap().remove(&token)
else {
return Err(
anyhow!("Terminal auth token not found")
.status_code(StatusCode::UNAUTHORIZED),
);
};
if komodo_timestamp() <= valid_until {
Ok(())
} else {
Err(
anyhow!("Terminal token is expired")
.status_code(StatusCode::UNAUTHORIZED),
)
}
}
}
pub async fn connect_terminal(
Query(query): Query<ConnectTerminalQuery>,
ws: WebSocketUpgrade,
@@ -336,10 +283,6 @@ async fn handle_terminal_websocket(
}))
}
/// Sentinels
const START_OF_OUTPUT: &str = "__KOMODO_START_OF_OUTPUT__";
const END_OF_OUTPUT: &str = "__KOMODO_END_OF_OUTPUT__";
pub async fn execute_terminal(
Json(ExecuteTerminalBody { terminal, command }): Json<
ExecuteTerminalBody,
@@ -352,7 +295,47 @@ pub async fn execute_terminal(
);
}
let terminal = get_terminal(&terminal).await?;
execute_command_on_terminal(&terminal, &command).await
}
pub async fn execute_container_exec(
Json(ExecuteContainerExecBody {
container,
shell,
command,
}): Json<ExecuteContainerExecBody>,
) -> serror::Result<axum::body::Body> {
if periphery_config().disable_container_exec {
return Err(
anyhow!("Container exec is disabled in the periphery config")
.into(),
);
}
if container.contains("&&") || shell.contains("&&") {
return Err(
anyhow!(
"The use of '&&' is forbidden in the container name or shell"
)
.into(),
);
}
// Create terminal (recreate if shell changed)
create_terminal(
container.clone(),
format!("docker exec -it {container} {shell}"),
TerminalRecreateMode::DifferentCommand,
)
.await
.context("Failed to create terminal for container exec")?;
execute_command_on_terminal(&container, &command).await
}
async fn execute_command_on_terminal(
terminal_name: &str,
command: &str,
) -> serror::Result<axum::body::Body> {
let terminal = get_terminal(terminal_name).await?;
// Read the bytes into lines
// This is done to check the lines for the EOF sentinal
@@ -399,43 +382,3 @@ pub async fn execute_terminal(
Ok(axum::body::Body::from_stream(TerminalStream { stdout }))
}
pin_project! {
struct TerminalStream<S> { #[pin] stdout: S }
}
impl<S> Stream for TerminalStream<S>
where
S:
Stream<Item = Result<String, tokio_util::codec::LinesCodecError>>,
{
// Axum expects a stream of results
type Item = Result<String, String>;
fn poll_next(
self: std::pin::Pin<&mut Self>,
cx: &mut std::task::Context<'_>,
) -> std::task::Poll<Option<Self::Item>> {
let this = self.project();
match this.stdout.poll_next(cx) {
Poll::Ready(None) => {
// This is if a None comes in before END_OF_OUTPUT.
// This probably means the terminal has exited early,
// and needs to be cleaned up
tokio::spawn(async move { clean_up_terminals().await });
Poll::Ready(None)
}
Poll::Ready(Some(line)) => {
match line {
Ok(line) if line.as_str() == END_OF_OUTPUT => {
// Stop the stream on end sentinel
Poll::Ready(None)
}
Ok(line) => Poll::Ready(Some(Ok(line + "\n"))),
Err(e) => Poll::Ready(Some(Err(format!("{e:?}")))),
}
}
Poll::Pending => Poll::Pending,
}
}
}
+132
View File
@@ -0,0 +1,132 @@
use std::{
fmt::Write,
path::{Path, PathBuf},
};
use anyhow::{Context, anyhow};
use formatting::format_serror;
use komodo_client::{
entities::{EnvironmentVar, Version, update::Log},
parsers::QUOTE_PATTERN,
};
pub async fn write_dockerfile(
build_path: &Path,
dockerfile_path: &str,
dockerfile: &str,
logs: &mut Vec<Log>,
) {
if let Err(e) = async {
if dockerfile.is_empty() {
return Err(anyhow!("UI Defined dockerfile is empty"));
}
let full_dockerfile_path = build_path
.join(dockerfile_path)
.components()
.collect::<PathBuf>();
// Ensure parent directory exists
if let Some(parent) = full_dockerfile_path.parent() {
if !parent.exists() {
tokio::fs::create_dir_all(parent)
.await
.with_context(|| format!("Failed to initialize dockerfile parent directory {parent:?}"))?;
}
}
tokio::fs::write(&full_dockerfile_path, dockerfile).await.with_context(|| {
format!(
"Failed to write dockerfile contents to {full_dockerfile_path:?}"
)
})?;
logs.push(Log::simple(
"Write Dockerfile",
format!(
"Dockerfile contents written to {full_dockerfile_path:?}"
),
));
anyhow::Ok(())
}.await {
logs.push(Log::error("Write Dockerfile", format_serror(&e.into())));
}
}
pub fn image_tags(
image_name: &str,
custom_tag: &str,
version: &Version,
additional: &[String],
) -> String {
let Version { major, minor, .. } = version;
let custom_tag = if custom_tag.is_empty() {
String::new()
} else {
format!("-{custom_tag}")
};
let additional = additional
.iter()
.map(|tag| format!(" -t {image_name}:{tag}{custom_tag}"))
.collect::<Vec<_>>()
.join("");
format!(
" -t {image_name}:latest{custom_tag} -t {image_name}:{version}{custom_tag} -t {image_name}:{major}.{minor}{custom_tag} -t {image_name}:{major}{custom_tag}{additional}",
)
}
pub fn parse_build_args(build_args: &[EnvironmentVar]) -> String {
build_args
.iter()
.map(|p| {
if p.value.starts_with(QUOTE_PATTERN)
&& p.value.ends_with(QUOTE_PATTERN)
{
// If the value already wrapped in quotes, don't wrap it again
format!(" --build-arg {}={}", p.variable, p.value)
} else {
format!(" --build-arg {}=\"{}\"", p.variable, p.value)
}
})
.collect::<Vec<_>>()
.join("")
}
/// <https://docs.docker.com/build/building/secrets/#using-build-secrets>
pub async fn parse_secret_args(
secret_args: &[EnvironmentVar],
build_dir: &Path,
) -> anyhow::Result<String> {
let mut res = String::new();
for EnvironmentVar { variable, value } in secret_args {
// Check edge cases
if variable.is_empty() {
return Err(anyhow!("secret variable cannot be empty string"));
} else if variable.contains('=') {
return Err(anyhow!(
"invalid variable {variable}. variable cannot contain '='"
));
}
// Write the value to file to mount
let path = build_dir.join(variable);
tokio::fs::write(&path, value).await.with_context(|| {
format!(
"Failed to write build secret {variable} to {}",
path.display()
)
})?;
// Extend the command
write!(
&mut res,
" --secret id={variable},src={}",
path.display()
)
.with_context(|| {
format!(
"Failed to format build secret arguments for {variable}"
)
})?;
}
Ok(res)
}
-711
View File
@@ -1,711 +0,0 @@
use std::{fmt::Write, path::PathBuf};
use anyhow::{Context, anyhow};
use command::{
run_komodo_command, run_komodo_command_multiline,
run_komodo_command_with_interpolation,
};
use formatting::format_serror;
use git::environment;
use komodo_client::entities::{
CloneArgs, FileContents, all_logs_success,
environment_vars_from_str,
stack::{
ComposeFile, ComposeService, ComposeServiceDeploy, Stack,
StackServiceNames,
},
to_path_compatible_name,
update::Log,
};
use periphery_client::api::{
compose::ComposeUpResponse,
git::{CloneRepo, PullOrCloneRepo, RepoActionResponse},
};
use resolver_api::Resolve;
use tokio::fs;
use crate::{
config::periphery_config, docker::docker_login,
helpers::parse_extra_args,
};
pub fn docker_compose() -> &'static str {
if periphery_config().legacy_compose_cli {
"docker-compose"
} else {
"docker compose"
}
}
/// If this fn returns Err, the caller of `compose_up` has to write result to the log before return.
pub async fn compose_up(
stack: Stack,
services: Vec<String>,
git_token: Option<String>,
registry_token: Option<String>,
res: &mut ComposeUpResponse,
core_replacers: Vec<(String, String)>,
) -> anyhow::Result<()> {
// Write the stack to local disk. For repos, will first delete any existing folder to ensure fresh deploy.
// Will also set additional fields on the reponse.
// Use the env_file_path in the compose command.
let (run_directory, env_file_path, periphery_replacers) =
write_stack(&stack, git_token, &mut *res)
.await
.context("Failed to write / clone compose file")?;
let replacers =
if let Some(periphery_replacers) = periphery_replacers {
core_replacers
.into_iter()
.chain(periphery_replacers)
.collect()
} else {
core_replacers
};
// Canonicalize the path to ensure it exists, and is the cleanest path to the run directory.
let run_directory = run_directory.canonicalize().context(
"Failed to validate run directory on host after stack write (canonicalize error)",
)?;
let file_paths = stack
.file_paths()
.iter()
.map(|path| {
(
path,
// This will remove any intermediate uneeded '/./' in the path
run_directory.join(path).components().collect::<PathBuf>(),
)
})
.collect::<Vec<_>>();
for (path, full_path) in &file_paths {
if !full_path.exists() {
res.missing_files.push(path.to_string());
}
}
if !res.missing_files.is_empty() {
return Err(anyhow!(
"A compose file doesn't exist after writing stack. Ensure the run_directory and file_paths are correct."
));
}
for (path, full_path) in &file_paths {
let file_contents = match fs::read_to_string(&full_path)
.await
.with_context(|| {
format!(
"failed to read compose file contents at {full_path:?}"
)
}) {
Ok(res) => res,
Err(e) => {
let error = format_serror(&e.into());
res
.logs
.push(Log::error("read compose file", error.clone()));
// This should only happen for repo stacks, ie remote error
res.remote_errors.push(FileContents {
path: path.to_string(),
contents: error,
});
return Err(anyhow!(
"failed to read compose file at {full_path:?}, stopping run"
));
}
};
res.file_contents.push(FileContents {
path: path.to_string(),
contents: file_contents,
});
}
let docker_compose = docker_compose();
let service_args = if services.is_empty() {
String::new()
} else {
format!(" {}", services.join(" "))
};
let file_args = if stack.config.file_paths.is_empty() {
String::from("compose.yaml")
} else {
stack.config.file_paths.join(" -f ")
};
// This will be the last project name, which is the one that needs to be destroyed.
// Might be different from the current project name, if user renames stack / changes to custom project name.
let last_project_name = stack.project_name(false);
let project_name = stack.project_name(true);
// Login to the registry to pull private images, if provider / account are set
if !stack.config.registry_provider.is_empty()
&& !stack.config.registry_account.is_empty()
{
docker_login(
&stack.config.registry_provider,
&stack.config.registry_account,
registry_token.as_deref(),
)
.await
.with_context(|| {
format!(
"domain: {} | account: {}",
stack.config.registry_provider, stack.config.registry_account
)
})
.context("failed to login to image registry")?;
}
let env_file = env_file_path
.map(|path| format!(" --env-file {path}"))
.unwrap_or_default();
let additional_env_files = stack
.config
.additional_env_files
.iter()
.fold(String::new(), |mut output, file| {
let _ = write!(output, " --env-file {file}");
output
});
// Uses 'docker compose config' command to extract services (including image)
// after performing interpolation
{
let command = format!(
"{docker_compose} -p {project_name} -f {file_args}{additional_env_files}{env_file} config",
);
let config_log = run_komodo_command(
"Compose Config",
run_directory.as_ref(),
command,
)
.await;
if !config_log.success {
res.logs.push(config_log);
return Err(anyhow!(
"Failed to validate compose files, stopping the run."
));
}
// Record sanitized compose config output
res.compose_config =
Some(svi::replace_in_string(&config_log.stdout, &replacers));
let compose =
serde_yaml::from_str::<ComposeFile>(&config_log.stdout)
.context("Failed to parse compose contents")?;
for (
service_name,
ComposeService {
container_name,
deploy,
image,
},
) in compose.services
{
let image = image.unwrap_or_default();
match deploy {
Some(ComposeServiceDeploy {
replicas: Some(replicas),
}) if replicas > 1 => {
for i in 1..1 + replicas {
res.services.push(StackServiceNames {
container_name: format!(
"{project_name}-{service_name}-{i}"
),
service_name: format!("{service_name}-{i}"),
image: image.clone(),
});
}
}
_ => {
res.services.push(StackServiceNames {
container_name: container_name.unwrap_or_else(|| {
format!("{project_name}-{service_name}")
}),
service_name,
image,
});
}
}
}
}
// Build images before deploying.
// If this fails, do not continue.
if stack.config.run_build {
let build_extra_args =
parse_extra_args(&stack.config.build_extra_args);
let command = format!(
"{docker_compose} -p {project_name} -f {file_args}{env_file}{additional_env_files} build{build_extra_args}{service_args}",
);
if stack.config.skip_secret_interp {
let log = run_komodo_command(
"Compose Build",
run_directory.as_ref(),
command,
)
.await;
res.logs.push(log);
} else if let Some(log) = run_komodo_command_with_interpolation(
"Compose Build",
run_directory.as_ref(),
command,
false,
&periphery_config().secrets,
&replacers,
)
.await
{
res.logs.push(log);
}
if !all_logs_success(&res.logs) {
return Err(anyhow!(
"Failed to build required images, stopping the run."
));
}
}
// Pull images before deploying
if stack.config.auto_pull {
// Pull images before destroying to minimize downtime.
// If this fails, do not continue.
let log = run_komodo_command(
"Compose Pull",
run_directory.as_ref(),
format!(
"{docker_compose} -p {project_name} -f {file_args}{env_file}{additional_env_files} pull{service_args}",
),
)
.await;
res.logs.push(log);
if !all_logs_success(&res.logs) {
return Err(anyhow!(
"Failed to pull required images, stopping the run."
));
}
}
// Pre deploy command
let pre_deploy_path =
run_directory.join(&stack.config.pre_deploy.path);
if let Some(log) = if stack.config.skip_secret_interp {
run_komodo_command_multiline(
"Pre Deploy",
pre_deploy_path.as_ref(),
&stack.config.pre_deploy.command,
)
.await
} else {
run_komodo_command_with_interpolation(
"Pre Deploy",
pre_deploy_path.as_ref(),
&stack.config.pre_deploy.command,
true,
&periphery_config().secrets,
&replacers,
)
.await
} {
res.logs.push(log);
}
if !all_logs_success(&res.logs) {
return Err(anyhow!(
"Failed at running pre_deploy command, stopping the run."
));
}
if stack.config.destroy_before_deploy
// Also check if project name changed, which also requires taking down.
|| last_project_name != project_name
{
// Take down the existing containers.
// This one tries to use the previously deployed service name, to ensure the right stack is taken down.
compose_down(&last_project_name, &services, res)
.await
.context("failed to destroy existing containers")?;
}
// Run compose up
let extra_args = parse_extra_args(&stack.config.extra_args);
let command = format!(
"{docker_compose} -p {project_name} -f {file_args}{env_file}{additional_env_files} up -d{extra_args}{service_args}",
);
let log = if stack.config.skip_secret_interp {
run_komodo_command("Compose Up", run_directory.as_ref(), command)
.await
} else {
match run_komodo_command_with_interpolation(
"Compose Up",
run_directory.as_ref(),
command,
false,
&periphery_config().secrets,
&replacers,
)
.await
{
Some(log) => log,
// The command is definitely non-empty, the result will never be None.
None => unreachable!(),
}
};
res.deployed = log.success;
// push the compose up command logs to keep the correct order
res.logs.push(log);
if res.deployed {
let post_deploy_path =
run_directory.join(&stack.config.post_deploy.path);
if let Some(log) = if stack.config.skip_secret_interp {
run_komodo_command_multiline(
"Post Deploy",
post_deploy_path.as_ref(),
&stack.config.post_deploy.command,
)
.await
} else {
run_komodo_command_with_interpolation(
"Post Deploy",
post_deploy_path.as_ref(),
&stack.config.post_deploy.command,
true,
&periphery_config().secrets,
&replacers,
)
.await
} {
res.logs.push(log)
}
if !all_logs_success(&res.logs) {
return Err(anyhow!(
"Failed at running post_deploy command, stopping the run."
));
}
}
Ok(())
}
pub trait WriteStackRes {
fn logs(&mut self) -> &mut Vec<Log>;
fn add_remote_error(&mut self, _contents: FileContents) {}
fn set_commit_hash(&mut self, _hash: Option<String>) {}
fn set_commit_message(&mut self, _message: Option<String>) {}
}
impl WriteStackRes for &mut ComposeUpResponse {
fn logs(&mut self) -> &mut Vec<Log> {
&mut self.logs
}
fn add_remote_error(&mut self, contents: FileContents) {
self.remote_errors.push(contents);
}
fn set_commit_hash(&mut self, hash: Option<String>) {
self.commit_hash = hash;
}
fn set_commit_message(&mut self, message: Option<String>) {
self.commit_message = message;
}
}
/// Either writes the stack file_contents to a file, or clones the repo.
/// Performs variable replacement on env and writes file.
/// Returns (run_directory, env_file_path, periphery_replacers)
pub async fn write_stack(
stack: &Stack,
git_token: Option<String>,
mut res: impl WriteStackRes,
) -> anyhow::Result<(
PathBuf,
Option<&str>,
Option<Vec<(String, String)>>,
)> {
let root = periphery_config()
.stack_dir()
.join(to_path_compatible_name(&stack.name));
let run_directory = root.join(&stack.config.run_directory);
// This will remove any intermediate '/./' in the path, which is a problem for some OS.
// Cannot use 'canonicalize' yet as directory may not exist.
let run_directory = run_directory.components().collect::<PathBuf>();
let (env_interpolated, env_replacers) =
if stack.config.skip_secret_interp {
(stack.config.environment.clone(), None)
} else {
let (environment, replacers) = svi::interpolate_variables(
&stack.config.environment,
&periphery_config().secrets,
svi::Interpolator::DoubleBrackets,
true,
)
.context(
"Failed to interpolate Periphery secrets into Environment",
)?;
(environment, Some(replacers))
};
match &env_replacers {
Some(replacers) if !replacers.is_empty() => {
res.logs().push(Log::simple(
"Interpolate - Environment (Periphery)",
replacers
.iter()
.map(|(_, variable)| format!("<span class=\"text-muted-foreground\">replaced:</span> {variable}"))
.collect::<Vec<_>>()
.join("\n"),
))
}
_ => {}
}
let env_vars = environment_vars_from_str(&env_interpolated)
.context("Invalid environment variables")?;
if stack.config.files_on_host {
// =============
// FILES ON HOST
// =============
let env_file_path = environment::write_file_simple(
&env_vars,
&stack.config.env_file_path,
run_directory.as_ref(),
res.logs(),
)
.await?;
Ok((
run_directory,
// Env file paths are expected to be already relative to run directory,
// so need to pass original env_file_path here.
env_file_path
.is_some()
.then_some(&stack.config.env_file_path),
env_replacers,
))
} else if stack.config.repo.is_empty() {
if stack.config.file_contents.trim().is_empty() {
return Err(anyhow!(
"Must either input compose file contents directly, or use files on host / git repo options."
));
}
// ==============
// UI BASED FILES
// ==============
// Ensure run directory exists
fs::create_dir_all(&run_directory).await.with_context(|| {
format!(
"failed to create stack run directory at {run_directory:?}"
)
})?;
let env_file_path = environment::write_file_simple(
&env_vars,
&stack.config.env_file_path,
run_directory.as_ref(),
res.logs(),
)
.await?;
let file_path = run_directory
.join(
stack
.config
.file_paths
// only need the first one, or default
.first()
.map(String::as_str)
.unwrap_or("compose.yaml"),
)
.components()
.collect::<PathBuf>();
let (file_contents, file_replacers) = if !stack
.config
.skip_secret_interp
{
let (contents, replacers) = svi::interpolate_variables(
&stack.config.file_contents,
&periphery_config().secrets,
svi::Interpolator::DoubleBrackets,
true,
)
.context("failed to interpolate secrets into file contents")?;
if !replacers.is_empty() {
res.logs().push(Log::simple(
"Interpolate - Compose file (Periphery)",
replacers
.iter()
.map(|(_, variable)| format!("<span class=\"text-muted-foreground\">replaced:</span> {variable}"))
.collect::<Vec<_>>()
.join("\n"),
));
}
(contents, Some(replacers))
} else {
(stack.config.file_contents.clone(), None)
};
fs::write(&file_path, &file_contents).await.with_context(
|| format!("Failed to write compose file to {file_path:?}"),
)?;
Ok((
run_directory,
env_file_path
.is_some()
.then_some(&stack.config.env_file_path),
match (env_replacers, file_replacers) {
(Some(env_replacers), Some(file_replacers)) => Some(
env_replacers.into_iter().chain(file_replacers).collect(),
),
(Some(env_replacers), None) => Some(env_replacers),
(None, Some(file_replacers)) => Some(file_replacers),
(None, None) => None,
},
))
} else {
// ================
// REPO BASED FILES
// ================
let mut args: CloneArgs = stack.into();
// Set the clone destination to the one created for this run
args.destination = Some(root.display().to_string());
let git_token = match git_token {
Some(token) => Some(token),
None => {
if !stack.config.git_account.is_empty() {
match crate::helpers::git_token(
&stack.config.git_provider,
&stack.config.git_account,
) {
Ok(token) => Some(token.to_string()),
Err(e) => {
let error = format_serror(&e.into());
res
.logs()
.push(Log::error("no git token", error.clone()));
res.add_remote_error(FileContents {
path: Default::default(),
contents: error,
});
return Err(anyhow!(
"failed to find required git token, stopping run"
));
}
}
} else {
None
}
}
};
let env_file_path = stack
.config
.run_directory
.parse::<PathBuf>()
.context("Invalid run_directory")?
.join(&stack.config.env_file_path)
.display()
.to_string();
let clone_or_pull_res = if stack.config.reclone {
CloneRepo {
args,
git_token,
environment: env_vars,
env_file_path,
// Env has already been interpolated above
skip_secret_interp: true,
replacers: Default::default(),
}
.resolve(&crate::api::Args)
.await
} else {
PullOrCloneRepo {
args,
git_token,
environment: env_vars,
env_file_path,
// Env has already been interpolated above
skip_secret_interp: true,
replacers: Default::default(),
}
.resolve(&crate::api::Args)
.await
};
let RepoActionResponse {
logs,
commit_hash,
commit_message,
env_file_path,
} = match clone_or_pull_res {
Ok(res) => res,
Err(e) => {
let error = format_serror(
&e.error.context("Failed to pull stack repo").into(),
);
res
.logs()
.push(Log::error("Pull Stack Repo", error.clone()));
res.add_remote_error(FileContents {
path: Default::default(),
contents: error,
});
return Err(anyhow!(
"Failed to pull stack repo, stopping run"
));
}
};
res.logs().extend(logs);
res.set_commit_hash(commit_hash);
res.set_commit_message(commit_message);
if !all_logs_success(res.logs()) {
return Err(anyhow!("Stopped after repo pull failure"));
}
Ok((
run_directory,
env_file_path
.is_some()
.then_some(&stack.config.env_file_path),
env_replacers,
))
}
}
async fn compose_down(
project: &str,
services: &[String],
res: &mut ComposeUpResponse,
) -> anyhow::Result<()> {
let docker_compose = docker_compose();
let service_args = if services.is_empty() {
String::new()
} else {
format!(" {}", services.join(" "))
};
let log = run_komodo_command(
"Compose Down",
None,
format!("{docker_compose} -p {project} down{service_args}"),
)
.await;
let success = log.success;
res.logs.push(log);
if !success {
return Err(anyhow!(
"Failed to bring down existing container(s) with docker compose down. Stopping run."
));
}
Ok(())
}
+114
View File
@@ -0,0 +1,114 @@
use std::path::PathBuf;
use anyhow::anyhow;
use command::run_komodo_command;
use komodo_client::entities::{
RepoExecutionArgs, repo::Repo, stack::Stack,
to_path_compatible_name,
};
use periphery_client::api::{
compose::ComposeUpResponse, git::PullOrCloneRepo,
};
use resolver_api::Resolve;
use crate::config::periphery_config;
pub mod up;
pub mod write;
pub fn docker_compose() -> &'static str {
if periphery_config().legacy_compose_cli {
"docker-compose"
} else {
"docker compose"
}
}
pub async fn down(
project: &str,
services: &[String],
res: &mut ComposeUpResponse,
) -> anyhow::Result<()> {
let docker_compose = docker_compose();
let service_args = if services.is_empty() {
String::new()
} else {
format!(" {}", services.join(" "))
};
let log = run_komodo_command(
"Compose Down",
None,
format!("{docker_compose} -p {project} down{service_args}"),
)
.await;
let success = log.success;
res.logs.push(log);
if !success {
return Err(anyhow!(
"Failed to bring down existing container(s) with docker compose down. Stopping run."
));
}
Ok(())
}
/// Only for git repo based Stacks.
/// Returns path to root directory of the stack repo.
///
/// Both Stack and Repo environment, on clone, on pull are ignored.
pub async fn pull_or_clone_stack(
stack: &Stack,
repo: Option<&Repo>,
git_token: Option<String>,
) -> anyhow::Result<PathBuf> {
if stack.config.files_on_host {
return Err(anyhow!(
"Wrong method called for files on host stack"
));
}
if repo.is_none() && stack.config.repo.is_empty() {
return Err(anyhow!("Repo is not configured"));
}
let (root, mut args) = if let Some(repo) = repo {
let root = periphery_config()
.repo_dir()
.join(to_path_compatible_name(&repo.name))
.join(&repo.config.path)
.components()
.collect::<PathBuf>();
let args: RepoExecutionArgs = repo.into();
(root, args)
} else {
let root = periphery_config()
.stack_dir()
.join(to_path_compatible_name(&stack.name))
.join(&stack.config.clone_path)
.components()
.collect::<PathBuf>();
let args: RepoExecutionArgs = stack.into();
(root, args)
};
args.destination = Some(root.display().to_string());
let git_token = crate::helpers::git_token(git_token, &args)?;
PullOrCloneRepo {
args,
git_token,
// All the extra pull functions
// (env, on clone, on pull)
// are disabled with this method.
environment: Default::default(),
env_file_path: Default::default(),
on_clone: Default::default(),
on_pull: Default::default(),
skip_secret_interp: Default::default(),
replacers: Default::default(),
}
.resolve(&crate::api::Args)
.await
.map_err(|e| e.error)?;
Ok(root)
}
+105
View File
@@ -0,0 +1,105 @@
use std::path::{Path, PathBuf};
use anyhow::{Context, anyhow};
use formatting::format_serror;
use komodo_client::entities::{
FileContents, stack::Stack, update::Log,
};
use periphery_client::api::compose::ComposeUpResponse;
use tokio::fs;
use crate::docker::docker_login;
pub async fn validate_files(
stack: &Stack,
run_directory: &Path,
res: &mut ComposeUpResponse,
) {
let file_paths = stack
.file_paths()
.iter()
.map(|path| {
(
path,
// This will remove any intermediate uneeded '/./' in the path
run_directory.join(path).components().collect::<PathBuf>(),
)
})
.collect::<Vec<_>>();
for (path, full_path) in &file_paths {
if !full_path.exists() {
res.missing_files.push(path.to_string());
}
}
if !res.missing_files.is_empty() {
res.logs.push(Log::error(
"Validate Files",
format_serror(
&anyhow!(
"Ensure the run_directory and file_paths are correct."
)
.context("A compose file doesn't exist after writing stack.")
.into(),
),
));
return;
}
for (path, full_path) in &file_paths {
let file_contents =
match fs::read_to_string(&full_path).await.with_context(|| {
format!(
"Failed to read compose file contents at {full_path:?}"
)
}) {
Ok(res) => res,
Err(e) => {
let error = format_serror(&e.into());
res
.logs
.push(Log::error("Read Compose File", error.clone()));
// This should only happen for repo stacks, ie remote error
res.remote_errors.push(FileContents {
path: path.to_string(),
contents: error,
});
return;
}
};
res.file_contents.push(FileContents {
path: path.to_string(),
contents: file_contents,
});
}
}
pub async fn maybe_login_registry(
stack: &Stack,
registry_token: Option<String>,
logs: &mut Vec<Log>,
) {
if !stack.config.registry_provider.is_empty()
&& !stack.config.registry_account.is_empty()
{
if let Err(e) = docker_login(
&stack.config.registry_provider,
&stack.config.registry_account,
registry_token.as_deref(),
)
.await
.with_context(|| {
format!(
"Domain: '{}' | Account: '{}'",
stack.config.registry_provider, stack.config.registry_account
)
})
.context("Failed to login to image registry")
{
logs.push(Log::error(
"Login to Registry",
format_serror(&e.into()),
));
}
}
}
+374
View File
@@ -0,0 +1,374 @@
use std::path::PathBuf;
use anyhow::{Context, anyhow};
use formatting::format_serror;
use komodo_client::entities::{
FileContents, RepoExecutionArgs, all_logs_success, repo::Repo,
stack::Stack, to_path_compatible_name, update::Log,
};
use periphery_client::api::{
compose::{ComposePullResponse, ComposeUpResponse},
git::{CloneRepo, PullOrCloneRepo},
};
use resolver_api::Resolve;
use tokio::fs;
use crate::{config::periphery_config, helpers};
pub trait WriteStackRes {
fn logs(&mut self) -> &mut Vec<Log>;
fn add_remote_error(&mut self, _contents: FileContents) {}
fn set_commit_hash(&mut self, _hash: Option<String>) {}
fn set_commit_message(&mut self, _message: Option<String>) {}
}
impl WriteStackRes for &mut ComposeUpResponse {
fn logs(&mut self) -> &mut Vec<Log> {
&mut self.logs
}
fn add_remote_error(&mut self, contents: FileContents) {
self.remote_errors.push(contents);
}
fn set_commit_hash(&mut self, hash: Option<String>) {
self.commit_hash = hash;
}
fn set_commit_message(&mut self, message: Option<String>) {
self.commit_message = message;
}
}
impl WriteStackRes for &mut ComposePullResponse {
fn logs(&mut self) -> &mut Vec<Log> {
&mut self.logs
}
}
/// Either writes the stack file_contents to a file, or clones the repo.
/// Asssumes all interpolation is already complete.
/// Returns (run_directory, env_file_path, periphery_replacers)
pub async fn write_stack<'a>(
stack: &'a Stack,
repo: Option<&Repo>,
git_token: Option<String>,
replacers: Vec<(String, String)>,
res: impl WriteStackRes,
) -> anyhow::Result<(
// run_directory
PathBuf,
// env_file_path
Option<&'a str>,
)> {
if stack.config.files_on_host {
write_stack_files_on_host(stack, res).await
} else if let Some(repo) = repo {
write_stack_linked_repo(stack, repo, git_token, replacers, res)
.await
} else if !stack.config.repo.is_empty() {
write_stack_inline_repo(stack, git_token, res).await
} else {
write_stack_ui_defined(stack, res).await
}
}
async fn write_stack_files_on_host(
stack: &Stack,
mut res: impl WriteStackRes,
) -> anyhow::Result<(
// run_directory
PathBuf,
// env_file_path
Option<&str>,
)> {
let run_directory = periphery_config()
.stack_dir()
.join(to_path_compatible_name(&stack.name))
.join(&stack.config.run_directory)
.components()
.collect::<PathBuf>();
let env_file_path = environment::write_env_file(
&stack.config.env_vars()?,
run_directory.as_path(),
&stack.config.env_file_path,
res.logs(),
)
.await;
if all_logs_success(res.logs()) {
Ok((
run_directory,
// Env file paths are expected to be already relative to run directory,
// so need to pass original env_file_path here.
env_file_path
.is_some()
.then_some(&stack.config.env_file_path),
))
} else {
Err(anyhow!("Failed to write env file, stopping run."))
}
}
async fn write_stack_linked_repo<'a>(
stack: &'a Stack,
repo: &Repo,
git_token: Option<String>,
replacers: Vec<(String, String)>,
mut res: impl WriteStackRes,
) -> anyhow::Result<(
// run_directory
PathBuf,
// env_file_path
Option<&'a str>,
)> {
let root = periphery_config()
.repo_dir()
.join(to_path_compatible_name(&repo.name))
.join(&repo.config.path)
.components()
.collect::<PathBuf>();
let mut args: RepoExecutionArgs = repo.into();
// Set the clone destination to the one created for this run
args.destination = Some(root.display().to_string());
let git_token = stack_git_token(git_token, &args, &mut res)?;
let env_file_path = root
.join(&repo.config.env_file_path)
.components()
.collect::<PathBuf>()
.display()
.to_string();
let on_clone = (!repo.config.on_clone.is_none())
.then_some(repo.config.on_clone.clone());
let on_pull = (!repo.config.on_pull.is_none())
.then_some(repo.config.on_pull.clone());
let clone_res = if stack.config.reclone {
CloneRepo {
args,
git_token,
environment: repo.config.env_vars()?,
env_file_path,
on_clone,
on_pull,
skip_secret_interp: repo.config.skip_secret_interp,
replacers,
}
.resolve(&crate::api::Args)
.await
.map_err(|e| e.error)?
} else {
PullOrCloneRepo {
args,
git_token,
environment: repo.config.env_vars()?,
env_file_path,
on_clone,
on_pull,
skip_secret_interp: repo.config.skip_secret_interp,
replacers,
}
.resolve(&crate::api::Args)
.await
.map_err(|e| e.error)?
};
res.logs().extend(clone_res.res.logs);
res.set_commit_hash(clone_res.res.commit_hash);
res.set_commit_message(clone_res.res.commit_message);
if !all_logs_success(res.logs()) {
return Ok((root, None));
}
let run_directory = root
.join(&stack.config.run_directory)
.components()
.collect::<PathBuf>();
let env_file_path = environment::write_env_file(
&stack.config.env_vars()?,
run_directory.as_path(),
&stack.config.env_file_path,
res.logs(),
)
.await;
if !all_logs_success(res.logs()) {
return Err(anyhow!("Failed to write env file, stopping run"));
}
Ok((
run_directory,
env_file_path
.is_some()
.then_some(&stack.config.env_file_path),
))
}
async fn write_stack_inline_repo(
stack: &Stack,
git_token: Option<String>,
mut res: impl WriteStackRes,
) -> anyhow::Result<(
// run_directory
PathBuf,
// env_file_path
Option<&str>,
)> {
let root = periphery_config()
.stack_dir()
.join(to_path_compatible_name(&stack.name))
.join(&stack.config.clone_path)
.components()
.collect::<PathBuf>();
let mut args: RepoExecutionArgs = stack.into();
// Set the clone destination to the one created for this run
args.destination = Some(root.display().to_string());
let git_token = stack_git_token(git_token, &args, &mut res)?;
let clone_res = if stack.config.reclone {
CloneRepo {
args,
git_token,
environment: Default::default(),
env_file_path: Default::default(),
on_clone: Default::default(),
on_pull: Default::default(),
skip_secret_interp: Default::default(),
replacers: Default::default(),
}
.resolve(&crate::api::Args)
.await
.map_err(|e| e.error)?
} else {
PullOrCloneRepo {
args,
git_token,
environment: Default::default(),
env_file_path: Default::default(),
on_clone: Default::default(),
on_pull: Default::default(),
skip_secret_interp: Default::default(),
replacers: Default::default(),
}
.resolve(&crate::api::Args)
.await
.map_err(|e| e.error)?
};
res.logs().extend(clone_res.res.logs);
res.set_commit_hash(clone_res.res.commit_hash);
res.set_commit_message(clone_res.res.commit_message);
if !all_logs_success(res.logs()) {
return Ok((root, None));
}
let run_directory = root
.join(&stack.config.run_directory)
.components()
.collect::<PathBuf>();
let env_file_path = environment::write_env_file(
&stack.config.env_vars()?,
run_directory.as_path(),
&stack.config.env_file_path,
res.logs(),
)
.await;
if !all_logs_success(res.logs()) {
return Err(anyhow!("Failed to write env file, stopping run"));
}
Ok((
run_directory,
env_file_path
.is_some()
.then_some(&stack.config.env_file_path),
))
}
async fn write_stack_ui_defined(
stack: &Stack,
mut res: impl WriteStackRes,
) -> anyhow::Result<(
// run_directory
PathBuf,
// env_file_path
Option<&str>,
)> {
if stack.config.file_contents.trim().is_empty() {
return Err(anyhow!(
"Must either input compose file contents directly, or use files on host / git repo options."
));
}
let run_directory = periphery_config()
.stack_dir()
.join(to_path_compatible_name(&stack.name))
.components()
.collect::<PathBuf>();
// Ensure run directory exists
fs::create_dir_all(&run_directory).await.with_context(|| {
format!(
"failed to create stack run directory at {run_directory:?}"
)
})?;
let env_file_path = environment::write_env_file(
&stack.config.env_vars()?,
run_directory.as_path(),
&stack.config.env_file_path,
res.logs(),
)
.await;
if !all_logs_success(res.logs()) {
return Err(anyhow!("Failed to write env file, stopping run"));
}
let file_path = run_directory
.join(
stack
.config
.file_paths
// only need the first one, or default
.first()
.map(String::as_str)
.unwrap_or("compose.yaml"),
)
.components()
.collect::<PathBuf>();
fs::write(&file_path, &stack.config.file_contents)
.await
.with_context(|| {
format!("Failed to write compose file to {file_path:?}")
})?;
Ok((
run_directory,
env_file_path
.is_some()
.then_some(&stack.config.env_file_path),
))
}
fn stack_git_token<R: WriteStackRes>(
core_token: Option<String>,
args: &RepoExecutionArgs,
res: &mut R,
) -> anyhow::Result<Option<String>> {
helpers::git_token(core_token, args).map_err(|e| {
let error = format_serror(&e.into());
res
.logs()
.push(Log::error("Missing git token", error.clone()));
res.add_remote_error(FileContents {
path: Default::default(),
contents: error,
});
anyhow!("failed to find required git token, stopping run")
})
}
+3
View File
@@ -51,6 +51,9 @@ pub fn periphery_config() -> &'static PeripheryConfig {
stats_polling_rate: env
.periphery_stats_polling_rate
.unwrap_or(config.stats_polling_rate),
container_stats_polling_rate: env
.periphery_container_stats_polling_rate
.unwrap_or(config.container_stats_polling_rate),
legacy_compose_cli: env
.periphery_legacy_compose_cli
.unwrap_or(config.legacy_compose_cli),
File diff suppressed because it is too large Load Diff
+759
View File
@@ -0,0 +1,759 @@
use std::collections::HashMap;
use anyhow::Context;
use bollard::query_parameters::{
InspectContainerOptions, ListContainersOptions,
};
use komodo_client::entities::docker::{
ContainerConfig, GraphDriverData, HealthConfig, PortBinding,
container::*,
};
use super::{DockerClient, stats::container_stats};
impl DockerClient {
pub async fn list_containers(
&self,
) -> anyhow::Result<Vec<ContainerListItem>> {
let containers = self
.docker
.list_containers(Some(ListContainersOptions {
all: true,
..Default::default()
}))
.await?;
let stats = container_stats().load();
let mut containers = containers
.into_iter()
.flat_map(|container| {
let name = container
.names
.context("no names on container")?
.pop()
.context("no names on container (empty vec)")?
.replace('/', "");
let stats = stats.get(&name).cloned();
anyhow::Ok(ContainerListItem {
server_id: None,
name,
stats,
id: container.id,
image: container.image,
image_id: container.image_id,
created: container.created,
size_rw: container.size_rw,
size_root_fs: container.size_root_fs,
state: convert_summary_container_state(
container.state.context("no container state")?,
),
status: container.status,
network_mode: container
.host_config
.and_then(|config| config.network_mode),
networks: container
.network_settings
.and_then(|settings| {
settings.networks.map(|networks| {
let mut keys =
networks.into_keys().collect::<Vec<_>>();
keys.sort();
keys
})
})
.unwrap_or_default(),
ports: container
.ports
.map(|ports| {
ports.into_iter().map(convert_port).collect()
})
.unwrap_or_default(),
volumes: container
.mounts
.map(|settings| {
settings
.into_iter()
.filter_map(|mount| mount.name)
.collect()
})
.unwrap_or_default(),
labels: container.labels.unwrap_or_default(),
})
})
.collect::<Vec<_>>();
let container_id_to_network = containers
.iter()
.filter_map(|c| Some((c.id.clone()?, c.network_mode.clone()?)))
.collect::<HashMap<_, _>>();
// Fix containers which use `container:container_id` network_mode,
// by replacing with the referenced network mode.
containers.iter_mut().for_each(|container| {
let Some(network_name) = &container.network_mode else {
return;
};
let Some(container_id) =
network_name.strip_prefix("container:")
else {
return;
};
container.network_mode =
container_id_to_network.get(container_id).cloned();
});
Ok(containers)
}
pub async fn inspect_container(
&self,
container_name: &str,
) -> anyhow::Result<Container> {
let container = self
.docker
.inspect_container(
container_name,
InspectContainerOptions { size: true }.into(),
)
.await?;
Ok(Container {
id: container.id,
created: container.created,
path: container.path,
args: container.args.unwrap_or_default(),
state: container.state.map(|state| ContainerState {
status: state
.status
.map(convert_container_state_status)
.unwrap_or_default(),
running: state.running,
paused: state.paused,
restarting: state.restarting,
oom_killed: state.oom_killed,
dead: state.dead,
pid: state.pid,
exit_code: state.exit_code,
error: state.error,
started_at: state.started_at,
finished_at: state.finished_at,
health: state.health.map(|health| ContainerHealth {
status: health
.status
.map(convert_health_status)
.unwrap_or_default(),
failing_streak: health.failing_streak,
log: health
.log
.map(|log| {
log
.into_iter()
.map(convert_health_check_result)
.collect()
})
.unwrap_or_default(),
}),
}),
image: container.image,
resolv_conf_path: container.resolv_conf_path,
hostname_path: container.hostname_path,
hosts_path: container.hosts_path,
log_path: container.log_path,
name: container.name,
restart_count: container.restart_count,
driver: container.driver,
platform: container.platform,
mount_label: container.mount_label,
process_label: container.process_label,
app_armor_profile: container.app_armor_profile,
exec_ids: container.exec_ids.unwrap_or_default(),
host_config: container.host_config.map(|config| HostConfig {
cpu_shares: config.cpu_shares,
memory: config.memory,
cgroup_parent: config.cgroup_parent,
blkio_weight: config.blkio_weight,
blkio_weight_device: config
.blkio_weight_device
.unwrap_or_default()
.into_iter()
.map(|device| ResourcesBlkioWeightDevice {
path: device.path,
weight: device.weight,
})
.collect(),
blkio_device_read_bps: config
.blkio_device_read_bps
.unwrap_or_default()
.into_iter()
.map(|bp| ThrottleDevice {
path: bp.path,
rate: bp.rate,
})
.collect(),
blkio_device_write_bps: config
.blkio_device_write_bps
.unwrap_or_default()
.into_iter()
.map(|bp| ThrottleDevice {
path: bp.path,
rate: bp.rate,
})
.collect(),
blkio_device_read_iops: config
.blkio_device_read_iops
.unwrap_or_default()
.into_iter()
.map(|iops| ThrottleDevice {
path: iops.path,
rate: iops.rate,
})
.collect(),
blkio_device_write_iops: config
.blkio_device_write_iops
.unwrap_or_default()
.into_iter()
.map(|iops| ThrottleDevice {
path: iops.path,
rate: iops.rate,
})
.collect(),
cpu_period: config.cpu_period,
cpu_quota: config.cpu_quota,
cpu_realtime_period: config.cpu_realtime_period,
cpu_realtime_runtime: config.cpu_realtime_runtime,
cpuset_cpus: config.cpuset_cpus,
cpuset_mems: config.cpuset_mems,
devices: config
.devices
.unwrap_or_default()
.into_iter()
.map(|device| DeviceMapping {
path_on_host: device.path_on_host,
path_in_container: device.path_in_container,
cgroup_permissions: device.cgroup_permissions,
})
.collect(),
device_cgroup_rules: config
.device_cgroup_rules
.unwrap_or_default(),
device_requests: config
.device_requests
.unwrap_or_default()
.into_iter()
.map(|request| DeviceRequest {
driver: request.driver,
count: request.count,
device_ids: request.device_ids.unwrap_or_default(),
capabilities: request.capabilities.unwrap_or_default(),
options: request.options.unwrap_or_default(),
})
.collect(),
kernel_memory_tcp: config.kernel_memory_tcp,
memory_reservation: config.memory_reservation,
memory_swap: config.memory_swap,
memory_swappiness: config.memory_swappiness,
nano_cpus: config.nano_cpus,
oom_kill_disable: config.oom_kill_disable,
init: config.init,
pids_limit: config.pids_limit,
ulimits: config
.ulimits
.unwrap_or_default()
.into_iter()
.map(|ulimit| ResourcesUlimits {
name: ulimit.name,
soft: ulimit.soft,
hard: ulimit.hard,
})
.collect(),
cpu_count: config.cpu_count,
cpu_percent: config.cpu_percent,
io_maximum_iops: config.io_maximum_iops,
io_maximum_bandwidth: config.io_maximum_bandwidth,
binds: config.binds.unwrap_or_default(),
container_id_file: config.container_id_file,
log_config: config.log_config.map(|config| {
HostConfigLogConfig {
typ: config.typ,
config: config.config.unwrap_or_default(),
}
}),
network_mode: config.network_mode,
port_bindings: config
.port_bindings
.unwrap_or_default()
.into_iter()
.map(|(k, v)| {
(
k,
v.unwrap_or_default()
.into_iter()
.map(|v| PortBinding {
host_ip: v.host_ip,
host_port: v.host_port,
})
.collect(),
)
})
.collect(),
restart_policy: config.restart_policy.map(|policy| {
RestartPolicy {
name: policy
.name
.map(convert_restart_policy)
.unwrap_or_default(),
maximum_retry_count: policy.maximum_retry_count,
}
}),
auto_remove: config.auto_remove,
volume_driver: config.volume_driver,
volumes_from: config.volumes_from.unwrap_or_default(),
mounts: config
.mounts
.unwrap_or_default()
.into_iter()
.map(|mount| ContainerMount {
target: mount.target,
source: mount.source,
typ: mount
.typ
.map(convert_mount_type)
.unwrap_or_default(),
read_only: mount.read_only,
consistency: mount.consistency,
bind_options: mount.bind_options.map(|options| {
MountBindOptions {
propagation: options
.propagation
.map(convert_mount_propogation)
.unwrap_or_default(),
non_recursive: options.non_recursive,
create_mountpoint: options.create_mountpoint,
read_only_non_recursive: options
.read_only_non_recursive,
read_only_force_recursive: options
.read_only_force_recursive,
}
}),
volume_options: mount.volume_options.map(|options| {
MountVolumeOptions {
no_copy: options.no_copy,
labels: options.labels.unwrap_or_default(),
driver_config: options.driver_config.map(|config| {
MountVolumeOptionsDriverConfig {
name: config.name,
options: config.options.unwrap_or_default(),
}
}),
subpath: options.subpath,
}
}),
tmpfs_options: mount.tmpfs_options.map(|options| {
MountTmpfsOptions {
size_bytes: options.size_bytes,
mode: options.mode,
}
}),
})
.collect(),
console_size: config
.console_size
.map(|v| v.into_iter().map(|s| s as i32).collect())
.unwrap_or_default(),
annotations: config.annotations.unwrap_or_default(),
cap_add: config.cap_add.unwrap_or_default(),
cap_drop: config.cap_drop.unwrap_or_default(),
cgroupns_mode: config
.cgroupns_mode
.map(convert_cgroupns_mode),
dns: config.dns.unwrap_or_default(),
dns_options: config.dns_options.unwrap_or_default(),
dns_search: config.dns_search.unwrap_or_default(),
extra_hosts: config.extra_hosts.unwrap_or_default(),
group_add: config.group_add.unwrap_or_default(),
ipc_mode: config.ipc_mode,
cgroup: config.cgroup,
links: config.links.unwrap_or_default(),
oom_score_adj: config.oom_score_adj,
pid_mode: config.pid_mode,
privileged: config.privileged,
publish_all_ports: config.publish_all_ports,
readonly_rootfs: config.readonly_rootfs,
security_opt: config.security_opt.unwrap_or_default(),
storage_opt: config.storage_opt.unwrap_or_default(),
tmpfs: config.tmpfs.unwrap_or_default(),
uts_mode: config.uts_mode,
userns_mode: config.userns_mode,
shm_size: config.shm_size,
sysctls: config.sysctls.unwrap_or_default(),
runtime: config.runtime,
isolation: config
.isolation
.map(convert_isolation_mode)
.unwrap_or_default(),
masked_paths: config.masked_paths.unwrap_or_default(),
readonly_paths: config.readonly_paths.unwrap_or_default(),
}),
graph_driver: container.graph_driver.map(|driver| {
GraphDriverData {
name: driver.name,
data: driver.data,
}
}),
size_rw: container.size_rw,
size_root_fs: container.size_root_fs,
mounts: container
.mounts
.unwrap_or_default()
.into_iter()
.map(|mount| MountPoint {
typ: mount
.typ
.map(convert_mount_point_type)
.unwrap_or_default(),
name: mount.name,
source: mount.source,
destination: mount.destination,
driver: mount.driver,
mode: mount.mode,
rw: mount.rw,
propagation: mount.propagation,
})
.collect(),
config: container.config.map(|config| ContainerConfig {
hostname: config.hostname,
domainname: config.domainname,
user: config.user,
attach_stdin: config.attach_stdin,
attach_stdout: config.attach_stdout,
attach_stderr: config.attach_stderr,
exposed_ports: config
.exposed_ports
.unwrap_or_default()
.into_keys()
.map(|k| (k, Default::default()))
.collect(),
tty: config.tty,
open_stdin: config.open_stdin,
stdin_once: config.stdin_once,
env: config.env.unwrap_or_default(),
cmd: config.cmd.unwrap_or_default(),
healthcheck: config.healthcheck.map(|health| HealthConfig {
test: health.test.unwrap_or_default(),
interval: health.interval,
timeout: health.timeout,
retries: health.retries,
start_period: health.start_period,
start_interval: health.start_interval,
}),
args_escaped: config.args_escaped,
image: config.image,
volumes: config
.volumes
.unwrap_or_default()
.into_keys()
.map(|k| (k, Default::default()))
.collect(),
working_dir: config.working_dir,
entrypoint: config.entrypoint.unwrap_or_default(),
network_disabled: config.network_disabled,
mac_address: config.mac_address,
on_build: config.on_build.unwrap_or_default(),
labels: config.labels.unwrap_or_default(),
stop_signal: config.stop_signal,
stop_timeout: config.stop_timeout,
shell: config.shell.unwrap_or_default(),
}),
network_settings: container.network_settings.map(|settings| {
NetworkSettings {
bridge: settings.bridge,
sandbox_id: settings.sandbox_id,
ports: settings
.ports
.unwrap_or_default()
.into_iter()
.map(|(k, v)| {
(
k,
v.unwrap_or_default()
.into_iter()
.map(|v| PortBinding {
host_ip: v.host_ip,
host_port: v.host_port,
})
.collect(),
)
})
.collect(),
sandbox_key: settings.sandbox_key,
networks: settings
.networks
.unwrap_or_default()
.into_iter()
.map(|(k, v)| {
(
k,
EndpointSettings {
ipam_config: v.ipam_config.map(|ipam| {
EndpointIpamConfig {
ipv4_address: ipam.ipv4_address,
ipv6_address: ipam.ipv6_address,
link_local_ips: ipam
.link_local_ips
.unwrap_or_default(),
}
}),
links: v.links.unwrap_or_default(),
mac_address: v.mac_address,
aliases: v.aliases.unwrap_or_default(),
network_id: v.network_id,
endpoint_id: v.endpoint_id,
gateway: v.gateway,
ip_address: v.ip_address,
ip_prefix_len: v.ip_prefix_len,
ipv6_gateway: v.ipv6_gateway,
global_ipv6_address: v.global_ipv6_address,
global_ipv6_prefix_len: v.global_ipv6_prefix_len,
driver_opts: v.driver_opts.unwrap_or_default(),
dns_names: v.dns_names.unwrap_or_default(),
},
)
})
.collect(),
}
}),
})
}
}
fn convert_summary_container_state(
state: bollard::secret::ContainerSummaryStateEnum,
) -> ContainerStateStatusEnum {
match state {
bollard::secret::ContainerSummaryStateEnum::EMPTY => {
ContainerStateStatusEnum::Empty
}
bollard::secret::ContainerSummaryStateEnum::CREATED => {
ContainerStateStatusEnum::Created
}
bollard::secret::ContainerSummaryStateEnum::RUNNING => {
ContainerStateStatusEnum::Running
}
bollard::secret::ContainerSummaryStateEnum::PAUSED => {
ContainerStateStatusEnum::Paused
}
bollard::secret::ContainerSummaryStateEnum::RESTARTING => {
ContainerStateStatusEnum::Restarting
}
bollard::secret::ContainerSummaryStateEnum::EXITED => {
ContainerStateStatusEnum::Exited
}
bollard::secret::ContainerSummaryStateEnum::REMOVING => {
ContainerStateStatusEnum::Removing
}
bollard::secret::ContainerSummaryStateEnum::DEAD => {
ContainerStateStatusEnum::Dead
}
}
}
fn convert_container_state_status(
state: bollard::secret::ContainerStateStatusEnum,
) -> ContainerStateStatusEnum {
match state {
bollard::secret::ContainerStateStatusEnum::EMPTY => {
ContainerStateStatusEnum::Empty
}
bollard::secret::ContainerStateStatusEnum::CREATED => {
ContainerStateStatusEnum::Created
}
bollard::secret::ContainerStateStatusEnum::RUNNING => {
ContainerStateStatusEnum::Running
}
bollard::secret::ContainerStateStatusEnum::PAUSED => {
ContainerStateStatusEnum::Paused
}
bollard::secret::ContainerStateStatusEnum::RESTARTING => {
ContainerStateStatusEnum::Restarting
}
bollard::secret::ContainerStateStatusEnum::EXITED => {
ContainerStateStatusEnum::Exited
}
bollard::secret::ContainerStateStatusEnum::REMOVING => {
ContainerStateStatusEnum::Removing
}
bollard::secret::ContainerStateStatusEnum::DEAD => {
ContainerStateStatusEnum::Dead
}
}
}
fn convert_port_type(
typ: bollard::secret::PortTypeEnum,
) -> PortTypeEnum {
match typ {
bollard::secret::PortTypeEnum::EMPTY => PortTypeEnum::EMPTY,
bollard::secret::PortTypeEnum::TCP => PortTypeEnum::TCP,
bollard::secret::PortTypeEnum::UDP => PortTypeEnum::UDP,
bollard::secret::PortTypeEnum::SCTP => PortTypeEnum::SCTP,
}
}
fn convert_port(port: bollard::secret::Port) -> Port {
Port {
ip: port.ip,
private_port: port.private_port,
public_port: port.public_port,
typ: port.typ.map(convert_port_type).unwrap_or_default(),
}
}
fn convert_health_status(
status: bollard::secret::HealthStatusEnum,
) -> HealthStatusEnum {
match status {
bollard::secret::HealthStatusEnum::EMPTY => {
HealthStatusEnum::Empty
}
bollard::secret::HealthStatusEnum::NONE => HealthStatusEnum::None,
bollard::secret::HealthStatusEnum::STARTING => {
HealthStatusEnum::Starting
}
bollard::secret::HealthStatusEnum::HEALTHY => {
HealthStatusEnum::Healthy
}
bollard::secret::HealthStatusEnum::UNHEALTHY => {
HealthStatusEnum::Unhealthy
}
}
}
fn convert_health_check_result(
check: bollard::secret::HealthcheckResult,
) -> HealthcheckResult {
HealthcheckResult {
start: check.start,
end: check.end,
exit_code: check.exit_code,
output: check.output,
}
}
fn convert_restart_policy(
policy: bollard::secret::RestartPolicyNameEnum,
) -> RestartPolicyNameEnum {
match policy {
bollard::secret::RestartPolicyNameEnum::EMPTY => {
RestartPolicyNameEnum::Empty
}
bollard::secret::RestartPolicyNameEnum::NO => {
RestartPolicyNameEnum::No
}
bollard::secret::RestartPolicyNameEnum::ALWAYS => {
RestartPolicyNameEnum::Always
}
bollard::secret::RestartPolicyNameEnum::UNLESS_STOPPED => {
RestartPolicyNameEnum::UnlessStopped
}
bollard::secret::RestartPolicyNameEnum::ON_FAILURE => {
RestartPolicyNameEnum::OnFailure
}
}
}
fn convert_mount_type(
typ: bollard::secret::MountTypeEnum,
) -> MountTypeEnum {
match typ {
bollard::secret::MountTypeEnum::EMPTY => MountTypeEnum::Empty,
bollard::secret::MountTypeEnum::BIND => MountTypeEnum::Bind,
bollard::secret::MountTypeEnum::VOLUME => MountTypeEnum::Volume,
bollard::secret::MountTypeEnum::IMAGE => MountTypeEnum::Image,
bollard::secret::MountTypeEnum::TMPFS => MountTypeEnum::Tmpfs,
bollard::secret::MountTypeEnum::NPIPE => MountTypeEnum::Npipe,
bollard::secret::MountTypeEnum::CLUSTER => MountTypeEnum::Cluster,
}
}
fn convert_mount_point_type(
typ: bollard::secret::MountPointTypeEnum,
) -> MountTypeEnum {
match typ {
bollard::secret::MountPointTypeEnum::EMPTY => {
MountTypeEnum::Empty
}
bollard::secret::MountPointTypeEnum::BIND => MountTypeEnum::Bind,
bollard::secret::MountPointTypeEnum::VOLUME => {
MountTypeEnum::Volume
}
bollard::secret::MountPointTypeEnum::IMAGE => {
MountTypeEnum::Image
}
bollard::secret::MountPointTypeEnum::TMPFS => {
MountTypeEnum::Tmpfs
}
bollard::secret::MountPointTypeEnum::NPIPE => {
MountTypeEnum::Npipe
}
bollard::secret::MountPointTypeEnum::CLUSTER => {
MountTypeEnum::Cluster
}
}
}
fn convert_mount_propogation(
propogation: bollard::secret::MountBindOptionsPropagationEnum,
) -> MountBindOptionsPropagationEnum {
match propogation {
bollard::secret::MountBindOptionsPropagationEnum::EMPTY => {
MountBindOptionsPropagationEnum::Empty
}
bollard::secret::MountBindOptionsPropagationEnum::PRIVATE => {
MountBindOptionsPropagationEnum::Private
}
bollard::secret::MountBindOptionsPropagationEnum::RPRIVATE => {
MountBindOptionsPropagationEnum::Rprivate
}
bollard::secret::MountBindOptionsPropagationEnum::SHARED => {
MountBindOptionsPropagationEnum::Shared
}
bollard::secret::MountBindOptionsPropagationEnum::RSHARED => {
MountBindOptionsPropagationEnum::Rshared
}
bollard::secret::MountBindOptionsPropagationEnum::SLAVE => {
MountBindOptionsPropagationEnum::Slave
}
bollard::secret::MountBindOptionsPropagationEnum::RSLAVE => {
MountBindOptionsPropagationEnum::Rslave
}
}
}
fn convert_cgroupns_mode(
mode: bollard::secret::HostConfigCgroupnsModeEnum,
) -> HostConfigCgroupnsModeEnum {
match mode {
bollard::secret::HostConfigCgroupnsModeEnum::EMPTY => {
HostConfigCgroupnsModeEnum::Empty
}
bollard::secret::HostConfigCgroupnsModeEnum::PRIVATE => {
HostConfigCgroupnsModeEnum::Private
}
bollard::secret::HostConfigCgroupnsModeEnum::HOST => {
HostConfigCgroupnsModeEnum::Host
}
}
}
fn convert_isolation_mode(
isolation: bollard::secret::HostConfigIsolationEnum,
) -> HostConfigIsolationEnum {
match isolation {
bollard::secret::HostConfigIsolationEnum::EMPTY => {
HostConfigIsolationEnum::Empty
}
bollard::secret::HostConfigIsolationEnum::DEFAULT => {
HostConfigIsolationEnum::Default
}
bollard::secret::HostConfigIsolationEnum::PROCESS => {
HostConfigIsolationEnum::Process
}
bollard::secret::HostConfigIsolationEnum::HYPERV => {
HostConfigIsolationEnum::Hyperv
}
}
}
+143
View File
@@ -0,0 +1,143 @@
use bollard::query_parameters::ListImagesOptions;
use komodo_client::entities::docker::{
ContainerConfig, GraphDriverData, HealthConfig,
container::ContainerListItem, image::*,
};
use super::DockerClient;
impl DockerClient {
pub async fn list_images(
&self,
containers: &[ContainerListItem],
) -> anyhow::Result<Vec<ImageListItem>> {
let images = self
.docker
.list_images(Option::<ListImagesOptions>::None)
.await?
.into_iter()
.map(|image| {
let in_use = containers.iter().any(|container| {
container
.image_id
.as_ref()
.map(|id| id == &image.id)
.unwrap_or_default()
});
ImageListItem {
name: image
.repo_tags
.into_iter()
.next()
.unwrap_or_else(|| image.id.clone()),
id: image.id,
parent_id: image.parent_id,
created: image.created,
size: image.size,
in_use,
}
})
.collect();
Ok(images)
}
pub async fn inspect_image(
&self,
image_name: &str,
) -> anyhow::Result<Image> {
let image = self.docker.inspect_image(image_name).await?;
Ok(Image {
id: image.id,
repo_tags: image.repo_tags.unwrap_or_default(),
repo_digests: image.repo_digests.unwrap_or_default(),
parent: image.parent,
comment: image.comment,
created: image.created,
docker_version: image.docker_version,
author: image.author,
architecture: image.architecture,
variant: image.variant,
os: image.os,
os_version: image.os_version,
size: image.size,
graph_driver: image.graph_driver.map(|driver| {
GraphDriverData {
name: driver.name,
data: driver.data,
}
}),
root_fs: image.root_fs.map(|fs| ImageInspectRootFs {
typ: fs.typ,
layers: fs.layers.unwrap_or_default(),
}),
metadata: image.metadata.map(|metadata| ImageInspectMetadata {
last_tag_time: metadata.last_tag_time,
}),
config: image.config.map(|config| ContainerConfig {
hostname: config.hostname,
domainname: config.domainname,
user: config.user,
attach_stdin: config.attach_stdin,
attach_stdout: config.attach_stdout,
attach_stderr: config.attach_stderr,
exposed_ports: config
.exposed_ports
.unwrap_or_default()
.into_keys()
.map(|k| (k, Default::default()))
.collect(),
tty: config.tty,
open_stdin: config.open_stdin,
stdin_once: config.stdin_once,
env: config.env.unwrap_or_default(),
cmd: config.cmd.unwrap_or_default(),
healthcheck: config.healthcheck.map(|health| HealthConfig {
test: health.test.unwrap_or_default(),
interval: health.interval,
timeout: health.timeout,
retries: health.retries,
start_period: health.start_period,
start_interval: health.start_interval,
}),
args_escaped: config.args_escaped,
image: config.image,
volumes: config
.volumes
.unwrap_or_default()
.into_keys()
.map(|k| (k, Default::default()))
.collect(),
working_dir: config.working_dir,
entrypoint: config.entrypoint.unwrap_or_default(),
network_disabled: config.network_disabled,
mac_address: config.mac_address,
on_build: config.on_build.unwrap_or_default(),
labels: config.labels.unwrap_or_default(),
stop_signal: config.stop_signal,
stop_timeout: config.stop_timeout,
shell: config.shell.unwrap_or_default(),
}),
})
}
pub async fn image_history(
&self,
image_name: &str,
) -> anyhow::Result<Vec<ImageHistoryResponseItem>> {
let res = self
.docker
.image_history(image_name)
.await?
.into_iter()
.map(|image| ImageHistoryResponseItem {
id: image.id,
created: image.created,
created_by: image.created_by,
tags: image.tags,
size: image.size,
comment: image.comment,
})
.collect();
Ok(res)
}
}
+89
View File
@@ -0,0 +1,89 @@
use std::sync::OnceLock;
use anyhow::anyhow;
use bollard::Docker;
use command::run_komodo_command;
use komodo_client::entities::{TerminationSignal, update::Log};
use run_command::async_run_command;
pub mod stats;
mod containers;
mod images;
mod networks;
mod volumes;
pub fn docker_client() -> &'static DockerClient {
static DOCKER_CLIENT: OnceLock<DockerClient> = OnceLock::new();
DOCKER_CLIENT.get_or_init(Default::default)
}
pub struct DockerClient {
docker: Docker,
}
impl Default for DockerClient {
fn default() -> DockerClient {
DockerClient {
docker: Docker::connect_with_defaults()
.expect("failed to connect to docker daemon"),
}
}
}
/// Returns whether build result should be pushed after build
#[instrument(skip(registry_token))]
pub async fn docker_login(
domain: &str,
account: &str,
// For local token override from core.
registry_token: Option<&str>,
) -> anyhow::Result<bool> {
if domain.is_empty() || account.is_empty() {
return Ok(false);
}
let registry_token = match registry_token {
Some(token) => token,
None => crate::helpers::registry_token(domain, account)?,
};
let log = async_run_command(&format!(
"echo {registry_token} | docker login {domain} --username '{account}' --password-stdin",
))
.await;
if log.success() {
Ok(true)
} else {
let mut e = anyhow!("End of trace");
for line in
log.stderr.split('\n').filter(|line| !line.is_empty()).rev()
{
e = e.context(line.to_string());
}
for line in
log.stdout.split('\n').filter(|line| !line.is_empty()).rev()
{
e = e.context(line.to_string());
}
Err(e.context(format!("Registry {domain} login error")))
}
}
#[instrument]
pub async fn pull_image(image: &str) -> Log {
let command = format!("docker pull {image}");
run_komodo_command("Docker Pull", None, command).await
}
pub fn stop_container_command(
container_name: &str,
signal: Option<TerminationSignal>,
time: Option<i32>,
) -> String {
let signal = signal
.map(|signal| format!(" --signal {signal}"))
.unwrap_or_default();
let time = time
.map(|time| format!(" --time {time}"))
.unwrap_or_default();
format!("docker stop{signal}{time} {container_name}")
}
+120
View File
@@ -0,0 +1,120 @@
use bollard::query_parameters::{
InspectNetworkOptions, ListNetworksOptions,
};
use komodo_client::entities::docker::{
container::ContainerListItem, network::*,
};
use super::DockerClient;
impl DockerClient {
pub async fn list_networks(
&self,
containers: &[ContainerListItem],
) -> anyhow::Result<Vec<NetworkListItem>> {
let networks = self
.docker
.list_networks(Option::<ListNetworksOptions>::None)
.await?
.into_iter()
.map(|network| {
let (ipam_driver, ipam_subnet, ipam_gateway) =
if let Some(ipam) = network.ipam {
let (subnet, gateway) = if let Some(config) = ipam
.config
.and_then(|configs| configs.into_iter().next())
{
(config.subnet, config.gateway)
} else {
(None, None)
};
(ipam.driver, subnet, gateway)
} else {
(None, None, None)
};
let in_use = match &network.name {
Some(name) => containers.iter().any(|container| {
container.networks.iter().any(|_name| name == _name)
}),
None => false,
};
NetworkListItem {
name: network.name,
id: network.id,
created: network.created,
scope: network.scope,
driver: network.driver,
enable_ipv6: network.enable_ipv6,
ipam_driver,
ipam_subnet,
ipam_gateway,
internal: network.internal,
attachable: network.attachable,
ingress: network.ingress,
in_use,
}
})
.collect();
Ok(networks)
}
pub async fn inspect_network(
&self,
network_name: &str,
) -> anyhow::Result<Network> {
let network = self
.docker
.inspect_network(
network_name,
InspectNetworkOptions {
verbose: true,
..Default::default()
}
.into(),
)
.await?;
Ok(Network {
name: network.name,
id: network.id,
created: network.created,
scope: network.scope,
driver: network.driver,
enable_ipv6: network.enable_ipv6,
ipam: network.ipam.map(|ipam| Ipam {
driver: ipam.driver,
config: ipam
.config
.unwrap_or_default()
.into_iter()
.map(|config| IpamConfig {
subnet: config.subnet,
ip_range: config.ip_range,
gateway: config.gateway,
auxiliary_addresses: config
.auxiliary_addresses
.unwrap_or_default(),
})
.collect(),
options: ipam.options.unwrap_or_default(),
}),
internal: network.internal,
attachable: network.attachable,
ingress: network.ingress,
containers: network
.containers
.unwrap_or_default()
.into_iter()
.map(|(container_id, container)| NetworkContainer {
container_id,
name: container.name,
endpoint_id: container.endpoint_id,
mac_address: container.mac_address,
ipv4_address: container.ipv4_address,
ipv6_address: container.ipv6_address,
})
.collect(),
options: network.options.unwrap_or_default(),
labels: network.labels.unwrap_or_default(),
})
}
}
+248
View File
@@ -0,0 +1,248 @@
use std::{
collections::HashMap,
sync::{Arc, OnceLock},
};
use anyhow::{Context, anyhow};
use arc_swap::ArcSwap;
use async_timing_util::wait_until_timelength;
use bollard::{models, query_parameters::StatsOptionsBuilder};
use futures::StreamExt;
use komodo_client::entities::docker::{
container::ContainerStats,
stats::{
ContainerBlkioStatEntry, ContainerBlkioStats, ContainerCpuStats,
ContainerCpuUsage, ContainerMemoryStats, ContainerNetworkStats,
ContainerPidsStats, ContainerStorageStats,
ContainerThrottlingData, FullContainerStats,
},
};
use run_command::async_run_command;
use crate::{config::periphery_config, docker::DockerClient};
pub type ContainerStatsMap = HashMap<String, ContainerStats>;
pub fn container_stats() -> &'static ArcSwap<ContainerStatsMap> {
static CONTAINER_STATS: OnceLock<ArcSwap<ContainerStatsMap>> =
OnceLock::new();
CONTAINER_STATS.get_or_init(Default::default)
}
pub fn spawn_polling_thread() {
tokio::spawn(async move {
let polling_rate = periphery_config()
.container_stats_polling_rate
.to_string()
.parse()
.expect("invalid stats polling rate");
update_container_stats().await;
loop {
let _ts = wait_until_timelength(polling_rate, 200).await;
update_container_stats().await;
}
});
}
async fn update_container_stats() {
match get_container_stats(None).await {
Ok(stats) => {
container_stats().store(Arc::new(
stats.into_iter().map(|s| (s.name.clone(), s)).collect(),
));
}
Err(e) => {
error!("Failed to refresh container stats cache | {e:#}");
}
}
}
pub async fn get_container_stats(
container_name: Option<String>,
) -> anyhow::Result<Vec<ContainerStats>> {
let format = "--format \"{{ json . }}\"";
let container_name = match container_name {
Some(name) => format!(" {name}"),
None => "".to_string(),
};
let command =
format!("docker stats{container_name} --no-stream {format}");
let output = async_run_command(&command).await;
if output.success() {
output
.stdout
.split('\n')
.filter(|e| !e.is_empty())
.map(|e| {
let parsed = serde_json::from_str(e)
.context(format!("failed at parsing entry {e}"))?;
Ok(parsed)
})
.collect()
} else {
Err(anyhow!("{}", output.stderr.replace('\n', " | ")))
}
}
impl DockerClient {
/// Calls for stats once, similar to --no-stream on the cli
pub async fn full_container_stats(
&self,
container_name: &str,
) -> anyhow::Result<FullContainerStats> {
let mut res = self.docker.stats(
container_name,
StatsOptionsBuilder::new().stream(false).build().into(),
);
let stats = res
.next()
.await
.with_context(|| format!("Unable to get container stats for {container_name} (got None)"))?
.with_context(|| format!("Unable to get container stats for {container_name}"))?;
Ok(FullContainerStats {
name: stats.name.unwrap_or(container_name.to_string()),
id: stats.id,
read: stats.read,
preread: stats.preread,
pids_stats: stats.pids_stats.map(convert_pids_stats),
blkio_stats: stats.blkio_stats.map(convert_blkio_stats),
num_procs: stats.num_procs,
storage_stats: stats.storage_stats.map(convert_storage_stats),
cpu_stats: stats.cpu_stats.map(convert_cpu_stats),
precpu_stats: stats.precpu_stats.map(convert_cpu_stats),
memory_stats: stats.memory_stats.map(convert_memory_stats),
networks: stats.networks.map(convert_network_stats),
})
}
}
fn convert_pids_stats(
pids_stats: models::ContainerPidsStats,
) -> ContainerPidsStats {
ContainerPidsStats {
current: pids_stats.current,
limit: pids_stats.limit,
}
}
fn convert_blkio_stats(
blkio_stats: models::ContainerBlkioStats,
) -> ContainerBlkioStats {
ContainerBlkioStats {
io_service_bytes_recursive: blkio_stats
.io_service_bytes_recursive
.map(convert_blkio_stat_entries),
io_serviced_recursive: blkio_stats
.io_serviced_recursive
.map(convert_blkio_stat_entries),
io_queue_recursive: blkio_stats
.io_queue_recursive
.map(convert_blkio_stat_entries),
io_service_time_recursive: blkio_stats
.io_service_time_recursive
.map(convert_blkio_stat_entries),
io_wait_time_recursive: blkio_stats
.io_wait_time_recursive
.map(convert_blkio_stat_entries),
io_merged_recursive: blkio_stats
.io_merged_recursive
.map(convert_blkio_stat_entries),
io_time_recursive: blkio_stats
.io_time_recursive
.map(convert_blkio_stat_entries),
sectors_recursive: blkio_stats
.sectors_recursive
.map(convert_blkio_stat_entries),
}
}
fn convert_blkio_stat_entries(
blkio_stat_entries: Vec<models::ContainerBlkioStatEntry>,
) -> Vec<ContainerBlkioStatEntry> {
blkio_stat_entries
.into_iter()
.map(|blkio_stat_entry| ContainerBlkioStatEntry {
major: blkio_stat_entry.major,
minor: blkio_stat_entry.minor,
op: blkio_stat_entry.op,
value: blkio_stat_entry.value,
})
.collect()
}
fn convert_storage_stats(
storage_stats: models::ContainerStorageStats,
) -> ContainerStorageStats {
ContainerStorageStats {
read_count_normalized: storage_stats.read_count_normalized,
read_size_bytes: storage_stats.read_size_bytes,
write_count_normalized: storage_stats.write_count_normalized,
write_size_bytes: storage_stats.write_size_bytes,
}
}
fn convert_cpu_stats(
cpu_stats: models::ContainerCpuStats,
) -> ContainerCpuStats {
ContainerCpuStats {
cpu_usage: cpu_stats.cpu_usage.map(convert_cpu_usage),
system_cpu_usage: cpu_stats.system_cpu_usage,
online_cpus: cpu_stats.online_cpus,
throttling_data: cpu_stats
.throttling_data
.map(convert_cpu_throttling_data),
}
}
fn convert_cpu_usage(
cpu_usage: models::ContainerCpuUsage,
) -> ContainerCpuUsage {
ContainerCpuUsage {
total_usage: cpu_usage.total_usage,
percpu_usage: cpu_usage.percpu_usage,
usage_in_kernelmode: cpu_usage.usage_in_kernelmode,
usage_in_usermode: cpu_usage.usage_in_usermode,
}
}
fn convert_cpu_throttling_data(
cpu_throttling_data: models::ContainerThrottlingData,
) -> ContainerThrottlingData {
ContainerThrottlingData {
periods: cpu_throttling_data.periods,
throttled_periods: cpu_throttling_data.throttled_periods,
throttled_time: cpu_throttling_data.throttled_time,
}
}
fn convert_memory_stats(
memory_stats: models::ContainerMemoryStats,
) -> ContainerMemoryStats {
ContainerMemoryStats {
usage: memory_stats.usage,
max_usage: memory_stats.max_usage,
stats: memory_stats.stats,
failcnt: memory_stats.failcnt,
limit: memory_stats.limit,
commitbytes: memory_stats.commitbytes,
commitpeakbytes: memory_stats.commitpeakbytes,
privateworkingset: memory_stats.privateworkingset,
}
}
fn convert_network_stats(
network_stats: models::ContainerNetworkStats,
) -> ContainerNetworkStats {
ContainerNetworkStats {
rx_bytes: network_stats.rx_bytes,
rx_packets: network_stats.rx_packets,
rx_errors: network_stats.rx_errors,
rx_dropped: network_stats.rx_dropped,
tx_bytes: network_stats.tx_bytes,
tx_packets: network_stats.tx_packets,
tx_errors: network_stats.tx_errors,
tx_dropped: network_stats.tx_dropped,
endpoint_id: network_stats.endpoint_id,
instance_id: network_stats.instance_id,
}
}
+155
View File
@@ -0,0 +1,155 @@
use bollard::query_parameters::ListVolumesOptions;
use komodo_client::entities::docker::{
PortBinding, container::ContainerListItem, volume::*,
};
use crate::docker::DockerClient;
impl DockerClient {
pub async fn list_volumes(
&self,
containers: &[ContainerListItem],
) -> anyhow::Result<Vec<VolumeListItem>> {
let volumes = self
.docker
.list_volumes(Option::<ListVolumesOptions>::None)
.await?
.volumes
.unwrap_or_default()
.into_iter()
.map(|volume| {
let scope = volume
.scope
.map(|scope| match scope {
bollard::secret::VolumeScopeEnum::EMPTY => {
VolumeScopeEnum::Empty
}
bollard::secret::VolumeScopeEnum::LOCAL => {
VolumeScopeEnum::Local
}
bollard::secret::VolumeScopeEnum::GLOBAL => {
VolumeScopeEnum::Global
}
})
.unwrap_or(VolumeScopeEnum::Empty);
let in_use = containers.iter().any(|container| {
container.volumes.iter().any(|name| &volume.name == name)
});
VolumeListItem {
name: volume.name,
driver: volume.driver,
mountpoint: volume.mountpoint,
created: volume.created_at,
size: volume.usage_data.map(|data| data.size),
scope,
in_use,
}
})
.collect();
Ok(volumes)
}
pub async fn inspect_volume(
&self,
volume_name: &str,
) -> anyhow::Result<Volume> {
let volume = self.docker.inspect_volume(volume_name).await?;
Ok(Volume {
name: volume.name,
driver: volume.driver,
mountpoint: volume.mountpoint,
created_at: volume.created_at,
status: volume.status.unwrap_or_default().into_keys().map(|k| (k, Default::default())).collect(),
labels: volume.labels,
scope: volume
.scope
.map(|scope| match scope {
bollard::secret::VolumeScopeEnum::EMPTY => {
VolumeScopeEnum::Empty
}
bollard::secret::VolumeScopeEnum::LOCAL => {
VolumeScopeEnum::Local
}
bollard::secret::VolumeScopeEnum::GLOBAL => {
VolumeScopeEnum::Global
}
})
.unwrap_or_default(),
cluster_volume: volume.cluster_volume.map(|volume| {
ClusterVolume {
id: volume.id,
version: volume.version.map(|version| ObjectVersion {
index: version.index,
}),
created_at: volume.created_at,
updated_at: volume.updated_at,
spec: volume.spec.map(|spec| ClusterVolumeSpec {
group: spec.group,
access_mode: spec.access_mode.map(|mode| {
ClusterVolumeSpecAccessMode {
scope: mode.scope.map(|scope| match scope {
bollard::secret::ClusterVolumeSpecAccessModeScopeEnum::EMPTY => ClusterVolumeSpecAccessModeScopeEnum::Empty,
bollard::secret::ClusterVolumeSpecAccessModeScopeEnum::SINGLE => ClusterVolumeSpecAccessModeScopeEnum::Single,
bollard::secret::ClusterVolumeSpecAccessModeScopeEnum::MULTI => ClusterVolumeSpecAccessModeScopeEnum::Multi,
}).unwrap_or_default(),
sharing: mode.sharing.map(|sharing| match sharing {
bollard::secret::ClusterVolumeSpecAccessModeSharingEnum::EMPTY => ClusterVolumeSpecAccessModeSharingEnum::Empty,
bollard::secret::ClusterVolumeSpecAccessModeSharingEnum::NONE => ClusterVolumeSpecAccessModeSharingEnum::None,
bollard::secret::ClusterVolumeSpecAccessModeSharingEnum::READONLY => ClusterVolumeSpecAccessModeSharingEnum::Readonly,
bollard::secret::ClusterVolumeSpecAccessModeSharingEnum::ONEWRITER => ClusterVolumeSpecAccessModeSharingEnum::Onewriter,
bollard::secret::ClusterVolumeSpecAccessModeSharingEnum::ALL => ClusterVolumeSpecAccessModeSharingEnum::All,
}).unwrap_or_default(),
secrets: mode.secrets.unwrap_or_default().into_iter().map(|secret| ClusterVolumeSpecAccessModeSecrets {
key: secret.key,
secret: secret.secret,
}).collect(),
accessibility_requirements: mode
.accessibility_requirements.map(|req| ClusterVolumeSpecAccessModeAccessibilityRequirements {
requisite: req.requisite.unwrap_or_default().into_iter().map(|map| map.into_iter().map(|(k, v)| (k, v.unwrap_or_default().into_iter().map(|p| PortBinding { host_ip: p.host_ip, host_port: p.host_port }).collect())).collect()).collect(),
preferred: req.preferred.unwrap_or_default().into_iter().map(|map| map.into_iter().map(|(k, v)| (k, v.unwrap_or_default().into_iter().map(|p| PortBinding { host_ip: p.host_ip, host_port: p.host_port }).collect())).collect()).collect(),
}),
capacity_range: mode.capacity_range.map(|range| ClusterVolumeSpecAccessModeCapacityRange {
required_bytes: range.required_bytes,
limit_bytes: range.limit_bytes,
}),
availability: mode.availability.map(|availability| match availability {
bollard::secret::ClusterVolumeSpecAccessModeAvailabilityEnum::EMPTY => ClusterVolumeSpecAccessModeAvailabilityEnum::Empty,
bollard::secret::ClusterVolumeSpecAccessModeAvailabilityEnum::ACTIVE => ClusterVolumeSpecAccessModeAvailabilityEnum::Active,
bollard::secret::ClusterVolumeSpecAccessModeAvailabilityEnum::PAUSE => ClusterVolumeSpecAccessModeAvailabilityEnum::Pause,
bollard::secret::ClusterVolumeSpecAccessModeAvailabilityEnum::DRAIN => ClusterVolumeSpecAccessModeAvailabilityEnum::Drain,
}).unwrap_or_default(),
}
}),
}),
info: volume.info.map(|info| ClusterVolumeInfo {
capacity_bytes: info.capacity_bytes,
volume_context: info.volume_context.unwrap_or_default(),
volume_id: info.volume_id,
accessible_topology: info.accessible_topology.unwrap_or_default().into_iter().map(|map| map.into_iter().map(|(k, v)| (k, v.unwrap_or_default().into_iter().map(|p| PortBinding { host_ip: p.host_ip, host_port: p.host_port }).collect())).collect()).collect(),
}),
publish_status: volume
.publish_status
.unwrap_or_default()
.into_iter()
.map(|status| ClusterVolumePublishStatus {
node_id: status.node_id,
state: status.state.map(|state| match state {
bollard::secret::ClusterVolumePublishStatusStateEnum::EMPTY => ClusterVolumePublishStatusStateEnum::Empty,
bollard::secret::ClusterVolumePublishStatusStateEnum::PENDING_PUBLISH => ClusterVolumePublishStatusStateEnum::PendingPublish,
bollard::secret::ClusterVolumePublishStatusStateEnum::PUBLISHED => ClusterVolumePublishStatusStateEnum::Published,
bollard::secret::ClusterVolumePublishStatusStateEnum::PENDING_NODE_UNPUBLISH => ClusterVolumePublishStatusStateEnum::PendingNodeUnpublish,
bollard::secret::ClusterVolumePublishStatusStateEnum::PENDING_CONTROLLER_UNPUBLISH => ClusterVolumePublishStatusStateEnum::PendingControllerUnpublish,
}).unwrap_or_default(),
publish_context: status.publish_context.unwrap_or_default(),
})
.collect(),
}
}),
options: volume.options,
usage_data: volume.usage_data.map(|data| VolumeUsageData {
size: data.size,
ref_count: data.ref_count,
}),
})
}
}
+94
View File
@@ -0,0 +1,94 @@
use std::path::PathBuf;
use command::run_komodo_command_with_sanitization;
use environment::write_env_file;
use interpolate::Interpolator;
use komodo_client::entities::{
EnvironmentVar, RepoExecutionResponse, SystemCommand,
all_logs_success,
};
use periphery_client::api::git::PeripheryRepoExecutionResponse;
use crate::config::periphery_config;
pub async fn handle_post_repo_execution(
mut res: RepoExecutionResponse,
mut environment: Vec<EnvironmentVar>,
env_file_path: &str,
mut on_clone: Option<SystemCommand>,
mut on_pull: Option<SystemCommand>,
skip_secret_interp: bool,
mut replacers: Vec<(String, String)>,
) -> anyhow::Result<PeripheryRepoExecutionResponse> {
if !skip_secret_interp {
let mut interpolotor =
Interpolator::new(None, &periphery_config().secrets);
interpolotor.interpolate_env_vars(&mut environment)?;
if let Some(on_clone) = on_clone.as_mut() {
interpolotor.interpolate_string(&mut on_clone.command)?;
}
if let Some(on_pull) = on_pull.as_mut() {
interpolotor.interpolate_string(&mut on_pull.command)?;
}
replacers.extend(interpolotor.secret_replacers);
}
let env_file_path = write_env_file(
&environment,
&res.path,
env_file_path,
&mut res.logs,
)
.await;
let mut res = PeripheryRepoExecutionResponse { res, env_file_path };
if let Some(on_clone) = on_clone {
if !on_clone.is_none() {
let path = res
.res
.path
.join(on_clone.path)
.components()
.collect::<PathBuf>();
if let Some(log) = run_komodo_command_with_sanitization(
"On Clone",
path.as_path(),
on_clone.command,
true,
&replacers,
)
.await
{
res.res.logs.push(log);
if !all_logs_success(&res.res.logs) {
return Ok(res);
}
}
}
}
if let Some(on_pull) = on_pull {
if !on_pull.is_none() {
let path = res
.res
.path
.join(on_pull.path)
.components()
.collect::<PathBuf>();
if let Some(log) = run_komodo_command_with_sanitization(
"On Pull",
path.as_path(),
on_pull.command,
true,
&replacers,
)
.await
{
res.res.logs.push(log);
}
}
}
Ok(res)
}
+18 -71
View File
@@ -1,19 +1,12 @@
use std::path::PathBuf;
use anyhow::{Context, anyhow};
use anyhow::Context;
use komodo_client::{
entities::{
CloneArgs, EnvironmentVar, SearchCombinator, stack::Stack,
to_path_compatible_name,
},
entities::{EnvironmentVar, RepoExecutionArgs, SearchCombinator},
parsers::QUOTE_PATTERN,
};
use periphery_client::api::git::PullOrCloneRepo;
use resolver_api::Resolve;
use crate::config::periphery_config;
pub fn git_token(
pub fn git_token_simple(
domain: &str,
account_username: &str,
) -> anyhow::Result<&'static str> {
@@ -24,7 +17,21 @@ pub fn git_token(
.and_then(|provider| {
provider.accounts.iter().find(|account| account.username == account_username).map(|account| account.token.as_str())
})
.with_context(|| format!("did not find token in config for git account {account_username} | domain {domain}"))
.with_context(|| format!("Did not find token in config for git account {account_username} | domain {domain}"))
}
pub fn git_token(
core_token: Option<String>,
args: &RepoExecutionArgs,
) -> anyhow::Result<Option<String>> {
if core_token.is_some() {
return Ok(core_token);
}
let Some(account) = &args.account else {
return Ok(None);
};
let token = git_token_simple(&args.provider, account)?;
Ok(Some(token.to_string()))
}
pub fn registry_token(
@@ -85,63 +92,3 @@ pub fn log_grep(
}
}
}
/// Returns path to root directory of the stack repo.
pub async fn pull_or_clone_stack(
stack: &Stack,
git_token: Option<String>,
) -> anyhow::Result<PathBuf> {
if stack.config.files_on_host {
return Err(anyhow!(
"Wrong method called for files on host stack"
));
}
if stack.config.repo.is_empty() {
return Err(anyhow!("Repo is not configured"));
}
let root = periphery_config()
.stack_dir()
.join(to_path_compatible_name(&stack.name));
let mut args: CloneArgs = stack.into();
// Set the clone destination to the one created for this run
args.destination = Some(root.display().to_string());
let git_token = match git_token {
Some(token) => Some(token),
None => {
if !stack.config.git_account.is_empty() {
match crate::helpers::git_token(
&stack.config.git_provider,
&stack.config.git_account,
) {
Ok(token) => Some(token.to_string()),
Err(e) => {
return Err(
e.context("Failed to find required git token"),
);
}
}
} else {
None
}
}
};
PullOrCloneRepo {
args,
git_token,
environment: vec![],
env_file_path: stack.config.env_file_path.clone(),
skip_secret_interp: stack.config.skip_secret_interp,
// repo replacer only needed for on_clone / on_pull,
// which aren't available for stacks
replacers: Default::default(),
}
.resolve(&crate::api::Args)
.await
.map_err(|e| e.error)?;
Ok(root)
}
+4 -1
View File
@@ -9,9 +9,11 @@ use axum_server::tls_rustls::RustlsConfig;
use config::periphery_config;
mod api;
mod build;
mod compose;
mod config;
mod docker;
mod git;
mod helpers;
mod ssl;
mod stats;
@@ -30,7 +32,8 @@ async fn app() -> anyhow::Result<()> {
info!("{:?}", config.sanitized());
}
stats::spawn_system_stats_polling_thread();
stats::spawn_polling_thread();
docker::stats::spawn_polling_thread();
let addr = format!(
"{}:{}",
+7 -3
View File
@@ -4,7 +4,7 @@ use async_timing_util::wait_until_timelength;
use komodo_client::entities::stats::{
SingleDiskUsage, SystemInformation, SystemProcess, SystemStats,
};
use sysinfo::{ProcessesToUpdate, System};
use sysinfo::{ProcessRefreshKind, ProcessesToUpdate, System};
use tokio::sync::RwLock;
use crate::config::periphery_config;
@@ -17,7 +17,7 @@ pub fn stats_client() -> &'static RwLock<StatsClient> {
/// This should be called before starting the server in main.rs.
/// Keeps the cached stats up to date
pub fn spawn_system_stats_polling_thread() {
pub fn spawn_polling_thread() {
tokio::spawn(async move {
let polling_rate = periphery_config()
.stats_polling_rate
@@ -74,7 +74,11 @@ impl StatsClient {
fn refresh(&mut self) {
self.system.refresh_cpu_all();
self.system.refresh_memory();
self.system.refresh_processes(ProcessesToUpdate::All, true);
self.system.refresh_processes_specifics(
ProcessesToUpdate::All,
true,
ProcessRefreshKind::everything().without_tasks(),
);
self.disks.refresh(true);
self.networks.refresh(true);
}
+101 -2
View File
@@ -1,15 +1,23 @@
use std::{
collections::{HashMap, VecDeque},
pin::Pin,
sync::{Arc, OnceLock},
task::Poll,
time::Duration,
};
use anyhow::{Context, anyhow};
use axum::http::StatusCode;
use bytes::Bytes;
use futures::Stream;
use komodo_client::{
api::write::TerminalRecreateMode, entities::server::TerminalInfo,
api::write::TerminalRecreateMode,
entities::{komodo_timestamp, server::TerminalInfo},
};
use pin_project_lite::pin_project;
use portable_pty::{CommandBuilder, PtySize, native_pty_system};
use rand::Rng;
use serror::AddStatusCodeError;
use tokio::sync::{broadcast, mpsc};
use tokio_util::sync::CancellationToken;
@@ -307,7 +315,7 @@ impl Terminal {
}
}
/// 1 MiB max history size per terminal
/// 1 MiB rolling max history size per terminal
const MAX_BYTES: usize = 1024 * 1024;
pub struct History {
@@ -344,3 +352,94 @@ impl History {
self.buf.read().unwrap().len() as f64 / 1024.0
}
}
/// Execute Sentinels
pub const START_OF_OUTPUT: &str = "__KOMODO_START_OF_OUTPUT__";
pub const END_OF_OUTPUT: &str = "__KOMODO_END_OF_OUTPUT__";
pin_project! {
pub struct TerminalStream<S> { #[pin] pub stdout: S }
}
impl<S> Stream for TerminalStream<S>
where
S:
Stream<Item = Result<String, tokio_util::codec::LinesCodecError>>,
{
// Axum expects a stream of results
type Item = Result<String, String>;
fn poll_next(
self: Pin<&mut Self>,
cx: &mut std::task::Context<'_>,
) -> Poll<Option<Self::Item>> {
let this = self.project();
match this.stdout.poll_next(cx) {
Poll::Ready(None) => {
// This is if a None comes in before END_OF_OUTPUT.
// This probably means the terminal has exited early,
// and needs to be cleaned up
tokio::spawn(async move { clean_up_terminals().await });
Poll::Ready(None)
}
Poll::Ready(Some(line)) => {
match line {
Ok(line) if line.as_str() == END_OF_OUTPUT => {
// Stop the stream on end sentinel
Poll::Ready(None)
}
Ok(line) => Poll::Ready(Some(Ok(line + "\n"))),
Err(e) => Poll::Ready(Some(Err(format!("{e:?}")))),
}
}
Poll::Pending => Poll::Pending,
}
}
}
/// Tokens valid for 3 seconds
const TOKEN_VALID_FOR_MS: i64 = 3_000;
pub fn auth_tokens() -> &'static AuthTokens {
static AUTH_TOKENS: OnceLock<AuthTokens> = OnceLock::new();
AUTH_TOKENS.get_or_init(Default::default)
}
#[derive(Default)]
pub struct AuthTokens {
map: std::sync::Mutex<HashMap<String, i64>>,
}
impl AuthTokens {
pub fn create_auth_token(&self) -> String {
let mut lock = self.map.lock().unwrap();
// clear out any old tokens here (prevent unbounded growth)
let ts = komodo_timestamp();
lock.retain(|_, valid_until| *valid_until > ts);
let token: String = rand::rng()
.sample_iter(&rand::distr::Alphanumeric)
.take(30)
.map(char::from)
.collect();
lock.insert(token.clone(), ts + TOKEN_VALID_FOR_MS);
token
}
pub fn check_token(&self, token: String) -> serror::Result<()> {
let Some(valid_until) = self.map.lock().unwrap().remove(&token)
else {
return Err(
anyhow!("Terminal auth token not found")
.status_code(StatusCode::UNAUTHORIZED),
);
};
if komodo_timestamp() <= valid_until {
Ok(())
} else {
Err(
anyhow!("Terminal token is expired")
.status_code(StatusCode::UNAUTHORIZED),
)
}
}
}
+1 -5
View File
@@ -45,8 +45,6 @@ services:
labels:
komodo.skip: # Prevent Komodo from stopping with StopAllContainers
restart: unless-stopped
logging:
driver: ${COMPOSE_LOGGING_DRIVER:-local}
# ports:
# - 5432:5432
volumes:
@@ -54,7 +52,7 @@ services:
environment:
POSTGRES_USER: ${KOMODO_DB_USERNAME}
POSTGRES_PASSWORD: ${KOMODO_DB_PASSWORD}
POSTGRES_DB: postgres
POSTGRES_DB: postgres # Do not change
ferretdb2:
# Recommended: Pin to a specific version
@@ -65,8 +63,6 @@ services:
restart: unless-stopped
depends_on:
- postgres2
logging:
driver: ${COMPOSE_LOGGING_DRIVER:-local}
# ports:
# - 27017:27017
volumes:
+3 -3
View File
@@ -160,14 +160,14 @@ pub enum BatchExecutionResponseItem {
Err(BatchExecutionResponseItemErr),
}
impl From<Result<Update, BatchExecutionResponseItemErr>>
impl From<Result<Box<Update>, BatchExecutionResponseItemErr>>
for BatchExecutionResponseItem
{
fn from(
value: Result<Update, BatchExecutionResponseItemErr>,
value: Result<Box<Update>, BatchExecutionResponseItemErr>,
) -> Self {
match value {
Ok(update) => Self::Ok(update),
Ok(update) => Self::Ok(*update),
Err(e) => Self::Err(e),
}
}
+4
View File
@@ -13,6 +13,7 @@ mod permission;
mod procedure;
mod provider;
mod repo;
mod schedule;
mod server;
mod stack;
mod sync;
@@ -33,6 +34,7 @@ pub use permission::*;
pub use procedure::*;
pub use provider::*;
pub use repo::*;
pub use schedule::*;
pub use server::*;
pub use stack::*;
pub use sync::*;
@@ -106,6 +108,8 @@ pub struct GetCoreInfoResponse {
pub github_webhook_owners: Vec<String>,
/// Whether to disable websocket automatic reconnect.
pub disable_websocket_reconnect: bool,
/// TZ identifier Core is using, if manually set.
pub timezone: String,
}
//
+32
View File
@@ -0,0 +1,32 @@
use derive_empty_traits::EmptyTraits;
use resolver_api::Resolve;
use serde::{Deserialize, Serialize};
use typeshare::typeshare;
use crate::{
deserializers::string_list_deserializer,
entities::{resource::TagQueryBehavior, schedule::Schedule},
};
use super::KomodoReadRequest;
/// List configured schedules.
/// Response: [ListSchedulesResponse].
#[typeshare]
#[derive(
Serialize, Deserialize, Debug, Clone, Resolve, EmptyTraits,
)]
#[empty_traits(KomodoReadRequest)]
#[response(ListSchedulesResponse)]
#[error(serror::Error)]
pub struct ListSchedules {
/// Pass Vec of tag ids or tag names
#[serde(default, deserialize_with = "string_list_deserializer")]
pub tags: Vec<String>,
/// 'All' or 'Any'
#[serde(default)]
pub tag_behavior: TagQueryBehavior,
}
#[typeshare]
pub type ListSchedulesResponse = Vec<Schedule>;

Some files were not shown because too many files have changed in this diff Show More