Another CLI tool for Alps.
Manta is a frontend cli to interact with Shasta, it uses mesa for all Shasta interaction.
Manta's goals:
- release operators from repetitive tasks.
- provide quick system feedback.
Manta aggregates information from multiple sources:
- Shasta Keycloak
- Shasta API
- Shasta K8s API
- local git repo
- Gitea API (Shasta VCS)
- Hashicorp Vault
- List and filter CFS configurations based on cluster name or configuration name
- List and filter CFS sessions based on cluster name or session name
- List and filter BOS session templates based on cluster name or session name
- List nodes in HSM groups
- Create CFS configuration and session (target dynamic) from local repository
- Create CFS configuration and session (target image) from CSCS SAT input file
- Watch logs of a CFS session
- Connect to a node's console
- Power On/Off or restart nodes individually, in a list or per cluster
- Restrict operations to nodes belonging to a specific HSM group
- Filter information to a HSM group
- Update node boot image based on CFS configuration name
- Audit/Log
Manta needs a configuration file in $HOME/.config/manta/config
like shown below
log = "error"
base_image_id = "c37fbd5f-0fe5-43ce-8cfe-337f3e78733e"
socks5_proxy = "socks5h://127.0.0.1:1080"
shasta_base_url = "https://api.cmn.alps.cscs.ch/apis"
keycloak_base_url = "https://api.cmn.alps.cscs.ch/keycloak"
gitea_base_url = "https://api.cmn.alps.cscs.ch/vcs"
k8s_api_url = "https://10.252.1.12:6442"
vault_base_url = "https://hashicorp-vault.cscs.ch:8200"
vault_role_id = "b15517de-cabb-06ba-af98-633d216c6d99" # vault in hashicorp-vault.cscs.ch
hsm_group = "psitds"
Manta logs user's operations in /var/log/manta/ folder, please make sure this folder exists and all users have rwx access to it
mkdir /var/log/manta
chmod 777 -R /var/log/manta
Name | mandatory | Type | Description | Example |
---|---|---|---|---|
log | no | config file | log details/verbosity | off/error/warn/info/debug/trace |
socks5_proxy | yes | config file | socks proxy to access the services (only needed if using manta from outside a Shasta management node. Need VPN. Need to ope your VPN IP in hashicorp vault approle) | socks5h://127.0.0.1:1080 |
keycloak_base_url | yes | config file | Keycloak base URL for authentication | https://api.cmn.alps.cscs.ch/keycloak |
gitea_base_url | yes | config file | Gitea base URL to fetch CFS layers git repo details | https://api.cmn.alps.cscs.ch/vcs |
k8s_api_url | yes | config file | Shasta k8s API URL | https://10.252.1.12:6442 |
vault_base_url | yes | config file | Hashicorp Vault base URL storing secrets to authenticate to external services | https://hashicorp-vault.cscs.ch |
vault_role_id | yes | config file | role id related to Hashicorp Vault base URL approle authentication | yes |
shasta_base_url | yes | config file | Shasta API base URL for Shasta related jobs submission | https://api-gw-service-nmn.local/apis |
hsm_group | no | config | If exists, then it will filter/restrict the hsm groups and/or xnames targeted by the cli command | psi-dev |
$ manta get session --most-recent
+----------------------------------------------+-------------------------+---------+---------------+---------------+---------------------+----------+-----------+------------------------------------------+
| Name | Configuration | Target | Target groups | Ansible limit | Start | Status | Succeeded | Job |
+==========================================================================================================================================================================================================+
| batcher-bab0cd68-5c61-4774-a685-bd57f744f62d | eiger-cos-config-3.0.24 | dynamic | | x1002c6s6b0n0 | 2022-10-29T15:50:19 | complete | true | cfs-cd39e25e-5b66-4ee9-be1c-027f5cd00683 |
+----------------------------------------------+-------------------------+---------+---------------+---------------+---------------------+----------+-----------+------------------------------------------+
$ manta log --session-name batcher-cef892ee-39af-444a-b32c-89478a100e4d --layer-id 0
[2022-09-27T12:41:49Z INFO manta::shasta_cfs_session_logs::client] Pod name: "cfs-b49cdc2b-d6cb-4477-b502-6be479472546-2jrlg"
Waiting for Inventory
Waiting for Inventory
Waiting for Inventory
Waiting for Inventory
Waiting for Inventory
Waiting for Inventory
Waiting for Inventory
Inventory generation completed
SSH keys migrated to /root/.ssh
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
HTTP/1.1 200 OK
content-type: text/html; charset=UTF-8
cache-control: no-cache, max-age=0
x-content-type-options: nosniff
date: Tue, 27 Sep 2022 12:18:16 GMT
server: envoy
transfer-encoding: chunked
Sidecar available
[WARNING]: Invalid characters were found in group names but not replaced, use
-vvvv to see details
PLAY [Compute] *****************************************************************
PLAY [Application] *************************************************************
skipping: no hosts matched
PLAY [Management_Worker] *******************************************************
skipping: no hosts matched
PLAY RECAP *********************************************************************
x1500c7s2b0n0 : ok=1 changed=0 unreachable=0 failed=0 skipped=33 rescued=0 ignored=0
$ manta apply session --repo-path /home/msopena/ownCloud/Documents/ALPSINFRA/vcluster_shasta_scripts/muttler/muttler_orchestrator/ --watch-logs --ansible-limit x1500c3s4b0n1
[2022-10-08T22:56:31Z INFO manta::create_session_from_repo] Checking repo /home/msopena/ownCloud/Documents/ALPSINFRA/vcluster_shasta_scripts/muttler/muttler_orchestrator/.git/ status
[2022-10-08T22:56:32Z INFO manta::create_session_from_repo] CFS configuration name: m-muttler-orchestrator
[2022-10-08T22:56:35Z INFO manta::create_session_from_repo] CFS session name: m-muttler-orchestrator-20221008225632
[2022-10-08T22:56:35Z INFO manta] cfs session: m-muttler-orchestrator-20221008225632
[2022-10-08T22:56:35Z INFO manta] Fetching logs ...
[2022-10-08T22:56:35Z INFO manta::shasta_cfs_session_logs::client] Pod for cfs session m-muttler-orchestrator-20221008225632 not ready. Trying again in 2 secs. Attempt 1 of 10
[2022-10-08T22:56:38Z INFO manta::shasta_cfs_session_logs::client] Pod name: cfs-f1588924-f791-4bb8-a565-f61563a4274b-n7bbn
[2022-10-08T22:56:38Z INFO manta::shasta_cfs_session_logs::client] Container ansible-0 not ready. Trying again in 2 secs. Attempt 1 of 10
[2022-10-08T22:56:40Z INFO manta::shasta_cfs_session_logs::client] Container ansible-0 not ready. Trying again in 2 secs. Attempt 2 of 10
[2022-10-08T22:56:42Z INFO manta::shasta_cfs_session_logs::client] Container ansible-0 not ready. Trying again in 2 secs. Attempt 3 of 10
Waiting for Inventory
Waiting for Inventory
Inventory generation completed
SSH keys migrated to /root/.ssh
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
HTTP/1.1 200 OK
content-type: text/html; charset=UTF-8
cache-control: no-cache, max-age=0
x-content-type-options: nosniff
date: Sat, 08 Oct 2022 22:56:49 GMT
server: envoy
transfer-encoding: chunked
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
Sidecar available
[WARNING]: Invalid characters were found in group names but not replaced, use
-vvvv to see details
PLAY [Compute:Application] *****************************************************
PLAY RECAP *********************************************************************
x1500c3s4b0n1 : ok=8 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
$ manta console x1500c2s4b0n1
[2022-10-30T02:14:44Z INFO manta::node_console] Alternatively run - kubectl -n services exec -it cray-console-node-2 -c cray-console-node -- conman -j x1500c2s4b0n1
[2022-10-30T02:14:44Z INFO manta::node_console] Connecting to console x1500c2s4b0n1
Connected to x1500c2s4b0n1!
Use &. key combination to exit the console.
<ConMan> Connection to console [x1500c2s4b0n1] opened.
<ConMan> Console [x1500c2s4b0n1] joined with <nobody@localhost> on pts/452 at 10-30 02:14.
<ConMan> Console [x1500c2s4b0n1] joined with <nobody@localhost> on pts/453 at 10-30 02:14.
<ConMan> Console [x1500c2s4b0n1] joined with <nobody@localhost> on pts/454 at 10-30 02:14.
<ConMan> Console [x1500c2s4b0n1] joined with <nobody@localhost> on pts/455 at 10-30 02:14.
<ConMan> Console [x1500c2s4b0n1] joined with <nobody@localhost> on pts/468 at 10-30 02:14.
<ConMan> Console [x1500c2s4b0n1] joined with <nobody@localhost> on pts/510 at 10-30 02:14.
<ConMan> Console [x1500c2s4b0n1] joined with <nobody@localhost> on pts/511 at 10-30 02:14.
nid003129 login:
$ manta apply node off --force "x1004c1s4b0n1"
$ manta apply node on "x1004c1s4b0n1"
This repo contains a Dockerfile to build a Container with manta cli.
docker build -t manta .
$ docker run -it --network=host -v ~:/root/ manta --help
Install Rust toolchain https://www.rust-lang.org/tools/install
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
Install cross to be able to complile on different platforms
cargo install cross
Generate binary
cross build --target x86_64-unknown-linux-gnu --release
Install 'cargo dist' and 'cargo release'
cargo install cargo-dist
cargo install cargo-release
Configure cargo-dist. Accept default options and only target linux assets
cargo dist init -t x86_64-unknown-linux-gnu
Then remove the assets for macos and windows
Make sure a github workflow is created in .github/workflows/release.yml
cargo dist patch --execute
If everything went well, then binary should be located in manta/target/x86_64-unknown-linux-gnu/release/manta
sudo sysctl -w kernel.perf_event_paranoid=-1
sudo apt-get install linux-tools-common linux-tools-generic linux-tools-`uname -r`
sudo sh -c " echo 0 > /proc/sys/kernel/kptr_restrict"
perf stat -ad -r 100 target/release/manta get session
perf record -g --call-graph=dwarf -F max target/release/manta get session
perf script -F +pid > manta.perf
Go to https://profiler.firefox.com/ and open manta.perf file
https://docs.rs/dhat/latest/dhat/ NOTE: lto in Cargo.toml needs to be disabled
cargo run -r --features dhat-heap -- get session