Mozart
If you struggle with maintaining a lot of scripts:
╰$ tree scripts
scripts
└── my-service
├── install
│ ├── install-component1.sh
│ ├── install-component2.sh
│ └── install-component3.sh
└── test
├── test-component1.sh
├── test-component2.sh
└── test-component3.sh
then think of Mozart! Mozart is a script orchestrator. It can help package all scripts to a single binary, along with a CLI and and UI to manage them, with ZERO lines of code!
CLI:
Available CLI commands:
mozart execute my-service
mozart execute my-service install
mozart execute my-service install install-component1
mozart execute my-service install install-component2
mozart execute my-service install install-component3
mozart execute my-service test
mozart execute my-service test test-component1
mozart execute my-service test test-component2
mozart execute my-service test test-component3
What is mozart?
Mozart is a simple drop-in (no go-coding required) utility to orchestrate and attach a CLI and a UI to your scripts, making your independent, messy bunch of scripts into a well defined, orchestrated program complete with a CLI and UI!
Within minutes, instead of having hundreds of different scripts, you can have a single binary, complete with a CLI and a UI, which includes all those scripts. All you need to do is to drop the scripts into a folder structure (explained below). That's it!
Note: No code changes required, it is a simple drop-in type utility for your scripts.
What is an orchestrator?
What Docker compose is to Docker containers, Mozart is to scripts.
- Manages packaging and distribution of scripts
- Manages modularization – breaking down of big scripts into smaller scripts
- Manages templating across scripts – runtime variable substitutions
- Manages DRY policy across scripts
- Manages all logs generated by scripts
- Manages state of execution of all scripts – success/failure/last-exec time
- Manages execution of all scripts – rerun/dry-run
- Provides a CLI, a UI, a REST interface to interact with the scripts
How mozart differs from Ansible
Mozart | Ansible |
---|---|
Easy learning curve | Tough learning curve |
No new software needed (only clone this repo) | Initial setup required |
Very less changes to existing scripts (simple drag and drop capability) | More effort in making ansible modules |
Working orchestrator within minutes | Takes time to create playbooks |
Provides single file packaging for all scripts | Doesn't provide packaging (AFAIK?) |
Useful for quick POC, testing, etc. | Suitable for heavy duty production deployments |
Simpler to use for single host (multiple hosts requires some changes) | Suitable for multiple hosts from start |
Table of Contents
- What is mozart?
- What is an orchestrator?
- How mozart differs from Ansible
- What exactly does an orchestrator do?
- Using your scripts with Mozart
- Mozart yaml file
- Using common snippets across scripts
- Adding custom resources
- CLI
- UI
- Helpful resources
What exactly does an orchestrator do?
Simply speaking, an orchestrator manages execution of all of your scripts.
Life without an orchestrator
Suppose you had 2 bash scripts which you use to test 2 different components. Without an orchestrator:
- You would have to ship both files to anyone who wants to use them. (imagine if you had 10 scripts, you would have to tar them up and send all).
- If running on a shared system, you can never know if another person already ran the scripts (unless actually going through the effort of seeing if the tests ran).
- Tendency to use lesser number of scripts for easy script execution management, but that makes each script huge. Breaking them down into multiple smaller scripts makes it easy to manage the code but it becomes harder to manage execution of all scripts.
- No way of using common variables across the scripts (like version of the component being tested).
- No way for one developer to look at the logs of a script executed by a second developer (unless the second developer explicitly routes the logs to an external file).
- No way of accidentally preventing execution of a script more than once (like prevent first script from running again if it already ran once).
- Manually execute each script through bash or python (no CLI or UI)
Benefits of using Mozart
- Simple migration to Mozart -
no Go code change necessary
. Just create a directory and dump your scripts in that - it's that easy. (Discussed in detail below) Single binary file which contains all your scripts
AND brings with it the CLI along with the UI - so no need to send in bunch of scripts to anyone anymore.- Lets you
modularize
the scripts, which means you can have more number of smaller scripts which do smaller tasks. No need to maintain huge bash files anymore. The smaller the scripts - the easier it is for you to manage and maintain them. - Ability to use
templating
capabilities - similar to helm. Values in ayaml
file are accessible by all scripts. Public visibility of the state
of execution of scripts, so everyone has a clear idea of whether scripts were executed or not.Central logs
for everyone to see.- Make sure scripts are
executed just once
- the orchestrator will not allow you to run the same script again without explicitly mentioning aRe-Run
flag, thereby preventing accidental execution of the same script. - Ready to use
CLI + UI
to manage the execution.
Using your scripts with Mozart
Let us walk through how to actually add your scripts. There is one term that is of prime importance to Mozart - Modules.
Modules
Mozart works with the concept of modules and not the scripts directly. Modules are nothing but directories, created with the intention of performing 1 simple task. Each module (aka directory) can consist of either scripts or more nested modules (nested directories).
So in short, all your scripts need to be in a module, and Mozart will help you control the execution of those modules (instead of the scripts themselves).
Sample module
There is already a sample module present called test-module
under resources/templates
, which you can use to reference.
Steps
1. Initial setup
- Install go - https://golang.org/doc/install
- Clone repo - https://github.com/countertenor/mozart
- Run the command
export PATH=$PATH:$(go env GOPATH)/bin
) (Please note - to make it persistent, you will have to add the command to your .bashrc) - Create a new directory inside
resources/templates
. This will be the base module under which all your modules will exist. - (Optional) Delete the existing
test-module
insideresources/templates
if you want a clean slate. That folder is only for reference. Leaving that folder as it is will not do any harm.
2. Modularize
-
Look through your scripts, and identify the most basic steps that the scripts are supposed to be performing. Suppose I want to install a component called
Symphony
. I have one huge bash script for that. Some basic steps inside that bash script could be:- Pre-requisite check.
- Installation of component.
- Validation of install.
- Uninstallation of component.
-
For each identified step, create a module(directory) within the base directory and add that part of the script within that directory. For example, continuing with the above example, the directory structure should look like this:
resources/templates ├── symphony (this is the base module) │ ├── 00-pre-req (first sub-module) │ │ └── pre-req.sh │ ├── 10-install (second sub-module) │ │ ├── 00-install-step1.sh │ │ └── 10-install-step2.sh │ ├── 20-validate (third sub-module) │ │ └── validate.sh │ └── 30-uninstall (fourth sub-module) │ └── uninstall.sh
The huge bash script is now broken down into smaller scripts, each in its own module. This makes the script easy to manage, while giving the option to add more scripts in the future as needed.
Note: the
xx-
prefix before a module or script name is an optional prefix, through this you can control the order of execution of scripts/modules within the module.
Note: In case you don't want to break down your script into smaller scripts, you can create only the base module and drop your script in that directory.
3. (Optional) Use templating
You can do something like this within any script:
echo "{{.values.value1}} {{.values.value2}}"
These values are going to be fetched from a yaml
file that you supply while invoking the CLI or the UI. (discussed later)
The yaml
file should have something like this for the example above:
values:
value1: hello
value2: world
When you execute the corresponding module that contains the above script, you will see
echo hello world
This is discussed in detail below.
4. Build the binary
Run make
- builds binaries for linux, mac and centOS environments, inside bin directory.
Voila! You have a single orchestrator binary with all your scripts in it.
Mozart yaml file
Providing an optional yaml file at runtime lets you enable certain templating features and configuration changes. If you do not need any changes, you can skip this section.
A sample blank configuration file can be generated from the binary itself, using the init
command.
./bin/mozart init
Generated sample file : mozart-sample.yaml
Templating
You can use the above yaml
file to help with templating. If you refer to the test-module
under resources/templates
, you can see some examples of templating.
This idea is similar to helm.
Example:
In the file step1.sh
, you see this:
#!/bin/bash
echo "{{.values.value1}} {{.values.value2}}"
The values in the brackets are the values that will be fetched from the yaml
at runtime. So if you want to substitute some values at runtime, you replace the values with the {{ }}
notation as you see above, and in the yaml
file, add:
values:
value1: hello
value2: world
Mozart will substitute these values at runtime.
Optional configuration parameters
There are certain configuration parameters also that you can change using the same yaml
file as above.
Log sub-directory
By default, log files are stored in /var/log/mozart
directory (For linux and centOS), but if for some reason you want to add a sub-directory, you can do so by adding one line to the yaml
file:
Example:
log_path: my-log-dir
Then all the logs will go to:
var/log/mozart/my-log-dir
Exec source
This lets you choose the execution environment of any type of script that you include.
The format is file_ext: source
Example:
exec_source:
py: /usr/bin/python
sh: /bin/bash
This lets Mozart know that if you place any file with the extension of .sh
, then run it using /bin/bash
. If you place any file with the extension .py
, then run it using /usr/bin/python
.
Note: These are the only 2 extensions added by default in Mozart. If you add any other type of script apart from python or bash, you will need to add the execution source in the yaml
.
Delims
This lets you change the default delimiters (default - {{
, }}
)
Example 1:
delims: ["[[", "]]"]
Adding this line in the yaml
file changes the delimiters to [[ ]]
. So after this, you can use templating like:
echo "[[.values.value1]] [[.values.value2]]"
Example 2:
delims: ["<<", ">>"]
Adding this line in the yaml
file changes the delimiters to << >>
. So after this, you can use templating like:
echo "<<.values.value1>> <<.values.value2>>"
Arguments
This lets you pass arguments to the scripts themselves when they are being executed.
Example:
args:
00-step1.sh: ["-s", "hello"]
This makes sure that whenever the 00-step1.sh
file is being executed by Mozart, the -s
and hello
arguments are passed to the file at runtime.
Using common snippets across scripts
There might be a scenario in which some scripts have a lot of common code. It is never a good idea to duplicate logic across scripts (DRY principle).
To tackle this, you can make use of the common
folder that is present under the resources
folder. This folder has one purpose and one purpose only - to hold common snippets of information that will be needed by more than one script.
You can take a look at the static/resources/templates/example-module/10-python-module/00-common-example/python-1.py
file for an example.
Example:
Suppose if you have a function that you want in more than one script, say
def my_func(str):
print('inside funct - ' + str)
Instead of having this function be duplicated across scripts, you add this function as its own file under the common
folder:
╰$ cat static/resources/common/python/my_func
def my_func(str):
print('inside funct - ' + str)
Once the templating engine parses the files, it creates a mapping:
"key" - the filename ("my_func" in this case)
"value" - the content of the file itself
You can then substitute this function in any script using the key:
{{.my_func}}
This will substitute the contents of the file.
More info
-
The files added under the
common
folder are also passes through the templating engine, so you can use templating in the files added to thecommon
folder as well, something like this:echo "{{.values.value1}} {{.values.value2}}"
These values will be parsed through the
yaml
file provided at runtime. -
Sometimes you might want to add indentation to the above substituted lines of code (It is essential in python scripts). You can do so by using
nindent
(courtesy of sprig functions){{.my_func | nindent 4}}
Gotchas
-
If you add a file under the
common
folder with an extension, for example,my_func.py
, the key still remainsmy_func
. That is because the templating engine gets confused when you try to access a key with a.
So in cases where the file has an extension, the engine strips off the extension for the "key" value. -
Names with
-
are not permitted to be used in go templating, therefore filenames with-
in them are not permitted.
Adding custom resources
Sometimes you might want to add files under certain modules which you don't want mozart to execute along with the module, instead these files will be used by your scripts.
In such cases, you can prefix !
to the file names (or prefix !
to an entire directory, in which case all files inside that directory will not be executed)
Example:
ls static/resources/templates/example-module/00-bash-module/02-external-example
!external_file.sh
step1.sh
As you see here, there's a file present with the name !external_file.sh
. Since this file is prefixed with a !
, this won't be executed by mozart when you run the module3
module.
If your script wants to execute this file, you can do so by adding these lines (example code in static/resources/templates/example-module/00-bash-module/02-external-example/step1.sh
):
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd -P )"
bash $DIR/!external_file.sh
OR
You can write this file to a custom location and execute it:
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd -P )"
cat $DIR/!external_file.sh > /your/location/file.sh
bash /your/location/file.sh
Note: The DIR command is needed since the execution directory for the script is not where the script resides. So unfortunately you cannot run a script that's present in the same folder using something like ./script_name
, since .
assumes current execution directory.
CLI
Once you build your binary, Mozart gives you a CLI:
Mozart commands
- init Generate a blank sample config yaml file for the orchestrator
- execute (executes all scripts in specified directory)
- state (displays execution state of all modules, accepts optional args [module name])
- server Starts the REST server
- version (displays version info for the application)
--json (gives output in JSON)
Global flags
-c (optional) (configuration file, defaults to 'mozart-sample.yaml')
-v (optional) (prints verbosely, useful for debugging)
-d, --dry-run (optional) shows what scripts will run, but does not run the scripts
-n, --no-generate (optional) do not generate bash scripts as part of execution, instead use the ones in generated folder. Useful for running local change to the scripts
-p, --parallel (optional) Run all scripts in parallel
-r, --re-run (optional) re-run script from initial state, ignoring previously saved state
Executing modules
Running the binary built in the earlier step, you will see something like this:
$ ./bin/mozart execute -h
Execute scripts inside any folder.
*****************************************
Available commands:
mozart execute symphony-module
mozart execute symphony-module pre-req
mozart execute symphony-module install
mozart execute symphony-module validate
mozart execute symphony-module uninstall
*****************************************
If you select a module that contains other modules, something like mozart execute symphony-module
, that's where the ordering of the sub-modules comes into play, which you control by adding the prefix. Or you can choose to execute a sub-module directly.
Note: Mozart automatically removes any prefix of the form xx-
before the module name.
Checking the state
Once you start the execution, the state
command shows you the current state of execution of the various modules within Mozart, along with other information.
State of all modules
$ ./bin/mozart state
State: {
"generated/symphony-module/00-pre-req": {
"pre-req.sh": {
"startTime": "2020-11-04T15:08:12.5981-08:00",
"timeTaken": "8.218745ms",
"lastSuccessTime": "2020-11-04 15:08:12.606306 -0800 PST m=+0.016747847",
"lastErrorTime": "",
"state": "success",
"logFilePath": "logs/2020-11-04--15-08-12.597-pre-req.log"
}
},
"generated/symphony-module/10-install": {
"00-install-step1.sh": {
"startTime": "2020-11-04T15:08:12.607053-08:00",
"timeTaken": "9.723926ms",
"lastSuccessTime": "2020-11-04 15:08:12.616754 -0800 PST m=+0.027195761",
"lastErrorTime": "",
"state": "success",
"logFilePath": "logs/2020-11-04--15-08-12.606-00-install-step1.log"
},
"10-install-step2.sh": {
"startTime": "2020-11-04T15:08:12.617443-08:00",
"timeTaken": "7.333338ms",
"lastSuccessTime": "2020-11-04 15:08:12.624767 -0800 PST m=+0.035208922",
"lastErrorTime": "",
"state": "success",
"logFilePath": "logs/2020-11-04--15-08-12.617-10-install-step2.log"
}
},
"generated/symphony-module/20-validate": {
"validate.sh": {
"startTime": "2020-11-04T15:08:12.625411-08:00",
"timeTaken": "7.542653ms",
"lastSuccessTime": "2020-11-04 15:08:12.632945 -0800 PST m=+0.043386468",
"lastErrorTime": "",
"state": "success",
"logFilePath": "logs/2020-11-04--15-08-12.625-validate.log"
}
},
"generated/symphony-module/30-uninstall": {
"uninstall.sh": {
"startTime": "2020-11-04T15:08:12.633649-08:00",
"timeTaken": "8.040003ms",
"lastSuccessTime": "2020-11-04 15:08:12.641679 -0800 PST m=+0.052120673",
"lastErrorTime": "",
"state": "success",
"logFilePath": "logs/2020-11-04--15-08-12.633-uninstall.log"
}
}
}
State of specific module
To get the state of a particular module:
./mozart state validate
State: {
"generated/symphony-module/20-validate": {
"validate.sh": {
"startTime": "2020-11-04T15:08:12.625411-08:00",
"timeTaken": "7.542653ms",
"lastSuccessTime": "2020-11-04 15:08:12.632945 -0800 PST m=+0.043386468",
"lastErrorTime": "",
"state": "success",
"logFilePath": "logs/2020-11-04--15-08-12.625-validate.log"
}
}
UI
Developed by @toshakamath
Helpful resources
Good links for templating
- https://golang.org/pkg/text/template/
- https://forum.golangbridge.org/t/referencing-map-values-in-text-template/6253/5
- https://medium.com/@IndianGuru/understanding-go-s-template-package-c5307758fab0
- https://goinbigdata.com/example-of-using-templates-in-golang/
- https://stackoverflow.com/questions/25689829/arithmetic-in-go-templates
- https://stackoverflow.com/questions/21305865/golang-separating-items-with-comma-in-template
- https://helm.sh/docs/chart_template_guide/functions_and_pipelines/
Bash scripts through go
- https://blog.kowalczyk.info/article/wOYk/advanced-command-execution-in-go-with-osexec.html
- https://stackoverflow.com/questions/18986943/in-golang-how-can-i-write-the-stdout-of-an-exec-cmd-to-a-file
- https://stackoverflow.com/questions/19965795/how-to-write-log-to-file
- https://yourbasic.org/golang/format-parse-string-time-date-example/
Versioning with go
Web sockets
- https://www.zupzup.org/io-pipe-go/
- https://gist.github.com/ifels/10392762
- https://godoc.org/github.com/gorilla/websocket
Command execution
- https://medium.com/@vCabbage/go-timeout-commands-with-os-exec-commandcontext-ba0c861ed738
- https://stackoverflow.com/a/58572436
- https://medium.com/@matryer/make-ctrl-c-cancel-the-context-context-bd006a8ad6ff
Error stack
- https://github.com/gruntwork-io/terratest/blob/master/modules/helm/template.go#L83
- https://godoc.org/github.com/go-errors/errors
- https://github.com/gruntwork-io/gruntwork-cli/blob/master/errors/errors.go
Embedding static content
- https://github.com/rakyll/statik/pull/101/files
- golang/go#41191
- https://blog.carlmjohnson.net/post/2021/how-to-use-go-embed/
- https://github.com/akmittal/go-embed (example for react app)
Go build tools
- https://blog.carlmjohnson.net/post/2021/how-to-use-go-embed/
- https://dave.cheney.net/2013/10/12/how-to-use-conditional-compilation-with-the-go-build-tool
- https://www.digitalocean.com/community/tutorials/customizing-go-binaries-with-build-tags
- List all go files:
go list -tags ui -f '{{.GoFiles}}' ./...