skx / marionette

Something like puppet, for the localhost only.

• marionette
• Installation & Usage
• Rule Definition
  • Dependency Management
  • Conditionals
  • Examples
  • Misc. Features
    • Command Execution
    • File Inclusion
    • Pre-Declared Variables
    • Outputs
• Module Types
  • directory
  • docker
  • edit
  • file
  • git
  • group
  • http
    • Outputs
  • link
  • log
  • package
  • shell
    • Outputs
  • sql
  • user
• Future Plans
  • See also
• Github Setup

marionette

marionette is a simple command-line application which is designed to carry out system automation tasks. It was designed to resemble the well-known configuration-management application puppet, albeit in a much simpler and more minimal manner.

marionette contains a small number of built-in modules, providing enough primitives to turn a blank virtual machine into a host running a few services:

• Cloning git repositories.
• Creating/modifying files/directories.
• Pulling Docker images from public container-registries.
• Installing and removing packages.
  • Debian GNU/Linux, and CentOS are supported, using apt-get, dpkg, and yum as appropriate.
• Executing shell commands.
• Making HTTP-requests.

In the future it is possible that more modules will be added, but this will require users to file bug-reports requesting them, contribute code, or the author realizing something is necessary.

Installation & Usage

Binaries for several systems are available upon our download page. If you prefer to use something more recent you can install directly from the repository via:

go install github.com/skx/marionette@latest

The main application can then be launched with the path to a set of rules, which it will then try to apply:

marionette [flags] ./rules.txt ./rules2.txt ... ./rulesN.txt

The following flags are supported:

• -debug
  • Show many low-level details when executing the supplied rules-file(s).
• -verbose
  • Show extra details when executing the supplied rules-file(s).
• -version
  • Show the released version number, and exit.

Typically a user would run with -verbose, and a developer might examine the output produced when -debug is specified.

In addition to the general-purpose flags -dp and -dl exist for developers, to dump the output of the parser and lexer, respectively.

Rule Definition

The general form of our rules looks like this:

$MODULE [triggered] {
            name  => "NAME OF RULE",
            arg_1 => "Value 1 ... ",
            arg_2 => [ "array values", "are fine" ],
            arg_3 => "Value 3 .. ",
}

Each rule starts by declaring the type of module which is being invoked, then there is a block containing "key => value" sections. Different modules will accept/expect different keys to configure themselves. (Unknown arguments will generally be ignored.)

A rule may also contain an optional triggered attribute. Rules which contain the triggered modifier are not executed unless explicitly invoked by another rule - think of it as a "handler" if you're used to ansible.

Here is an example rule which executes a shell-command:

# Run a command, unconditionally
shell {
        command => "uptime > /tmp/uptime.txt"
}

Another simple example to illustrate the available syntax might look like the following, which ensures that I have ~/bin/ owned by myself:

directory {
             target  => "/home/${USER}/bin",
             mode    => "0755",
             owner   => "${USER}",
             group   => "${USER}",
}

There are four magical keys which can be supplied to all modules:

Name	Usage
require	This is used for dependency management
notify	This is used for dependency management
if	This is used to make a rule conditional
unless	This is used to make a rule conditional

Dependency Management

There are two keys which can be used to link rules together, to handle dependencies:

• require
  • This key contains either a single rule-name, or a list of any rule-names, which must be executed before this one.
• notify
  • A list of any number of rules which should be notified, if the given rule resulted in a state-change.

Note You only need to give rules names to link them for the purpose of managing dependencies.

Imagine we wanted to create a new directory, and write a file there. We could do that with a pair of rules:

• One to create a directory.
• One to generate the output file.

We could wing-it and write the rules in the logical order, but it would be far better to link the two rules explicitly.

There are two ways we could implement this. The simplest way would be this:

shell { command   => "uptime > /tmp/blah/uptime",
        require   => "Create /tmp/blah" }

directory{ name   => "Create /tmp/blah",
           target => "/tmp/blah" }

The alternative would have been to have the directory-creation trigger the shell-execution rule via an explicit notification:

# This command will notify the "Test" rule, if it creates the directory




# because it was not already present.
directory{ target => "/tmp/blah",
           notify => "Test"
}




# Run the command, when triggered/notified.
shell triggered { name         => "Test",
                  command      => "uptime > /tmp/blah/uptime",
}

The difference in these two approaches is how often things run:

• In the first case we always run uptime > /tmp/blah/uptime
  • We just make sure that before that the directory has been created.
• In the second case we run the command only once.
  • We run it only after the directory is created.
  • Because the directory-creation triggers the notification only when the rule changes (i.e. the directory goes from being "absent" to "present").

You'll note that any rule which is followed by the token triggered will only be executed when it is triggered by name. If there is no notify key referring to that rule it will never be executed.

Conditionals

Rules may be made conditional, via the magical keys if and unless.

The following example runs a command, using apt-get, only if a specific file exists upon the filesystem:

shell { name    => "Upgrade",
        command => "apt-get dist-upgrade --yes --force-yes",
        if      => exists("/usr/bin/apt-get") }

For the reverse, running a rule unless something is true, we can use the unless key:

let arch = `/usr/bin/arch`

file { name   => "Create file",
       target => "/tmp/foo",
       unless => equal( "x86_64", "${arch}" ) }

Here we see that we've used two functions equal and exists, these are both built-in functions which do what you'd expect.

The following list shows all the built-in functions that you may use (but only within the if or unless keys):

• contains(haystack, needle)
  • Returns true if the first string contains the second.
• exists( /some/file )
  • Return true if the specified file/directory exists.
• equal( foo, bar )
  • Return true if the two values are identical.
• nonempty(string|variable)
  • Return true if the string/variable is non-empty.
  • set is a synonym.
• prompt(string|variable)
  • Prompt the user for input, at run-time, and return it.
• on_path(string|variable)
  • Return true if a binary with the given name is on the users' PATH.
• empty(string|variable)
  • Return true if the string/variable is empty (i.e. has zero length).
  • unset is a synonym.
• success(string)
  • Returns true if the command string is executed and returns a non-error exit-code (i.e. 0).
  • Output is discarded, and not captured.
• failure(string)
  • Returns true if the command string is executed and returns an error exit-code (i.e. non-zero 0).
  • Output is discarded, and not captured.

More conditional primitives may be added if they appear to be necessary, or if users request them.

Conditionals may also be applied to variable assignments and file inclusion:

# Include a file of rules, on a per-arch basis
include "x86_64.rules" if equal( "${ARCH}","x86_64" )
include "i386.rules"   if equal( "${ARCH}","i386" )




# Setup a ${cmd} to download something, depending on what is present.
let cmd = "curl --output ${dst} ${url}" if on_path("curl")
let cmd = "wget -O ${dst} ${url}"       if on_path("wget")

In addition to these conditional functions the following primitives are built in, and may be freely used:

• field(txt,index)
  • Split the given text on whitespace, and return the specified field by index.
  • 0 is the first field, 1 is the second, etc.
• gt(a,b)
  • Return true if a>b
• gte(a,b)
  • Return true if a>=b
• lt(a,b)
  • Return true if a<b
• lte(a,b)
  • Return true if a<=b
• len(txt)
  • Return the length of the given value.
• lower(txt)
  • Converts the given string to lower-case.
• matches(text, regexp)
  • Return true if the text matches the specified regular expression.
• rand(min,max,seed)
  • Return a random integer between min and max. Optionally set a seed value.
• md5sum(txt)
  • Returns the MD5-digest of the given value.
• sha1sum(txt)
  • Returns the SHA1-digest of the given value.
• upper(txt)
  • Converts the given string to upper-case.

Examples

You can find a small set of example recipes beneath the examples directory:

• examples/

Misc. Features

Command Execution

Backticks can be used to execute commands, in variable-assignments and in parameters to rules.

For example we might determine the system architecture like this:

let arch = `/usr/bin/arch`

shell { name    => "Show arch",
        command => "echo We are running on an ${arch} system" }

Here ${arch} expands to the output of the command, as you would expect, with any trailing newline removed.

Note ${ARCH} is available by default, as noted in the pre-declared variables section. This was just an example of command-execution.

Using commands inside parameter values is also supported:

file { name    => "set-todays-date",
       target  => "/tmp/today",
       content => `/usr/bin/date` }

The commands executed with the backticks have any embedded variables expanded before they run, so this works as you'd expect:

let fmt   = "+%Y"

file { name    => "set-todays-date",
       target  => "/tmp/today",
       content => `/bin/date ${fmt}` }

Include Files

You can break large rule-files into pieces, and include them in each other:

# main.in

let prefix="/etc/marionette"

include "foo.in"
include "${prefix}/test.in"

To simplify your recipe writing including other files may be made conditional, just like our rules:

# main.in

include "x86_64.rules" if equal( "${ARCH}","x86_64" )
include "i386.rules"   if equal( "${ARCH}","i386" )

Pre-Declared Variables

The following variables are available by default:

Name	Value
${ARCH}	The system architecture (as taken from sys.GOARCH).
${HOMEDIR}	The home directory of the user running marionette.
${HOSTNAME}	The hostname of the local system.
${OS}	The operating system name (as taken from sys.GOOS).
${USERNAME}	The username of user running marionette.

There are additionally two "magic" variables available which will always have values based upon the current rule-file being processed, whether that is a file specified upon the command-line, or as a result of an include statement:

Name	Value
${INCLUDE_DIR}	The absolute directory path of the current file being processed.
${INCLUDE_FILE}	The absolute path of the current file being processed.

Outputs

Some modules will set "outputs" after they've executed, and those outputs will be documented explicitly in the later list of available modules.

When a module creates an output it will be available for subsequent modules to use, prefixed with the name of the rule which created it.

Here is an example showing the use of the stdout output which the shell module produces:

# Run a command - This will produce "${user.stdout}" and "${user.stderr}"




# variables which can be used later.
shell {
           name => "user",
        command => "/usr/bin/whoami"
}


log {
    message => "STEVE!",
    if      => equal( "${user.stdout}", "skx" )
}
log {
    message => "ROOT!",
    if      => equal( "${user.stdout}", "root" )
}

NOTE: The output stdout is available here as ${user.stdout} - the "user." prefix comes from the name of the rule which invoked the shell. So here we'd be able to process the result of ${kernel-version.output}

shell {
          name => "kernel-version",
          command => "uname -r",
}

Module Types

Our primitives are implemented in 100% pure golang, and are included with our binary, these are now described briefly:

directory

The directory module allows you to create a directory, or change the permissions of one.

Example usage:

directory {  name    => "My home should have a binary directory",
             target  => "/home/steve/bin",
             mode    => "0755",
}

Valid parameters are:

• target is a mandatory parameter, and specifies the directory, or directories, to operate upon.
• owner - Username of the owner, e.g. "root".
• group - Groupname of the owner, e.g. "root".
• mode - The mode to set, e.g. "0755".
• state - Set the state of the directory.
  • state => "absent" remove it.
  • state => "present" create it (this is the default).

docker

This module allows fetching a container from a remote registry.

docker { image => "alpine:latest" }

The following keys are supported:

• image - The image/images to fetch.
• force
  • If this is set to true then we fetch the image even if it appears to be available locally already.

NOTE: We don't support private registries, or the use of authentication.

edit

This module allows minor edits to be applied to a file:

• Removing lines matching a given regular expression.
• Appending a line to the file if missing.

edit { name => "Remove my VPN hosts",
       target => "/etc/hosts",
       remove_lines => "\.vpn" }

The following keys are supported:

• target - Mandatory filename to edit.
• remove_lines - Remove any lines of the file matching the specified regular expression.
• append_if_missing - Append the given text if not already present in the file.
• search
• replace
  • If both search and replace are non-empty then they will be used to update the content of the specified file.
  • search is treated as a regular expression, for added flexibility.

An example of changing a file might look like this:

edit { target  => "/etc/ssh/sshd_config",
       search  => "^PasswordAuthentication",
       replace => "# PasswordAuthentication",
}

fail

The fail-module is designed to terminate processing, if you find a situation where the local environment doesn't match your requirements. For example:

let path = `which useradd`

fail {
   message => "I can't find a working useradd binary to use",
   if      => empty(path)
}

The only valid parameter is message.

See also log, which will log a message but then continue execution.

file

The file module allows a file to be created, from a local file, or via a remote HTTP-source.

Example usage:

file {  name       => "fetch file",
        target     => "/tmp/steve.txt",
        source_url => "https://steve.fi/",
}

file {  name     => "write file",
        target   => "/tmp/name.txt",
        content  => "My name is Steve",
}

target is a mandatory parameter, and specifies the file to be operated upon.

There are four ways a file can be created:

• content - Specify the content inline.
• source_url - The file contents are fetched from a remote URL.
• source - Content is copied from the existing path.
• template - Content is produced by rendering a template from a path.

Other valid parameters are:

• owner - Username of the owner, e.g. "root".
• group - Groupname of the owner, e.g. "root".
• mode - The mode to set, e.g. "0755".
• state - Set the state of the file.
  • state => "absent" remove it.
  • state => "present" create it (this is the default).

Where template is used, the template file is rendered using the text/template Go package

git

Clone a remote repository to a local directory.

Example usage:

git { path => "/tmp/xxx",
      repository => "https://github.com/src-d/go-git",
}

Valid parameters are:

• repository Contain the HTTP/HTTPS repository to clone.
• path - The location we'll clone to.
• branch - The branch to switch to, or be upon.
  • A missing branch will not be created.

If this module is used to notify another then it will trigger such a notification if either:

• The repository wasn't present, and had to be cloned.
• The repository was updated. (i.e. Remote changes were pulled in.)

group

The group module allows you to add or remove local Unix groups to your system.

Example:

group { group => "sysadmin",
        state => "present" }

• elevate is an optional parameter, which should contain the path to "sudo", or similar program to grant root-privileges.
• group is a mandatory parameter.
• state should be one of absent or present, depending upon whether you want to add or remove the group.

http

The http module allows you to make HTTP requests.

Example:

http { url => "https://api.github.com/user",
       method => "PATCH",
       headers => [
         "Authorization: token ${AUTH_TOKEN}",
         "Accept: application/vnd.github.v3+json",
       ],
       body => "{\"name\":\"new name\"}",
       expect => "200" }

Valid parameters are:

• url is a mandatory parameter, and specifies the URL to make the HTTP request to.
• method - If this is set, the HTTP request will use this method, otherwise a GET request is made.
• headers - If this is set, the headers provided will be sent with the request.
• body - If this is set, this will be the body of the HTTP request.
• expect - If this is set, an error will be triggered if the response status code does not match the expected status code.
  • If expect is not set, an error will be triggered for any non 2xx response status code.

The http module is always regarded as having made a change on a successful request.

http Outputs

The following outputs will be set:

• body
  • The body returned from the request.
• code
  • The HTTP-status code of the response.
• status
  • The HTTP-status line of the response.

link

The link module allows you to create a symbolic link.

Example usage:

link { name => "Symlink test",
       source => "/etc/passwd",  target => "/tmp/password.txt" }

Valid parameters are:

• target is a mandatory parameter, and specifies the location of the symlink to create.
• source is a mandatory parameter, and specifies the item the symlink should point to.

log

The log module allows you to output a message, but continue execution.

Example usage:

log { message => "I'm ${USER} running on ${HOSTNAME}" }

The only valid parameter is message, which may be either a single value, or an array of values.

See also fail, which will log a message but then terminate execution.

package

The package-module allows you to install or remove a package from your system, via the execution of apt-get, dpkg, and yum, as appropriate.

Example usage:

# Install a single package
package { name    => "Install bash",
          package => "bash",
          state   => "installed",
        }




# Uninstall a series of packages
package { package => [ "nano", "vim-tiny", "nvi" ],
          state => "absent" }

Valid parameters are:

• elevate is an optional parameter, which should contain the path to "sudo", or similar program to grant root-privileges.
• package is a mandatory parameter, containing the package, or list of packages.
• state - Should be one of installed or absent, depending upon whether you want to install or uninstall the named package(s).
• update - If this is set to true then the system will be updated prior to installation.
  • In the case of a Debian system, for example, apt-get update will be executed.

sql

The SQL-module allows you to run arbitrary SQL against a database. Two parameters are required driver and dsn, which are used to open the database connection. For example:

sql {
       driver => "sqlite3",
       dsn    => "/tmp/sql.db",

       # Memory based-SQLite could be used like so:
       #  dsn   => "/tmp/foo.db?mode=memory",

       # MySQL connection would look like this:
       #
       #   driver => "mysql",
       #   dsn    => "user:password@(127.0.0.1:3306)/",
}

We support the following drivers:

• mysql
• postgres
• sqlite3

To specify the query to run you should set one of the following two parameters:

• sql
  • Literal SQL to execute.
• sql_file
  • A file to read and execute in one execution.

NOTE: You may find you need to append multiStatements=true to your DSN to ensure correct operation when reading SQL from a file.

shell

The shell module allows you to run shell-commands, complete with redirection and pipes.

Example:

shell { name    => "I touch your file.",
        command => "touch /tmp/blah/test.me"
      }

command is the only mandatory parameter. Multiple values can be specified:

shell { name    => "restart the aplication",
        command => [
                     "systemctl stop  foo.service",
                     "systemctl start foo.service",
                   ]
      }

By default commands are executed directly, unless they contain redirection-characters (">", or "<"), or the use of a pipe ("|"). If special characters are used then we instead invoke the command via /bin/bash:

• bash -c "${command}"

You may specify shell => true to force the use of a shell, despite the lack of redirection/pipe characters:

shell { shell   => true,
        command => "sed .. /etc/file.txt"
      }

shell Outputs

The following outputs will be set:

• stdout
  • Anything the command wrote to STDOUT.
• stderr
  • Anything the command wrote to STDERR.

user

The user module allows you to add or remove local Unix users to your system.

Example:

user { login => "steve",
       state => "present" }

• elevate is an optional parameter, which should contain the path to "sudo", or similar program to grant root-privileges.
• login is a mandatory parameter.
• shell is an optional parameter to use for the users' shell.
• state should be one of absent or present, depending upon whether you want to add or remove the user.

Future Plans

• Gathering "facts" about the local system, and storing them as variables would be useful.
  • At the moment we just have a small number of pre-declared variables.

See Also

• There are some brief notes on our implementation contained within HACKING.md.
  • This gives an overview of the structure.
• There are some brief-notes on fuzz-testing the parser.
  • This should ensure that there is no input that can crash our application.

Github Setup

This repository is configured to run tests upon every commit, and when pull-requests are created/updated. The testing is carried out via .github/run-tests.sh which is used by the github-action-tester action.

Releases are automated in a similar fashion via .github/build, and the github-action-publish-binaries action.

Steve