Here be dragons!
A baby sitter for your computer system.
Provides a platform for watching your host and performing any corrective actions you wish.
Runs natively.
Configuration consists of a series of TOML files - each file represents a “sytter”. A sytter watches some particular aspect of the host and performs operations based on changes to what the sytter observes. This can be used to automate remediation or track the status of something.
Most operations here are shell operations, but Sytter will allow plugins that make some operations incredibly easy.
Sometimes you’ll see some process that eats up loads of resources. One such
example is com.apple.safari.History. We need to move aside the history files
and restart the service (really just kill the service).
name = "History de-peg"
description = "Sometimes com.apple.safari.History eats a full CPU. Move SQLite
DB files and kill the process."
[trigger]
cron = "0 1 * * * *"
[condition]
process_name = "com.apple.safari.History"
process_resources = "cpu > 90%"
duration = "5m"
[execute]
shell = "mkdir -p ~/Library/Safari/old-history &&
mv ~/Library/Safari/History* ~/Library/Safari/old-history/ &&
kill -9 $(pgrep com.apple.Safari.History)
"
[failure]
shell = "mail -s 'History is pegged but remediation failed' $USER < $errFile"Using a contrived interface-changed executable, track the status of VPN
connectivity with an event listener. Your prompt will need to poll
~/.vpn-status to determine what to display.
name = "Show VPN Connection Status"
description = "Check if we're connected to the VPN and record it for use in our prompt."
[trigger]
trigger = "stdout"
# Some arbitrary executable that prints to stdout when the network interface
# changes.
exec = "interface-changed 'utun*'"
# Pass the stdout to this handler. Sets $sytter_state for use in our [execute]
# section based on what we see from the stdout.
state = "grep added && echo 'online' || echo 'offline'"
[execute]
shell = "echo \"$sytter_state\" > ~/.vpn-status"
[failure]
# This goes to a ~/.sytter-failures and ensures a unique account of this one
# sytter.
list-failure = trueLet’s say $WORK has some weird network segmentation policy in effect, and you
can only communicate with hosts if you’re authenticated via a “captive portal”
system. Since this is $WORK you also have to be on the VPN for it to be
relevant. If there is a failure, use sendmail to send it. We only need one
email per consecutive error.
name = "Captive Portal Authentication"
description = "Periodically authenticate via Captive Portal when on the VPN."
[trigger]
cron = "0 1 * * * *"
[condition]
shell = "[[ \"$(cat $VPN_FILE)\" == true ]]"
[execute]
shell = "~/bin/captivate.sh"
[failure]
repeat = false
shell = "printf '%s\n' 'Subject: Capitive Portal auth failed' \
'' \
'Captive Portal authentication is failing! See $LOG for details.' \
| sendmail $USER"The Shell Sytter component allows shell invocations to do virtually any task.
Shell components expose context variables via environment variables prefixed
with sytter_.
Sytter aims to be a sort of IFTTT that uses standard posix/unix tooling and can be managed via version control. Sytter’s primary goal is to provide a platform with which system health can be monitored and assured, but its uses can be extended beyond baby-sitting systems as a more general automation system (though it could be argued most of these will be some form of a baby-sitter for the system anyways).
Its responsibilities will overlap with many other tools in the adjacent problem space, and indeed could fill their roles. For example, Sytter could serve as a make-shift Puppet agent. Its goal is not to succeed Puppet, however.
Sytter has some principal qualities, and the reasons why they are held as principal qualities:
- Sytter runs natively. a. Runtime changes cannot break Sytter’s core functionality. b. Static linking means Sytter still works across major operating system upgrades. c. No need for tuning a garbage collector, or debugging issues with garbage collectors.
- Sytter structural configuration is very ergonomic. a. Structural configuration is defined as things such as: a. Logging. b. Waiting for consecutive failures. c. Notifications. d. Exponential backoff. b. Structural configuration is desirable across all Sytters and thus must be expressed succinctly in a Sytter configuration. On the axes of simple vs complex and easy vs hard, this should be easy.
Upon startup, sytter reads from --config-dir, $SYTTER_CONFIG_DIR, or
~/.config/sytter for Sytters in that order. Execution of cron statements do
not happen immediately but instead wait for the schedule. Unscheduled operations
happen immediately, and have a soft intention of executing in lexicographical
order. No guarantees are made about this order.
See Order Dependent Sytters for examples of how to handle Sytters that need to execute in a controlled order.
A Sytter declaration is made manifest via various Sytter Components that the Sytter calls upon. These components fall under a few basic categories:
triggerconditionexecutorfailure
A Sytter Component can be stateful. All Sytters components may write to a shared
context. Different kinds of components can be intermixed. For example, using a
ShellCondition does not lock one into using a ShellExecutor.
A Start allows a Sytter to setup initial state. It is run once during the
Sytter’s initialization. Sytter ships with a ShellStart.
A Sytter trigger is some event in which a Sytter is executed. A file could be written to, some resource may become available, or the timer on a polling mechanism may fire. Each of these would be a trigger.
Sytter ships with a cron based trigger and a shell based trigger.
Sytter conditions evaluate the circumstances in which action is required. In the true condition, the Sytter’s executor will be executed. In the false condition, nothing additional happens.
Sytter ships with a shell based condition which (by default) uses a 0 exit code as true and anything else as false.
Sytter executors simply run some piece of functionality. By the time this occurs, the event for the trigger has fired and the condition has evaluated to true.
Sytter ships with a shell based executor.
Sytter failure components describe what the Sytter should do in the case of a failure. Failure is described as some clear error that has occurred at any phase of the Sytter lifecycle. This can include problems setting up the trigger, the condition check failing (error instead of true/false), or the executor fails its operation.
Sytter ships with a shell based failure component.
Structural configuration can be thought of as parts of Sytter which aren’t componentized but instead generalized across all components. Examples of structural configuration include:
- Logging.
- Waiting for consecutive failures.
- Notifications.
- Exponential backoff.
For example, logging is not part of a Sytter component but instead something all components may wish to use. Triggers can universally be configured to wait for a certain number of consecutive failures or some other pattern in the rate at which failures occur.
Sytter supports a shared state and a Sytter can read from and write to this shared state.
All values are Strings and must be parsed for non-String values.
The Shell components provide some helper shell functions for reading and
writing state. At time of writing these functions only support Bash. Other
shells can be supported by default as well as provide a generalized mechanism in
which one can provide their own helpers for unsupported shells.
The sytter-vars function takes a variable number of variable names. Any
renaming must be done in the shell steps themselves.
The example below reads the sytter_bluetooth_enabled variable into an
environment variable of the same name (sytter_bluetooth_enabled). The value
is then printed.
sytter-vars sytter_bluetooth_enabled
echo "Bluetooth enabled? $sytter_bluetooth_enabled"The sytter-write function takes a variable name and a value.
The example below writes the results of blueutil --power to the
sytter_bluetooth_enabled variable.
sytter-write sytter_bluetooth_enabled $(blueutil --power)SIGTERM should begin a graceful shutdown. Listeners are shut down, and the
process waits for any outstanding executions to complete.
SIGQUIT and SIGINT are less graceful. Shut down listeners but immediately
give up on executors.
SIGHUP reloads configuration and Sytters.
Provide other machinery for daemonization. This could mean adding logs, a log destination, additional configuration, etc.
We need a way for Sytters to govern their own state and possibly a global state. This way Sytter components can have a decoupled means of communicating with each other.
We also need to provide a standardized set of variables for common activities, such as a variable for log location, the Sytter information itself, etc.
A Sytter should have some means of being enabled/disabled via some conditions
(evaluated upon startup or SIGHUP).
Additionally, we should allow Sytters to trigger other Sytters to be enabled. This should be carefully thought out. Is it done via variables? Do variable changes cause the enabled field to be re-evaluated?
I was hoping to steer away from yet-another-system-management-tool-powered-by-yaml but TOML is proving to be a little too simple for our uses. It is difficult to express a Sytter with idiomatic TOML and harder still to deserialize the various components. Switching to YAML may prove more useful.
Somehow we need to be able to dynamically load new components. I have no idea how to do this in Rust.
I can be convinced of the value of unit tests in Rust, since they are local to the method and double as documentation. But we also need to have some kind of integration level tests.
We need a build pipeline that produces executables for the big 3 (Linux, macOS, and Windows). Windows I am completely unable to test, so someone else will have to handle that.
We need installers for the big three (Linux, macOS, Windows). I don’t have Windows available to test, so someone else will have to contribute or verify that.
Installers include:
- Homebrew (macOS) - I know there are others and will be happy to support them if interest is expressed.
- Chocolatety (for Windows?).
- An RPM/yum package.
- A deb/apt package.
- A nix derivation.
- Home manager would be neat!
Each of these should support the ability to run daemonized. So that means LaunchServices, a systemd unit file, etc.
How do you author your own components? How do they get added? How can you test them?
What does contribution look like?
More examples of doing cool stuff. Examples can double as a trove of great tools.
All configurations should load fine with symlinks. I feel like this should be a given, but I have seen far too many modern tools that give symlinks special treatment, and thus support is poor to outright refused. As a mission statement we support symlinks. Integration tests will include a symlink test.
This can be done via file watches but really we can just listen for SIGHUP.
More importantly for this item: We need to be able to tear down Sytters and
stand them up again. We should take an MD5 sum of each Sytter and use that as a
basis of whether or not we should attempt a reload for that particular Sytter.