AlphaHydrae / deploy

A (magic) shell script to deploy Git repositories

deploy

A (magic) shell script to deploy Git repositories

Read the annotated source

Repository: AlphaHydrae/deploy

Shamelessly inspired by: visionmedia/deploy

• Usage
• Installation
• Requirements
• Configuration file
  • Single-value and multiple-value keys
  • Environment inheritance
• Hooks
• Configuration file
  • Project
  • SSH connection
  • Host environment variables
• Command line options & environment variables
  • General options
  • Project options
  • SSH options
  • Self-update options
• Command line options format
  • Short option format
  • Long option format
  • Invalid options
  • Changing the working directory
• Default command
• Sub-commands
  • <env> setup
  • <env> rev [rev]
  • [env] config [key]
  • <env> config-all <key>
  • [env] config-section
  • <env> console [path]
  • <env> cleanup
  • <env> exec <cmd>
  • <env> list
  • update [rev]
• Cleaning up old releases

Usage

deploy is a shell script to deploy your Git projects through SSH. Add a deploy.conf file in your project's directory. Here's an example for a Node.js project:

# deploy.conf
[production]
repo  https://github.com/me/my-app
host  my.server.com
user  deploy
path  /var/www/app

# describe how to deploy your app
env          NODE_ENV=production
deploy       npm install --production
post-deploy  npm start

deploy is language-agnostic. For a Rails project, you could replace the last 3 lines with:

env          RAILS_ENV=production
deploy       bundle install --without development test
deploy       rake assets:precompile
post-deploy  rails server

Now run deploy!

deploy production setup
deploy production rev master

It will:

• Connect to my.server.com as the deploy user through SSH
• The setup command will set up a deployment structure and clone your repository
• The second rev command will checkout the latest version of your master branch and run the deployment hooks you defined (deploy and post-deploy in the configuration file)

Read on to learn what to write in the configuration file or how to use each sub-command.

Installation

With npm:

npm install -g bash-deploy

With curl:

PREFIX=/usr/local/bin \
  FROM=https://raw.githubusercontent.com \
  && curl -sSLo $PREFIX/deploy \
  $FROM/AlphaHydrae/deploy/master/bin/deploy \
  && chmod +x $PREFIX/deploy

With wget:

PREFIX=/usr/local/bin \
  FROM=https://raw.githubusercontent.com \
  && wget -qO $PREFIX/deploy \
  $FROM/AlphaHydrae/deploy/master/bin/deploy \
  && chmod +x $PREFIX/deploy

Or download it yourself.

Requirements

deploy is a one-file bash script which requires the following commands: cat, cut, date, git, grep, ls, mkdir, readlink, sed, ssh, tail, tar and wc. Most bash shells have all of those out of the box except perhaps Git.

It also optionally requires the chmod, cp and mktemp commands to update itself.

Configuration file

deploy reads its main configuration from a deploy.conf file in the current directory. It can also be customized with environment variables and command line options.

The configuration file is basically a series of sections containing key/value pairs:

# deploy.conf
[staging]
host  staging.example.com
user  test
# how to deploy
post-deploy  ./run-test.sh

[production]
host 192.168.1.42
user root
# how to deploy
env           NODE_ENV=production
deploy        npm install --production
deploy        npm run build
post-deploy   pm2 start pm2.json

Each named section, delimited by [name], represents an environment (i.e. a host machine) to deploy to, in this example the staging and production environments.

Lines beginning with # are comments and are ignored.

Other lines are key/value pairs. A key is a sequence of characters containing no whitespace, followed by at least one tab or space.

For example, in the line deploy npm run build, the key is deploy and its value is npm run build.

Single-value and multiple-value keys

Some keys like repo, host, port or user are simple configuration properties that have one value per environment. If multiple values are found, the last one is used.

These keys can be overriden through an environment variable of the same name in uppercase and prefixed by DEPLOY_. For example, the environment variable to override the repo key is DEPLOY_REPO.

Other keys like setup, deploy or post-deploy are multiple-value keys used for hooks. They can be present multiple time in the same environment and all their values will be used in order.

Environment inheritance

When deploying your project to multiple environments, there will probably be some configuration properties that are identical for multiple environments. To avoid repetition, an environment can inherit from one or multiple other environments:

# deploy.conf
[common]
user dev
deploy do-stuff
deploy do-more-stuff

[secure]
user sekret
deploy do-sensitive-stuff

[production]
inherits common
inherits secure
user root
deploy just-do-it

In this example, the production environment inherits from common and secure (in that order).

The value of a single-value key like user will be the last value found in the environment inheritance tree. In this case, it will be root for the production environment (the values inherited from the common and secure environments are overwritten).

A multiple-value key like deploy will include all values found in the entire inheritance tree. In this case, it will have 4 values for the production environments, taken in order from the common, secure and production sections:

do-stuff
do-more-stuff
do-sensitive-stuff
just-do-it

For convenience in simple use cases, you can also add a default environment by including a nameless section in your configuration file:

# deploy.conf
[]
user dev
deploy do-stuff
deploy do-more-stuff

[development]
deploy do-it-somehow

[production]
user root
deploy do-it-seriously

All environments that have no inherits key automatically inherit from the default environment.

Hooks

Hooks are user-defined commands that can be run during a deployment phase. There are currently two phases defined: setup and deploy. The setup phase happens when you run the setup command, while the deploy phase happens when you run the rev command.

There are various hooks for each phase: some that run before, some during and some after deploy does its thing. These are the currently available hooks:

pre-setup
post-setup

pre-deploy
deploy
post-deploy

Hooks are multiple-value keys that are optional and can be used as many times as you want. They will all be run in order.

Each hook is run in a specific working directory and has access to various environment variables. $DEPLOY_PATH is always exported and indicates the deployment directory (which is not necessarily the same as the hook's working directory). Additional user-defined variables may also be made available (see environment variables).

Here's an example of how you could use hooks to cache your node_modules directory after every deployment to shorten future installation times:

# restore cache (if present)
deploy       tar -xzf $DEPLOY_PATH/cache.gz -C . || exit 0
# install new dependencies
deploy       npm install --production
# update cache
post-deploy  tar -czf $DEPLOY_PATH/cache.gz node_modules

See the setup and rev commands to learn exactly when and where hooks are executed.

Configuration file

Project

• repo <url> (or the $DEPLOY_REPO variable) defines the Git URL from which your repository will be cloned at setup time.

• path <dir> (or the $DEPLOY_PATH variable) defines the directory into which your project will be deployed on the host.

• keep <n> (or the $DEPLOY_KEEP variable) defines how many releases to keep after deploying. Older releases will be deleted. It must be either all (the default), or an integer greater than zero.

  If it's a number, it indicates the number of releases that should be kept after successful deployment. For example, with keep set to 3, if there are 5 old releases deployed before running the script, 3 will be deleted (so that 2 old releases and the new one being deployed, 3 in total, remain).

  It's also possible to manually trigger cleanup with the cleanup command.

SSH connection

Various SSH options can be specified through the configuration file or environment variables:

• host <address> (or the $DEPLOY_HOST variable) is mandatory for all environments. It indicates which host or IP address to connect to.

• user <name> (or the $DEPLOY_USER variable) specifies the user to connect as. By default, you connect with the same name as your local user.

• identity <file> (or the $DEPLOY_IDENTITY variable) specifies a file from which the private key for public key authentication is read.

• forward-agent yes (or the $DEPLOY_FORWARD_AGENT variable) enables agent forwarding.

• port <number> (or the $DEPLOY_PORT variable) specifies the host port to connect to.

• tty yes (or the $DEPLOY_TTY variable) forces pseudo-terminal allocation.

Commands executed on the host through SSH will be logged to the console.

For some commands, the output may be retrieved and displayed in a more user-friendly manner.

Host environment variables

You can define environment variables that will be exported on the host when executing hooks.

The $DEPLOY_PATH variable is always exported and indicates the deployment directory configured by the user with the path config key or the local $DEPLOY_PATH variable.

• env <NAME>=<VALUE>... defines one or multiple environment variables to export on the host when running hooks.

  # deploy.conf
  env FOO=BAR
  env BAZ=QUX CORGE=GRAULT
  

• forward-env <NAME>... defines the name(s) of one or multiple local environment variables to export on the host when running hooks.

  # deploy.conf
  forward-env FOO
  forward-env BAR BAZ QUX
  

If you have a .env file in your local project directory, it will automatically be sourced. This can be handy to create local variables that you can forward to the host.

# .env
export FOO=BAR
export YEAR=$(date "+%Y")

(Note that if you use the -C, --chdir command line option or the $DEPLOY_CHDIR variable, the .env file is read after the working directory is changed, so setting $DEPLOY_CHDIR in the file has no effect.)

If you have an environment file named after the environment, it will also be sourced. For example, when deploying in the "production" environment, a .env.production file would be sourced if present in the local project directory.

If both .env and .env.production are present, they are sourced in that order.

Command line options & environment variables

The properties in the configuration file can also be specified through command line options or environment variables.

For example, the repo property in the configuration file can also be specified:

• With the -r, --repo <url> command line option.
• With the $DEPLOY_REPO environment variable.

Their precedence is as follows:

• The command line option (e.g. -r, --repo <url>) always takes precedence if specified.
• Next, the environment variable (e.g. $DEPLOY_REPO) takes precedence over the configuration file.
• Otherwise, the value in the configuration file (e.g. repo) is used.

General options

These options are valid for most sub-commands:

• -C, --chdir <dir> (or the $DEPLOY_CHDIR variable) changes deploy's local working directory before loading the configuration file.

• --color always|never|auto (or the $DEPLOY_COLOR variable) enables/disables colors in the output of the script.

  This defaults to auto, which only enables colors if the current terminal is interactive.

• -c, --config <file> (or the $DEPLOY_CONFIG variable) allows you to set a custom path for the configuration file (defaults to ./deploy.conf).

• --help prints usage information and exits.

• -v, -V, --version prints the current version and exits.

• -y, --yes (or the $DEPLOY_YES variable) will automatically accept all confirmation prompts (only valid for the rev, cleanup and update sub-commands).

  Use with caution: old releases may be deleted with the rev or cleanup sub-commands if a keep option is configured.

Project options

These options configure where and how the project is deployed on the remote host:

• -r, --repo <url> (or the $DEPLOY_REPO variable) sets the Git URL to fetch the project's source code from when deploying.

• -P, --path <dir> sets the remote path to deploy the project to on the host (this is the path to the directory managed by deploy, not to the current release).

• -k, --keep <n> (or the $DEPLOY_KEEP variable) changes the number of old releases that are kept after successful deployment (see the keep option).

• -f, --force (or the $DEPLOY_FORCE variable) forces deployment to proceed even when you have uncommitted changes.

SSH options

These options are valid for all commands which connect to the remote host through SSH (setup, rev, console, exec, list and cleanup):

• -A, --forward-agent (or the $DEPLOY_FORWARD_AGENT variable) enables forwarding of the authentication agent connection.

  Use with caution. Users with the ability to bypass file permissions on the remote host (for the agent's UNIX-domain socket) can access the local agent through the forwarded connection. An attacker cannot obtain key material from the agent, however they can perform operations on the keys that enable them to authenticate using the identities loaded into the agent.

• -H, --host <address> (or the $DEPLOY_HOST variable) sets the host to connect to.

• -i, --identity <file> (or the $DEPLOY_IDENTITY variable) selects a file from which the identity (private key) for public key authentication is read.

  The default is ~/.ssh/id_dsa, ~/.ssh/id_ecdsa, ~/.ssh/id_ed25519 and ~/.ssh/id_rsa.

• -p, --port <n> (or the $DEPLOY_PORT variable) sets the port to connect to.

• -t, --tty (or the $DEPLOY_TTY variable) forces pseudo-terminal allocation.

  This can be used to execute arbitrary screen-based programs on a remote machine, which can be very useful, e.g. when implementing menu services.

• -u, --user <name> (or the $DEPLOY_USER variable) sets the remote user to connect as.

Self-update options

These options can be used by deploy to update itself with the update sub-command:

• --update-path <file> (or the $DEPLOY_UPDATE_PATH variable) sets the path to save the updated script when using the update command.

  This overrides the --update-prefix or $DEPLOY_UPDATE_PREFIX options.

• --update-prefix <dir> (or the $DEPLOY_UPDATE_PREFIX variable) sets the base directory when using the update command.

  The updated script will be saved in bin/deploy relative to this path. (Note that setting --update-path or $DEPLOY_UPDATE_PATH overrides this option.)

Command line options format

Note that all of deploy's command line options can be used anywhere in a command. The following 3 commands are equivalent:

• deploy -u deploy production --keep 3 --yes master
• deploy -u deploy --keep 3 --yes production master
• deploy production master -u deploy --keep 3 --yes

Short option format

Some options have a short form which may be specified with one hyphen, e.g. -h to display usage or -f to force deployment with uncommitted local changes.

If a short option takes a value, it may be provided either as the next argument (e.g. -k 5) or with an equal sign (e.g. -k=5).

Long option format

All options have a long form which may be specified with two hyphens, e.g. --help or --force.

If a long option takes a value, it may be provided either as the next argument (e.g. --keep 5) or with an equal sign (e.g. --keep=5).

Invalid options

The following errors will cause the script to be interrupted and print an error:

• Badly formatted or unknown options (e.g. ---foo, -bar, --baz).

• Options that take no value specified with a value (e.g. --help=3).

• Options that take a value specified without their value.

• Duplicate options (e.g. both -k 5 and --keep 3).

Changing the working directory

By default, deploy looks in the current directory to find its configuration file and optional environment files. This can be customized by setting the -c, --chdir <dir> command line option or the $DEPLOY_CHDIR variable.

If this option is set, deploy will first move to that directory before sourcing the environment files and reading the configuration file.

Default command

deploy is usually used this way:

• deploy prod setup
• deploy prod list
• deploy prod rev master

However, since the most often used sub-command is rev, it is also the default command. The following 2 commands are equivalent:

• deploy prod master
• deploy prod rev master

Sub-commands

These are the commands you will use to set up and deploy your projects.

Note that most (but not all) of deploy's sub-commands require an environment to be specified before the actual command name, for example:

deploy setup             # error! no environment
deploy production setup  # all good

<env> setup

Perform first-time setup tasks on the host before deployment. You should only have to run this once before deploying the first time.

deploy production setup

deploy will create the following structure for you in the deployment directory defined by the path config key (or the $DEPLOY_PATH variable):

• $DEPLOY_PATH/releases will contain each deployment's files
• $DEPLOY_PATH/repo will be a bare clone of your Git repository
• $DEPLOY_PATH/tmp will be used to store temporary files during deployment

This phase has 3 hooks that are all executed in the deployment directory. Note that this directory must already exist on the host and be writable by the user you connect as.

This is what deploy will do during setup:

• Run user-defined pre-setup hooks (if any).

• Create the releases and a tmp directories if they don't exist already.

• Clone the repository into the repo directory if it isn't already there.

• Run user-defined post-setup hooks (if any).

<env> rev [rev]

Deploy a new release from the latest changes in the Git repository.

You must provide a Git revision, i.e. a commit, branch or tag to deploy, either as the [rev] argument, with the rev config key or the $DEPLOY_REV variable.

If your Git repository is private, make sure that the deployment user has access to it.

For each deployment, a new release directory will be created in releases in the deployment directory. The name of a release directory is the current date and time in the YYYY-MM-DD-HH-MM-SS format. (You can list deployed releases with the list command.)

Note that deploy will refuse to deploy unless all your local changes are committed and pushed. You can override this behavior with the -f|--force option (or the $DEPLOY_FORCE variable).

This is what deploy will do during deployment:

• If a keep option is configured, all previously deployed releases are listed, with those to be deleted in red, and those to be kept and the current release in green, and confirmation is asked that they may be deleted if the deployment is successful.

• User-defined pre-deploy hooks (if any) are run in the directory of the previous release. You may define pre-deploy hooks to perform any task you might want to do before the actual deployment (e.g. you might want to stop the currently running version or put it into maintenance mode).

• The latest changes are fetched from the repository.

• The new release directory is created and the source is extracted at the specified revision into it.

• User-defined deploy hooks (if any) are run in the new release directory. You should define deploy hooks to build your application or install its dependencies at this stage.

• A symlink of the new release directory is created as current in the deployment directory.

• User-defined post-deploy hooks (if any) are run in the current release directory (which is now the same as the new release directory that was just created).

  You should define post-deploy hooks to execute or start your application at this stage.

• Old releases are deleted according to the keep option (if configured and any were found).

[env] config [key]

Print values from your deploy.conf configuration file.

• If called with no environment and no argument, it prints the whole configuration file:

  $> deploy config
  []
  user dev
  
  [production]
  user root
  port 222

• If called with the environment and a key, it prints the last value of that key for that environment (only the last value is printed even for a multiple-value key):

  $> deploy production config user
  root

  Exits with status 1 if no value is found for the key.

<env> config-all <key>

Print all values of a config key in the current environment and its inherited environments (all values are printed even for a single-value key):

$> deploy production config-all user
dev
root

Exits with status 1 if no value is found for the key.

[env] config-section

Print all values of a config section "as is" (with no inheritance). If no environment is specified, the default config section is printed.

Exits with status 1 if the config section is not found (including the default one).

<env> console [path]

Launch an interactive ssh session on the host.

• If an absolute path is specified, the session starts there.

  deploy production console /var/www

• If no path is specified, the session starts in the deployment directory.

  deploy production console

• If a relative path is specified, it starts there relative to the deployment directory.

  deploy production console current

<env> cleanup

Deletes old releases based on the keep option.

The number of releases to keep is determined from the keep configuration file option, the $DEPLOY_KEEP variable or the -k|--keep command line option.

The releases to delete will be listed and confirmation will be asked before they are actually deleted.

<env> exec <cmd>

Execute the specified command on the host.

deploy production exec ps -ef

<env> list

List the releases that have been deployed on the host.

$> deploy production list
2016-12-24-17-45-23
2017-01-01-02-03-43
2017-04-01-00-00-00

update [rev]

Updates deploy to the latest version by downloading it from the Git repository and installing it at /usr/local/bin/deploy (by default).

In addition to the basic requirements, this sub-command also requires chmod, cp and mktemp to be available in the shell.

The installation path must be a writable file or not exist. If the installation path does not already exist, its parent must be a writable directory.

To perform the update, deploy will download its Git repository into a temporary directory that will be cleaned up when the update is done (or fails).

The correct revision of the script is then copied to the installation path and made executable.

Cleaning up old releases

deploy creates a new release directory for each deployment. By default, it keeps these directories forever, leaving you responsible for cleaning them up.

If given a keep option, it can do it for you automatically after each deployment.

deploy will list the contents of the releases directory on the host.

If no old release is found, deployment will proceed normally.

If the number of old releases is equal to or greater than the keep number, deploy will print the list of releases that will be deleted in red.

Old releases that will be kept will be shown in green.

During deployment, the current release being deployed is also indicated in green (as it counts towards the keep number).

If there are any releases to delete, deploy will ask for confirmation (unless the -y, --yes command line option or the $DEPLOY_YES variable is specified).

Responding "no" interrupts the deployment and prints a help message explaining how to configure the keep number.

The actual cleanup of old releases is always performed after successful deployment.

A single rm command is executed on the host through SSH, deleting all appropriate old release directories in one go.