git.watch

Trigger commands when your Git repository is updated

Usage
Use cases
Architecture
- Technical choices
- Security
Recipes
- For a service
Other notes
Licence

Usage

Supported: GitHub, BitBucket and GitLab

Go to 'WebHooks' section of your GitHub, BitBucket or GitLab repository.
Add the WebHook URL https://git.watch/.
On your target machine, run npm install -g gitwatch-client.
On your target machine, go to the clone of your repository and run git-watch -- ./push.sh.
Make a commit and push to your Git repository.
You should see push.sh executed automatically!

Use git-watch --help to get full run options.

Use cases

I use it for continuous deployment from master branch for:

ActionFPS and Git Work: on push to master, there are two options:
- Fast deployment: Build the project with sbt and deploy it with systemd on a bare metal server.
- Instant deployment: rsync the HTML templates if they are the only change in the branch.
- Example deployment scripts: ActionFPS push.sh, Git Watch push.sh, Git Work push.
ActionFPS:
- Redeploy a simple background Python app on a bare metal server.
- Build a game server binary and deploy it with systemd on a bare metal server.
ScalaWilliam.com (previously):
- Build the static site and deploy it automatically to Firebase.
- However, now I deploy to Firebase via Travis-CI (See a YouTube tutorial) in order to have fewer servers to manage. Unfortunately, deploy speeds are poorer than git watch.
- Before Git Watch and Travis, I used a custom PHP WebHook receiver. However there was too much management involved in getting it to work: Setting up PHP-FPM, nginx, etc, just for simple deployments.
- Git Watch only needs HTTPS access and that's it!
This would also work very well for deploying changes to your VPS/private server for your Wordpress, PHP and other pages. Super easy and simple - just use git pull as the executable command and that's it!

So typical characteristics when you would use Git Watch would be:

You're experimenting
Project is lightweight
You want rapid iterations
You don't need "The Cloud" yet
You have minimal scaling needs
You want to deploy where you build
You want instant content redeploys
You don't need reproducibility right now
You don't want to invest time in a proper CI right now

Why other solutions did not work for my use cases:

I use my own server and find it's easier to debug and tinker with things directly via SSH
My apps tend to use the filesystem and UDP networking
I didn't like using 'The Cloud' due to slow deployments by nature of having to publish and download artifacts. For basic content changes it's a long wait.
I can still use Travis/Circle for automated testing but not for deployment.
From experience Jenkins is quite heavyweight but scalable in the longer run - and I don't need this scalability.
Using WebHooks was painful enough with all the configuration work needed including setting up an nginx and a backend to process for each small service.
Not using continuous deployment is out of question for me. I like receiving immediate feedback. Read: Why Continuous Deployment matters to business. There's no benefit for me to be deploying manually at all.
Non-critical projects, and so deploying straight from master branch is Okay.

Architecture

Your Git repository is updated on the Git host.
The Git host sends a webhook to the Git Watch server.
The Git Watch server broadcasts the repository url to the event stream at https://git.watch/events/.
A Git Watch client listens to the stream, receives the event and then triggers your custom command.

Technical choices

User experience

Linking out to README for getting started: I tried includign a tutorial on the homepage but this meant having to synchronise the README and index.html. I even tried embedding the README as HTML using GitHub's Content API (see the code) but this clearly became cumbersome. So I took the single-source approach.
Embedding a Tweet: Single source of marketing. Links to my Twitter account and I'd like to get more Twitter followers who are into tech and Git Watch sort of thing.
Homepage links to glossaries and definitions of what a Git repository and what commands are: To make it easier for somebody non-technical to understand what is going on. It might still be quite confusing but much better than NOT having any links.
NOT installing the hook automatically: I did this before. It complicated the workflow too much and abstracted away an important thing that the user should find out what they need.; This also required having OAuth code for very little work. Included downloading the list of available repositories etc meaning quite complex and unnecessary code, taking us away from the USP of this software.; In fact, there's room for a dedicated service for selecting repositories.; The extra complication was that we have different providers anyway.
Auto-discovering the URL based on current Git repository: I previously had to enter the URL for each repository explicitly and that was just too bothersome. Why bother if Git can allow you to compute the URL yourself.
Not passing the commit hash to the user: A Git push event is a bit more than just the latest hash. There's a lot of this information and it requires a lot of custom parsing that can't be re-used. Instead of this approach, I realised all we need is just a trigger, and then the user can decide how to deal with the new information available in Git.; Extracting too much information became a scope creep that introduced significant complexity in both the client and the server.

Server side

EventSource HTTP/S transport for passing events from server to client: Easier to use than WebSockets because we can get the result via CURL and basic HTTP. WebSockets require extra application strength on the client side and server side configuration; We only need a one-way flow, which is exactly what EventSource provides.
Play Framework and Scala programming language: I am specialised in Scala and this problem is for me easiest to solve with Play & Scala<./dd>; Play provides us with native EventSource support.; We use ScalaTest as the de facto Scala testing framework.; See: Scala for 2017 talk; See: An Introduction to Scala (2014)
SBT for building the project: Natural fit for Scala and Play.; See: Essential SBT
Code formatting with scalafmt and IntelliJ scalafmt plugin: Helps to make code more readable and consistent.
Recommended IDE: IntelliJ IDEA: Works very well with SBT and Scala. De facto Scala IDE.; Make sure to install the Scala Plugin.

Client side

Node.js: Comes with the most standards-compliant non-web EventSource client. I tried Python, Scala, Jersey, Go, Bash clients which proved unsatisfactory. The Node library worked out the best.; Lightweight and most common language for web developers
yargs for argument parsing: I tried argv but it's a bit unfriendly to use; Allows to pass a full command after --
console-stamp: Realised it's troublesome to look at the logs and not see when an event happened
Mocha and Chai: Documented well enough for me to use without difficulty.; Appears to be the most standard way of testing in Node.js

Security

URLs of updated repositories are sent to the public event stream.

Source IP addresses are checked for BitBucket and GitHub to prevent DOS.

Recipes

For a service

push.sh could be:

#!/bin/bash
git fetch
if [[ $(git diff --name-only HEAD origin/master) != "" ]]; then
    pkill -F .pid || true
    git pull origin master
    pip install --user -r requirements.txt || true
    python -m app &
    echo "$!" > .pid
fi

Now run git-watch with this inside a tmux or screen session and you have an automated process restart.

Other notes

To regenerate TOC we use markdown-toc:

$ npm install -g markdown-toc
$ markdown-toc -i README.md

Licence

Client: MIT, Server: GPLv3.

iamnut / git-watch