ansible ansible-vault btrbk btrfs homelab-automation nextcloud

🖴 mipro98/infra 🖴

An Ansible playbook for a docker-based homelab with automatic maintenance and monitoring.

Flexible docker-compose templates for various services like Nextcloud, Vaultwarden, Gitea, Prometheus+Grafana,...

All services are set up ready-to-use with a Traefik reverse proxy and automatic TLS certificates.

Unattended server maintenance with auto updates, S.M.A.R.T monitoring, BTRFS snapshots and email notifications.

Features

This playbook includes tasks for:

Setting up the host system (packages, user, mounts,...).
Deploying all docker services according to their docker-compose templates.
Setting up fully automatic maintenance tasks like:
- BTRFS snapshots and backups to a different drive using the btrbk utility
- BTRFS scrubbing
- S.M.A.R.T. monitoring with email notifications
- automatic system and container updates.

Various configuration can be done using variables defined in group_vars or in vault.yml. For almost all files, templates are used to craft configuration files or docker-compose.yml files according to set variables.

This repo contains ready-to-use container setups for:

Nextcloud (with Mariadb, Redis, Cron, Collabora, Memories incl. hardware transcoding)
A server monitoring stack with Prometheus, Grafana and node_exporter.
Dokuwiki
Gitea
Homer
Vaultwarden
ulogger
OwnTracks (recorder + frontend)
Planka
Traefik v2 as the central reverse proxy.

All these services are setup to be exposed through Traefik on different domains by simply providing domain names for the services in vault.yml.

Quick start

Clone the repo.
Run ./git-init.sh
(optionally: run source unlock-bw.sh)
Deploy (see below)

Deploy everything:

make deploy

Deploy only host os system setup: make system
Deploy only docker-compose changes: make containers
Deploy only maintenance changes: make maintenance

Ansible-Vault

This repo uses ansible-vault to encrypt secret variables used by Ansible (passwords, domain names, etc.). This affects the file vars/vault.yml. A template for vault.yml can be found in vars/vault.yml.template The password for the encryption is obtained by bitwarden-cli in the script vault-pass.sh. Therefore you can encrypt and decrypt by just entering your bitwarden master password by just writing:

make encrypt
make decrpyt

Always remember to encrypt the vault before commiting! Also, absolutely run the script git-init.sh after cloning to install a pre-commit hook which prevents committing the unencrypted vault!

Roles

The repo contains 3 ansible roles:

system
containers
maintenance

`system` role

installs all packages defined in extra_packages
sets correct permissions for docker
sets fstab mount points for a "NAS" and a "BACKUP" drive
sets up smartd with a sane monitoring configuration and email alerts

Note: SSH configuration is skipped for now.

`container` role

Creates the folder structure for docker-compose files according to enabled services
Deploys the docker-compose.yml templates for the services in their respective folders
Makes some special configuration for nextcloud, if nextcloud is enabled

`maintenance` role

At the heart of the maintenance role is the script mpserver-maintenance.sh which gets deployed by ansible. Additionally:

A systemd timer is set up which triggers above script every night at 2 A.M.
The btrbk.conf file is deployed
An uptime-monitor.sh script is deployed which checks continously whether the server is online. When it is offline, it optionally executes an ssh command (I use it to restart my router). After successful reconnect, an E-Mail is sent about the downtime.
A systemd service is installed to start the uptime-monitor.sh script on boot.

Refer to section Automatic Maintenance for more details.

Automatic Maintenance

The custom bash script mpserver-maintenance.sh gets triggered every night at 2 A.M. by systemd. Based on the time (or command line flags), it decides what actions to take:

monthly tasks (when script is triggered on last sunday of a month)
weekly tasks (when script is triggered on a sunday except the last sunday of a month)
daily tasks (when triggered all other days between 0 A.M. and 6 A.M.)

The script can be triggered manually for daily, monthly or weekly tasks. Refer to sudo ./mpserver-maintenance.sh --help.

Usage: ./mpserver-maintenance.sh [<options>] [<command>]

Options:
	--dry-run	Don't execute anything, only show what would be executed
	--no-email	Don't send an email with the log
	--help		Show this help message and exit.

Commands:
	daily		Run daily maintenance tasks
	weekly		Run weekly maintenance tasks
	monthly	Run monthly maintenance tasks

	If no command is passed, the script will determine the correct schedule and run the appropriate tasks.

The tasks are defined in the bash functions weekly, monthly, daily. Each trigger writes a detailed log into ~/maintenance/logs which also gets emailed for monthly and weekly triggers.

Daily tasks:

Snapshot the subvolume "Daten" through btrbk

Weekly tasks:

backup ~/dockerdata onto a backup location through rsync
snapshot and backup all configured subvolumes according to btrbk.conf
update the host system
send an email with a the detailed log.

Monthly tasks:

backup ~/dockerdata onto a backup location through rsync
snapshot and backup all configured subvolumes according to btrbk.conf
update all containers (docker-compose pull all services)
prune old docker images and volumes
btrfs-scrub both the NAS and the BACKUP drive.
update the host system.
send an email with a the detailed log.

Nextcloud stack

The Nextcloud stack is carefully fine-tuned for performance and simplicity using:

A custom built Nextcloud image with ffmpeg, imagemagick and ghostscript for media previews.
MariaDB for a performant database
Redis for fast memory caching and Transactional File Locking (I personally use APCu for caching and Redis only for File Locking)
A dedicated go-vod container with access to /dev/dri for hardware acceleration in Nextcloud Memories.
A dedicated Nextcloud cron container with mounted crontab file so you can easily add cronjobs without messing on the host system's cron/systemd.
A dedicated Collabora container for performant Nextcloud Office integration.

Notes:

The nextcloud container will not auto-update unless you run docker-compose build --pull nextcloud inside ~/docker/nextcloud because the container is built using ~/docker/nextcloud/Dockerfile in order to have ffmpeg support. Since Nextcloud updates often require manual intervention and can easily be discovered through the admin panel, this isn't that much of an issue.
I use Nextcloud Memories alongside Preview Generator and Recognize without issues and with hardware transcoding using the seperate go-vod instance. To make it work, adjust the following in the Memories Admin GUI:
- enable "Images", "HEIC" and "Videos" under "File Support"
- define /usr/bin/ffmpeg / /usr/bin/ffprobe as ffmpeg / ffprobe path
- enable "external transcoder" and just write go-vod:47788 under "Connection address" (The Traefik DNS will resolve go-vod correctly within the docker proxy network).
- tick "Enable acceleration with VA-API" to on. (ignore the warning "VA-API device not found")
- (note that I added the cronjob for preview generator in the mounted crontab-www-data file.)
To enter the Nextcloud container (e.g. to run occ commands), run:
```
docker exec -itu www-data nextcloud bash
```

Notes & Tips

for ulogger:
- The container is setup to use sqlite
- Before the first run: Either use a named volume¹ or start the container once with a bind mount other than /data to copy the container contents of /data onto the host to bootstrap the database.
The makefile supports shorthands for often used Ansible commands. For example, when only modifying mpserver-maintenance.sh, just run make script and it will only replace the script on the server.
The ~/docker/docker-compose.yml, serves as a "master" docker-compose.yml file with all the activated services included in the file. You can use it to execute actions on all containers at once just with one docker-compose <up|down|...> command.
~~Make sure to also re-deploy the maintenance script when adding/removing docker services in group_vars since the script will change according to the enabled services.~~²
Enable debug_ports_open: true to open ports on the host bypassing the reverse proxy. This can be used for debugging, e.g. Traefik metrics and node_exporter is usually not accessible outside of traefik-net.

TODOs

Credits

A large portion of the repository is inspired by Alex Kretzschmar's infra repo as well as Wolfgang notthebee's infra repo. Also, both BTRFS maintenance and the system setup was largely inspired by zilexa's Homeserver Guide.

Thanks also goes to Jeff Geerling's security role where I took almost all of my (not yet activated) SSH configuration.

https://stackoverflow.com/q/65176940 ↩
Not needed anymore since we can use the new include mechanism to generate one "master" docker-compose.yml. ↩

About

My IaC configuration for my homelab.