Managing my personal server in 2020
Summary
This document is going to describe how I manage my personal server in 2020. It will talk about
- Management of secrets with SOPS and a GPG key
- Automatic management of DNS record
- Configuration of Debian and installation of Kubernetes k3s
- Setup Nginx ingress with let's encrypt for automatic TLS certificate
- Deployment of postfix + dovecot for deploying an email server
- Install Nextcloud to get your personal cloud in the sky
- Putting backup in place
- Using Wireguard to create a private network and WsTunnel to bypass firewalls
- Adding a Raspberry Pi to the K3s cluster
My goals for this setup are:
- Simple to deploy, manage and update
- Everything should live inside the git repository
- Automating as much as possible with free tier service (GitHub actions) but reproducible locally
Table of Contents
- The road so far
- Creating GPG key
- Encrypting secrets with Sops
- Generating a new ssh key
- Automating installation with a Makefile
- Chose a server provider
- Secure and Automate installation of the base machine
- Chose your registrar for DNS
- Automate your DNS record update
- Installing Kubernetes K3S
- Nginx as Ingress controller for Kubernetes
- CertManager with let's encrypt for issuing TLS certificates
- Mail Server with Postfix + Dovecot + Fetchmail + SpamAssassin
- Automating build and push of our images with GitHub actions
- Hosting your own cloud with nextcloud
- Backups
- [TODO] Monitoring with netdata
- VPN with Wireguard
- Bypass firewalls with WsTunnel
- Raspberry Pi as k8S node using your Wireguard VPN
- Deploying PiHole on your Raspberry Pi
- Conclusion
- If you want more freedom
The road so far
<iframe width="560" height="315" src="https://www.youtube.com/embed/P5ZJui3aPoQ" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>It has been more than 15 years now that I manage my own dedicated server, I am in my thirties, and it all started thanks to a talk from Benjamin Bayart Internet libre, ou minitel 2.0 or for the non-French "Free internet or minitel 2.0". For those who don't know what is a minitel, let me quote for you Wikipedia.
The Minitel was a videotex online service accessible through telephone lines, and was the world's most successful online service prior to the World Wide Web. It was invented in Cesson-Sévigné, near Rennes in Brittany, France.
In the essence, the talk is about creating awareness in 2007 that the internet is starting to lose its decentralized nature and look like more to a minitel 2.0 due to our reliance on centralized big corp for about everything on the Web. This warning ring even louder nowadays with the advent of the Cloud where our computers are now just a fancy screen for accessing data/compute remotely.
I went from hardcore extremist, by hosting a server made from scrap materials behind my parent house telephone line, using Gentoo to recompile everything, control every USE flags and have the satisfying pleasure of adding -mtune=native to the compilation command line. Some years later, after being fed up with having to spend nights to recompile everything on an old Intel Pentium 3 because I missed a USE flag that was mandatory to try this new software, I switched to Debian.
At that point I thought I had the perfect setup, just do an apt-get install and you have your software installed in a few minutes, is there anything more than that really ?
It was at that time also that I switched from hosting my server at my parent house to a hosting company. I was out for college and calling my parents to ask them to reboot the machine because it froze due to aging components was taking too much time. I was living in the constant fear of losing some emails and friends on IRC were complaining that the archive/history of the channel that my server was providing was not accessible anymore. So as hard as the decision had been, especially since everything was installed by hand without configuration management, I went to see my parents to tell them that I am removing the server from their care to host it on online.net and that they should expect even fewer calls from me from now on.
Rich of this new available bandwidth and after porting my manual deployments to Ansible, I really thought this time I had the perfect setup. Easy to install and configured management ! Is there anything more than that really ?
I had found my cruise boat and sailed peacefully with it until the dependencies monsters knocked me off board. When you try to CRAM everything (mail, webserver, gitlab, pop3, imap, torrent, owncloud, munin, ...) into a single machine on Debian, you ultimately end-up activating unstable repository to get the latest version of packages and end-up with conflicting versions between softwares to the point that doing an apt-get update && apt-get upgrade is now your nemesis.
While avoiding system upgrade, I spent some time playing with Kvm/Xen, FreeBSD jails, Illumos, micro-kernel (I thought it will be the future :x) and the new player in town Docker ! I ended-up using Docker due to being too busy/lazy to reinstall everything on something new and Docker allowed me to progressively isolate/patch the software that were annoying me. Hello Python projects !
This hybrid setup worked for a while, but it felt clunky to manage, especially with Ansible in the mix. I ended-up moving everything into containers, not without hassle, and now Ansible was felling at odd and the integration between systemd and Docker weird, I was just spending time gluing trivial things.
So I spent a bit of time this month to create and share with you my perfect new setup for managing a personal server in 2020 !
Creating a GPG key
So let's start. The first step is to create a GPG key. This key will serve to encrypt every secret we have in order to be able to commit them inside the git repository. With secrets inside git, our repository will be able be standalone and portable across machines. We will be able to do a git clone and get working !
gpg --full-generate-keyThis GPG key will be the guardian of your infrastructure, if it leaks, anybody will be able to access your infra. So store it somewhere safe
gpg --armor --export erebe@erebe.eu > pub.asc
gpg --armor --export-secret-key erebe@erebe.eu > private.ascEncrypting secrets with sops
Now that we have a PGP key, we will use the wonderful tool SOPS to encrypt our secrets with it.
Sops is not very well known, but very practical and easy to use, which is a great plus for once in a security tool.
To use it, create a config file at the root of your repository
❯ cat .sops.yaml
creation_rules:
- gpg: >-
YOUR_PGP_FINGERPRINT_WITHOUT_SPACEAfter that just invoke sops to create a new secret with your GPG key. Sops force the use of YAML, so your file need to be a valid YAML.
❯ mkdir secrets secrets_decrypted
❯ sops secrets/foobar.yml
* editor with default values *
❯ cat secrets/foorbar.yml # content of file is encrypted now
hello: ENC[AES256_GCM,data:zpzQz+siZxcshJjmi4PBvX2GMm3sWibxRPCgil2mi+c6AQ0uXEBLM2lL0o+BBg==,
...To decrypt your secrets just do a
sops -d --output secrets_decrypted/foobar.yml secrets/foorbar.ymlThere are other commands that allow you to avoid dumping your decrypted secrets onto the file system. If you are interested in this feature look at
sops exec-env
# or
sops exec-fileGenerating a new ssh key
Now that we are able to store secrets securely within our repository, it is time to generate a new ssh key in order to be able to log to our future next server.
We are going to set a passphrase to our ssh keys and use a ssh-agent/keychain in order to avoid typing it every time
# Don't forget to set a strong passphrase and change the default name for your key from id_rsa to something else, it will be usefull later on
ssh-keygen
# To add your ssh ke into the keyring
eval $(keychain --eval --agents ssh ~/.ssh/your_private_key)We are going to commit this ssh key into the repository with sops
sops secrets/ssh.yml
# edit the yaml file to get 2 section for your private and public ssh key
# paste the content of your keys in this section
git add secrets/ssh.yml
git commit -m 'Adding ssh key'Automating installation with a Makefile
Now we want for this repository to be self-contained and easily portable across machines. A valid approach would have been to use Ansible in order to automate our deployment. But we will not tap a lot into the full power of a configuration management in this setup, so I chose to use a simple makefile to automate the deployment.
❯ mkdir config
❯ cat Makefile
.PHONY: install
install:
sops -d --extract '["public_key"]' --output ~/.ssh/erebe_rsa.pub secrets/ssh.yml
sops -d --extract '["private_key"]' --output ~/.ssh/erebe_rsa.key secrets/ssh.yml
chmod 600 ~/.ssh/erebe_rsa.*
grep -q erebe.eu ~/.ssh/config > /dev/null 2>&1 || cat config/ssh_client_config >> ~/.ssh/config
mkdir ~/.kube || exit 0
The installation section is decrypting the ssh keys, install them and looking into my ~/.ssh/config to see if I already have a section for my server in order to add it if missing. With that I will be able to do a ssh my-server and get everything setup correctly
Chose a server provider
We have a git repository with our ssh keys, so now is the time to use those keys and get a real server behind it.
Personally I use a 1st tier dedibox from online.net now renamed into scaleway for 8€ per month. Their machine is rock solid, cheap and never had an issue with them since more than 15 years. You are free to chose whatever provider you want but here my recommendations for the thing to look at
- Disk space: Using containers consume a lot of disk space. So take a machine with at least 60G of space.
- Public bandwidth limitation: All hosting company throttle public bandwidth to avoid issue with torrent seedbox. So the higher you get for the same price, the better it is (i.e: scaleway provide 250Mbits/s while OVH only 200Mbits)
- Free backup storage: At some point we will have data to backup, so look if they provide some external storage for backups
- IPv6: They should provide IPv6, not mandatory but it is 2020
- Domain name/Free mail account: If you plan to use them as a registrar for your domain name, look if they can provide you email account storage in order to configure them as fallback to not lose mail
Once you have your server provider, do the installation and choose Debian for the OS. At some point we will ask you for your ssh key, so provide the one you created earlier. If you have the possibility to select your filesystem use XFS instead of ext4 as it provides good support
If everything is installed correctly you should be able to do a
ssh root@ip.of.the.serverSecure and automate installation of base machine
The machine is in place and reachable to the outside world. First thing to do is secure it ! We want to :
- Allow automatic security update
- Tighten SSH server access
- Restrict network access
Enable automatic security update
Let's start by enabling the automatic security update of Debian. In our Makefile
HOST=${my-server}
.PHONY: install package
# ...
package:
ssh ${HOST} 'apt-get update && apt-get install -y curl htop mtr tcpdump ncdu vim dnsutils strace
linux-perf iftop'
# Enable automatic security Updates
ssh ${HOST} 'echo "unattended-upgrades unattended-upgrades/enable_auto_updates boolean true" | de
bconf-set-selections && apt-get install unattended-upgrades -y'
With that the machine will install security update by its own, without requesting us to type manually apt-get update && apt-get upgrade
Secure SSH server
Next is improving the security of our ssh server. We are going to disable password authentication and allowing only public key authentication. As our ssh keys are encrypted in our repository, they will be always available to us if needed, as long as we have the GPG key.
The main config options for your sshd_config
PermitRootLogin prohibit-password
PubkeyAuthentication yes
AllowUsers erebe root
X11Forwarding no
StrictModes yes
IgnoreRhosts yesAs I don't use any configuration management (i.e: Ansible), It is kind of tedious to use a normal user and leverage privilege escalation (sudo) to do stuff in root. So I allow root login on the SSH server to make things easier to manage. If you plan to use a configuration management system, disable the root login authentication.
Now let's use again our Makefile to automate the deployment of the config
Warning Be sure that you are correctly able to log with your ssh key before doing that or you will need to reinstall your machine/use the rescue console of your hosting provider to fix things
❯ Makefile
.PHONY: install package ssh
#...
# Check if the file is different from our git repository and if it is the case re-upload and restart the ssh server
ssh:
ssh ${HOST} "cat /etc/ssh/sshd_config" | diff - config/sshd_config \
|| (scp config/sshd_config ${HOST}:/etc/ssh/sshd_config && ssh ${HOST} systemctl restartif you want to go a step further, you can
- Change default ssh server port
- Disallow root authentication
- Enable 2-factor authentication with google-authentificator (it does not contact google)
Secure Network access
Last part of the plan is to secure the network by putting in place firewall rules.
I want to stay close to the real things, so I use directly iptables to create my firewall rules. This is at the cost of having to duplicate the rules for IPv4 and IPv6.
If you want to simplify you the task please use UFW - Uncomplicated Firewall
We want for our deployment of iptables rules to be idempotent, so we are going to create a custom chain to avoid messing with the default one.
Also, I am going to use iptables command directly instead of iptable-restore, because iptables-restore files need to be holistic and does not compose well when programs manage only a subpart of the firewall rules. As we are going to install Kubernetes later on, it will allow us to avoid messing with proxy rules.
#!/bin/sh
# Execute only when it is for our main NIC
[ "$IFACE" != "enp1s0" ] || exit 0
# In order to get an IPv6 lease from online.net
sysctl -w net.ipv6.conf.enp1s0.accept_ra=2
###########################
# IPv4
###########################
# Reset our custom chain
iptables -P INPUT ACCEPT
iptables -D INPUT -j USER_CUSTOM
iptables -F USER_CUSTOM
iptables -X USER_CUSTOM
iptables -N USER_CUSTOM
# Allow loopback interface
iptables -A USER_CUSTOM -i lo -j ACCEPT
# Allow wireguard interface
iptables -A USER_CUSTOM -i wg0 -j ACCEPT
# Allow Kubernetes interfaces
iptables -A USER_CUSTOM -i cni0 -j ACCEPT
iptables -A USER_CUSTOM -i flannel.1 -j ACCEPT
# Allow already accepted connections
iptables -A USER_CUSTOM -p tcp -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A USER_CUSTOM -p udp -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A USER_CUSTOM -p icmp -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
# Accept incoming ICMP - Server provider use ping to monitor the machine
iptables -A USER_CUSTOM -p icmp -j ACCEPT
# Allow ssh
iptables -A USER_CUSTOM -p tcp --dport 22 -j ACCEPT
# Allow http/https
iptables -A USER_CUSTOM -p tcp --dport 80 -j ACCEPT
iptables -A USER_CUSTOM -p tcp --dport 443 -j ACCEPT
# Allow SMTP and IMAP
iptables -A USER_CUSTOM -p tcp --dport 25 -j ACCEPT
iptables -A USER_CUSTOM -p tcp --dport 993 -j ACCEPT
# Allow wireguard
iptables -A USER_CUSTOM -p udp --dport 995 -j ACCEPT
# Allow kubernetes k3S api server
iptables -A USER_CUSTOM -p tcp --dport 6443 -j ACCEPT
# Add our custom chain
iptables -I INPUT 1 -j USER_CUSTOM
# DROP INCOMING TRAFFIC by default if nothing match
iptables -P INPUT DROP
#######
#IPv6
#######
#do the same thing with ip6tables instead of iptables
# Accept incoming ICMP
ip6tables -A USER_CUSTOM -p icmpv6 -j ACCEPT
# Allow ipv6 route auto configuration if your provider support it
ip6tables -A USER_CUSTOM -p udp --dport 546 -j ACCEPT
ip6tables -A USER_CUSTOM -p icmpv6 --icmpv6-type router-advertisement -j ACCEPT
ip6tables -A USER_CUSTOM -p icmpv6 --icmpv6-type router-solicitation -j ACCEPT
ip6tables -A USER_CUSTOM -p icmpv6 --icmpv6-type neighbour-advertisement -j ACCEPT
ip6tables -A USER_CUSTOM -p icmpv6 --icmpv6-type neighbour-solicitation -j ACCEPT
ip6tables -A USER_CUSTOM -p icmpv6 --icmpv6-type echo-request -j ACCEPT
ip6tables -A USER_CUSTOM -p icmpv6 --icmpv6-type echo-reply -j ACCEPTI don't rate limit ssh connections, as most of the time it is me that is hit by that limit. Nowadays most bot scanning ssh servers are smart enough to time their attempts to avoid being rate limited. Even if we are only allowing public key authentication, some bots are going to try endlessly to connect to our SSH server, in hope that someday a breach appears. Like waves crashing tirelessly on the shore.
If you want to enable it anyway, you need to add those rules
-A USER_CUSTOM -p tcp -m conntrack --ctstate NEW --dport 22 -m recent --set --name SSH
-A USER_CUSTOM -p tcp -m conntrack --ctstate NEW --dport 22 -m recent --update --seconds 60 --hitcount 4 --rttl --name SSH -j DROP
-A USER_CUSTOM -p tcp -m conntrack --ctstate NEW --dport 22 -j ACCEPTWe are going to deploy those rules in the if-pre-up to restore them automatically when the machine reboots. As those rules are idempotent we force their execution when invoked to be sure they are in place.
iptables:
scp config/iptables ${HOST}:/etc/network/if-pre-up.d/iptables-restore
ssh ${HOST} 'chmod +x /etc/network/if-pre-up.d/iptables-restore && sh /etc/network/if-pre-up.d/iptables-restore'Chose your registrar (DNS)
Now that we have a server provisioned and a bit more secure, we want to assign it a cute DNS name instead of just its IP address. If you don't know what is a DNS please refer to :
Like for the server provider, you are free to chose whatever you want here.
I personally use GANDI.net, as they provide free mailbox with a domain name. While I run a postfix/mail server on my server to receive and store emails, to avoid having to set up and manage a tedious DKIM I use GANDI SMTP mail server to send my emails and be trusted/not end up as spams. More on that later in setup your mail server.
If you don't know which one to take, here the point I look for:
- Provide an API to manage DNS record
- Propagation should be fast enough (if you plan to use let's encrypt DNS challenge for wildcard)
- Provide DNSSEC (I don't use it personally)
Beside that every registrar are the same. I recommend you using
- Your hosting company for your registrar in order to centralize things
- Use Cloudflare if you plan to setup a blog later on
Automate your DNS record update
This one is simple, just need to get the information of how to use the API of your registrar.
For gandi, we will use their cli gandi and manage our zone in a plain text file.
In our Makefile, it gives something like
dns:
sops -d --output secrets_decrypted/gandi.yml secrets/gandi.yml
GANDI_CONFIG='secrets_decrypted/gandi.yml' gandi dns update erebe.eu -f dns/zones.txtwith our zones.txt file looking
@ 10800 IN SOA ns1.gandi.net. hostmaster.gandi.net. 1579092697 10800 3600 604800 10800
@ 10800 IN A 195.154.119.61
@ 10800 IN AAAA 2001:bc8:3d8f::cafe
@ 10800 IN MX 1 mail.erebe.eu.
@ 10800 IN MX 10 spool.mail.gandi.net.
@ 10800 IN MX 50 fb.mail.gandi.net.
api 10800 IN A 195.154.119.61
api 10800 IN AAAA 2001:bc8:3d8f::cafe
...
Depending from your registrar, FAI and the TTL you set on your records, it can take quite some times for the new record to be propagated/updated everywhere, so be patient !
Installing Kubernetes k3s
We now have a server secured, with a domain name attached, and that we can re-deploy at ease.
The next step is to install Kubernetes on it. The choice of Kubernetes can be a bit controversial for only using it on a single machine. Kubernetes is a container orchestrator, so you can only leverage its full power when managing a fleet of servers. In addition, running vanilla Kubernetes require installing ETCD and other heavyweight components, plus some difficulties configuring every module for them to work correctly together.
Luckily for us an alternative to this heavy and challenging vanilla installation exists.
Meet K3S, a trimmed and packaged Kubernetes cluster in a single binary. This prodigy is bought to us by rancher labs, one of the big player in the container operator world. They took the decision for you (replacing ETCD by SQLite, network overlay, load balancer, ...) in order for k3s to be the smaller possible and easy to setup, while being a 100% compliant Kubernetes cluster.
The main benefit of having Kubernetes installed on my server, is that it allow me to have a standard interface for all my deployments, have everything store in git and allows me to leverage other tools like skaffold when I am developing my projects.
Warning With everything installed, just having the Kubernetes server components running add a 10% CPU on my Intel(R) Atom(TM) CPU C2338 @ 1.74GHz. So if you are already CPU bound, don't use it or scale up your server.
Let's start, to install K3S nothing more complicated than
kubernetes_install:
ssh ${HOST} 'export INSTALL_K3S_EXEC=" --no-deploy servicelb --no-deploy traefik --no-deploy local-storage"; \
curl -sfL https://get.k3s.io | sh -'We are disabling some more components as we don't need them. Specifically:
-
servicelbEverything will live on the same machine, so there is no need to load balance, most of the time we are going to avoid the network overlay also, by using the host network directly as much as possible -
traefikI have more experience with Nginx/HAProxy for reverse-proxy, so I am going to use nginx ingress controller in place of Traefik -
local-storagethis application is for creating automatically local volume (PV) for your hosts, as we have only one machine, we will bypass this complexity and just useHostPathvolume
After running this command, you can ssh on your server and do a
sudo kubectl get nodes
# logs are available with
# sudo journalctl -feu k3sand check that your server is in Ready state (it can take some time). If it is the case, congrats ! You have a Kubernetes control plane working !
Now that this is done, we need to automate the setup of the kubeconfig installation.
On your server copy the content of the kube config file /etc/rancher/k3s/k3s.yaml and encrypt it with sops under secrets/kubernetes-config.yml. Be sure to Replace 127.0.0.1 in the config by the ip/domain name of your server.
After that, in your Makefile add in the install section
install:
...
mkdir ~/.kube || exit 0
sops -d --output ~/.kube/config secrets/kubernetes-config.ymlIf you made things correctly and that you have kubectl installed on your local machine, you should be able to do a
kubectl get nodes
and see your server ready !
Nginx as Ingress controller for Kubernetes
I have many small pet projects exposing an HTTP endpoint that I want to expose to the rest of the internet, as I have blocked every ingoing traffic other than for port 80 and 443, I need to multiplex every application under those two. For that I need to install a reverse proxy that will also do TLS termination.
As I have disabled Traefik, the default reverse-proxy, during the k3s installation, I need to install my own. My choice went to Nginx. I know it well with HaProxy, knows it is reliable and it is the most mature between the two on Kubernetes.
To install it on your K3s cluster either use the Helm chart or directly with a kube apply. Refer for the installation guide for baremetal
WARNING: Don't copy-paste directly from the documentation nginx-ingress annotations, the '-' is not a real '-' and your annotation will not be recognized 🤦
I am going to install it directly from the YAML files available at
https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.40.2/deploy/static/provider/baremetal/deploy.yaml
To avoid having to manage also helm deployment.
I am going also to edit the deployment in order for the Nginx reverse proxy to use HostNetwork and avoid going thought the network overlay.
In the above YAML file, replace DNS policy value by ClusterFirstWithHostNet and add a new entry hostNetwork: true for the container to use directly your network card instead of a virtual interface.
# Source: ingress-nginx/templates/controller-deployment.yaml
apiVersion: apps/v1
kind: Deployment
...
spec:
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
containers:
...If you are using the helm chart, there is a variable/flag to toggle the usage of host network. Save your YAML file in your repository and update your Makefile to deploy it
k8s:
# If you use helm
#helm3 repo add stable https://kubernetes-charts.storage.googleapis.com/
#helm3 repo update
# helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
# helm install ingress-nginx ingress-nginx/ingress-nginx --set controller.hostNetwork=true
kubectl apply -f k8s/ingress-nginx-v0.40.2.yml
kubectl wait --namespace ingress-nginx \
--for=condition=ready pod \
--selector=app.kubernetes.io/component=controller \
--timeout=120sFor more information regarding Nginx as ingress, please refer to the Documentation
If you setup everything correctly you should see something like
❯ kubectl get pods -o wide -n ingress-nginx
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ingress-nginx-admission-create-gzpvj 0/1 Completed 0 9d 10.42.0.106 erebe-server <none> <none>
ingress-nginx-admission-patch-hs457 0/1 Completed 0 9d 10.42.0.107 erebe-server <none> <none>
ingress-nginx-controller-5f89b4b887-5wxmd 1/1 Running 0 8d 195.154.119.61 erebe-server <none> <none>with the IP of your ingress-nginx-controller being the ip of your main interface
erebe@erebe-server:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
...
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:08:a2:0c:63:4e brd ff:ff:ff:ff:ff:ff
inet 195.154.119.61/24 brd 195.154.119.255 scope global enp1s0
valid_lft forever preferred_lft forevererebe@erebe-server:~$ sudo ss -lntp | grep -E ':(80|443) '
LISTEN 0 128 0.0.0.0:80 0.0.0.0:* users:(("nginx",pid=32448,fd=19),("nginx",pid=22350,fd=19))
LISTEN 0 128 0.0.0.0:80 0.0.0.0:* users:(("nginx",pid=32448,fd=11),("nginx",pid=22349,fd=11))
LISTEN 0 128 0.0.0.0:443 0.0.0.0:* users:(("nginx",pid=32448,fd=21),("nginx",pid=22350,fd=21))
LISTEN 0 128 0.0.0.0:443 0.0.0.0:* users:(("nginx",pid=32448,fd=13),("nginx",pid=22349,fd=13))
LISTEN 0 128 [::]:80 [::]:* users:(("nginx",pid=32448,fd=12),("nginx",pid=22349,fd=12))
LISTEN 0 128 [::]:80 [::]:* users:(("nginx",pid=32448,fd=20),("nginx",pid=22350,fd=20))
LISTEN 0 128 [::]:443 [::]:* users:(("nginx",pid=32448,fd=14),("nginx",pid=22349,fd=14))
LISTEN 0 128 [::]:443 [::]:* users:(("nginx",pid=32448,fd=22),("nginx",pid=22350,fd=22))to test that everything is working you can deploy those resources and check that you can access your http://domain.name with the list of files of the container displayed
apiVersion: apps/v1
kind: Deployment
metadata:
name: test
labels:
app: test
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: test
template:
metadata:
labels:
app: test
spec:
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: webserver
image: python:3.9
imagePullPolicy: IfNotPresent
command: ["python"]
args: ["-m", "http.server", "8083"]
ports:
- name: http
containerPort: 8083
---
apiVersion: v1
kind: Service
metadata:
name: test
spec:
selector:
app: test
ports:
- protocol: TCP
port: 8083
clusterIP: None
type: ClusterIP
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: test-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- http:
paths:
- path: /
backend:
serviceName: test
servicePort: 8083
This deployment, will start a python simple HTTP server on port 8083 on host network, a service will reference this deployment and the ingress (the configuration for our reverse proxy) will be configured to point toward it on path /
To debug you can check
# To investigate python simple http server pod issue
kubectl describe pod test
# To see endpoints listed by the service
kubectl describe service test
# To see ingress issue
kubectl describe service test
# To check the config of nginx
kubectl exec -ti -n ingress-nginx ingress-nginx-controller-5f89b4b887-5wxmd -- cat /etc/nginx/nginx.confCertManager with let's encrypt for issuing TLS certificates
We have our reverse proxy working, now we want our k3s cluster to be able to generate on the fly certificates for our deployments. For that we are going to use the standard CertManager with let's encrypt as a backend/issuer.
to install it simply do a
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.0.4/cert-manager.yamlto automate the deployment we are going to add it in the repository and add a few lines in our Makefile
k8s:
...
kubectl apply -f k8s/cert-manager-v1.0.4.ymlYou can verify the installation with
$ kubectl get pods --namespace cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-5c6866597-zw7kh 1/1 Running 0 2m
cert-manager-cainjector-577f6d9fd7-tr77l 1/1 Running 0 2m
cert-manager-webhook-787858fcdb-nlzsq 1/1 Running 0 2mOnce Cert-Manager is deployed we need to configure an Issuer that is going to generate valid TLS certificates. For that we are going to use the free let's encrypt !
To do that simply deploy a new resource on the cluster
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
namespace: cert-manager
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: email@your_domain.name
privateKeySecretRef:
name: letsencrypt-prod
solvers:
- http01:
ingress:
class: nginxIt will tell cert manager, that we are going to use the acme HTTP challenges of let's encrypt and use nginx as ingress for that.
The issuer is configured for the whole cluster (with kind: ClusterIssuer) so it is going to
- Watch on all namespaces for the annotation `cert-manager.io/cluster-issuer: "letsencrypt-prod"generate TLS certificate
- Request a challenge from let's encrypt to (re-)generate TLS certificate
- Create a secret with those new certificate upon challenge success
In our Makefile
k8s:
...
kubectl apply -f k8s/lets-encrypt-issuer.ymlWarning: Be sure that your DNS name is valid/pointing to the correct machine before doing that, as it is easy to be blacklisted/throttled by let's encrypt. Especially if you are using DNS challenge for getting wildcard certificates.
If you configured everything correctly editing our previous ingress and adding a cluster issuer annotation, a TLS section in the spec and a host in the rules is enough.
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: test-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: "letsencrypt-prod" #Here
spec:
tls: # here
- hosts:
- domain.name
secretName: test-tls
rules:
- host: domain.name # here
http:
paths:
- path: /
backend:
serviceName: test
servicePort: 8083After applying the new version of the ingress, the cert manager should detect the annotation and launch a challenge. (You can check the pods for ACME challenge being spawned) and after a few minutes getting your TLS certificate deployed as a secret.
$ kubectl get secrets test-tls
If everything is ok, simply use your browser and visit https://domain.name to see our simple python backend being deployed with a valid TLS certificate !!!
Mail Server with Postfix + Dovecot + Fetchmail + SpamAssassin
Now I am going to install a mail server on my machine with a few caveats.
I am not going to use this SMTP server as an outgoing mail server, because nowadays it supposes to setup and maintain DKIM, SPF, DMARC and even when I have done so, sometimes my emails were ending-up in spam. The cost is not worth it, so I am using my registrar gandi.net SMTP server as relay to send my emails.
I am not going to enter into the details of how to configure postfix + dovecot + fetchmail + spamassassin as there are already plenty of guides available for that on the internet. My goal is to explain how I use Kubernetes to make them all work together.
For more information you can refer to my repository to look into the detail. The high level overview is :
- Cert-Manager issue valid TLS certificate that are used by dovecot and postfix
- Postfix is configured with virtual alias to allow emails from
*@my_domain.name - Postfix does not use any database (so no MySQL)
- Every mail are redirected to a single user, that run
procmailwith a custom program hmailfilter to triage automatically my email (Warning procmail is since a few years unmaintained and contains CVEs) - Emails are stored in the
maildirformat - Dovecot and postfix communicate by sharing this single maildir by mounting the same hostPath volume in both container
- I don't tag my custom container images, GitHub action is configured on every push to rebuild the image of {postfix, dovecot} and to publish them under
latest - I use trunk deployment for my images. I simply delete the current pod and let it recreate itself with the use of
imagePullPolicy: Alwaysto get the latest version
So let's start, first update your MX DNS record to point to your server
@ 10800 IN MX 1 mail.erebe.eu.
@ 10800 IN MX 10 spool.mail.gandi.net.
@ 10800 IN MX 50 fb.mail.gandi.net.In my setup I add my registrar SMTP server as a safety net in case my server is down. Fetchmail is configured to retrieve from it any emails it may have received for me.
Next step is to create a valid TLS certificate for both:
- Postfix as we want to support STARTTLS/SSL
- Dovecot, I only allow IMAPs and don't want self-signed certificate warning pop-ups
For that we simply use Kubernetes cert-manager and create a Certificate resource
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: dovecot-tls
spec:
# Secret names are always required.
secretName: dovecot-tls
duration: 2160h # 90d
renewBefore: 720h # 30d
subject:
organizations:
- erebe
# The use of the common name field has been deprecated since 2000 and is
# discouraged from being used.
isCA: false
privateKey:
algorithm: RSA
encoding: PKCS1
size: 2048
usages:
- server auth
- client auth
# At least one of a DNS Name, URI, or IP address is required.
dnsNames:
- mail.your_domain.name
# Issuer references are always required.
issuerRef:
name: letsencrypt-prod
# We can reference ClusterIssuers by changing the kind here.
# The default value is Issuer (i.e. a locally namespaced Issuer)
kind: ClusterIssuer
With that the cert-manager will issue a certificate under the secret dovecot-tls signed by let's encrypt.
After a few minutes, your secret will be available
❯ kubectl describe secret dovecot-tls
Name: dovecot-tls
Namespace: default
Labels: <none>
Annotations: cert-manager.io/alt-names: mail.erebe.eu
cert-manager.io/certificate-name: dovecot-tls
cert-manager.io/common-name: mail.erebe.eu
cert-manager.io/ip-sans:
cert-manager.io/issuer-group:
cert-manager.io/issuer-kind: ClusterIssuer
cert-manager.io/issuer-name: letsencrypt-prod
cert-manager.io/uri-sans:
Type: kubernetes.io/tls
Data
====
tls.crt: 3554 bytes
tls.key: 1679 bytesafter that we can inject those certificate in the container thanks to volumes in our deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: dovecot
labels:
app: dovecot
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: dovecot
template:
metadata:
labels:
app: dovecot
spec:
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: mail
image: erebe/dovecot:latest
imagePullPolicy: Always
ports:
- containerPort: 993
volumeMounts:
- name: dovecot-tls
mountPath: /etc/ssl/dovecot/
readOnly: true
- name: dovecot-users-password
mountPath: /etc/dovecot/users/
readOnly: true
- name: mail-data
mountPath: /data
volumes:
- name: dovecot-tls
secret:
secretName: dovecot-tls
- name: dovecot-users-password
secret:
secretName: dovecot-users-password
- name: mail-data
hostPath:
path: /opt/mail/data
type: DirectoryIn my deployments:
- We are using host network
- All the data are stored on the host file system under
/opt/xxxin order to easily backup it - All the container use the same created user ID 1000 for writing data to avoid conflicting rights
- Password are stored as Kubernetes secrets and committed inside the repository thanks to sops
apiVersion: v1
kind: Secret
metadata:
name: dovecot-users-password
type: Opaque
stringData:
users: 'erebe:{MD5-CRYPT}xxxxx.:1000:1000::::/bin/false::'In the end, deploying dovecot from our makefile is a simple
dovecot:
sops -d --output secrets_decrypted/dovecot.yml secrets/dovecot.yml
kubectl apply -f secrets_decrypted/dovecot.yml
kubectl apply -f dovecot/dovecot.ymlFor postfix it is the same, and we are reusing the previous created TLS certificate for providing STARTTLS/SSL support for the SMTP server
apiVersion: apps/v1
kind: Deployment
metadata:
name: postfix
labels:
app: postfix
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: postfix
template:
metadata:
labels:
app: postfix
spec:
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: postfix
image: erebe/postfix:latest
imagePullPolicy: Always
ports:
- containerPort: 25
volumeMounts:
- name: dovecot-tls
mountPath: /etc/ssl/postfix/
readOnly: true
- name: mail-data
mountPath: /data
- name: fetchmail
mountPath: /etc/fetchmail
volumes:
- name: dovecot-tls
secret:
secretName: dovecot-tls
- name: mail-data
hostPath:
path: /opt/mail/data
type: Directory
- name: fetchmail
configMap:
name: fetchmail
items:
- key: fetchmailrc
path: fetchmailrcour Makefile
postfix:
sops -d --output secrets_decrypted/fetchmail.yml secrets/fetchmail.yml
kubectl apply -f secrets_decrypted/fetchmail.yml
kubectl apply -f postfix/postfix.ymlAutomating build and push of our images with GitHub actions
When I do a change, I want my custom images to be rebuilt and push automatically on docker hub. To achieve it I rely on a 3rd party, GitHub actions !
#.github/workflows/docker-dovecot.yml
name: Publish Dovecot Image
on:
push:
paths:
- 'dovecot/**'
jobs:
buildAndPush:
name: Build And Push docker images
runs-on: ubuntu-latest
steps:
- name: Check out the repo
uses: actions/checkout@v2
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1
- name: Login to Github container repository
uses: docker/login-action@v1
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.CR_PAT }}
- name: Dovecot
id: docker_build_dovecot
uses: docker/build-push-action@v2
with:
context: dovecot
file: dovecot/Dockerfile
push: true
tags: ghcr.io/erebe/dovecot:latest
- name: Dovecot Image digest
run: echo Dovecot ${{ steps.docker_build_dovecot.outputs.digest }}
When a file under dovecot/** is modified during a commit, the Github Actions CI will trigger the job that re-build the docker image and push it to the GitHub container registry.
I use GitHub container registry in order to centralize things as much as possible and avoid adding docker hub as another external dependencies.
The part left to do yet, is automatic deployment when a new image is build. Ideally, I would like to avoid having to store my kubeconfig inside GitHub secrets and code an app that support web hook in order to trigger a new deployment. But for now I am still thinking of how to do that properly, so I am left to delete manually my pod to re-fetch the latest image until then ¯\(ツ)/¯
Hosting your own cloud with Nextcloud
Backups
My backups are simplistic, as I store all the data under /opt of the host machine and that I am not running any dedicated database.
The Backup of the data consist of:
- Running a cron-job every night inside Kubernetes that is spawning a container
- Mounting the whole
/optfolder inside the container as a volume - Creating a tar of
/opt - Pushing the tarball to the ftp server that my hosting company provide me
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: backup
spec:
schedule: "0 4 * * *"
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 2
jobTemplate:
spec:
template:
spec:
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: backup
image: alpine
args:
- /bin/sh
- -c
- apk add --no-cache lftp; tar -cvf backup.tar /data; lftp -u ${USER},${PASSWORD} dedibackup-dc3.online.net
-e 'put backup.tar -o /backups/backup_neo.tar; mv backups/backup_neo.tar backups/backup.tar; bye'
env:
- name: USER
valueFrom:
secretKeyRef:
name: ftp-credentials
key: username
- name: PASSWORD
valueFrom:
secretKeyRef:
name: ftp-credentials
key: password
volumeMounts:
- name: data
mountPath: /data
restartPolicy: OnFailure
volumes:
- name: data
hostPath:
path: /opt
type: Directory
[TODO] Monitoring with netdata
Virtual Private Network with Wireguard
My next step is to setup a VPN with wireguard to :
- Remove the access of the kube api server from internet
- Connect machines (Raspberry Pi) that can't be reached from internet
- Manage my Raspberry Pi as simple nodes inside the k3s cluster
- Route my traffic toward a safe network when in café, airports, etc (almost never...)
We are not going to install WireGuard a Kubernetes deployment as it requires a kernel module in order to work correctly. The only way is to install it directly on the host machine !
Follow this guide in order install and configure WireGuard for Debian.
The only change I made is to add postUp and postDown rules to the wg0.conf in order to forward and masquerade traffic that are targeting network outside the VPN. This setup allows me to route all my local machine traffic through the VPN (i.e: When using my phone) when I want to.
wstunnel !!!
#wg0.conf
[Interface]
Address = 10.200.200.1/24
ListenPort = 995
PrivateKey = __SERVER_PRIVATE_KEY__
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o enp1s0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -o enp1s0 -j MASQUERADE
[Peer]
PublicKey = __RASPBERRY_PUBLIC_KEY__
AllowedIPs = 10.200.200.2/32
[Peer]
PublicKey = __PHONE_PUBLIC_KEY__
AllowedIPs = 10.200.200.3/32
[Peer]
PublicKey = __LAPTOP_PUBLIC_KEY__
AllowedIPs = 10.200.200.4/32
On my phone for example, to route all the traffic trough the VPN. I am going to have a setup like this one
[Interface]
# Client
PrivateKey = xxx
Address = 10.200.200.2/32
[Peer]
# Server
PublicKey = xxxx
## Allow all the traffic to flow throught the VPN
AllowedIPs = 0.0.0.0/0The make file to automate the deployment of the config
wireguard:
sops exec-env secrets/wireguard.yml 'cp wireguard/wg0.conf secrets_decrypted/; for i in $$(env | grep _KEY | cut -d = -f1); do sed -i "s#__$${i}__#$${!i}#g" secrets_decrypted/wg0.conf ; done'
ssh ${HOST} "cat /etc/wireguard/wg0.conf" | diff - secrets_decrypted/wg0.conf \
|| (scp secrets_decrypted/wg0.conf ${HOST}:/etc/wireguard/wg0.conf && ssh ${HOST} systemctl restart wg-quick@wg0)
ssh ${HOST} 'systemctl enable wg-quick@wg0'Bypass firewalls with WsTunnel
Sometimes is it not possible to connect to my VPN due to some firewalls, because Wireguard uses UDP traffic and it is not allowed, or because the port 995 (POP3s) I bind it on is forbiden.
To bypass those firewalls and allow me to reach my private network I use WsTunnel, a websocket tunneling utility that I wrote. Basically, wstunnel leverage Websocket protocol that is using HTTP in order to tunnel TCP/UDP traffic through it. With that, 99.9% of the time I can connect to my VPN network, at the cost of 3 layer of encapsulation (data -> WebSocket -> Wireguard -> Ip) :x
Check the readme for more information
# On the client
wstunnel -u --udpTimeout=-1 -L 1995:127.0.0.1:995 -v ws://ws.erebe.eu
# in your wg0.conf point the peer address to 127.0.0.1:995 instead of domain.nameOn the server
apiVersion: apps/v1
kind: Deployment
metadata:
name: wstunnel
labels:
app: wstunnel
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: wstunnel
template:
metadata:
labels:
app: wstunnel
spec:
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: wstunnel
image: erebe/wstunnel:latest
imagePullPolicy: Always
args:
- "--server"
- "ws://0.0.0.0:8084"
- "-r"
- "127.0.0.1:995"
ports:
- containerPort: 8084
---
apiVersion: v1
kind: Service
metadata:
name: wstunnel
spec:
selector:
app: wstunnel
ports:
- protocol: TCP
port: 8084
name: http
clusterIP: None
type: ClusterIP
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: wstunnel-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/proxy-read-timeout: "1800"
nginx.ingress.kubernetes.io/proxy-send-timeout: "1800"
nginx.ingress.kubernetes.io/connection-proxy-header: "upgrade"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
tls:
- hosts:
- ws.erebe.eu
secretName: wstunnel-tls
rules:
- host: ws.erebe.eu
http:
paths:
- path: /
backend:
serviceName: wstunnel
servicePort: httpInstalling K3S on our Raspberry Pi using your Wireguard VPN
I want my Raspberry Pi that is living inside my home network to be manageable like a simple node inside the Kubernetes cluster. For that I am going to setup Wireguard on my Raspberry Pi and install the k3s agent on it.
- Installation Raspbian on your raspberry - Tutorial
- Setup Wireguard on the raspberry - Tutorial
- Configure Wireguard
[Interface]
PrivateKey = xxx
## Client ip address ##
Address = 10.200.200.2/32
[Peer]
PublicKey = xxxx
AllowedIPs = 10.200.200.0/24
## Your Debian 10 LTS server's public IPv4/IPv6 address and port ##
Endpoint = domain.name:995
## Key connection alive ##
PersistentKeepalive = 20- Start wireguard and test it is working properly
- On the
/boot/cmdline.txtof the Raspberry Pi add those 2 boot options
cgroup_memory=1 cgroup_enable=memory- Reboot the raspberry
- On the server edit
/etc/systemd/system/k3s.serviceand add the argument
--advertise-address 10.200.200.1
- Install k3S on the raspberry
# ssh on the raspberry, endpoint should be the ip of the server on the VPN network
# token can be found on the server at cat /var/lib/rancher/k3s/server/token
curl -sfL https://get.k3s.io | K3S_URL=https://10.200.200.1:6443 INSTALL_K3S_EXEC="--node-ip 10.200.200.2 --node-taint 'kubernetes.io/hostname=raspberrypi:NoSchedule'" K3S_TOKEN="xxxx" sh -
# --node-ip is to force using interface of wireguard to communicate with the node
# --node-taint is to disallow random container to end up on the raspberry, a toleration need to target it specifically- Check that your raspberry is in Ready state with its IP
Deploying PiHole on your Raspberry Pi
We have our raspberry Ready to use now inside our Kubernetes cluster. It is time now to use it by deploying PiHole as a DNS server inside our home local network. PiHole allows blocking trackers by not responding to DNS requests. It is like having ad-block on your network instead of your browser.
This is a standard deployment, with only 3 specificity:
- We need to set the container in privileged mode as it needs to bind on port 53 (DNS)
securityContext:
privileged: true- It has a
nodeAffinityand a toleration for ourtaintin order to allow and force the deployment on the raspberry
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- raspberrypi
...
tolerations:
- key: "kubernetes.io/hostname"
operator: "Equal"
value: "raspberrypi"
effect: "NoSchedule"- We had a other toleration for the state
unreachableto let the container live on the raspberry even if we lose connectivity with the cluster. The pihole container will be deployed and will stay there until manually deleted
tolerations:
- key: "node.kubernetes.io/unreachable"
operator: "Exists"
effect: "NoExecute"Full YAML
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: pihole
labels:
app: pihole
spec:
replicas: 1
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
selector:
matchLabels:
app: pihole
template:
metadata:
labels:
app: pihole
spec:
hostNetwork: true
dnsPolicy: "None"
dnsConfig:
nameservers:
- 127.0.0.1
- 1.1.1.1
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- raspberrypi
containers:
- name: pihole
image: pihole/pihole:v5.1.2
imagePullPolicy: IfNotPresent
env:
- name: TZ
value: "Europe/Paris"
- name: WEBPASSWORD
value: "pihole"
- name: CONDITIONAL_FORWARDING
value: "true"
- name: CONDITIONAL_FORWARDING_IP
value: "192.168.1.254"
- name: CONDITIONAL_FORWARDING_DOMAIN
value: "lan"
ports:
- containerPort: 80
name: http
protocol: TCP
- containerPort: 53
name: dns
protocol: TCP
- containerPort: 53
name: dns-udp
protocol: UDP
securityContext:
privileged: true
volumeMounts:
- name: pihole-etc-volume
mountPath: "/etc/pihole"
- name: pihole-dnsmasq-volume
mountPath: "/etc/dnsmasq.d"
tolerations:
- key: "kubernetes.io/hostname"
operator: "Equal"
value: "raspberrypi"
effect: "NoSchedule"
- key: "node.kubernetes.io/unreachable"
operator: "Exists"
effect: "NoExecute"
volumes:
- name: pihole-etc-volume
hostPath:
path: /opt/pihole/etc
type: Directory
- name: pihole-dnsmasq-volume
hostPath:
path: /opt/pihole/dnsmasq
type: DirectoryConclusion
- Everything is automated and idempotent
- Installation of the machine
- DNS
- Reverse proxy
- Certificate generation
- docker image build
- Everything is centralized inside the git repository
- Secrets are not separated from the code and derived from a GPG key
- Configuration and deployment are together thanks to container and kubernetes
- Deployment can be triggered from the makefile
- A single interface/control plane to control several machines
If you want more Freedom
- Host the machine at Home, if you have fiber you may have more bandwidth than a provider
- Manage dkim to not rely on the SMTP relay
- Host Gitlab to avoid relying on Github for git
- Can be used to mirror your github repository
- Can be used to have your own git actions
- Can be used to have your own Docker repositry