Document how to rollback properly to the last stable version / pin to an stable/older version
alexanderadam opened this issue · comments
A recent comment in one of the mailcow issues made me think that there is probably an easy way to rollback a mailcow release in case there's some issue.
Docker & docker compose support this anyway (you can usually use an image tag with an older version) so I thought there is probably something.
I wasn't able to find something in the docs but at least the update is described.
It seems that there are no releases or proper git tags neither so for now it seems like someone would have to tangle through merges and commits by trial and error for now.
PS: Not quite related but maybe it would be useful to track the last used revisions in a separate file when calling the update script (obviously before a newer revision is checked out) as a first step in some improvement of the file.
The issue was related to Ubuntu 16.04 with Linux 4.4 and a bug with overlayfs in combination with Docker. It existed before, if you revert to a previous image, you still have this bug. It was also related to the strange behavior of ClamAV on these systems. It was just coincidence, that it broke Dovecot and people noticed and couldn't work it around like they do with SKIP_CLAMD (or just ignore a restarting clamd-mailcow). :)
The update script creates a commit "before update", that can be used. We don't delete old image tags from Docker Hub. You should be able to run docker-compose down on the broken stack, revert to "before update XYZ", run docker-compose pull and docker-compose up -d.
This can fail, as we might have introduced a database change, that cannot be easily reverted. You need to backup /var/lib/docker/volumes/mailcowdockerized_mysql-vol-1/ and /var/lib/docker/volumes/mailcowdockerized_redis-vol-1/ (keep the file permissions! use rsync -a or cp -rp **while the stack is down!).
So, it can look like this (you should always have a snapshot available anyway - this example also copies the mailcow-dockerized folder):
cd /opt/mailcow-dockerized
docker-compose down
DATE=$(date +"%Y-%m-%d-%H-%M-%S")
rsync -avh /var/lib/docker/volumes/mailcowdockerized_redis-vol-1/ /var/lib/docker/volumes/mailcowdockerized_redis-vol-1_backup_${DATE}/
rsync -avh /var/lib/docker/volumes/mailcowdockerized_mysql-vol-1/ /var/lib/docker/volumes/mailcowdockerized_mysql-vol-1_backup_${DATE}/
rsync -avh /opt/mailcow-dockerized /opt/mailcow-dockerized_backup_${DATE}
./update.sh
If you have trouble updating a mailcow or crashed it, there are support packages and a Telegram group available.
You should always have a backup concept and be able to figure out how to revert to X, when you upgrade a system. That's part of the server administration. :)
It may also be required to roll back more than just SQL and Redis. It depends on the last commits. But that's very rare. As I said previously: you should always know how to reset your system anyway.
The script above is not part of a "normal" mailcow update. You should always backup your system on your own. We cannot provide these tasks in a mail server package.
Hi André, thank you for answering so fast.
I guess what you wrote would make a good base for documentation, wouldn't it?
Not really. Everyone should have a backup concept on their own. It is just a quick-and-dirty way to backup mailcow (not even complete). You should just do regular backups, rsyncs, snapshots, database backups etc. :)
This issue is not only about having a backup concept though (this is what the backup_and_restore.sh would be for anyway, isn't it?).
See, the logic and/or code of modern application stacks is usually versioned properly and can therefore easily rolled back. Docker or modern package managers allow this.
Meaning that it's usually only necessary to backup databases & assets and making backups also usually easier (and I guess smaller as well).
If mailcow and it's dependencies are already properly versioned it would be nice if the backup_and_restore.sh script would warn and suggest using the same versions dependencies when the backup was made (or is this done already and I just missed it?).
I'm not sure whether this would have helped in some of the problems of issue 3115.
This way also a proper migration path can be ensured.
Or would you rather say that the current backup_and_restore.sh script is complete, robust and enough anyway?