Need a scheme to backup files long-term to Amazon Glacier. The steps and scripts used to make up this collection is an amalgemation of information from a number of different sources.
In general there are three steps to archiving a ZFS dataset to Amazon Glacier. These tools are designed to help with 2 and 3.
- Take a snapshot of the ZFS dataset
- Create a dump of that ZFS dataset snapshot
- Upload the dump to Amazon Glacier
Since Amazon Glacier, and most other backup services, charge for the amount of data that is archived it is assumed that compression should be employed. Also, it assumed that the backup system you are going to employ may not be properly secured. As a result the data MUST be encrypted before sending over the wire to the 3rd party network. Because of these requirements I decided to benchmark different ways of encrypting/compressing data before sending to a backup client. My two optimizing parameters were time and resulting file-size. I used the experiment script to test real dataset snapshots to see which order of operations would lead to the smallest file in the shortest amount of time.
[mysql] Plain
real 0.24
user 0.00
sys 0.08
[mysql] Bytes: 45M
[mysql.bz2] AES > Bzip2
real 12.56
user 12.89
sys 0.20
[mysql.bz2] Bytes: 46M
[mysql.pbz2] AES > PBzip2
real 4.70
user 14.86
sys 0.38
[mysql.pbz2] Bytes: 46M
[mysql.bz2.aes.bz2] Bzip2 > AES > Bzip2
real 6.19
user 6.47
sys 0.12
[mysql.bz2.aes.bz2] Bytes: 5.0M
[mysql.pbz2.aes.bz2] PBzip2 > AES > PBzip2
real 2.63
user 5.88
sys 0.24
[mysql.pbz2.aes.bz2] Bytes: 5.1M
I ran the tests on two datasets. The first was 45MB and the second 16GB. After running the experiments, it became clear that first compressing the dataset, then encrypting the dataset, and compressing that result 1 final time resulted in both the smallest file and shortest execution time. The lessons learned by this experiment have been applied to the encrypt/decrypt scripts in this toolset.
- Parallel bzip2 implementation
apt-get install pbzip2
mkkey key
zfs-encrypt key stream/mysql@zfs-auto-snap_frequent-2013-12-19-1545 > mysql@2013-12-19-1545.bz2.aes.bz2