This is part 2 of a 2 part series, if you have not read part one yet you can do so here: Backing up of KVMs with BorgBackup and BlockFuse Pt. 1
We use a lot of Virtual Machines (VM) to run ours infrastructure, these are on multiple host machines running Logical Volume Management (LVM) with each VM on its on its own Logical Volume (LV). After faffing around with multiple backup strategies and programs I settled on using BorgBackup (short: Borg) due to its special handling of block devices along with its deduplication and compression capabilities.
Borgs’ documentation is vast and can be found here: https://borgbackup.readthedocs.io/en/stable/
BlockFuse is quite an ancient program and was written by Eric Wheeler initially to do rdiff-backups, but works perfectly in this case as well to mount blockdevice. More information can be found here: https://www.linuxglobal.com/blockfuse-to-the-rescue-rdiff-backup-of-lvm-snapshots-and-block-devices/
BlockFuse:
BlockFuse is a relatively old piece of code which hasn’t been maintained for some time, it was originally created for use with rdiff-backup but works great with Borg as well.
BlockFuse takes two arguments:
# ./block-fuse usage: ./block-fuse /dev/directory /mnt/point
BlockFuse is quite simple: it enumerates the content of the mount-source directory, and exports all block devices with non-zero size as a file with 0400 permissions, owned by your fuse user (probably root for this).
Notes:
- BlockFuse uses the mount-time as the modification time (st_mtime) for the mounted filesystem. This will force Borg to scan the block devices for changes. Therefore you must unmount and re-mount your BlockFuse filesystem after updating your snapshots. If you do not, Borg will skip the “files” because their modification time-stamp had not changed since the last backup.
Backup Process:
The first task in the backup process was to check for and mount your Logical Volumes:
if [[ ! -d /mnt/block-devices ]]; then
sudo mkdir -p /mnt/block-devices
cd /etc/block-fuse
./block-fuse /dev/mapper /mnt/block-devices
else
cd /etc/block-fuse
./block-fuse /dev/mapper /mnt/block-devices
fi
In our backup process we are backing up to an external drive so we need to check to see if that is mounted as well:
if [[ ! -d /mnt/borgbackups ]]; then
sudo mkdir -p /mnt/borgbackups
sudo mount -o rw,$KEY -t cifs [your_external_storage_address] /mnt/borgbackups
else
sudo mkdir -p /mnt/borgbackups
sudo mount -o rw,$KEY -t cifs [your_external_storage_address] /mnt/borgbackups
fi
We are backing up multiple vmhosts so we have to verify that there is a separate backup folder for this machine:
if [[ ! -d /mnt/borgbackups/$HOSTNAME ]]; then sudo mkdir -p /mnt/borgbackups/$HOSTNAME fi
The only machines we want to back up are those in service, so we will make an array to pass through of all the running instances on the host:
name(){
virsh list | awk '( $1 ~ /^[0-9]+$/ ) { print $2 }'
}
arr=($( name ))
Now by passing-over the array we can make the magic happen!:
for v in ${arr[*]}; do
if sudo lvdisplay | grep -q $v; then
#Take snapshots of VMs
lvcreate --size 5G -s -n $v-snap /dev/main-vg/$v
borgrepo=/mnt/borgbackups/$HOSTNAME/$v-bbyo
#Check to see if vm has been backed up before and then create backup
if [[ ! -d $borgrepo ]]; then
borg init --encryption=none $borgrepo
borg create -C zlib,6 "$borgrepo::$v_{now:%Y-%m-%d}" /mnt/block-devices/main--vg-$v--snap
else
borg create -C zlib,6 "$borgrepo::$v_{now:%Y-%m-%d}" /mnt/block-devices/main--vg-$v--snap
fi
#Remove snapshot
lvremove -f /dev/main-vg/$v-snap
fi
done
And finally we will clean up our workspace in preparation for the next scheduled back up:
umount /mnt/block-devices umount /mnt/borgbackups #Finish echo "VM backup complete"
In the next part we will look at how we can recover our VMs.
