- Shell 74.6%
- Makefile 25.4%
The CI section still named the ubuntu-latest label and the debian:13 and archlinux:base-devel images. The jobs run on the runner labels "debian" (node:24-trixie on this instance) and "arch" (custom arch-ci:latest image); describe those and what each image needs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Alexander Diewald <a.diewald@diewald-net.com> |
||
|---|---|---|
| .forgejo/workflows | ||
| bin | ||
| config | ||
| debian | ||
| docs | ||
| examples/systemd | ||
| packaging/arch | ||
| .gitattributes | ||
| .gitignore | ||
| Makefile | ||
| README.md | ||
backup-diewald
Backup toolchain for libvirt hosts (workstations and servers). A wrapper that opens a LUKS-encrypted external disk, optionally runs btrbk for system subvolumes, and then backs up every VM defined under qemu:///system with an application-consistent snapshot into its own Borg repo.
Key properties:
- Automatic VM discovery: adding a new VM to the backups = dropping one yaml file under
config/vms/. - Failures of individual VMs do not abort the whole run. Results per component are listed in the summary mail.
--quiesce/fs-freeze via the qemu-guest-agent, with a clean fallback to crash-consistent if the agent does not respond.- Overlay cleanup per VM right after its Borg run (
blockcommit --active --pivot), no global state window. - Single-instance lock via
flock, safe against duplicate trigger events. - Idempotent: broken runs leave an overlay + state file behind, and the next run cleans them up.
Tested against qemu:///system. For user-session VMs (qemu:///session), the way forward is to migrate them to system libvirt first.
Repo layout
backup-diewald/
├── Makefile # make deb | make arch | make install | ...
├── bin/
│ ├── run-backup.sh # Wrapper: LUKS, mount, VM iteration, mail
│ └── borg-vm-hook.sh # prepare/cleanup per VM
├── config/
│ ├── backup.conf.example # Template for /etc/backup/backup.conf
│ └── vm-borgmatic-template.yaml # Template for /etc/backup/vms/<vm>.yaml
├── debian/ # Debian source package (building the .deb)
├── packaging/arch/ # PKGBUILD + .install (building the Arch package)
├── docs/
│ └── recovery.md # Recovery guide without borgmatic
├── examples/
│ └── systemd/
│ ├── run-backup.service.example # oneshot, started by the timer (server)
│ ├── run-backup.timer.example
│ └── backup-extern.service.example # oneshot, started by a .device unit (workstation)
├── .gitignore
└── README.md
Building
Build prerequisites:
- Debian/Ubuntu:
sudo apt install build-essential debhelper dh-make devscripts - Arch: the
base-develgroup (sudo pacman -S --needed base-devel)
make deb # builds dist/backup-diewald_<version>-1_all.deb
make arch # builds dist/backup-diewald-<version>-1-any.pkg.tar.zst
make all # both
make clean # removes dist/ and build artifacts
Which files are installed where is defined only in the install target of the Makefile. debian/rules (with PREFIX=/usr) and the PKGBUILD (additionally SBINDIR=/usr/bin) call exactly this target, so add new files only there.
The version is kept in sync in Makefile, debian/changelog and packaging/arch/PKGBUILD; when bumping, update all three (see ## Bumping the version below).
Installing from a package (recommended)
Debian 13 / Trixie, from the APT repository: Every release (v* tag) is published automatically to the Debian package registry of the Forgejo organization diewald. Set it up once per host; after that, new versions arrive with apt upgrade:
sudo install -d -m 0755 /etc/apt/keyrings
sudo curl -fsSL https://git.diewald-net.com/api/packages/diewald/debian/repository.key \
-o /etc/apt/keyrings/diewald-forgejo.asc
sudo tee /etc/apt/sources.list.d/diewald-forgejo.sources >/dev/null <<'EOF'
Types: deb
URIs: https://git.diewald-net.com/api/packages/diewald/debian
Suites: trixie
Components: main
Signed-By: /etc/apt/keyrings/diewald-forgejo.asc
EOF
sudo apt update
sudo apt install backup-diewald
If the organization is private, APT also needs credentials: a token with scope read:package in /etc/apt/auth.conf.d/diewald-forgejo.conf (mode 0600), containing machine git.diewald-net.com login <user> password <token>.
Debian 13 / Trixie, locally built package:
sudo apt install ./dist/backup-diewald_<version>-1_all.deb
Arch, from the pacman repository: Every release is likewise published to the Arch registry of the organization diewald (repository main). Set it up once per host; after that, new versions arrive with pacman -Syu:
curl -fsSL https://git.diewald-net.com/api/packages/diewald/arch/repository.key -o /tmp/diewald.gpg
sudo pacman-key --add /tmp/diewald.gpg
sudo pacman-key --lsign-key diewald@noreply.git.diewald-net.com
sudo tee -a /etc/pacman.conf >/dev/null <<'EOF'
[diewald.main.git.diewald-net.com]
SigLevel = Required
Server = https://git.diewald-net.com/api/packages/diewald/arch/main/$arch
EOF
sudo pacman -Syu backup-diewald
If the organization is private, put the credentials (a token with scope read:package) into the Server URL: https://<user>:<token>@git.diewald-net.com/.... pacman.conf is world-readable, so in that case move the repo block into a separate file with mode 0600 and pull it in with Include =.
Arch, locally built package:
sudo pacman -U dist/backup-diewald-<version>-1-any.pkg.tar.zst
Both packages create:
/usr/sbin/run-backup.sh,/usr/sbin/borg-vm-hook.sh(Debian) or/usr/bin/(Arch)./usr/share/doc/backup-diewald/withbackup.conf.example,vm-borgmatic-template.yaml,README.md,recovery.md,examples/./etc/backup/vms/,/var/log/backup/,/var/lib/borg-prep/as empty directories.
The post-install hook reports that /etc/backup/backup.conf is still missing. Then:
sudo cp /usr/share/doc/backup-diewald/backup.conf.example /etc/backup/backup.conf
sudo chmod 0600 /etc/backup/backup.conf
sudoedit /etc/backup/backup.conf
And for each VM, derive an /etc/backup/vms/<vm>.yaml from vm-borgmatic-template.yaml in the same way (see below).
Manual installation (without a package)
Prerequisites: cryptsetup, btrfs-progs, borg, borgmatic, libvirt, msmtp (if you want mail), qemu-guest-agent in every Windows/Linux guest (for clean snapshots).
Conveniently via the Makefile:
sudo make install # into /usr/local/sbin, /usr/local/share/doc
# or into a staging tree:
sudo make install DESTDIR=/tmp/stage
Or directly:
sudo install -m 0755 bin/run-backup.sh /usr/local/sbin/run-backup.sh
sudo install -m 0755 bin/borg-vm-hook.sh /usr/local/sbin/borg-vm-hook.sh
sudo mkdir -p /etc/backup/vms /var/log/backup
sudo install -m 0600 config/backup.conf.example /etc/backup/backup.conf
sudoedit /etc/backup/backup.conf
sudo cp config/vm-borgmatic-template.yaml /etc/backup/vms/<vmname>.yaml
sudoedit /etc/backup/vms/<vmname>.yaml
Adding a VM to the backups
vm=my-vm
# Find the VM's paths
virsh -c qemu:///system domuuid "$vm"
virsh -c qemu:///system domblklist "$vm" --details
virsh -c qemu:///system dumpxml "$vm" | grep -E '<nvram|<tpm|source file='
# Copy the template, fill in the paths
sudo cp config/vm-borgmatic-template.yaml "/etc/backup/vms/${vm}.yaml"
sudoedit "/etc/backup/vms/${vm}.yaml"
# Bootstrap the Borg repo once
sudo cryptsetup open /dev/disk/by-uuid/<LUKS_UUID> backup_ext \
--key-file /etc/keys/backup.key
sudo mount /mnt/btr_backup_pool
sudo mkdir -p /mnt/btr_backup_pool/@backup-kvm
sudo borg init --encryption=repokey-blake2 \
"/mnt/btr_backup_pool/@backup-kvm/borg-${vm}"
sudo umount /mnt/btr_backup_pool
sudo cryptsetup close backup_ext
# Test run
sudo run-backup.sh
sudo tail -n 200 /var/log/backup/backup-latest.log
VMs without a matching yaml are marked NO CONFIG and skipped, not treated as errors.
Triggers
Two common patterns, see examples/:
- Workstation: plug in the external disk → a oneshot service pulled in by a systemd device unit starts the backup. Deliberately not via udev
RUN+=: udev runs such processes in a heavily restricted context (permissions, environment, tight time budget), which causes failures in practice. The.device-triggered service runs in the full system context. - Server: permanently attached external target, nightly via a timer. Copy
run-backup.service.exampleandrun-backup.timer.examplefromexamples/systemd/to/etc/systemd/system/(without the.examplesuffix) and activate withsystemctl enable --now run-backup.timer.
Setting up the workstation trigger:
# 1. Generate the device unit name(s) of the disk(s) (do NOT escape by hand):
systemd-escape -p --suffix=device /dev/disk/by-uuid/<LUKS-UUID>
# -> dev-disk-by\x2duuid-....device
# 2. Copy the service, enter the UUID(s) in After=/WantedBy=:
sudo cp examples/systemd/backup-extern.service.example \
/etc/systemd/system/backup-extern.service
sudoedit /etc/systemd/system/backup-extern.service
sudo systemctl daemon-reload
sudo systemctl enable backup-extern.service # creates the .wants symlinks per device unit
# 3. Test: plug in the disk, then
journalctl -u backup-extern.service -f
For off-site rotation, add one After= and one WantedBy= line per LUKS UUID to the service (see the comments in the example file).
Both triggers can be enabled at the same time; flock prevents collisions.
Multiple backup disks (off-site rotation)
For redundancy at two locations, simply list several LUKS UUIDs in LUKS_UUID (whitespace-separated). During a run, run-backup.sh opens whichever disk is currently attached as backup_ext. Everything below that (fstab mount, repo paths, borgmatic configs) stays identical, because the mapper name is constant.
# /etc/backup/backup.conf
LUKS_UUID="1111...-disk-A 2222...-disk-B"
Each disk is a complete, standalone backup with its own Borg repos. You plug in whichever is available and the script backs up to it; the other one stays at the second location.
One-time setup per disk:
# 1. Add the shared keyfile to the LUKS header of the second disk
sudo cryptsetup luksAddKey /dev/disk/by-uuid/<UUID-disk-B> /etc/keys/backup.key
# 2. Create btrfs with the same layout, then bootstrap the repos (as for disk A)
sudo cryptsetup open /dev/disk/by-uuid/<UUID-disk-B> backup_ext --key-file /etc/keys/backup.key
sudo mount /mnt/btr_backup_pool # fstab maps /dev/mapper/backup_ext
sudo mkdir -p /mnt/btr_backup_pool/@backup-kvm
sudo borg init --encryption=repokey-blake2 /mnt/btr_backup_pool/@backup-kvm/borg-<vmname> # per VM
sudo umount /mnt/btr_backup_pool
sudo cryptsetup close backup_ext
# 3. Add disk B's UUID to LUKS_UUID; in backup-extern.service add one
# After= and one WantedBy= line per UUID (see examples/systemd/), then
# `systemctl daemon-reload && systemctl enable backup-extern.service`.
Two things to keep in mind: retention (keep_daily etc.) counts archives per disk. Each disk only "sees" the host while it is plugged in, so plan retention accordingly. And if both disks are (accidentally) attached at the same time, the script uses the first one it finds and logs a warning.
Behavior on failure
| Failure | Effect |
|---|---|
prepare <vm> fails |
This VM is skipped, cleanup is attempted, the next VM runs. |
borgmatic <vm> fails |
Cleanup runs anyway (the overlay is closed), VM marked FAILED, the next VM runs. |
cleanup <vm> fails (blockcommit) |
Overlay + state remain; the next prepare cleans them up. |
btrbk fails |
Marked in the results; the VM stage runs anyway. |
| LUKS/mount fails | Hard abort, mail with rc≠0. |
Exit code: 0 = all OK, 1 = at least one component had errors (details in the log/mail).
Recovery
See docs/recovery.md. Important: keep the repo passphrase (or the keyfile for keyfile-* encryption) and the LUKS keyfile outside the host being backed up. Otherwise even the best recovery documentation won't help you.
Security notes
LUKS_KEY(/etc/keys/backup.key) and the Borg passphrase do not belong in this repo..gitignoreprotectsconfig/backup.confandconfig/vms/so that agit add -Adoes not pull secrets into the public branch. If the repo lives on a self-hosted Git server (Gitea, Forgejo, GitLab, …) with access control and you want to version your configs, you can comment out the corresponding lines in.gitignore, but think it through deliberately.- Alternatives for versioned but encrypted configs:
git-crypt,sops,ansible-vault. Not required; for a purely private repo behind a VPN/SSH, plain text is often acceptable.
Automatic build via Forgejo Actions
The workflow .forgejo/workflows/build.yml runs on every push to main, on every PR against main, on every v* tag and manually (workflow_dispatch).
Two parallel jobs:
build-deb: runner labeldebian(Debian 13),make check-version,make deb, uploaded as artifactbackup-diewald-deb.build-arch: runner labelarch,make arch(as the unprivilegedbuilderuser, because makepkg refuses to run as root), uploaded asbackup-diewald-arch.
A third job, release, runs only on tag pushes and attaches both packages to the automatically created Forgejo release. The job uses https://code.forgejo.org/actions/forgejo-release@v2, provided your Forgejo instance allows actions from the code.forgejo.org registry (usually the default). If not, either enable that registry in app.ini or the runner config, or remove the release job; the artifacts remain available on the workflow run either way.
Runner requirements: a Forgejo Actions runner (act_runner) with the Docker executor that provides two labels (the release job also runs on debian):
debian: a Debian 13 image. On this instance it maps tonode:24-trixie; the image needs Node.js becauseactions/checkoutand the artifact actions are JavaScript actions.arch: an Arch Linux image. On this instance it maps to the custom imagearch-ci:latest. The job installsgitandsudoand creates thebuilderuser if the image doesn't already provide them.
In the runner config, labels map to images in the form debian:docker://node:24-trixie.
Creating a release:
# Bump the version in three places (see below), commit, then:
git tag -a v<version> -m "Release <version>"
git push origin main --tags
Pushing the tag triggers a build; after ~2 min the .deb and .pkg.tar.zst are attached to the release.
In addition, the release job uploads the .deb to the Debian registry and the .pkg.tar.zst to the Arch registry (repository main) of the organization diewald (for usage, see Installing from a package). This uses the bot account ci-packages, whose token with scope write:package is stored as the repo secret PACKAGE_TOKEN. The registry rejects a version that was already uploaded with 409 Conflict; to upload it again, first delete the version under diewald → Packages.
Bumping the version
The version is kept in three places, and all three must stay consistent:
Makefile: the lineVERSION := 1.1.0debian/changelog: a new entry at the top with version, date and comment (RFC 2822 date format)packaging/arch/PKGBUILD:pkgver=1.1.0, resetpkgrel=1if needed
Then:
make check-version # checks that all three match
make clean
make all
git tag -a v1.1.1 -m "Release 1.1.1"
make check-version TAG=v1.1.1 # optional: check the tag against the version
CI runs make check-version on every build, and on tag pushes also with the tag. A tag that does not match the version in the files therefore fails the build before anything is published.
For future automation, the version could be derived from git describe --tags. For now it is deliberately static so that local builds work without tag history.