No description
  • Shell 74.6%
  • Makefile 25.4%
Find a file
Alexander Diewald 19b849e104
All checks were successful
Build packages / Build .deb (Debian 13) (push) Successful in 1m2s
Build packages / Build .pkg.tar.zst (Arch) (push) Successful in 25s
Build packages / Publish release on tag (push) Has been skipped
README: describe the actual CI runner labels
The CI section still named the ubuntu-latest label and the debian:13 and
archlinux:base-devel images. The jobs run on the runner labels "debian"
(node:24-trixie on this instance) and "arch" (custom arch-ci:latest image);
describe those and what each image needs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Alexander Diewald <a.diewald@diewald-net.com>
2026-10-20 21:36:24 +02:00
.forgejo/workflows CI: publish the Arch package to the Forgejo Arch registry 2026-10-19 23:48:37 +02:00
bin Locate borg-vm-hook.sh relative to run-backup.sh 2026-10-19 23:33:23 +02:00
config Locate borg-vm-hook.sh relative to run-backup.sh 2026-10-19 23:33:23 +02:00
debian Release 1.1.0 2026-10-19 23:50:00 +02:00
docs docs: translate README and recovery guide to English 2026-10-20 00:20:37 +02:00
examples/systemd examples: translate systemd unit comments to English 2026-10-20 00:20:38 +02:00
packaging/arch Release 1.1.0 2026-10-19 23:50:00 +02:00
.gitattributes Install through the Makefile in both packages 2026-10-19 23:42:56 +02:00
.gitignore feature/rename-backup-diewald-ci (#1) 2026-07-19 21:41:06 +02:00
Makefile Release 1.1.0 2026-10-19 23:50:00 +02:00
README.md README: describe the actual CI runner labels 2026-10-20 21:36:24 +02:00

backup-diewald

Backup toolchain for libvirt hosts (workstations and servers). A wrapper that opens a LUKS-encrypted external disk, optionally runs btrbk for system subvolumes, and then backs up every VM defined under qemu:///system with an application-consistent snapshot into its own Borg repo.

Key properties:

  • Automatic VM discovery: adding a new VM to the backups = dropping one yaml file under config/vms/.
  • Failures of individual VMs do not abort the whole run. Results per component are listed in the summary mail.
  • --quiesce/fs-freeze via the qemu-guest-agent, with a clean fallback to crash-consistent if the agent does not respond.
  • Overlay cleanup per VM right after its Borg run (blockcommit --active --pivot), no global state window.
  • Single-instance lock via flock, safe against duplicate trigger events.
  • Idempotent: broken runs leave an overlay + state file behind, and the next run cleans them up.

Tested against qemu:///system. For user-session VMs (qemu:///session), the way forward is to migrate them to system libvirt first.


Repo layout

backup-diewald/
├── Makefile                       # make deb | make arch | make install | ...
├── bin/
│   ├── run-backup.sh              # Wrapper: LUKS, mount, VM iteration, mail
│   └── borg-vm-hook.sh            # prepare/cleanup per VM
├── config/
│   ├── backup.conf.example        # Template for /etc/backup/backup.conf
│   └── vm-borgmatic-template.yaml # Template for /etc/backup/vms/<vm>.yaml
├── debian/                        # Debian source package (building the .deb)
├── packaging/arch/                # PKGBUILD + .install (building the Arch package)
├── docs/
│   └── recovery.md                # Recovery guide without borgmatic
├── examples/
│   └── systemd/
│       ├── run-backup.service.example         # oneshot, started by the timer (server)
│       ├── run-backup.timer.example
│       └── backup-extern.service.example      # oneshot, started by a .device unit (workstation)
├── .gitignore
└── README.md

Building

Build prerequisites:

  • Debian/Ubuntu: sudo apt install build-essential debhelper dh-make devscripts
  • Arch: the base-devel group (sudo pacman -S --needed base-devel)
make deb        # builds dist/backup-diewald_<version>-1_all.deb
make arch       # builds dist/backup-diewald-<version>-1-any.pkg.tar.zst
make all        # both
make clean      # removes dist/ and build artifacts

Which files are installed where is defined only in the install target of the Makefile. debian/rules (with PREFIX=/usr) and the PKGBUILD (additionally SBINDIR=/usr/bin) call exactly this target, so add new files only there.

The version is kept in sync in Makefile, debian/changelog and packaging/arch/PKGBUILD; when bumping, update all three (see ## Bumping the version below).


Debian 13 / Trixie, from the APT repository: Every release (v* tag) is published automatically to the Debian package registry of the Forgejo organization diewald. Set it up once per host; after that, new versions arrive with apt upgrade:

sudo install -d -m 0755 /etc/apt/keyrings
sudo curl -fsSL https://git.diewald-net.com/api/packages/diewald/debian/repository.key \
     -o /etc/apt/keyrings/diewald-forgejo.asc

sudo tee /etc/apt/sources.list.d/diewald-forgejo.sources >/dev/null <<'EOF'
Types: deb
URIs: https://git.diewald-net.com/api/packages/diewald/debian
Suites: trixie
Components: main
Signed-By: /etc/apt/keyrings/diewald-forgejo.asc
EOF

sudo apt update
sudo apt install backup-diewald

If the organization is private, APT also needs credentials: a token with scope read:package in /etc/apt/auth.conf.d/diewald-forgejo.conf (mode 0600), containing machine git.diewald-net.com login <user> password <token>.

Debian 13 / Trixie, locally built package:

sudo apt install ./dist/backup-diewald_<version>-1_all.deb

Arch, from the pacman repository: Every release is likewise published to the Arch registry of the organization diewald (repository main). Set it up once per host; after that, new versions arrive with pacman -Syu:

curl -fsSL https://git.diewald-net.com/api/packages/diewald/arch/repository.key -o /tmp/diewald.gpg
sudo pacman-key --add /tmp/diewald.gpg
sudo pacman-key --lsign-key diewald@noreply.git.diewald-net.com

sudo tee -a /etc/pacman.conf >/dev/null <<'EOF'

[diewald.main.git.diewald-net.com]
SigLevel = Required
Server = https://git.diewald-net.com/api/packages/diewald/arch/main/$arch
EOF

sudo pacman -Syu backup-diewald

If the organization is private, put the credentials (a token with scope read:package) into the Server URL: https://<user>:<token>@git.diewald-net.com/.... pacman.conf is world-readable, so in that case move the repo block into a separate file with mode 0600 and pull it in with Include =.

Arch, locally built package:

sudo pacman -U dist/backup-diewald-<version>-1-any.pkg.tar.zst

Both packages create:

  • /usr/sbin/run-backup.sh, /usr/sbin/borg-vm-hook.sh (Debian) or /usr/bin/ (Arch).
  • /usr/share/doc/backup-diewald/ with backup.conf.example, vm-borgmatic-template.yaml, README.md, recovery.md, examples/.
  • /etc/backup/vms/, /var/log/backup/, /var/lib/borg-prep/ as empty directories.

The post-install hook reports that /etc/backup/backup.conf is still missing. Then:

sudo cp /usr/share/doc/backup-diewald/backup.conf.example /etc/backup/backup.conf
sudo chmod 0600 /etc/backup/backup.conf
sudoedit /etc/backup/backup.conf

And for each VM, derive an /etc/backup/vms/<vm>.yaml from vm-borgmatic-template.yaml in the same way (see below).


Manual installation (without a package)

Prerequisites: cryptsetup, btrfs-progs, borg, borgmatic, libvirt, msmtp (if you want mail), qemu-guest-agent in every Windows/Linux guest (for clean snapshots).

Conveniently via the Makefile:

sudo make install                # into /usr/local/sbin, /usr/local/share/doc
# or into a staging tree:
sudo make install DESTDIR=/tmp/stage

Or directly:

sudo install -m 0755 bin/run-backup.sh   /usr/local/sbin/run-backup.sh
sudo install -m 0755 bin/borg-vm-hook.sh /usr/local/sbin/borg-vm-hook.sh
sudo mkdir -p /etc/backup/vms /var/log/backup
sudo install -m 0600 config/backup.conf.example /etc/backup/backup.conf
sudoedit /etc/backup/backup.conf
sudo cp config/vm-borgmatic-template.yaml /etc/backup/vms/<vmname>.yaml
sudoedit /etc/backup/vms/<vmname>.yaml

Adding a VM to the backups

vm=my-vm

# Find the VM's paths
virsh -c qemu:///system domuuid    "$vm"
virsh -c qemu:///system domblklist "$vm" --details
virsh -c qemu:///system dumpxml    "$vm" | grep -E '<nvram|<tpm|source file='

# Copy the template, fill in the paths
sudo cp config/vm-borgmatic-template.yaml "/etc/backup/vms/${vm}.yaml"
sudoedit "/etc/backup/vms/${vm}.yaml"

# Bootstrap the Borg repo once
sudo cryptsetup open /dev/disk/by-uuid/<LUKS_UUID> backup_ext \
    --key-file /etc/keys/backup.key
sudo mount /mnt/btr_backup_pool
sudo mkdir -p /mnt/btr_backup_pool/@backup-kvm
sudo borg init --encryption=repokey-blake2 \
    "/mnt/btr_backup_pool/@backup-kvm/borg-${vm}"
sudo umount /mnt/btr_backup_pool
sudo cryptsetup close backup_ext

# Test run
sudo run-backup.sh
sudo tail -n 200 /var/log/backup/backup-latest.log

VMs without a matching yaml are marked NO CONFIG and skipped, not treated as errors.


Triggers

Two common patterns, see examples/:

  • Workstation: plug in the external disk → a oneshot service pulled in by a systemd device unit starts the backup. Deliberately not via udev RUN+=: udev runs such processes in a heavily restricted context (permissions, environment, tight time budget), which causes failures in practice. The .device-triggered service runs in the full system context.
  • Server: permanently attached external target, nightly via a timer. Copy run-backup.service.example and run-backup.timer.example from examples/systemd/ to /etc/systemd/system/ (without the .example suffix) and activate with systemctl enable --now run-backup.timer.

Setting up the workstation trigger:

# 1. Generate the device unit name(s) of the disk(s) (do NOT escape by hand):
systemd-escape -p --suffix=device /dev/disk/by-uuid/<LUKS-UUID>
#   -> dev-disk-by\x2duuid-....device

# 2. Copy the service, enter the UUID(s) in After=/WantedBy=:
sudo cp examples/systemd/backup-extern.service.example \
        /etc/systemd/system/backup-extern.service
sudoedit /etc/systemd/system/backup-extern.service
sudo systemctl daemon-reload
sudo systemctl enable backup-extern.service   # creates the .wants symlinks per device unit

# 3. Test: plug in the disk, then
journalctl -u backup-extern.service -f

For off-site rotation, add one After= and one WantedBy= line per LUKS UUID to the service (see the comments in the example file).

Both triggers can be enabled at the same time; flock prevents collisions.


Multiple backup disks (off-site rotation)

For redundancy at two locations, simply list several LUKS UUIDs in LUKS_UUID (whitespace-separated). During a run, run-backup.sh opens whichever disk is currently attached as backup_ext. Everything below that (fstab mount, repo paths, borgmatic configs) stays identical, because the mapper name is constant.

# /etc/backup/backup.conf
LUKS_UUID="1111...-disk-A 2222...-disk-B"

Each disk is a complete, standalone backup with its own Borg repos. You plug in whichever is available and the script backs up to it; the other one stays at the second location.

One-time setup per disk:

# 1. Add the shared keyfile to the LUKS header of the second disk
sudo cryptsetup luksAddKey /dev/disk/by-uuid/<UUID-disk-B> /etc/keys/backup.key

# 2. Create btrfs with the same layout, then bootstrap the repos (as for disk A)
sudo cryptsetup open /dev/disk/by-uuid/<UUID-disk-B> backup_ext --key-file /etc/keys/backup.key
sudo mount /mnt/btr_backup_pool          # fstab maps /dev/mapper/backup_ext
sudo mkdir -p /mnt/btr_backup_pool/@backup-kvm
sudo borg init --encryption=repokey-blake2 /mnt/btr_backup_pool/@backup-kvm/borg-<vmname>   # per VM
sudo umount /mnt/btr_backup_pool
sudo cryptsetup close backup_ext

# 3. Add disk B's UUID to LUKS_UUID; in backup-extern.service add one
#    After= and one WantedBy= line per UUID (see examples/systemd/), then
#    `systemctl daemon-reload && systemctl enable backup-extern.service`.

Two things to keep in mind: retention (keep_daily etc.) counts archives per disk. Each disk only "sees" the host while it is plugged in, so plan retention accordingly. And if both disks are (accidentally) attached at the same time, the script uses the first one it finds and logs a warning.


Behavior on failure

Failure Effect
prepare <vm> fails This VM is skipped, cleanup is attempted, the next VM runs.
borgmatic <vm> fails Cleanup runs anyway (the overlay is closed), VM marked FAILED, the next VM runs.
cleanup <vm> fails (blockcommit) Overlay + state remain; the next prepare cleans them up.
btrbk fails Marked in the results; the VM stage runs anyway.
LUKS/mount fails Hard abort, mail with rc≠0.

Exit code: 0 = all OK, 1 = at least one component had errors (details in the log/mail).


Recovery

See docs/recovery.md. Important: keep the repo passphrase (or the keyfile for keyfile-* encryption) and the LUKS keyfile outside the host being backed up. Otherwise even the best recovery documentation won't help you.


Security notes

  • LUKS_KEY (/etc/keys/backup.key) and the Borg passphrase do not belong in this repo. .gitignore protects config/backup.conf and config/vms/ so that a git add -A does not pull secrets into the public branch. If the repo lives on a self-hosted Git server (Gitea, Forgejo, GitLab, …) with access control and you want to version your configs, you can comment out the corresponding lines in .gitignore, but think it through deliberately.
  • Alternatives for versioned but encrypted configs: git-crypt, sops, ansible-vault. Not required; for a purely private repo behind a VPN/SSH, plain text is often acceptable.

Automatic build via Forgejo Actions

The workflow .forgejo/workflows/build.yml runs on every push to main, on every PR against main, on every v* tag and manually (workflow_dispatch).

Two parallel jobs:

  • build-deb: runner label debian (Debian 13), make check-version, make deb, uploaded as artifact backup-diewald-deb.
  • build-arch: runner label arch, make arch (as the unprivileged builder user, because makepkg refuses to run as root), uploaded as backup-diewald-arch.

A third job, release, runs only on tag pushes and attaches both packages to the automatically created Forgejo release. The job uses https://code.forgejo.org/actions/forgejo-release@v2, provided your Forgejo instance allows actions from the code.forgejo.org registry (usually the default). If not, either enable that registry in app.ini or the runner config, or remove the release job; the artifacts remain available on the workflow run either way.

Runner requirements: a Forgejo Actions runner (act_runner) with the Docker executor that provides two labels (the release job also runs on debian):

  • debian: a Debian 13 image. On this instance it maps to node:24-trixie; the image needs Node.js because actions/checkout and the artifact actions are JavaScript actions.
  • arch: an Arch Linux image. On this instance it maps to the custom image arch-ci:latest. The job installs git and sudo and creates the builder user if the image doesn't already provide them.

In the runner config, labels map to images in the form debian:docker://node:24-trixie.

Creating a release:

# Bump the version in three places (see below), commit, then:
git tag -a v<version> -m "Release <version>"
git push origin main --tags

Pushing the tag triggers a build; after ~2 min the .deb and .pkg.tar.zst are attached to the release.

In addition, the release job uploads the .deb to the Debian registry and the .pkg.tar.zst to the Arch registry (repository main) of the organization diewald (for usage, see Installing from a package). This uses the bot account ci-packages, whose token with scope write:package is stored as the repo secret PACKAGE_TOKEN. The registry rejects a version that was already uploaded with 409 Conflict; to upload it again, first delete the version under diewald → Packages.


Bumping the version

The version is kept in three places, and all three must stay consistent:

  1. Makefile: the line VERSION := 1.1.0
  2. debian/changelog: a new entry at the top with version, date and comment (RFC 2822 date format)
  3. packaging/arch/PKGBUILD: pkgver=1.1.0, reset pkgrel=1 if needed

Then:

make check-version               # checks that all three match
make clean
make all
git tag -a v1.1.1 -m "Release 1.1.1"
make check-version TAG=v1.1.1    # optional: check the tag against the version

CI runs make check-version on every build, and on tag pushes also with the tag. A tag that does not match the version in the files therefore fails the build before anything is published.

For future automation, the version could be derived from git describe --tags. For now it is deliberately static so that local builds work without tag history.