Design
Two layers
- Whole-VM backups. Each Proxmox host runs a nightly
vzdumpof its guests to local storage, keeping one copy there. - Off-host copy. A vzdump hook script copies each finished backup to an Apple Time Capsule on the LAN over SMB. It keeps four copies per guest there and prunes older ones.
The production database gets a third layer. Inside the VM that runs the Aegis API, a nightly pg_dump runs shortly before the host's vzdump. Every VM backup therefore contains a consistent database dump, not only a crash-consistent disk image of a running Postgres.
- Hosts
- 2-node cluster + 1 standalone
- Guests covered
- 7 (VMs and a container)
- Local copies
- 1 per guest
- Time Capsule copies
- 4 per guest
- DB dump
- nightly, 14 days kept in-VM
- Credentials
- root-only file in
/etc/pve/priv
Part 01 · the failure
98.8% full, and every backup failed
On 2026-10-13 the 02:30 job on the first cluster node failed for every guest with No space left on device. The local backup directory was 98.8% full. Local copies had been piling up, and the host was carrying weeks of full VM images.
Local retention was cut to one copy per guest, which brought the disk down to about 50%. That fixed the symptom. The next question was why the Time Capsule wasn't showing the same growth, and it was.
Part 02 · the bug
Pruning that never pruned
The Time Capsule held about 705 GB of backups, far more than four copies of seven guests should take. The hook pushed every backup and logged success, but it never deleted anything.
The prune step looked up the guest's ID in an environment variable called $VMID. vzdump doesn't provide one. It passes the guest ID to a hook as the third positional argument, after the phase and the mode. The variable was always empty, so no file ever matched the prune pattern. No error was raised either: an empty pattern that matches nothing looks the same as "nothing to prune".
phase="$1"; mode="$2"; vmid="$3" # how vzdump calls a hook # before: the prune pattern used $VMID from the environment — always empty # after: it reads the ID from $3
The fix went to all three hosts, with the previous script kept beside it as .bak. A guard belongs next to it as well: a prune step that finds an empty ID should stop and log, not quietly do nothing or, worse, match everything.
◆ Lesson
Verify retention, not just the backup. "The backup ran" and "the copy arrived" were both checked. "Old copies get deleted" was not, because its failure is silent until a disk fills. Every retention rule needs a check that counts what's actually there.
Part 03 · the new host
Adding the production VM
The third Proxmox host took on the production API on 2026-10-13, so it needed the same treatment. Its nightly job runs at 03:30. The first push of the production VM was checked byte for byte on both sides: 5,094,183,873 bytes, identical.
That also retired a GitHub Actions database-backup workflow that had been failing every night since 2026-08-04, when the AWS cluster it backed up was deleted. It wasn't removed while it was the only off-host plan on paper. It was removed once something real replaced it.
Part 04 · verification
The overnight check
The fixes were checked the next morning, against what was actually on disk rather than what the logs said should be there:
| Check | Result |
|---|---|
| All 7 guests backed up and pushed | ✓ 0 errors |
| First node's 02:30 job (failed the night before) | ✓ all 4 guests; disk at 55% |
| Old copies pruned | ✓ 65 removed, each logged keeping 4 |
| Copies per guest on the Time Capsule | ✓ exactly 4 (2 on the new host, which started a day earlier) |
| Time Capsule usage | ✓ ~705 GB → ~175 GB; 1.63 TB free |
| Production DB dump | ⚠ missed one night, expected (below) |
▲ The dump that didn't run
The in-VM pg_dump is scheduled in UTC. That night's run fell inside the host outage described in the migration write-up, and the VM booted two minutes after its slot. The VM backup still held a fresh dump taken during a deploy earlier that evening, so nothing was lost. A catch-up job that runs after boot if the newest dump is older than about 20 hours would close the gap.
What's still open
Same building, no encryption
- Off-site. The Time Capsule covers disk and host loss, not losing the building. A copy to object storage elsewhere is the next step.
- Encryption. The backups on the Time Capsule aren't encrypted. Encrypting the dumps and VM images before they leave the host comes before any off-site copy.
- Restore drills. The database dump has been restore-tested. A full VM restore from the Time Capsule hasn't been yet.
