The Great Migration: Aegis production comes home

Background

Three homes in three months

Aegis CloudOps and Fishing Rod Terminal share one backend: a Python API with Postgres and Redis. The frontends are on Vercel and only rewrite /api/* to it. The backend has moved twice:

• AWS, until early August. It cost about $99 a month for a product with no paying customers yet, so the estate was hibernated.
• A DigitalOcean droplet, about $24 a month plus backups. It ran the same docker compose stack: Caddy in front, then the API, Postgres and Redis, with no ports published except Caddy's.
• The homelab, from 2026-10-13. It's a VM on the third Proxmox host, reached the same way this site is: through an outbound Cloudflare Tunnel.

Hosting it here wasn't about the money alone. This site has run behind a tunnel since September (how that started), and the pattern had proved itself. The droplet was the last piece of rented infrastructure in the stack.

Design

The request path

browser → Vercel (aegiscloudops.io / fishingrodterminal.io, rewrites /api/*)
        → api.aegiscloudops.io        # Cloudflare DNS, proxied
        → Cloudflare Tunnel           # cloudflared dials OUT from the VM
        → cloudflared → 127.0.0.1     # loopback only
        → Caddy (security headers) → backend → postgres / redis

• Nothing is forwarded on the router. A new home IP changes nothing, and the home IP is never published.
• The tunnel origin is loopback-only. Caddy's tunnel site isn't published to the LAN; only cloudflared on the same box can reach it.
• DNS moved too. aegiscloudops.io's nameservers moved from the registrar to Cloudflare so the tunnel hostname could live in the same zone. Fishing Rod's domain didn't need to move: its API calls already go through the Aegis hostname.

Part 01 · the cutover

Moving it without losing data

1. Copied the compose stack and the backend onto the VM. Secrets were copied by hand and never went near git.
2. Stopped the droplet's backend, dumped its Postgres and restored it on the VM.
3. Brought the stack up on the VM and checked it on loopback before anything public pointed at it.
4. Added the tunnel's public hostname and removed the old A record.
5. Turned the droplet's Caddy into a forwarder to the new home, so clients still holding the old DNS answer kept working.

The droplet's containers were stopped with restart policy no rather than deleted, which leaves a rollback path: restore a dump there, start three containers and put the record back. Deleting the droplet is a separate decision for once the old DNS traffic has died away.

Part 02 · the surprise

Port 8080 was already taken

The plan put the tunnel origin on port 8080. Something was already listening there. It was a MicroK8s API server from an old experiment, started with --insecure-bind-address=0.0.0.0: an unauthenticated Kubernetes API on every interface of the machine about to hold production data.

It was removed rather than reconfigured, since nothing used it. The tunnel origin moved to a different port anyway, so the next leftover can't collide with it either.

Part 03 · deploys

From 90 seconds of 503s to none

The first deploy on the new box produced about 90 seconds of public 503s. Caddy checked the backend's health every 30 seconds, so after a restart it kept the upstream marked down long after the API was ready, and it failed every request in the meantime.

reverse_proxy backend:8000 {
	health_uri /health
	health_interval 5s       # was 30s
	lb_try_duration 30s      # hold a request while the upstream comes back
	lb_try_interval 500ms
}

The next deploy was measured with a probe every 2 seconds: 89 responses, all 200, and one client-side timeout while Caddy held a request. Not one 503.

Part 04 · the outage

25 minutes down, and why it took that long

The same night the internet connection dropped and the host rebooted. The root cause was outside the lab, but the recovery was slower than it needed to be. All six VMs on the host booted at once, and Docker on the production VM took about three minutes to come up while they fought over the disk. Total public outage: about 25 minutes. Nothing alerted on it; it was found by hand.

qm set 101 --startup order=1,up=60   # production first, alone, 60 s head start
qm set 100 --startup order=2         # everything else after it

After a reboot, production now starts first and alone. An external uptime check on the API's /health is next, so the next outage is reported by a monitor rather than noticed by a person.

Part 05 · honesty

The claims the move made false

A migration changes more than where packets go. Three public statements were true on AWS and stopped being true on the night of the move:

• The Data Handling page said production ran in AWS us-east-1.
• The landing page and the Security page said data at rest was on encrypted volumes.
• The security questionnaire sent to prospects said both.

The new disks aren't encrypted at rest. Re-encrypting means rebuilding the VM or its storage, which isn't a change to make unattended, so the claims were corrected instead. The pages now say production runs on company-operated servers in Canada, encrypted in transit, with encryption at rest being re-established. Each questionnaire line is marked verified or gap.

Status

Where it stands

The nightly backups for the new box, and the bug that had been quietly filling a disk on another host, are in their own write-up. Still open: encryption at rest, an uptime alert, and deleting the droplet once its forwarder stops seeing traffic.