Migrating a live product's OS, orchestrator, and database in one window — without losing a row

How I moved a real, user-facing application from k3s-on-Rocky-Linux/SQLite to Talos/PostgreSQL as a single blue/green cutover, with zero data loss and a few minutes of planned downtime.

Most migration war stories are about one layer: a database engine swap, or a Kubernetes distribution change, or an OS rebuild. This one moved all three at once — Rocky Linux + k3s + SQLite became Talos Linux + upstream Kubernetes + PostgreSQL — in a single cutover window, for a web app with real users. The interesting part isn't that it worked; it's why doing all three together was the safer choice, and how the cutover was designed so that the failure mode was always "nothing happened," never "we lost data."

The whole migration in one view: the Cloudflare Tunnel repoints from blue (k3s + SQLite) to green (Talos + CloudNativePG); the database moves exactly once, from an inert SQLite snapshot into PostgreSQL. Continuous WAL archiving to R2 is new on green.

The system, and the failure that forced the move

The application is Cartarch, a self-hosted Magic: The Gathering collection manager. It ran on a four-VM k3s v1.34.6 cluster (Rocky Linux 9.7) on a single Unraid host, with the app's data in a SQLite file on a Longhorn volume, fronted by a Cloudflare Tunnel into Traefik.

The trigger wasn't ambition — it was data loss. Over two days in late May 2026, two filesystem corruptions hit: the app's ext4 volume ("Resize inode not valid," May 28) and then Grafana's SQLite database ("database disk image is malformed," May 29). The root cause was structural: when a stateful pod reschedules, Longhorn detaches and reattaches its volume, and an unclean detach leaves a single-writer filesystem inconsistent. Crucially, Longhorn's replica count does not protect against this — every replica faithfully mirrors the same corrupt bytes. Redundancy at the storage layer is not durability at the database layer.

There was a second, independent problem: after a routine host reboot, the cluster couldn't reliably reassemble itself. A VM boot-order and Longhorn-readiness race below the GitOps layer left it half-up and needing manual kubectl rescue. A rebuild of the same stack wouldn't fix that — it would faithfully reproduce the race.

So the problem statement was concrete: a product serving real users was losing data to a storage/OS failure class that replication couldn't fix, on a cluster that couldn't recover itself after a reboot. PostgreSQL solves the first (crash-safe WAL survives unclean shutdown by design); Talos solves the second (node lifecycle and boot ordering become platform-managed instead of hand-wired). The two fixes pointed at the same move.

There was one false start worth admitting: a premature PostgreSQL bring-up on the old cluster got reverted a day later so the transition could be planned properly. Stepping back cost a day and saved a bad foundation.

The constraints

Before any design, the non-negotiables:

Real users (about six at the time). A few minutes of planned downtime was acceptable; data loss was not.
Exactly one writer at all times. Old SQLite is the sole writer until cutover; new PostgreSQL after. In between, the migration source is an inert point-in-time snapshot, never a live-replicated second writer. No split-brain, ever.
Zero data loss. Rollback had to mean "point back at the untouched SQLite; not one write lost."
The old stack keeps serving the whole time. Parallel build, no teardown until after a confident cutover.
Change one major layer per rebuild. Talos + upstream Kubernetes + PostgreSQL was already a large surface, so Longhorn was deliberately held constant — keeping "is it the storage layer?" a question experience could still answer.
Prove equivalence, don't assume it. The app was kept engine-agnostic specifically so its test suite could run green on both SQLite and PostgreSQL — behavioural equivalence as a gate, not a hope.

Why Talos, and why fold it into the database move

Talos Linux is an immutable, API-managed OS with no shell and no package manager — you configure it declaratively and it reconciles. That directly addresses the reboot-race failure class: node lifecycle, boot ordering, and component readiness stop being hand-wired systemd concerns and become platform-managed. OS updates are atomic and roll back cleanly, instead of "dnf update, reboot, and pray." And in practice nearly all operations were already kubectl against the API — a shell-having host OS was buying very little.

I considered the alternatives explicitly. Staying on Rocky/k3s didn't address the reboot race and kept the hand-wired substrate. Running k3s on Talos made no sense — Talos already provisions Kubernetes; layering k3s on top just re-adds the opinionated substitutions (its bundled Traefik, its ServiceLB) that upstream Kubernetes avoids.

Folding the PostgreSQL migration into the cluster rebuild was itself a decision, not an accident. The database cutover I needed anyway and the resilient-cutover mechanism I wanted were the same move: build a second cluster in parallel, migrate onto it from an inert snapshot, and switch traffic when confident. Doing them separately would have meant two risky windows instead of one.

The target: Talos v1.13.3 running Kubernetes v1.35.4 — pinned deliberately below Talos's bundled 1.36, which exceeded Longhorn v1.11.2's tested ceiling. Three control-plane nodes (stacked etcd, an HA API VIP) plus one worker, all VMs on the one Unraid host. The database is CloudNativePG running PostgreSQL 18 as a three-instance cluster (one primary, two replicas with pod anti-affinity), with continuous WAL archiving and daily base backups to Cloudflare R2 via the Barman Cloud plugin. Ingress on the new cluster is deliberately minimal — a NodePort with cloudflared pointing at it — because the traffic switch is a tunnel repoint, not a load-balancer dance.

Moving the data safely

The migration is a cross-engine conversion (SQLite → PostgreSQL), so streaming replication doesn't apply — the source is a snapshot, converted once.

The first real decision was tooling: I rejected pgloader, the obvious choice. Once I'd made Alembic the authoritative schema owner, pgloader's core value — schema translation — was gone, and what remained actively fought the design. It drops and recreates foreign keys (which would clobber the ON DELETE topology I'd carefully recovered), it doesn't topologically sort inserts (a probe failed on exactly that: a child row referencing a not-yet-loaded parent), and it's effectively unmaintained. The replacement was a short script that loads through the application's own SQLAlchemy table definitions, so booleans, timestamps, and cache columns coerce correctly by construction rather than by guesswork. It loads parent-before-child with foreign-key enforcement temporarily off (session_replication_role = replica), then resets every sequence. The guiding principle: the rehearsal artifact and the cutover artifact are the same script — only the connection strings differ.

The load ran as four ordered, single-shot Kubernetes Jobs, deliberately not wired into a single kustomization so they couldn't all fire at once:

Schema — alembic upgrade head builds the baseline into the empty database.
Load — the SQLite→PostgreSQL copy (28 tables: 21 ORM-managed + 7 raw-SQL).
Sweep — remediate known foreign-key orphans per the schema's declared ON DELETE intent.
Validate — a hard GO/NO-GO gate (below); exit 0 means go.

One detail from this stage is the kind of thing that only shows up in a rehearsal. SQLite in WAL mode keeps recent writes in a side file; copying the main database file without first checkpointing captures stale, un-flushed data. That actually bit a verification run and produced phantom orphans until I traced it. The fix is to run PRAGMA wal_checkpoint(TRUNCATE) on the live pod — and to do it before freezing the app, because a scaled-to-zero pod can't be exec'd into. The snapshot's integrity was then chained end to end: PRAGMA integrity_check, plus a SHA-256 recorded when it left the old pod and re-verified byte-identical after it landed on the new cluster.

Schema safety: Alembic as the single owner

The schema itself needed hardening before it could survive the engine change. The Alembic baseline was autogenerated against SQLite and then hand-corrected for PostgreSQL. The most important class of fix: boolean server_defaults. Autogenerate emits integer literals (0/1), which are valid in SQLite but fail outright on PostgreSQL ("column is of type boolean but default is integer"). Beyond that, diffing the generated baseline against the live production schema recovered eight invariants a pure model-derived baseline would have silently dropped — six foreign- key ON DELETE rules and three partial-unique indexes. Reading what was actually running in production, not just what the models implied, is what kept those from vanishing.

Two operational rules make schema changes safe from here on. First, migrations run before the app rolls — an ArgoCD PreSync hook runs alembic upgrade head ahead of the new pods, so old code never meets a schema it doesn't understand. Second, the app no longer builds schema on PostgreSQL at all; Alembic is the sole owner, with the ORM's create_all gated to the dev/SQLite path only. The discipline is additive, migrate-before-deploy — new columns land nullable and ahead of the code that needs them. (It is not a formal two-phase expand-then-contract process, and I don't claim it is.)

Cutover and rollback

The rollback model was chosen for the user count, not for a whitepaper. At six users, a full write-freeze was correct: the app is briefly down, nothing writes to either database during the window, so aborting means pointing the tunnel back at the untouched SQLite with zero writes lost. Dual-write and reverse-sync were explicitly rejected as over-engineering at this scale.

The traffic switch is a Cloudflare Tunnel repoint — seconds, no DNS TTL to wait on. Before, the tunnel targeted the old cluster's Traefik; after, the new cluster's NodePort. Rollback is the same edit in reverse.

The executed sequence: checkpoint and copy the snapshot while the old app is still running → freeze the old app (scale to zero) → deliver the snapshot to the new cluster and verify its SHA-256 → run the four Jobs (schema, load, sweep, validate-gate) → bring the app up on the new cluster → smoke-test through a port-forward → repoint the tunnel → smoke-test through the tunnel, including a create-and-delete to prove writes land in PostgreSQL.

The run logs show two entries a few minutes apart, which is worth explaining so it doesn't read as a failure: the first was a deliberate --preflight check (verify everything, change nothing, exit); the second was the real run, gated on a human typing yes, which ran to completion. Rollback was never used.

User-facing downtime was about two minutes and forty seconds — the interval between freezing the old app (scaling it to zero, 14:06:40) and the tunnel repoint completing (14:09:19), per the cutover log's timestamps. With only a handful of active users at the time and no formal outage announcement, the window passed without anyone reporting an interruption.

Proving it worked

Two independent validations, one for the code and one for the data.

For the code, the whole point of keeping the app engine-agnostic paid off: the same test suite runs against either backend via one environment variable, so a green run on both SQLite and PostgreSQL 18 proves behavioural equivalence rather than assuming it. That count grew as fixes landed — 263 tests at rehearsal, 278 by the final pre-cutover build, passing on both. A dedicated foreign-key harness deserves a mention: it's metadata-driven, so adding a foreign key forces you to account for its delete behaviour, and it runs every parent-delete path across both foreign-key postures on both backends. On a fresh production pull it found 22 orphaned rows actively accruing, and one genuinely dangerous case: a foreign key set to NO ACTION meant that under PostgreSQL's enforcement, deleting any user who'd ever recorded a game would abort the entire delete with a violation — a crash, not a silent orphan. The fix (change it to SET NULL and snapshot the player's name onto the game) went into the baseline before cutover.

For the data, the fourth Job was a five-check GO/NO-GO gate: zero foreign-key orphans; per-table row-count parity; a boolean round-trip run through the app's real query path (to catch the 0/1- versus-native-boolean hazard end to end); a sequence-collision probe; and a byte-identical round- trip of the app's Scryfall cache seam. Only an all-pass returned exit 0.

And before betting production data on any of it, the backup/restore path was exercised repeatedly — a throwaway restore from R2 (objects independently listed with rclone, not trusted from status), a fresh cluster recovered from R2 to healthy in about a minute, and a data-fidelity restore on cutover morning that matched row counts and preserved booleans exactly.

What happened

The cutover shipped on June 22, 2026. No rollback, no data loss, and faster than the twenty minutes I'd budgeted. The new Talos cluster became the sole production environment and the old k3s stack was decommissioned — which also resolved a memory-oversubscription problem that had existed only while both clusters ran. The app has run a steady, auto-deployed release train since, with Alembic as the sole schema owner via the PreSync hook. The migration didn't just land a database; it left a repeatable schema-change process behind it.

Two rough edges, kept in because they're the truth and they read as judgment rather than polish. Standing up three PostgreSQL instances while the old cluster was still running briefly put more VM memory in play than the host physically had, starved the host's management plane, and forced a hard power-down during the build — with zero data loss, because production never moved during that phase and the new cluster was still empty. Root cause was honest planning error: I'd sized the VMs against scheduling headroom, not host-backable memory. And the day before cutover, an unpinned dependency drifted a minor version on rebuild and broke an entire class of routes in production; the "pre-existing benign test failures" I'd been waving through turned out to be the suite correctly detecting exactly that. I pinned the whole stack and the suite went fully green. The lesson I took: a failing test is a hypothesis about a real defect until proven benign. The migration went onto a foundation that was, finally, still enough to move.

What I'd change at real scale

This worked because it's small, and the honest version of the story says so.

The single-host SPOF is real. Redundancy here is at the Kubernetes and Longhorn layer, not the hardware layer — one physical host failing takes the product down. It's an accepted risk at this size; at an employer it's a second physical host, redundant networking, or managed infrastructure.
The cutover pattern doesn't scale. A full write-freeze is fine for six users and a few minutes. At real scale you'd need logical replication or dual-write with reverse-sync for a genuine zero-downtime cutover, and a backfill-and-verify rather than a single frozen snapshot.
Single app replica, no connection pooler. In scope by choice here; at scale it's a multi-replica app, PgBouncer in front of PostgreSQL, and a real HA ingress controller instead of one NodePort and one cloudflared.
DR is now measured at the data layer, not yet at the disaster layer. I run a non-destructive restore drill: prod's R2 backup chain is restored into an isolated throwaway cluster and validated against production. The most recent run recovered the full database from object storage in 99 seconds (single instance, apply-to-healthy) with effectively zero RPO — the restore was byte-current with prod (identical schema head, row counts, and latest-transaction timestamp), because WAL archiving is continuous. What's still not drilled is the full-disaster path: rebuilding the cluster from bare metal and then restoring, measured end to end against an RTO target. At an employer that's a scheduled game-day with an SLO, not an aspiration — but "I restore production from backups on a cadence and measure it" is now a fact, not a hope.
Observability shipped thin. The new cluster went live with app-metrics scraping deferred. Enterprise-grade means dashboards, alerts, and SLOs before cutover, not after.

The gap between a personal platform and a production one isn't the parts list — it's the redundancy, the rehearsed recovery, and the automation of the human gates. Knowing exactly where those gaps are, and why each was an acceptable trade at this size, is the actual deliverable.