Moving bitspur to the new line (design)
Status: open design (SHC-1061). Nothing here is built and nothing has moved. It answers what the simplification design §6 left to SHC-1061: the order of apps, how the shared databases and mailcow move, and what happens to the rig VMs on the Gorilla boxes. It keeps §6’s mechanics (the archive is what crosses, the manifest rewrite, the write barrier, takeover then reconcile for DNS) and changes its rollback (section 5.8). No app moves before the cutover gate, scheduler §7 (SHC-1165), passes on the rig.
The shape, ruled by Clay on 2026-10-01:
- Fresh nodes. compose01 is retired, never upgraded in place.
- App by app, with both estates live. A deployment is stopped on the old estate, brought up on the new one, verified, and its DNS switched.
- Each app’s data arrives by restoring its restic backup, taken fresh right before its move. Volumes are never copied off compose01. The move proves the backups.
- The new nodes are VMs on both Gorilla boxes under Proxmox, docker runtime. Bare Incus on those hosts comes after the cutover; so does the two-tunnel mesh (SHC-1201). The move rides today’s Nebula plus fabric.
- compose01 stays untouched for two weeks after the last app moves. Switching back is a DNS change plus starting the old deployment.
The migration is a one-off script outside the product. promanager is frozen and not touched.
Evidence: a read-only inventory of bitspur taken 2026-10-01 19:19Z to 19:58Z,
attached to SHC-1061 (inventory.md, inventory.json, the raw listings).
Every number below comes from it unless it names another source. A Codex
judge pass (high effort) read this design against that evidence and blocked
the first draft on 18 findings; every one is answered below.
1. What moves
Section titled “1. What moves”| deployments | 68 (corp 61, master 4, shc 3), plus 5 plugins with no data of their own |
| data in restic | 88.0 GiB in the newest scheduled snapshots, every one dated 2026-09-30 or 2026-10-01 |
| data on compose01’s disk | 84.2 GiB under /var/lib/shc/deployments; swarm01 and swarm02 refuse our key, so GitLab and ollama (swarm01) and penpot and pokemon-kc (swarm02) are measured through their backups only |
| data outside restic | 26 Hetzner hel1 buckets bound by 9 deployments (Nextcloud’s primary store was 41,030 objects, 57.1 GiB on 09-30; GitLab’s registry, artifacts, LFS and packages; peertube, pulp, huly, penpot, openreplay, affine, supabase) |
| memory | compose01’s containers hold 9.4 GiB resident and 29.3 GiB in swap; with GitLab, ollama and penpot the estate’s working set is about 50 GB |
| sources | 47 from the catalog, 4 deb-shipped builtins in master, 16 path-pinned (Nextcloud, nc-db, wazuh, keygen, keygen-db, license-broker, ivatar, onlyoffice, immich, postiz, marketplace, ses, openhands, gitlab-huly, mpo-storybook, pokemon-kc) and 1 git-pinned to the shc repo on git.corp (pokemon-web), each carried as a local package |
| DNS | 70 A records through the cloudflare connection, all 2.29.3.100, TTL 300; 66 belong to a deployment, 4 do not (auth.corp.bitspur.com, shc.hel1.hetzner.bitspur.com, traefik.corp.bitspur.com, and docs.corp.bitspur.com, which is not desired); MX of bitspur.com and flyingfoxlabs.in is 10 mail.bitspur.com and is not an SHC record |
| vault | about 5,600 entries across 74 deployment scopes (09-30 count), copied key by key, never re-minted |
Not covered by any backup, or not proven to be:
- The 26 buckets.
corp/hz-s3hasbackup.bucketsempty on purpose (arming it once filled compose01’s disk). They are bound again, not copied, and nothing protects them against deletion or a provider loss, before or after the move. GitLab’s registry is one of them (SHC-812). - LibreChat’s conversations.
corp/librechat-dbholds 456 MiB of MongoDB on disk and its backups are under 1 MiB, and the librechat template says its own cross-boundarymongodumpwas removed. Until a rehearsal restore shows LibreChat’s users and known conversations, treat them as not backed up; librechat does not move before that. corp/mysqlbacks up 17 MiB against 2.6 GiB on disk. Its four consumers’ databases are in that dump or nowhere; the rehearsal restore has to show each one’s tables and rows. (sonarqube’s 704 MiB is its Elasticsearch index, which rebuilds; its database is in corp/postgres.)- 57 anonymous Docker volumes attached to running containers. None holds
data that matters: Frappe
sites/assets(rebuilt), pulp’s/var/lib/containers(a cache), image-declaredVOLUMEs of client containers whose real data sits in a bind mount (checked for every postgres, mysql and mongo container), jitsi and coturn config. shc/ffx-frappe:based93, the image ffx runs, exists only in compose01’s Docker store with no registry digest. A fresh node cannot pull it, so it is pushed to the corp registry and pinned by digest before wave 0’s rehearsal (SHC-691).- Control-plane state (tenants, config, connections, grants, schedules, retention, DNS ownership, the licence) is not app data. The script recreates it on the new cluster from the old API, per §6.
Not moved, by recommendation (open question 7): corp/nc-db has no
consumer left since Nextcloud moved to the shared postgres, and ffx’s mount
bench.pre-based93 (2.7 GiB) is a stale copy beside the live bench. Both
stay on compose01 and go with it. docs.corp.bitspur.com is already not
desired and is not moved. traefik.corp.bitspur.com is the old Traefik’s
dashboard (dashboardHost); the new line’s per-node Traefiks have no shared
dashboard name, so it is retired with compose01 by decision, not because
the record is stale.
2. The node plan
Section titled “2. The node plan”| node | host | VMID | vCPU | RAM | disk | address | role |
|---|---|---|---|---|---|---|---|
| shc01.odg.bitspur.com | pve02 | 110 | 16 | 28 GiB + 16 GiB swap | 500 GB | 146.71.124.78/29 | voter, workloads, mail |
| shc02.odg.bitspur.com | pve03 | 120 | 16 | 28 GiB + 16 GiB swap | 500 GB | 146.71.124.68/29 | voter, workloads |
| shc03.odg.bitspur.com | pve03 | 130 | 2 | 4 GiB | 40 GB | 146.71.124.69/29 | voter only |
One workload VM per box. A box is the failure domain, so a workload node per box keeps “one replica per node” honest: two workload VMs on one box would let the engine believe two replicas are apart when one power cut takes both. It also fits the addresses: pve02 has only .78 left while the rig holds .75 to .77, pve03 has .68 to .70 (all four measured silent on 10-01).
Sizes. The estate’s working set is about 50 GB; two workload nodes at 28 GiB plus swap carry it with room, and both boxes keep the rig running beside them (pve02 44 of 60 GB with the rig’s three 4 GB nodes, pve03 50 of 60 GB with shc03, far01 and the SHC-1199 and SHC-1202 proof VMs). After the rig goes, each workload node grows to 44 GiB, one at a time with a drain, which is also what lets one box carry the whole estate while the other is reinstalled as a bare Incus host. Disk: 84 GiB of deployment data plus images and restore staging fits 500 GB on each, with 370 GB of each box left free.
Three voters on two boxes, and what that does not buy. With two nodes
the control plane has one voter, and losing it stops every lease-held app
on the survivor
(simplification §4.9).
So shc03 is a small voter-only VM in a reserved fence no app tolerates: it
runs rqlite, shcd and the per-node platform set and nothing else. The ruling
puts the new nodes on pve02 and pve03, so shc03 sits on pve03 beside shc02.
That tolerates losing pve02 or any one VM, and does not tolerate losing
pve03: two of three voters go with it, lease-held apps on shc01 stop
renewing, and recovery is repair --force on shc01 or pve03 coming back.
Before pve03 is ever taken down on purpose (the Incus reinstall), the
voter-only role moves first: a voter-only VM joins on pve02, then shc03 is
removed while all three voters are up, so pve02 holds two of the three
voters when pve03 goes down. Adding a fourth node alone would not do it:
four members still give three voters, and the engine may leave two of them
on pve03. A voter-only VM on pve01, the third GorillaServers box
in the same datacenter (0.5 ms from pve02, measured), would make any single
box survivable; that needs Clay to widen the ruling (open question 1).
Fences.
| fence | kind | members |
|---|---|---|
control-plane | control-plane | shc01, shc02, shc03 |
gorillaservers/ogden | location | shc01, shc02, shc03 |
gorillaservers | provider | shc01, shc02, shc03 |
voter-only | reserved | shc03 |
One location, because both boxes sit in one datacenter. With one workload node per box, “one per node” already spreads across boxes; naming a box a location would mislead every later estate.
Cluster identity. The new cluster is a new generation of the same
cluster: slug shc-hel1-hetzner-bitspur-com, DNS heritage
shc/owner=shc-hel1-hetzner-bitspur-com-a995f257. The takeover only adopts
a sibling generation’s claim, and the licence is bound to
shc.hel1.hetzner.bitspur.com and auth.corp.bitspur.com (installed on the
new cluster by file, its fallback carrier). During the move the old console
keeps shc.hel1.hetzner.bitspur.com; the new cluster is driven by the Mac
CLI at shc.odg.bitspur.com, a new record the new cluster owns, and the
console name flips in the last wave. The hel1 names are renamed after the
move, with a re-minted licence (open question 5).
The SHC-1123 rig (VMs 200 to 203). It stays through the move. The cutover gate is proven there, and each wave is rehearsed there first against its own small old-line cluster. It uses at most half of each box and does not collide with production (VMIDs 200 to 209, addresses .75 to .77 and .67). After the last wave, and after SHC-1201’s mesh rehearsal, which needs a standing two-site cluster, the rig is torn down (lock released, module block deleted) and the production nodes grow into the room.
3. The order: move units and waves
Section titled “3. The order: move units and waves”3.1 The graph
Section titled “3.1 The graph”The inventory holds 119 plug edges. What decides the order is whether a
consumer reaches its provider by a public hostname or privately (a
socket field such as internalUrl, hostname, apiUrl, a database
socket, or a container network alias). A private edge cannot cross the
estates, and in most templates the socket value wins over any var, so it
cannot be pointed at a public hostname either.
| edge | how it is reached | consequence |
|---|---|---|
29 oidc, 3 saml to master/keycloak | the public issuer https://auth.corp.bitspur.com, except identity-broker’s API calls (socket.hostname, port; overridable by vars.oidc.apiUrl) and both mailbox reconcilers (http://${KC_HOSTNAME}:${KC_PORT}, no override) | 29 cross by hostname; identity-broker gets oidc.apiUrl; the reconcilers pause (3.2) |
10 smtp, 3 imap to mailcow | mail.bitspur.com:587/993, which mailcow’s network aliases onto postfix and dovecot | public from either estate once the old containers are stopped (the alias goes with them) |
3 mailcow admin API, matrix, matrixAppservice, matrixAdmin, 4 nextcloud via internalUrl, affine, fizzy, identityBroker, gitlab, llm, frappe, keygen, goalert, clickhouse, redis, mongodb, every mysql/postgres edge | private | the two ends share a move unit |
| Nextcloud and office | office calls Nextcloud back through a private tenant alias (its template says so) and Nextcloud calls office | one unit |
3 webhook to hookshot | hookshot has no hostname; GitLab, glitchtip and wazuh post to it privately | glitchtip joins hookshot’s unit; GitLab and wazuh lose notifications for a day or two (3.4) |
nextcloud from bit, ffx | read by a Frappe addon, not the compose | which URL it uses is read on the rig; if private, wave 6 runs the night after wave 4 and DAV sync pauses that day |
9 s3 to hz-s3 | the hel1 endpoint, public | no constraint |
3.2 Move units
Section titled “3.2 Move units”A move unit is a connected component of private edges. Its members share one write barrier: every member stops (consumers first), every member is backed up, every backup is restored, providers start before consumers, every member is verified, and only then do their DNS records switch together. A unit is never left half-moved: if anything fails the whole unit rolls back (5.8). Most units are one deployment.
A deployment whose private providers sit in two different units stays
stopped from the first of those units’ barriers until the last has moved,
then moves with the last. Two need it: gitlab-fizzy (fizzy in wave 4,
GitLab in wave 5) and both mailbox reconcilers (Keycloak in wave 1, mailcow
in wave 4). While they are stopped, GitLab-to-Fizzy sync stops and new
Keycloak users get their mailbox by hand. A paused deployment’s fresh
backup is the one taken at its pause, through the running-app path right
before it stops; it does not change while stopped, so that verified capture
stands in for the scheduled-backup freshness check of section 7, with the
script’s journal and the container state as proof it stayed stopped.
The reconcilers also cannot resume as they are: on the new line Keycloak is
in another tenant, so their private http://${KC_HOSTNAME}:${KC_PORT} is
refused. They move at a new package version that reads a public API base
(vars.oidc.apiUrl, the override identity-broker already has), filed
as a catalog change before wave 4 and proven in rehearsal. It is the one
deliberate version change in the move.
3.3 Reaching back between waves
Section titled “3.3 Reaching back between waves”Only by public hostname. There is no tunnel, Nebula link or shared network
between the estates. 47 deployments are LAN-only (exposure: lan),
Keycloak among them, and a request from one estate to the other’s public
address arrives from that estate’s egress, which neither LAN allowlist
admits today. So each side’s tenant lanAllowlistCidrs admits the other
side’s node addresses for the duration (old: 2.29.3.100, 2.29.2.229,
2.29.10.60; new: 146.71.124.78, 146.71.124.68, 146.71.124.69). It is written
on the old cluster before wave 1 and stays through the two weeks, because a
rollback needs it.
3.4 The waves
Section titled “3.4 The waves”A wave is a set of units moved one unit at a time in one window, in the order listed. Window estimates use tonight’s capture times, a pull of about 1 GiB a minute from hel1 (17 MB/s single stream measured from pve02; restic runs several streams), and 5 minutes fixed per deployment; the rig rehearsal replaces them with measured numbers. Section 7 lists when no window may run.
| wave | units, in order (one unit per brace) | why here | window |
|---|---|---|---|
| 0 build | nothing moves: cluster, platform set, connections, tenants, config, vault, repos and local packages, the licence; empty engines corp/postgres, corp/mysql, corp/mariadb at the old majors and image digests; the hz-s3 and ses adapters; every image by digest; every certificate by DNS-01; bucket versioning; every unit’s rehearsal restore in isolation (5.1) | no traffic, no DNS change | days, no downtime |
| 1 identity | {master/postgres, master/keycloak, the record auth.corp.bitspur.com}; {master/openobserve} | the new cluster’s shcd, forward-auth and every restored app need the identity provider it administers (6.3) | about 20 min of no logins; openobserve about 75 min of no new logs |
| 2 self-contained | {wellknown}, {lookout}, {mpo-storybook}, {openhands}, {crowdsec}, {uptime}, {marketplace}, {workflows-redis, workflows}, {shc/keygen-db, keygen, license-broker}, {pokemon-db, pokemon-kc, pokemon-bench, pokemon-web}, {ollama, librechat-db, librechat} once LibreChat’s data is proven | no private edge leaves its unit; small units prove the runbook; license-broker also answers promanager’s six-hourly licence reverify, so its unit’s window stays under 15 minutes | about 3 h over two nights |
| 3 single consumers | {ciso}, {docuseal}, {dtrack}, {ivatar}, {openreplay-ch, openreplay}, {peertube}, {penpot}, {photos}, {postiz}, {pulp}, {sonarqube}, {superset}, {supabase}, {blog with its two plugins}, {fleet}, {mautic} | each reaches only its databases, Keycloak, mail and S3 | about 5 h over two or three nights |
| 4 mail and collaboration | one unit: {mailcow, mailbox-reconciler, mailbox-reconciler-ffx, identity-broker, matrix, hookshot, glitchtip, nextcloud with its plugins mail and integration_paperless, office, pdf, projects, affine, fizzy, crm} | one component of private edges (identity-broker reaches mailcow, matrix and Nextcloud; projects reaches matrix, Nextcloud and identity-broker; office and Nextcloud call each other; Stirling is wired to Nextcloud by its plug) | about 3 h, every member down for all of it |
| 5 GitLab | {gitlab, gitlab-huly, gitlab-fizzy} | GitLab alone is 12.9 GiB plus its database; the bridges reach it privately | about 1.5 h, CI paused |
| 6 Frappe and SIEM | {bit, ffx with ffx/live}; {wazuh, oncall} | bit and ffx share the mariadb engine; wazuh posts alerts to oncall privately | about 2 h |
| 7 VPN and console | {netbird}; the record shc.hel1.hetzner.bitspur.com | moving NetBird re-points the VPN path to LAN-only apps; last, so every LAN app is already where the VPN lands | about 45 min |
| hold | nothing stops on compose01, the old engines included, for two weeks; then the old cluster stops with its disks kept until Clay signs off | rollback needs every old provider running |
A wave may stop after any unit and resume the next night.
4. The shared database engines
Section titled “4. The shared database engines”| engine | consumers, with every database each one uses |
|---|---|
| corp/postgres | 23 consumers, 25 databases: affine (affine_rs), ciso (ciso_assistant), docuseal, dtrack, gitlab (gitlabhq_production, role gitlab), glitchtip, identity-broker (broker), ivatar, matrix (synapse and synapse_mas), netbird, nextcloud, oncall (goalert), openreplay, peertube, penpot, photos (immich), pokemon-kc (postgres, role postgres), postiz (postiz and postiz_temporal), projects (huly), pulp, sonarqube, superset, workflows (activepieces) |
| corp/mysql | blog, crm (monica), fleet, mautic |
| corp/mariadb | bit, ffx |
| single-consumer engines | master/postgres (keycloak), corp/pokemon-db (pokemon-bench, frappe), corp/librechat-db (librechat, LibreChat), corp/workflows-redis (workflows), shc/keygen-db (keygen), corp/openreplay-ch (openreplay) |
Recommendation: stand up each shared engine empty in wave 0 and move its consumers in their own units, each with its own databases. Do not move an engine with all its consumers in one wave.
Moving corp/postgres whole means the engine and all 23 consumers down in one window (the 09-30 dry run put 26 deployments, 92.5 GiB and a 148-minute write barrier in that group): the big bang the ruling rules out, and rolling back any one of them rolls back all of them.
It costs nothing new, because the engine’s backup already holds one dump per
database: the postgres addon writes databases/<n>.sql.gz with a .name
beside it and globals.sql for roles (ce addons/postgres/backup.yaml),
and the mysql and mariadb apps write the same layout with
system-mysql.sql.gz for users and grants. For each unit:
- Write barrier on the unit (3.2).
shc backup create postgreson the old cluster (5.2 minutes for corp/postgres tonight), once per unit whatever the number of consumers in it. The unit is stopped, so its databases are quiet.- The script pulls that
.bakfrom the restic repository, extracts every database of every consumer in the unit (all of them, from the table above) and their roles fromglobals.sql, creates the roles and databases on the new engine and replays them, then checks each table count against its dump, the check the addon’s own restore makes. pokemon-kc lives in the maintenance databasepostgresas the superuserpostgres: its dump replays into the existingpostgresdatabase with no create, and the role already exists with the password the new engine carries from the copied vault. (A consumer on the superuser is its own follow-up.) - Each consumer’s own backup, also taken in the barrier, is restored into its new deployment, and it starts against the new engine.
The data still arrives from restic, so the ruling holds, and every unit proves more of the engine’s dump. The old engine keeps serving the consumers that have not moved and keeps running through the two weeks, so a rolled-back consumer finds its old database where it left it. Engines match majors on both sides (the restore refuses otherwise) and upgrade after the move. Single-consumer engines move with their consumer, whole.
5. The runbook
Section titled “5. The runbook”Every unit, in its wave, through the script (section 7):
5.1 Prepare (days before, wave 0)
Section titled “5.1 Prepare (days before, wave 0)”- Rehearsal in isolation. Each unit’s newest backup is restored once
into a throwaway tenant
rehearsalon the new cluster, with its hosts unrouted, its S3 and SMTP relay credentials blanked, no plug job run against any production provider (Keycloak, mailcow, the buckets), and outbound mail and webhooks dropped. It proves the archive restores and the data is there (row and file counts against the source), not that the app runs against production. The rehearsal tenant is deleted after. This is gate condition 9 for bitspur. - Images on the new nodes by digest; certificates for the unit’s hostnames issued by DNS-01; local packages uploaded; vault keys copied.
- Images pulled ahead of their wave are unused images. On a docker
node an install counts the bytes unused images hold as room and removes
the least recently used of them before it pulls (SHC-2621), and
image_prune_cronremoves any older than a day. An image is safe only while a container, running or stopped, is on it, a deployment on the node names it or keeps it in its rollback reserve, or the node cannot pull it back. So pull a unit’s images in that unit’s own prepare, or install the unit stopped first. Never clear images by hand (docker image prune -a,docker rmi).shc/ffx-frappe:based93is pushed and pinned by digest before it reaches a new node (section 1); a copy that cannot be pulled back is kept. - Bucket versioning on every bucket the unit binds, proven by deleting and recovering a test object, so a rollback can return the bucket to its state at the barrier. The buckets are in no backup, so a bucket that cannot be versioned makes its unit forward-only from 5.4: no rollback once the new copy has started, and Clay is told before that unit’s window.
- The new deployment is installed and left stopped, at the same package and
image digests, vars, overrides,
--hostand--portmappings, and the same plugs except SSO (section 8). Plug jobs that would change something a running old copy also uses (a Keycloak client’s redirect URIs, a mailbox, a bucket policy) are deferred to after the barrier; the copied vault means the rest find their credentials and change nothing.
5.2 TTL
Section titled “5.2 TTL”Every record is already TTL 300 (the DNS module’s default) and the old cluster’s DNS reconcile would rewrite a hand-lowered TTL, so it stays 300. Clients with a cached address reach the old ingress for up to five minutes after the switch and get 503 for that host.
5.3 Write barrier and fresh backup
Section titled “5.3 Write barrier and fresh backup”Cut the old copies’ ingress, drain (for office, open documents force-save first), quiesce background writers the dry run lists, take each member’s final backup through the old line’s running-app path plus the engine backup of section 4, verify them, stop the old copies.
5.4 Restore
Section titled “5.4 Restore”Into the new deployments (manifest rewrite, shc backup import, restore,
per §6), then run the deferred plug jobs and start providers before
consumers. This is the first write on the new side; from here a rollback
is no longer free (5.8).
5.5 Verify
Section titled “5.5 Verify”Through the new cluster, before DNS moves, from test clients that resolve
the unit’s hostnames to the new node (curl --resolve, a browser profile
with host-resolver rules, extra_hosts on a test runner), and confirm in
the new Traefik’s access log that the new side answered. Apps on the new
side that call a same-unit public name (Nextcloud to office, any app to
mail.bitspur.com in wave 4, anything to auth.corp.bitspur.com in
wave 1) would still resolve the old address, so for 5.5 the script pins the
unit’s hostnames to the new node for the unit’s own members only and drops
the pin at 5.6. Every other deployment, moved or not, keeps resolving public
DNS and reaches nothing of the unit until the switch. Until then the new
ingress admits the unit’s hosts only from the test clients’ addresses and
from the unit’s own members, whose pin points at the node’s ingress over
the fabric, so ingress sees their deployment’s fabric sources, not a node
address every deployment shares. No production request reaches a new copy
before 5.6:
| class | check |
|---|---|
| web app | its health path; a login through SSO completes, callbacks included |
| database consumer | per-database table counts equal the dumps’; the app reads a record created before the barrier |
| object-store bound | lists a known object and round-trips a test file |
| identity | discovery document and JWKS; a token for a test client; a gated app’s login |
banners on 25, 465, 587 and 993 with the mail.bitspur.com certificate; IMAP login; a message sent straight to the new node’s 25 is delivered; one sent out leaves through SES (its Received chain); DKIM verifies at the receiver | |
| git | git ls-remote over HTTPS and over SSH on 2222, both with the address pinned; docker pull of a known tag; one CI job after the switch, in the watch |
| Frappe | bench --site <site> doctor; a login; a document opens with its CSS |
| VPN | a peer connects; LAN-only apps answer through the route; split DNS answers the new addresses |
5.6 DNS switch
Section titled “5.6 DNS switch”The new cluster takes over every record of the unit’s deployments, plus any
platform record the unit carries (auth.corp.bitspur.com with Keycloak,
shc.hel1.hetzner.bitspur.com in wave 7), and its DNS reconcile points them
at the new node.
5.7 Watch
Section titled “5.7 Watch”24 hours: errors in OpenObserve, the Uptime Kuma monitor, the logs. That night the new cluster takes each member’s first new-line backup and verifies it; the unit is not done until those backups exist.
5.8 Rollback
Section titled “5.8 Rollback”Always the whole unit, never one member, and back to the state each member had before the unit’s barrier: a deployment that was already paused (3.2) stays paused. Every old provider a rolled-back unit needs is still running, because nothing on compose01 stops during the move or the two weeks except the units being moved and the paused ones. Every rollback also restores the split-DNS answers for the unit’s own hosts (6.8), from the values the script journalled, and checks a VPN client resolves and reaches the old copy. The NetBird routes to the new nodes are shared by every moved unit, so a rollback never removes them; they go only when no moved unit is left on a node.
- Before 5.4 (no write on the new side): start the old copies, restore their ingress. Nothing public changed. Free.
- After 5.4, before 5.6, for a unit with no background work that acts on the outside world: stop the new copies, return each bound bucket to its state at the barrier from its versions, start the old copies. Only the new copies’ own migrations and verification traffic are discarded. A unit whose apps process on their own once started (mailcow’s queue, GitLab’s CI and mirrors, Huly, wazuh, workflows, postiz, the Frappe schedulers; the dry run lists them, as it lists background writers for 5.3) has no step 2: from its first start its state is production state, and rollback is step 3. A forward-only unit (5.1) has no step 2 or 3.
- After 5.6 (users wrote to the new side): rollback is a reverse restore: quiesce the new unit, take its new-line backups, restore every database and volume of it into the old copies (the archive format is the same on both lines during the move, which wave 0 rehearses once per app class), leave each bucket at its current versions (the restored databases are current too, so the objects they name exist), then switch DNS back by the old cluster’s own takeover and reconcile, then start the old copies. If the reverse restore has not been rehearsed for that class, or fails, there is no rollback after the switch: the fix goes forward on the new side. Starting the old copy on its pre-move data while users have written to the new one is never done, because it silently drops their writes.
6. The hard ones
Section titled “6. The hard ones”6.1 mailcow
Section titled “6.1 mailcow”Mail must not bounce. Moving mailcow changes one record, the A record of
mail.bitspur.com; MX stays 10 mail.bitspur.com for both domains, and
SPF’s mx follows the A record by itself.
- No bounce. While the old postfix is stopped and before the record flips, a sender gets a refused connection on 25, a temporary failure: it queues and retries for days. Nothing answers 5xx. The five-minute TTL tail lands on the stopped old postfix and retries the same way.
- The relay credentials trap. Outbound goes through Amazon SES
(
relayHost,relayUsername,relayPassword). They are persisted overrides now, withmailDomain,keycloak.version,hostnameandapppass.audience, and the new install carries all seven. A mailcow install without them falls back to direct delivery and reports healthy; 5.5 reads theReceivedchain for SES. - The certificate on raw TLS ports. 465, 993, 995, 587 and 143 serve
CN=mail.bitspur.comfrom Let’s Encrypt (issued 09-29, read on the live ports). The new cluster issues it by DNS-01 in wave 0, mailcow’s certificate sync places it for dovecot and postfix, and 5.5 reads it withopenssl s_clientagainst 146.71.124.78 on every port. - IP reputation. 146.71.124.78 is not on Spamhaus ZEN, Barracuda, SORBS
or PSBL (checked 10-01 from the Mac; rechecked from the node’s own
resolver before wave 4). Outbound relays through SES, so the address only
matters if the relay is lost. Its PTR is set to
mail.bitspur.comwith GorillaServers anyway (today’s PTR is Hetzner’s generic name). Outbound 25 from pve02 is open (measured). - Placement. Its ports are published on the node that runs it and
mail.bitspur.commust resolve to exactly that node, so mailcow is pinned to shc01 until multi-replica--portis ruled. - After the switch mail lands on the new side only. Rolling back after that is a reverse restore of the whole mailcow backup (mailboxes, database, configuration and queue) into the old mailcow, never imapsync alone. This is the least reversible step of the migration.
6.2 GitLab
Section titled “6.2 GitLab”- Data. 12.9 GiB in restic (repositories through
gitlab-backup, run withSKIP=db),gitlabhq_productionin corp/postgres (section 4), and ten hel1 buckets: artifacts, packages, LFS, uploads, pages, terraform state, secure files, dependency proxy, external diffs and the registry (about 100 GiB, in no backup, SHC-812), bound again, never copied. - Single writer on the buckets. The old GitLab is stopped before the new one starts, always in that order, and the rehearsal never boots GitLab against them (5.1). Two GitLabs on one registry bucket can garbage-collect each other’s blobs.
- The backup that freezes it. The old line’s capture pauses the whole
stack while
gitlab-backup-createruns (13 minutes tonight). In the move that pause is the write barrier. Do not restart GitLab while it is frozen. - SSH. Today Traefik publishes 2222 and routes it to GitLab’s sshd. On
the new line the gitlab app must declare 22 (it declares only 80 and 5050
at next) and the deployment gets
--port 2222:gitlab:22. That catalog change is filed before wave 5. - Webhooks. Its hooks to gitlab-huly and gitlab-fizzy move in its own
unit. Its hooks to hookshot fail between waves 4 and 5; GitLab disables a
hook after repeated failures, so 5.5 re-enables any hook with
disabled_untilset and checks one delivery. - CI. build01’s runner on pve01 reaches
git.corp.bitspur.com; it is paused for the window. After the move GitLab and its runner share a datacenter. - Images. Nine deployments run images from
registry.git.corp.bitspur.com(fizzy, fleet, gitlab-fizzy, mailcow, marketplace, nextcloud, pokemon-kc, pokemon-web, pulp), and pokemon-web’s package is a git URL on the same GitLab. Wave 0 pulls every image by digest and uploads every package, so no install depends on GitLab being up; nothing re-pulls during wave 5.
6.3 Keycloak and SSO
Section titled “6.3 Keycloak and SSO”Every gated deployment depends on it. It moves first (wave 1), with
master/postgres, its only database, and the platform record
auth.corp.bitspur.com, which no deployment owns (Keycloak has no host
binding; its name comes from system.apps.keycloak.host). The script takes
that one name over explicitly in 5.6 and gives it back explicitly in a
rollback.
- The new cluster’s shcd reaches Keycloak through its node-aware dialer, its forward-auth decides from tokens Keycloak signs, and each restored app’s SSO client must exist in the Keycloak its cluster administers. Running the new cluster against the old Keycloak would keep compose01 in every login path for the whole move.
- 29 of 32 consumers use the public issuer, so after the switch the old side’s apps follow DNS to the new Keycloak. The realm’s signing keys are in the restored database, so tokens validate on both sides.
- identity-broker’s API calls get
oidc.apiUrl=https://auth.corp.bitspur.combefore wave 1 (an update that replays its persisted overrides). The two mailbox reconcilers build their URL from the socket’s private hostname with no override, so they stop at wave 1’s barrier and move with mailcow (3.2). They use their own confidential clients, so nothing else changes. - The risk this order carries: the old line’s shcd dials Keycloak for
its own backchannel (forward-auth’s code exchange, JWKS, the console)
through the same node-aware dialer (ce
modules/auth/service.go). If that dialer keeps sendingauth.corp.bitspur.comto compose01’s own ingress once the old Keycloak is stopped, the 17 gated deployments still on the old side lose login until they move. The rig proves this first, against an old-line cluster. If it fails, Keycloak moves last instead: the new side uses the old Keycloak by its public hostname (admitted per 3.3), and wave 1 is openobserve alone (open question 2).
6.4 Nextcloud
Section titled “6.4 Nextcloud”- It runs a path-pinned package (
/var/lib/shc/_ncimport/load/adapted/nc, pluginsmailandintegration_paperlessfrom the same tree). The script uploads that tree as a local package, so the new deployment runs the same bytes. Repointing it onto the catalog app comes after the move. - Its primary storage is
cloud-bitspur-comin hel1, bound again: 57 GiB of objects do not move, and every file read crosses 157 ms to hel1 (measured from pve02) until the bucket moves (open question 3). - Its local data (5.7 GiB in restic) and its database cross normally, in
wave 4’s unit with everything that reaches it privately: office (both
directions), Stirling (
pdf, wired by Nextcloud’s stirling plug, not by a plug edge), crm, fizzy, identity-broker and projects. - The files_stirling plug job is run by hand on the path-pinned deployment; 5.5 opens a PDF in Stirling and saves it back.
cloud.corp,cloud.bitspur.comandcloud.hel1.hetznermove together.
6.5 Huly
Section titled “6.5 Huly”projectsruns the huly fork (images on registry.gitlab.com), its databasehulyin corp/postgres, blobs inhuly-corp-bitspur-com, 839 MiB in restic; the fulltext index rebuilds.- It reaches matrix, Nextcloud and identity-broker privately, so it is in wave 4’s unit. gitlab-huly reaches Huly by its public URL and GitLab privately, so it moves with GitLab in wave 5.
- Huly is the tracker every agent writes to. During wave 4 no ticket write lands; wave 4 runs when no agent wave does.
6.6 Frappe benches
Section titled “6.6 Frappe benches”- bit and ffx run the catalog Frappe app with image overrides
(
images.frappe), their databases in the shared corp/mariadb, and/home/frappe/frappe-benchas a host bind mount that the backup carries as a host-bind tar (2.65 GiB for bit, 2.57 GiB for ffx). The image matters only at seed time; the restored bench is the truth. - ffx’s image
shc/ffx-frappe:based93is pushed and digest-pinned before wave 0 (section 1). sites/assetsare anonymous volumes and are rebuilt; 5.5 checks a page’s CSS.- marketplace (path-pinned,
_w34) bundles its own MariaDB and moves whole in wave 2. pokemon-bench’s database is the separate deploymentcorp/pokemon-db, in its unit.
6.7 supabase
Section titled “6.7 supabase”It resolves from the catalog now (2.0+34); the _platform path pin is gone.
What still matters from its import is the seven vault secrets seeded from
its source estate: a fresh install would mint credentials that cannot open
its data. The vault copy carries them key by key, and 5.5 checks that
/auth/v1/ and /rest/v1/ answer with key-auth and Studio sits behind the
gate. Its bucket is bound again.
6.8 NetBird
Section titled “6.8 NetBird”NetBird is how VPN users reach LAN-only apps, so moving it can cut that access, never ours: every node of both estates keeps public SSH from the admin addresses, the out-of-band path for every wave.
- Today the route
estate-lanadvertises10.236.16.0/20through the peer on swarm01, with masquerade, and dnsmasq on swarm01 answers*.corp.bitspur.comwith compose01’s private 10.236.16.8. From wave 1, as each unit switches, dnsmasq answers its hosts with the new node’s address, and the route gains 146.71.124.78/32, .68/32 and .69/32 through the same peer, so VPN traffic to a moved app arrives from swarm01’s egress, which 3.3 admits. Those are edits on swarm01’s hand-made dnsmasq and in NetBird’s own settings, not on compose01, and the script journals each previous answer so a rollback restores it (5.8). The routes are added once per node and stay while any moved unit runs there. - Wave 7 moves the management, signal (
--port 33073:netbird-signal:80) and relay (--port 33080:netbird-relay:33080), their database and volumes. Peers reconnect tovpn.corp.bitspur.comandapi.vpn.corp.bitspur.comafter the switch. A routing peer on shc01 takes over the routes and the split DNS, and swarm01’s peer is removed.vpnandvpn-apistay public in the split DNS, or a client cannot reach its own control plane on reconnect. - The second VPN path, the pfSense in hel1, keeps working because its clients’ egress is admitted by 3.3; where it lives after compose01 goes is open question 9.
6.9 The backup repositories
Section titled “6.9 The backup repositories”- The source. Everything is restored from
shc-backup-primary-bitspur-comin hel1. The script reads it with the old estate’s restic password and S3 key, taken from the old vault at run time, never written down. Only the old cluster’s own schedules write to it. - The new cluster’s repository is a new bucket, never the old one and never the old copy bucket: pointing a new repository or a mirror at a bucket holding another generation is how the copy bucket’s history was destroyed on 08-22. Recommendation: a new hel1 bucket on the estate’s existing credential for the move, so no new secret is needed, and a US repository later (open question 4). Each unit’s first new-line backups are taken in its watch night, which spreads the first full upload (about 90 GiB at the 6 MiB/s pve01 measured to hel1) across the waves.
- The final pre-move archive of each deployment is also imported into
the new repository by the move itself (§6’s
shc backup import), so it outlives the old repository’s retention. - The old schedules keep firing on compose01 and will try the stopped deployments every night. Untouched means they are left alone; the alerts for stopped deployments are expected noise for the two weeks, and retention (7 daily, 4 weekly, 12 monthly) keeps the pre-move snapshots.
7. The one-off script
Section titled “7. The one-off script”It extends what exists: bitspur/ops terraform/cluster/migration/
(migrate.sh, merged 09-30 as !198), which today runs the read-only dry run
against compose01 through an allowlist of reads and refuses any promanager
host. It stays outside shc and ce and is deleted once bitspur is across.
It does:
inventory,render,plan,dry-runas today, the plan rebuilt from this design: the move units (3.2), the waves and their order (3.4), every database of every unit (section 4), the platform records each unit carries, and the preconditions below.prepare <wave>: section 5.1 for the wave’s units.move <unit>: 5.3 to 5.6, then records the watch.rollback <unit>: 5.8, from whichever phase the unit reached.status: one line per unit with its phase.
The only writes it makes on the old cluster: stopping and starting the
units it moves and cutting and restoring their ingress; taking their
backups; the LAN allowlist cross-admission (3.3); oidc.apiUrl on
identity-broker (6.3); stopping gitlab-fizzy and the two reconcilers until
their unit moves (3.2). Outside compose01 it also edits swarm01’s dnsmasq
and NetBird’s routes as units switch (6.8); swarm01 refuses the Mac’s key,
so that needs the cluster key from the ops Terraform state. It never deletes anything on either side, copies a
volume off compose01, moves a bucket, mints, rotates or prints a secret
(it reads them from the old vault at run time), or touches promanager.
When it refuses to run. No move starts before the SHC-1165 gate is recorded as passed. No barrier may overlap a backup on either side: the old estate’s corp tenant run (03:30 to about 05:20 UTC tonight), master’s openobserve (04:00), postgres (04:30) and keycloak (05:00), traefik on Sundays (05:30) and the shc tenant (23:00 to about 23:05), each widened by 30 minutes and by its last measured duration. The script checks for a running backup job on both clusters and refuses a unit whose estimated window would reach the next scheduled run.
Inputs. ssh to compose01 ([email protected], the estate key) and the
old CLI there; the new cluster’s CLI context on the Mac
(shc.odg.bitspur.com); the plan file it generates; ce and shc checkouts at
the commit the new cluster runs, for the render check.
Dry run. Every command takes --dry-run, which prints each API call and
remote command it would run and the checks it would make, and runs only the
reads. move refuses unless the unit’s preconditions hold: rehearsal
restore passed, images present, certificates issued, the new installs exist
and are stopped, every member’s last scheduled backup is verified and under
26 hours old (or, for a paused deployment, its verified pause-time capture,
3.2), its wave’s earlier units are done, and the window is clear.
State and resume. One JSON file per unit under
~/.drover/state/shc-1061-migration/units/<unit>.json (mode 600, outside the
repo): its members, phase (prepared, barrier, backed-up, restored,
verified, switched, watching, done, rolled-back), each member’s old
final snapshot id, the engine snapshot id, the new backup ids, the records
moved, the rollback prerequisites (old providers that must be running), and
every check’s result with its time. Each phase is idempotent and is written
before the next starts, so a killed run resumes with the same move command
from the last completed phase, and a unit can never be marked done with a
member behind.
8. Cross-tenant edges
Section titled “8. Cross-tenant edges”Every connection on bitspur whose two ends are in different tenants:
| consumer | plug | provider |
|---|---|---|
| corp/affine, bit, blog, ciso, dtrack, ffx, fizzy, gitlab, glitchtip, identity-broker, ivatar, librechat, mailbox-reconciler, mailbox-reconciler-ffx, mailcow, matrix, netbird, nextcloud, oncall, openreplay, peertube, penpot, photos, postiz, projects, supabase, superset, wazuh, workflows | oidc | master/keycloak |
| corp/fleet, mautic, sonarqube | saml | master/keycloak |
32 edges, all corp to master/keycloak. Nothing else crosses: no corp
deployment uses master/postgres or openobserve’s store directly, and the
shc tenant only reaches inside itself. At the network level 32 corp
containers sit on master_default_keycloak_public and 51 on
master_default_traefik_public (ingress, which is the system tier).
The new line refuses a plug across tenants where it is declared
(X400372FABXTN, SHC-1202, merged on next), with one carve-out: the shared
keycloak at master/default is the one cross-tenant edge the check and
the fabric let through (Clay, 2026-10-02; ce modules/app/integration_tenant.go,
integration_magic.go). These 32 are all edges to that keycloak, so as the
code stands they reinstall as plugs through it. The resolution below was
written before the carve-out. Named resolution: on the new line
SSO stops being a plug. A deployment’s SSO client becomes a platform auth
registration (/api/auth, the module that already provisions clients),
which renders the same values the templates read today (clientId,
clientSecret, issuer, realm, the SAML entity) without a plug
declaration, and the script carries each client’s id, secret, redirect URIs
and grants from the restored Keycloak and the copied vault. The network path
is the public issuer through ingress, so no fabric accept is needed. This
has to be ruled and built before wave 0 (open question 8); it is product
work, not script work.
9. Ports published from compose today
Section titled “9. Ports published from compose today”Under the publication ruling the host side of ports: is not honoured, so
each needs --port at install. Grammar
[<conn>@]<external>[:<service>][:<container>][/udp] (ce
modules/app/port_mappings.go).
| deployment | today | --port at install |
|---|---|---|
| corp/mailcow | 25, 465, 587 on postfix-mailcow; 110, 143, 993, 995, 4190 on dovecot-mailcow | --port 25:postfix-mailcow:25 --port 465:postfix-mailcow:465 --port 587:postfix-mailcow:587 --port 110:dovecot-mailcow:110 --port 143:dovecot-mailcow:143 --port 993:dovecot-mailcow:993 --port 995:dovecot-mailcow:995 --port 4190:dovecot-mailcow:4190 |
| corp/netbird | 33080 on netbird-relay; 33073 to netbird-signal’s 80 | --port 33080:netbird-relay:33080 --port 33073:netbird-signal:80 |
| corp/gitlab | 2222 through Traefik’s raw TCP entrypoint to sshd | --port 2222:gitlab:22, once the app declares 22 (6.2) |
| master/traefik | 80, 443 | not moved: the new line’s per-node Traefik binds them |
| shc-collector | 4318 on docker0 | not moved: the per-node collector is the new line’s own |
All three run single-replica on a pinned node until multi-replica --port
is ruled, and their hostnames resolve to that node only.
10. Risks, what cannot be undone, and windows
Section titled “10. Risks, what cannot be undone, and windows”Cannot be undone without a reverse restore (5.8 step 3):
- Mail delivered to the new mailcow (6.1).
- Writes on the new side after a switch: GitLab pushes and merges, Huly tickets, Nextcloud files (their objects sit in the shared bucket, their rows in the new database), Keycloak password and user changes, Frappe documents.
- Anything in a forward-only unit (an unversioned bucket, 5.1) after 5.4.
Cannot be undone at all: Matrix notifications GitLab and wazuh sent to hookshot while it sat in another wave (a day or two), and the GitLab-to-Fizzy sync and mailbox provisioning skipped while their bridges were stopped (3.2). Data in GitLab, wazuh and oncall is unaffected.
Top risks:
- Keycloak first strands old-side SSO if the old shcd’s dialer pins to its own ingress (6.3). Proven on the rig before wave 1; the fallback order is written.
- The cross-tenant refusal blocks every SSO app until SSO is a platform registration (section 8). Product work on the critical path.
- LibreChat’s conversations and corp/mysql’s databases may not be in their backups (section 1). The rehearsal decides; nothing moves on belief.
- pve03 holds two voters. Losing that box stops lease-held apps until a manual repair (section 2).
- The ocean. Every bucket and the backup repository stay in hel1, 157 ms away. Restores pull at about 1 GiB a minute; Nextcloud, the GitLab registry and peertube serve every object across that link until the buckets move.
- compose01 is fragile (91% disk, 29 GiB swap). Each fresh backup
stages on
/mnt/shc-staging; no barrier runs beside another heavy job, and the units leaving take load off it. - Single-writer buckets. Old copies stop before new ones start, and rehearsals never touch them.
- Wave 4 is one long barrier: mail and every collaboration app down together for about three hours, and Huly’s downtime stops agent ticket writes.
- The licence and the takeover need the old cluster’s identity; a rename before the move breaks both.
Windows: wave 1 about 20 minutes of no logins; waves 2 and 3 a few minutes per unit, over four or five nights; wave 4 about three hours; wave 5 about 1.5 hours of GitLab; wave 6 about two hours; wave 7 about 45 minutes of VPN. Roughly ten windows over two to three weeks, then the two-week hold.
11. Open questions for Clay
Section titled “11. Open questions for Clay”| # | question | recommended answer |
|---|---|---|
| 1 | The ruling puts every new node on pve02 and pve03, so the third voter shares pve03 and losing pve03 loses quorum. Allow a voter-only VM elsewhere? | Yes: a 2 vCPU, 4 GiB voter-only VM on pve01 (same datacenter, its own box), taken from the lane farm’s budget. Then any one box can fail. |
| 2 | Keycloak first or last? | First (wave 1), if the rig shows the old line’s shcd reaches it by public DNS once the old copy stops; last otherwise. |
| 3 | Do the 26 buckets stay in hel1? | Yes for the move, bound again, no copy. Move them to US object storage afterwards, Nextcloud’s first, one app at a time. |
| 4 | Where does the new cluster’s backup repository live? | A new bucket in hel1 on the existing credential for the move, never an old bucket. A US provider afterwards, with credentials you mint. |
| 5 | Keep the cluster’s identity (slug, owner heritage, shc.hel1.hetzner.bitspur.com, the licence) through the move? | Yes: the takeover and the licence depend on it. Rename afterwards with a re-minted licence, with the other *.hel1.hetzner.* names. |
| 6 | Restore openobserve’s 24 GiB of logs, or start it empty? | Restore it: every app moves from its backup, and it costs one 75-minute window with nothing depending on it. |
| 7 | Move corp/nc-db (no consumer) and ffx’s stale bench.pre-based93? | No. Both stay on compose01 and go with it. |
| 8 | SSO on the new line: a platform auth registration instead of a cross-tenant plug? | Yes, with the refusal kept unconditional. Rule it now; it gates wave 0. |
| 9 | Where does the pfSense VPN path land after compose01 goes? | Keep the hel1 pfSense through the move with its egress admitted; afterwards pfsense01 on pve01 takes it over, so both paths end in Ogden. |
| 10 | SHC-1061 still has the criterion “the pfSense HAProxy edge points at the new cluster’s Traefiks”. | Drop it. That HAProxy is empty and the public names resolve straight to the nodes; the DNS switch is the re-point. |
| 11 | The writes on compose01 this design needs (LAN allowlist cross-admission, identity-broker’s oidc.apiUrl, gitlab-fizzy and the reconcilers stopped between waves)? | Yes, all before wave 1, kept through the two weeks. |
| 12 | Wave 4 as one three-hour barrier for mail and every collaboration app? | Yes. Splitting it needs public endpoints for six private edges whose templates have no override, which is catalog work on six apps for one night’s saving. |
References
Section titled “References”- The new line, deleting first, §3.13 policy, §3.14 publication, §4.9 suffrage, §6 the one-time migration.
- The SHC scheduler, §3.5 reserved fences, §3.25 stopped-app backups, §7 the cutover gate.
- Multi-region fences.
- bitspur/ops
terraform/cluster/migration/(the dry run and its 09-30 SUMMARY) andterraform/gorillaservers/ogden/(pve01 to pve03 and the rig). - SHC-1061 (this), SHC-1165 (the gate), SHC-1123 (the rig), SHC-1202 (policy), SHC-1201 (mesh), SHC-812 (registry backup), SHC-691 (ffx image).