Skip to content
SHC Docs

Clone/restore lineage is not rename-aware

Status: open design. Clone and restore still replay the name a stack had when the backup was taken. This page records a live finding from the Roll 2.1 close so the class has a name and a home. The fix belongs to the backup/clone machinery and earns its own lane; nothing in this page changes behaviour.

bitspur corp/prod, 2026-08-22. The stack now named paperless was renamed from docs earlier in its life. shc clone paperless failed with X500603CLNRCV, and the RecoveryOrchestrator’s plan shows why: it tried to install stack=docs, the pre-rename identity, three times, drawing X503010NDUNRC (mesh routing), X500960SWMDPL (swarm deploy) and X409006RTMOWN (runtime ownership: the live paperless swarm objects sit under the project the resurrected docs render folds onto).

The restore chain carries the OLD name. The archives, their manifests and the recovery plan were written under the identity the stack had at capture time, and nothing in the clone/restore path re-maps that lineage through the rename. A renamed stack’s clone therefore tries to resurrect its former self from stale state, and the failure surfaces as routing/deploy/ownership errors that name everything except the rename.

The adjacent class: import-origin divergence

Section titled “The adjacent class: import-origin divergence”

The rename is one member of a wider class: the identity or path a deployment RUNS under today diverges from the identity its durable lineage was recorded under. Known members on the live estates:

  • docs → paperless (this incident): rename, restore lineage keeps the old stack name.
  • supabase on the imported estate runs from the _platform path, an estate-import artifact shc update replays; its lineage is pathed, not named, the way a native install would be.
  • imported uptime (Uptime Kuma): same import-origin shape, lineage recorded under the origin identity.

Anything that replays recorded lineage (clone, recover, move’s restore leg, possibly scheduled-backup subject resolution) can hit a member of this class. The sidecar-not-where-the-render-looks refusal (X409010DEPNOS / X409011DEPELS) is the same divergence seen from the node axis instead of the name axis.

Shape of a fix (for the future lane, not here)

Section titled “Shape of a fix (for the future lane, not here)”

The capture side should record a stable deployment identity that survives rename (or the rename verb should rewrite/annotate lineage), and the replay side should resolve lineage THROUGH the current row rather than trusting the archived name. Fail closed stays the rule: a replay that cannot prove which present-day deployment its lineage names should refuse and say both names, not install the older one.

A clone/recover of a renamed or imported stack is suspect. If it fails with a plan naming an identity you no longer use, that is this issue; do not retry into the old name; the retries are what stacked three distinct errors on one incident.

Addendum 2026-08-22: the same-day clone-to-new-target failures are FIXED, and they were a different member

Section titled “Addendum 2026-08-22: the same-day clone-to-new-target failures are FIXED, and they were a different member”

Root-causing the day’s other two failures (shc clone paperless docs and shc clone immich photos, both corp/prod, both dying X500603CLNRCV … restore from file [X404401STKNOF] stack <target> not found after a completed capture) landed on a scope bug, not the rename lineage above: the clone restore leg’s target scaffold read the SOURCE sidecar from the scaffolder’s baseline-tenant/default-env fallback, so on any estate whose deployments live outside that scope it silently no-oped and the restore resolved a target that was never materialised. That is why immich → photos failed with no rename anywhere in its history, and why the shc recover -b <bak>:<target> workaround succeeded (recover installs its target before restoring).

Fixed in ce (daemon/services_seams_clone.go, ensureCloneRestoreTarget: read the source from the input’s own scope, fall back to a bare target scaffold, fail closed by name), ce MR: https://gitlab.com/bitspur/selfhosted-cloud/ce/-/merge_requests/203, with the regression pinned in this repo’s corpus (tests/_ce/daemon/clone_recovery_target_scaffold_test.go).

The capture-side finding this page records (archives and recovery plans carrying a pre-rename identity, and the wider import-origin divergence class) REMAINS OPEN. A clone/recover of a renamed or imported stack is still suspect exactly as described above; what is no longer suspect is a plain clone to a fresh target name in a non-default scope.

Addendum 2026-08-22: the rename’s DNS gap: operator note, and why it stays

Section titled “Addendum 2026-08-22: the rename’s DNS gap: operator note, and why it stays”

A renamed host is unresolvable AFTER it is healthy

Section titled “A renamed host is unresolvable AFTER it is healthy”

shc uninstall <old> releases the deployment’s managed DNS records and the shc recover -b <archive>:<new> that follows re-creates them, so the hostname does not exist for the length of the gap between the two commands. A recursive resolver that queries inside that window caches the NXDOMAIN for the ZONE’S NEGATIVE TTL (the SOA minimum, 3600 seconds on bitspur.com) and keeps answering nothing long after the stack is up and serving. Measured three times on 2026-08-22 (paperless→docs, immich→photos, huly→projects): the authoritative NS and 8.8.8.8 answered the new A record correctly while other recursors (the estate node’s own Hetzner resolvers included) still returned nothing.

Before calling it an outage, check the two things that skip the cache:

Terminal window
dig +short @<zone NS> <host> # authoritative
curl -sI --resolve <host>:443:<node-ip> https://<host>/ # the node itself

A correct authoritative answer plus a 200 with a valid cert means the stack is fine and you are waiting out somebody else’s negative cache. Nothing on the estate shortens it.

Verdict: the preserving rename already exists, and it is shc move

Section titled “Verdict: the preserving rename already exists, and it is shc move”

Investigated whether the rename path could keep the records instead of delete-and-recreate. It can, through the verb the incident did not use.

Where the delete happens. ExecuteComposeDown releases every hostname the manifest persisted (ce modules/app/install_runner.go:3643-3672), latching the per-deployment retire tombstone first (RetireDeploymentDNS, modules/app/dns_retire.go:52-61) so the leader’s proxy_reconcile cannot re-publish into the window. It runs on EVERY uninstall by design: a record is per-deployment state, and one left behind orphans in the zone and survives cluster destroy (2026-07-23 cloud shakedown, daemon bug #2, install_runner.go:3646-3649).

Why KeepDNS is move-only. A managed record is keyed by HOSTNAME ALONE; a zone carries no tenant/stack/environment (modules/app/install_runner.go:4212-4215, modules/clone/types.go:801-808). Suppressing the release is therefore safe only when another deployment is ALREADY serving that exact hostname, and the only caller that can prove it is the move orchestrator: recoverPhase brings the target up and re-registers the source’s hosts verbatim (daemon/services_clone_remote.go:484-485) BEFORE uninstallPhase tears the source down (modules/clone/service.go:798, then :880), which is where keepDNS=true is passed (modules/clone/service.go:2458) and mapped onto the teardown (daemon/services_seams_stack.go:807-810, :832). That ordering IS the invariant. Hand the flag to a bare uninstall and the record dangles over an address the cloud is free to recycle.

A rename is not distinguishable at the deletion. shc uninstall <old> and shc recover -b <archive>:<new> are two independent operations with nothing joining them. The uninstall has no forward knowledge that a recover is coming and no way to acquire it, so no fix belongs on the uninstall side.

The recover side is already an idempotent upsert, and ownership does not block it. CreateDeploymentDNSRecords → syncHostRecords → SyncRecordsForHostByPurpose is a sync, not a create (modules/app/dns_lifecycle.go:135, :331-342), and it SELF-ADOPTS an unowned host, minting the ownership TXT and correcting the targets in the same pass (modules/dns/ownership.go:42-50). Ownership is per-CLUSTER (dns.owner_id) and hostname-scoped, never per-deployment, so a record this cluster published under the OLD stack name reads ours under the new one and the upsert is a no-op decision; a recover onto a still-live record converges cleanly. The X403049DNSPTO refusals in the same night’s reap are a different axis: a predecessor claim from another cluster INSTANCE sharing the host slug (modules/dns/predecessor_gate.go:42-59). It cannot arise inside one cluster’s rename.

The gap belongs to the workaround, not to the code. shc move <old> <new> is the rename that preserves DNS, and has been since the cross-key path existed. It was unusable on this estate the night of the incident: a swarm source took the in-place clone.Rename fast path, which renames nothing on swarm and reported success anyway (X400635CLNSWR, modules/clone/errors.go:500-530, measured on the same 2026-08-22 corp/prod shc move paperless docs). That is fixed in ce main: a swarm source is routed onto the backup+restore path (modules/clone/service.go:1466-1492), which is the path that carries keepDNS. clone → uninstall → recover was a workaround for a broken verb, and the negative-TTL tail is a property of the workaround.

The fix is on main, not on the estates. As of 2026-08-22 the fleet runs a pre-fix daemon; v0.0.61 was never tagged, so nothing carrying the swarm routing has been rolled. Until it is, shc move on a swarm source still takes the fast path, the clone → uninstall → recover sequence remains the only working rename, and the operator note above is live guidance rather than history. Re-read this section after the roll; the verb is the answer the moment the daemon carries it.

No behaviour change taken here. The preservation is implemented and pinned (tests/_ce/modules/clone/move_teardown_test.go:72, :116; tests/_ce/modules/app/teardown_dns_release_test.go; tests/_ce/daemon/services_seams_stack_ownership_test.go). Weakening the uninstall release to make the workaround cheaper would trade a bounded negative-cache tail for an unbounded dangling record, which is the wrong direction. Use shc move; when you cannot, the operator note above is the answer.

Test the case that breaks, not the case that passes. A rename exercises vault-secret handling differently depending on how the app declares its secrets: a ref+vault://key?autogen regenerates silently and sails through, while an operator-provisioned ref+vault://key fails closed when the deployment-scoped row is purged. The same app can differ per estate: on 2026-08-22 bitspur’s copy of huly carried an operator-provisioned huly_s3_secret_key while promanager’s carried ref+vault://s3_secret_key?autogen&length=40, so the identical rename breaks on one estate and passes on the other. Any regression test for the vault-ordering fix must rename a stack whose secret is OPERATOR-PROVISIONED; one that renames an autogen stack proves nothing and reports a false all-clear.