Skip to content
SHC Docs

The deployment binding model (proposal)

Status: open design. Not ruled and not built. The Swarm cell it describes went with the new line (SHC-1092), and nobody has ruled on whether the bound and unbound predicate still applies. Written 2026-08-02, from three parallel read-only audits of ce’s volume, placement and enforcement surfaces, re-verified against ce and shc at 2815168 (shc pins ce at v0.0.12-0.20260731190259-5da1b0869c7a). No production estate was contacted.

The owner’s framing, which this document makes precise:

“Let’s not have ‘this deployment is on this node’, but ‘this deployment has critical state on this node’ or ‘this deployment is bound to this node’. Unless it’s moved or becomes unbound via unpinning, that node cannot be removed. However where there are other potentially available nodes, or you could add another node, it would simply start working there, of course only in a swarm environment. I don’t want to ever be in a scenario where you can lose state.”

Two sharpenings carried throughout:

  1. Bound-ness is a property of the DEPLOYMENT, not of current topology. An unbound deployment on a one-node cluster is still unbound. Removing that node makes it UNSCHEDULABLE (availability), not LOST (data). Those deserve different treatment: losing state is a hard refusal, losing availability is a warning an operator may accept.
  2. Swarm does not make a deployment safe; shared storage does. A swarm service with a critical node-local volume is the most dangerous configuration in the system, because swarm reschedules the task and starts it against an empty volume.

This document consumes, and must not contradict, the node addressing model. Its slice 11 (“destroy refuses, drain relocates”) is the mechanism at the terraform boundary; this document supplies the predicate that refusal should test. Its Q11 is refined here rather than re-asked. For the runtime-neutral cross-node data plane this model must eventually express, see network-fabric.md.


A word this project has already spent three times

Section titled “A word this project has already spent three times”

Before anything else, because the design is unreadable otherwise. “Bind” has three unrelated meanings in SHC today, and one of them means close to the opposite of the owner’s:

spellingtoday’s meaningscope
bindwhere the daemon LISTENS: bind_ip, internal_api.bind, SHC_RQLITE_BIND_ADDRa node’s own socket
bindinga compose mount that has an x-shc-volumes block-volume row: volume.ParseVolumeBindings, “a BOUND swarm DATA mount” (ce/modules/app/install_runner.go:3089)a mount
boundthe owner’s: a deployment whose critical data can be served only by named nodesa deployment

The second is the dangerous one. In today’s code a “bound” mount is a block volume, which is one of the LEAST node-bound things in the system, because a location-scoped block volume detaches and reattaches to any node in its location. A reader who has internalised the code’s spelling will read the owner’s sentence backwards.

Proposal, and it is cheap: the code sense is renamed. x-shc-volumes mounts become provisioned (they carry a volume row) versus plain; hasBinding becomes hasVolume. bind keeps its addressing-model meaning unchanged (that glossary is authoritative and this document does not touch it). bound / unbound / bound_to are reserved for this model, at deployment scope only.


SHC cannot answer “would removing this node lose data”. It can answer “how many deployment rows point at this node”, and it treats that as the same question. It is not.

Three facts, each verified:

1. The gates count deployments, not state. shc cluster rm refuses on X400017NODDEP: “node has stacks, rerun with force=true to override” (shc/modules/cluster/admin_router.go:171-183), a pure COUNT over deployment.node_id. It cannot tell a stateless nginx from the only copy of a database, and force=true bypasses it entirely, after which DeploymentRepo.DeleteByNode (ce/modules/app/deployment_repo.go:162-180) sweeps the rows so the ON DELETE RESTRICT foreign key stops objecting. The bytes stay orphaned on a disk nothing points at any more. shc node forget has no deployment gate at all; its handler calls Remove(ctx, m.ID, true) with force hardcoded (shc/modules/cluster/lifecycle_router.go:305), and that is the verb the terraform provider’s Delete invokes.

2. The predicate already exists, correct, and is reachable from one verb. ClassifyMountDecision (ce/daemon/clone_move_adapters.go:213-266) reduces per-mount facts to Ephemeral | Rebindable | NotRebindable | CrossProvider, keying the block case on a topology predicate (a non-empty persisted location) rather than a vendor name. ClassifyDeadMountDecision (ce/daemon/clone_dead_move_adapters.go:353-379) goes further and names the owner’s condition outright: DeadMountLost is “a local-only persistent mount; the data lives ONLY on the dead node’s disk and does NOT survive. Refused unless the operator passes --discard-local” (ce/modules/clone/move_presumed_dead.go:73-77), enforced by X400619CLNDLM. That is the owner’s model, already implemented, already fail-closed, already with an explicit override, and it is reachable only from shc move --presumed-dead. Install does not consult it. Node removal does not consult it. cluster repair does not consult it.

3. The one configuration that loses state silently is the default. Detailed in the decision table. In summary: ApplyDeployPlacement returns the compose unchanged when no node was requested (ce/modules/app/swarm_placement.go:33-36), and an unbound swarm DATA mount is deliberately kept as a docker-managed named volume that “swarm provisions per-node on whichever node the task lands” (ce/modules/app/install_runner.go:3066-3090). So shc install <db-app> --runtime swarm yields critical state in a per-node volume with no placement constraint. Twelve lines below that branch, the same file refuses the equivalent configuration for a provisioned mount: X400324VOLSWMNB, because it “would shadow the attached device with an empty volume (medium #18 data-loss case), and on swarm there is no single-node guarantee”. The unbound case does exactly what the bound case is forbidden from doing.

And the guard an operator would expect to catch this does not exist. shc/docs/reference/cli/storage.md:233 states “Swarm-mode stacks require a shared provider (nfs or cephfs); local is rejected for Swarm”, and the architecture doc names the error code X400044SWRLCL and the function ResolveDefaultStorage. Grep across ce and shc: the code, the function and the error code do not exist. They are a description of Python-era behaviour that was never ported.


Two independent per-mount properties. One derived per-deployment verdict. Nothing stored.

Durability: would losing these bytes be data loss?

Section titled “Durability: would losing these bytes be data loss?”
durability(m) ∈ { critical, ephemeral }

Resolved by a fixed three-step precedence, and every step is reportable:

  1. Declared. The app’s backup.yaml gains a durability: block naming mounts explicitly. Author intent wins.
  2. Inferred, structurally: ephemeral when the mount is tmpfs, an anonymous volume, a socket / device node / FIFO, a read-only bind, under a pseudo-root (/proc /sys /dev /run /var/run /tmp), a RUNTIME mount, or a FILE mount re-derived from the app source on every render. Every one of these judgements is already written and already exercised against 73 apps: ce/modules/backup/hostbind.go:100, :210, :222, :269-281, :382-385 and ce/core/volume/resolve.go:93.
  3. Otherwise critical. Fail-closed. Unknown binds.

Anchor: which nodes can serve these bytes without copying them?

Section titled “Anchor: which nodes can serve these bytes without copying them?”
anchor(m) ⊆ Nodes, or UNKNOWN
mount shapeanchor
compose DATA mount (host bind under <deployDir>/mounts/<name>){ the node holding the deployment dir }, a singleton
absolute host bind{ that node }, a singleton
swarm docker-managed named volume, service PINNED{ the pinned node }
swarm docker-managed named volume, service UNPINNEDUNKNOWN
RWX shared filesystem (nfs / cephfs), verified mounted on NN
location-scoped block volume (volume.location != ""){ n : n.location == volume.location }
block volume with no location stamp{ current attachment node }

Note the asymmetry that makes anchor the right primitive rather than a shared/node-local label: a location-scoped block volume in a single-node location is a singleton anchor and binds, while the same provider in a two-node location does not. The topology decides, and ClassifyMountDecision already computes exactly this reduction.

A deployment D is BOUND to node n if and only if D has a mount m where durability(m) == critical and anchor(m) == { n }.

critical(D) = { m ∈ mounts(D) : durability(m) == critical }
bound_to(D) = { n : ∃ m ∈ critical(D) with anchor(m) = {n} }
unanchored(D) ⟺ ∃ m ∈ critical(D) with anchor(m) = UNKNOWN
unbound(D) ⟺ bound_to(D) = ∅ and not unanchored(D)

bound_to is a set, which is the owner’s plural and it is load-bearing: a deployment with critical local state on two nodes (after a partial move, or with two mounts whose singleton anchors differ) is bound to both, and neither may be removed. A block-volume deployment in a two-node location is bound to neither, because neither singleton condition holds.

What is NOT a binding factor, and why the owner’s list needs trimming

Section titled “What is NOT a binding factor, and why the owner’s list needs trimming”

The owner’s candidates were: explicit pin; a runtime that cannot reschedule (compose, podman); any critical volume on a node-local provider; host paths and bind mounts. Two of those four are not data bindings, and conflating them is what makes the removal gate either unsound or useless.

An explicit --node pin does not bind. It is a placement declaration. If every critical mount of a pinned deployment sits on a shared filesystem, removing the pinned node loses availability, not bytes; clear the pin, or shc move, and it runs elsewhere with the same data. A pin is nonetheless load-bearing in the other direction: a pin is what makes an otherwise dangerous configuration safe, because it turns anchor = UNKNOWN into anchor = { pinned node }. Pins are the cure, not the disease.

“Compose and podman cannot reschedule” does not bind either. It is a scheduling fact, and it is a consequence of how compose renders, not an independent cause. What actually binds a compose deployment is that ResolveVolume rewrites every DATA mount to a host bind under <StateHome>/deployments/<t>/<s>/<e>/mounts/<name> (ce/core/volume/resolve.go:148-155, ce/core/paths/paths.go:712) on the node that ran the install, so its critical bytes are on that node’s disk. A compose deployment with no critical mounts (a stateless proxy) is unbound, and the fact that compose will not move it by itself makes it UNSCHEDULABLE elsewhere until an operator acts. shc move resolves that in seconds with no data copy. That is a warning, not a refusal.

Trimming these two is not pedantry. If “pinned” and “compose” both counted as bindings, then on both live estates every deployment would be hard-unremovable, the refusal would fire constantly for reasons that are not data loss, and operators would learn to reach for the override. A refusal that fires when nothing is at stake is how a refusal stops being believed.

A binding factor the owner did not list, and it is the sharpest one. anchor(m) = UNKNOWN. A swarm service with an unpinned docker-managed named volume has its bytes on exactly one node’s disk; SHC simply cannot say which. SwarmServiceStates shells docker stack services --format {{json .}} and decodes {Name, Replicas, Image} (ce/modules/app/swarm_driver.go:82-94); there is no node field, and nothing writes observed placement back. Meanwhile stampNode records the node that handled the install request (ce/modules/app/store.go:160-176) and never updates it. So the deployment row asserts a location the scheduler is free to ignore, and after one reschedule the row is simply wrong.

This deserves its own state name because it is not a kind of bound-ness; it is the absence of the fact bound-ness is computed from.

statemeaningremoving a node in scope
UNBOUNDevery critical mount has an anchor larger than one node, or there are no critical mountsavailability question: WARN
BOUNDsome critical mount’s anchor is a known singleton; bound_to names those nodesdata question: REFUSE
UNANCHOREDsome critical mount’s anchor is UNKNOWNnot a design state; a defect state

UNANCHORED is not a third posture an operator may choose. It is the configuration in which the question “would this removal lose data” has no answer, and the whole of § enforcement turns on making it unrepresentable.


Critical versus temporal: one property, three sources

Section titled “Critical versus temporal: one property, three sources”

The owner asked for “a concept of critical volumes versus temporal volumes; only the critical ones bind”. Resolved as one property with three sources, not two properties, and deliberately not by reusing the backup signal as-is.

Why backup-inclusion is the wrong signal to reuse directly

Section titled “Why backup-inclusion is the wrong signal to reuse directly”

The correlation is high and it is causal: the backup layer exists to preserve exactly the bytes whose loss is unacceptable, and 23 of 73 apps already reason about it in prose, well. apps/highlight/backup.yaml excludes kafka-data / zookeeper-data / redis-data as “pure ingest-queue + cache state, NOT durable user data”; apps/huly/backup.yaml keeps elastic despite it being derivable because “rebuilding is O(every document), not O(delta)”. Those are criticality arguments, in the right place, by the right person.

Three reasons they still cannot be the binding signal:

It inverts on the most critical mounts. ShouldBackupMount returns false when a converter owns the mount (ce/modules/backup/archive.go:1302-1310); the database dump replaces the raw files. A naive “backed up implies critical” therefore classifies a Postgres data directory as NOT critical. That is the single worst misclassification available.

It is a per-run runtime fact, not a static property. Backup inclusion depends on which capture path ran, whether a converter fired, and whether --path narrowed the snapshot. Per sharpening 1, bound-ness must be a property of the deployment. A property that changes because a backup ran with different flags cannot gate a node removal.

exclude.mounts is an author preference with a different meaning. An author who excludes a 100k-file feed to keep snapshots small has said “don’t snapshot this”. They have not said “you may destroy the node holding it”. Overloading one key with a second, destructive meaning is precisely the trap ce/modules/backup/types.go:709-729 already refuses elsewhere, where include.files / exclude.files were deleted because one key could only reach one of two capture paths.

durability is declared in backup.yaml in a separate durability: block, alongside exclude.mounts and never derived from it. That keeps the reasoning where the reasoning already lives, and keeps the two questions apart:

apps/greenbone/backup.yaml
exclude:
mounts: [vt-data] # too big to snapshot; a re-feed is cheaper
durability:
critical: [vt-data] # ...but losing the node still costs a re-feed
apps/highlight/backup.yaml
exclude:
mounts: [kafka-data, zookeeper-data, redis-data]
durability:
ephemeral: [kafka-data, zookeeper-data, redis-data] # genuinely disposable

ephemeral means “I accept losing this without being warned”. That is the whole contract, and it is why the middle case (expensive to rebuild but not truly lost) classifies critical by default. An author who wants the rebuild to be silent says so explicitly.

Two cross-checks fall out for free and should be reported by shc app lint: a mount declared ephemeral but not excluded from backup is a smell (you are snapshotting something you call disposable), and a mount excluded from backup with no durability declaration is where an author most likely meant one.

The classifier is promoted, not duplicated

Section titled “The classifier is promoted, not duplicated”

The judgements above are made in six independent implementations today, each re-parsing the compose with its own notion of a source: DetectMountType (ce/core/volume/resolve.go:93), backup’s DiscoverHostBindSources + ExcludeHostBindMode (ce/modules/backup/hostbind.go), ShouldBackupMount (ce/modules/backup/archive.go:1302), DiscoverUnboundNamedVolumes (ce/modules/backup/namedvolume.go), PersistentMountsFromCompose (ce/daemon/clone_move_adapters.go:551), and DeployedStackVolumes (ce/modules/app/volumes_deployed.go:47). They agree by code review, not by construction, and they already disagree in one concrete place: an absolute host bind is CRITICAL to backup (“host-bind data must MOVE with the app, not merely be warned about”, hostbind.go:23-31) and EPHEMERAL to move (“classified ephemeral; SHC data never lives there by convention”, clone_move_adapters.go:551).

They collapse into one classifier in core/volume, with backup, move, dead-move, the HUD and the new binding predicate as consumers. Collapsing six parsers to one is worth doing on its own; it is a precondition here.

An operator must be able to see a classification before it refuses something, and the refusal must be reproducible as a query. Two verbs:

  • shc <stack> binding: per mount: name, kind, durability and which of the three sources decided it, anchor (the node set), and a one-line reason. Then the rollup: state, and bound_to.
  • shc node bindings <node>: the inverse. Every deployment bound to this node, with the mounts that bind it named. Runnable before attempting a removal, not discovered from a refusal message.

Both render the same reasons the refusal renders, from the same function. The nearest thing today is shc node info <name>, which joins the node row with shc.node.deployments and returns {stack, env, status, runtime} (ce/modules/node/deployments_router.go:26-31): placement, no binding, no reason.

Misclassification fails safe in both directions. Unknown durability is critical; unknown anchor is UNANCHORED, which refuses. The cost of over-binding is one shc move an operator did not need. The cost of under-binding is a restore, or nothing to restore from.


The owner’s question: with swarm, do you have to move or not?

You have to move under exactly the same condition as compose: when a critical mount’s anchor is a singleton. Swarm changes nothing about whether you must move. It changes what happens if you don’t. On compose, not moving means the deployment stops, loud, recoverable, the bytes are still there. On swarm, not moving means the task restarts on another node against an empty volume, silent, and the bytes are stranded on a disk the running app is no longer looking at.

So swarm is strictly more dangerous than compose for the identical configuration, which inverts the intuition that swarm is the highly available runtime. Swarm buys availability only for deployments that were already unbound.

#runtimepincritical mount’s storagewhat happens when that node goes awaymust move?state
1compose / podmanimplicitnode-local diskdeployment stops; row still points at the dead nodeYESBOUND
2compose / podmanimplicitshared filesystemunreachable today; see belown/awould be UNBOUND
3compose / podmanimplicitlocation-scoped block volumevolume survives detached; move fences and rebindsnoUNBOUND in-location
4compose / podmanimplicitnone criticaldeployment stopsnoUNBOUND, unschedulable
5swarm--nodenode-local named volumetask goes Pending: no suitable node, LOUD, bytes intactYESBOUND
6swarmnonenode-local named volumetask reschedules; docker creates a fresh EMPTY volume; the app comes up blankYES⚠ UNANCHORED
7swarmeithershared filesystemunreachable todayn/awould be UNBOUND
8swarmnoneprovisioned RWO block volumeApplyVolumePlacement pins to the attachment node and clamps replicas: 1; fenced rebind on movenoUNBOUND in-location
9swarmnonenone criticalreschedules cleanlynoUNBOUND

Every cell is critical durability except 4 and 9; the ephemeral variant of any row is UNBOUND by definition and is not worth nine more rows.

Cell 6 is the one that loses state silently, and it is today’s default

Section titled “Cell 6 is the one that loses state silently, and it is today’s default”

Reachable by shc install <any stateful app> --runtime swarm with no other flag. Nothing refuses it, nothing warns, and shc ls reports a node. Cell 8 (the safe swarm shape) requires the app to declare x-shc-volumes, and grep over shc/apps/ returns x-shc-volumes zero times across all 73 catalogue apps (x-shc-scale appears in 63 of them, so the extension mechanism is well understood; this particular one is simply unused). So every stateful swarm install of every shipped app lands in cell 6.

Two further hazards on the same shape:

  • shc scale <stack> <svc> N has no volume guard at all (ce/modules/app/service.go:859-897). N replicas against one node-local volume name is N forked datasets. The replicas: 1 clamp exists only for provisioned RWO (ce/modules/app/swarm_placement.go:104-161).
  • FILE mounts fail the opposite way. SHC never converts a swarm DATA mount to a host bind, so the loud “bind source does not exist → task Rejected” failure does not apply to DATA. It applies to FILE, and that is an INFO log (X200995PMTFIL). DATA fails silently-empty; FILE fails loudly-unschedulable. Exactly inverted from what safety wants.

Cells 2 and 7 are unreachable, and that is the second finding

Section titled “Cells 2 and 7 are unreachable, and that is the second finding”

The owner’s model requires an escape hatch: “become unbound” by moving critical data onto a shared provider. SHC cannot execute it.

DeploymentDir is unconditionally <StateHome>/deployments/<t>/<s>/<e> (ce/core/paths/paths.go:712) and MountsDir unconditionally <deployDir>/mounts (ce/modules/app/install_runner.go:3023). The storage provider is never consulted on that path. storageapi.Provider.BasePath() and CreateDeploymentFolder (the interface that would place a deployment’s data on a provider) have zero production callers in either repo. NFS and CephFS providers are real: they are registered, linked to nodes, and genuinely mounted at /mnt/shc/nfs and /mnt/shc/cephfs. Nothing routes a deployment’s mounts onto them.

--storage <name> is therefore an unvalidated label. Nothing checks that the provider exists, that the target node is linked to it, or that it is legal (ce/modules/app/service.go:475-478 stores it verbatim; install_refusal.go has no storage gate). It changes no placement. And it reaches exactly two consumers, one of which is dangerous:

The move classifier trusts it. ClassifyDeployment falls back to deployment.storage as the provider for any mount without a volume row, which is every mount in every shipped app (ce/daemon/clone_move_adapters.go:117-120, defaulting to "local" when empty). ClassifyMountDecision then returns Rebindable for nfs / cephfs whenever the target node carries a store_node link row, a row AttachAllNodes writes even when the node-side client mount failed, warned as X503085SHFSMT. runRebindMove then skips the rebind entirely, on the reasoning that “a rebindable mount without a volume id is an RWX shared FS already visible on the target; nothing to detach/attach” (ce/modules/clone/move_strategy.go:429-445), re-renders on the target and starts.

So shc install nextcloud --storage nfs (the storage documentation’s own worked example, shc/docs/reference/cli/storage.md:229) records "nfs", places the bytes on local disk anyway, and arms a later shc move to relocate the deployment onto a node where those bytes are not. The only thing on that path is a warning.

The same unverified claim is trusted by the dead-node classifier (ce/daemon/clone_dead_move_adapters.go:205-208), where DeadMountShared means “the data lives on the share”, on the path the code itself calls “the most dangerous path in the product”.


Three points, and the ordering between them is the argument.

1. Install: make UNANCHORED unrepresentable, do not refuse the workload

Section titled “1. Install: make UNANCHORED unrepresentable, do not refuse the workload”

The install-time guard is what makes the removal-time guard usable. This is the whole case for enforcing at install, and it is structural rather than a matter of taste.

The removal guard must be fail-closed: if SHC cannot prove a removal is safe, it refuses. But anchor = UNKNOWN on an unpinned swarm service means the bytes might be on any node, so a sound fail-closed removal guard must refuse every node removal for as long as one UNANCHORED deployment exists anywhere in the cluster. That leaves exactly two options, and both are bad: allow the removal anyway (unsound: it loses data), or block all removals (useless: the cluster becomes unrotatable, which directly contradicts the addressing model’s rotation goal).

Making UNANCHORED unrepresentable at install is the only way out. It is also the only enforcement point that helps an operator who never runs node rm at all, including the case in § when a bound node dies, which is the case that most directly threatens the owner’s bar.

The mechanism is a pin, not a refusal. At render, a swarm deployment with any critical mount whose anchor would be a singleton is stamped with node.hostname==<holder> (whether or not --node was passed), reusing the mechanism ApplyVolumePlacement already implements for provisioned RWO, including its replicas: 1 clamp. Cell 6 becomes cell 5. Nothing is refused, no catalogue app breaks, and no operator action is required.

Why pin rather than refuse: refusing cell 6 makes stateful swarm installs impossible for all 73 catalogue apps, since cell 8 needs x-shc-volumes that none of them declare and cell 7 does not work. That is a product regression with no safety gain over pinning; the pin is exactly the fact the deployment already implies, made explicit.

Refusal is reserved for requests with no safe interpretation:

requestverdict
swarm + critical local mount, no pinauto-pin (derived), report it
explicit --unbound / --storage <shared> that cannot be honouredrefuse: the ask is unsatisfiable, and today it is silently accepted
shc scale <stack> <svc> N, N > 1, on a BOUND deploymentrefuse: N replicas, N forked datasets
clearing a derived pin while the mount is still node-localrefuse: that manufactures UNANCHORED on purpose
clearing an operator pin on an UNBOUND deploymentallow: that is what unpin is for

The derived pin must be distinguishable from an operator’s --node, so that shc unpin can clear the second and refuse the first. That is one boolean alongside the placement, and without it “unpin” has no coherent definition.

One honest cost of auto-pinning, stated plainly: it makes swarm’s non-availability for stateful workloads silent unless bound-ness is visible. That is the entire justification for the two inspection verbs being part of the same slice, not a later nicety. A pinned deployment must read BOUND in shc ls, or SHC has quietly replaced a data-loss bug with an availability lie.

2. Node removal: split DATA LOSS from AVAILABILITY

Section titled “2. Node removal: split DATA LOSS from AVAILABILITY”

Today --force collapses both into one bypass. Splitting them is the change that makes “I don’t want to ever be in a scenario where you can lose state” enforceable rather than aspirational.

conditionverdictescape
n ∈ bound_to(D) for some DHARD REFUSAL: not --force-ableshc move D --node <other> (relocates the bytes), or an explicit shc D abandon --discard-local naming the mounts being given up
any D is UNANCHOREDREFUSE, fail-closed: the message names the uncertainty, not a deploymentmake it anchored (pin or move), then retry
D unbound, n was its last runnable nodeWARN: availability, not dataplain --force
otherwiseallownone

The escape verb is not new machinery: --discard-local is the existing precedent from move_presumed_dead.go, and it reads right: the operator names what they are abandoning rather than passing a generic override.

Non---force-able matches EnsureVoterRemovable’s posture, which is already the house pattern for “this guard exists to stop you wedging something you cannot un-wedge” (ce/modules/cluster/service.go:1715-1810).

Four removal paths must all consult this, and today they behave four different ways:

pathtodayunder this model
shc cluster rm <node>X400017NODDEP counts deployments; --force bypassessplit verdict; the data leg is not bypassable
shc node forget <node>force hardcoded, no deployment gate at all (lifecycle_router.go:305)keeps force for a node that is already gone, but reports what it is orphaning
shc cluster repairevicts any node unreachable or unseen 24h, force hardcodedmust refuse for a node in any bound_to; may sweep the rest
terraform destroy of shc_nodecalls forget, so no gate at allthe addressing model’s slice 11 routes it to the gated removal; this model supplies the predicate

shc cluster repair’s 24-hour sweep is the sharpest instance of the owner’s concern in the current code: a day-long partition silently evicts every deployment row on a node, plus its node-local backup rows.

Both the transitions and the temptation are real. Bound-ness can change under a running deployment at least four ways:

  1. shc storage nfs disable deletes the provider row through the raw repository path, deliberately bypassing DeleteProvider’s in-use guard (ce/modules/storage/shared_manager.go:245-249, comment: “Bypass the IsManaged guard… this orchestrator is the only legal entry point”). Every deployment naming that provider silently rebinds. The guard it drives around already implements the right join (ce/modules/storage/service.go:388-420).
  2. shc storage unlink <provider> <node>: a shared anchor shrinks; nothing re-classifies.
  3. shc <stack> update: a catalogue bump can introduce a new DATA mount, and update re-renders while dropping --node (ce/modules/app/commands/stack_lifecycle.go:691-693). An UNBOUND deployment silently becomes BOUND, or on swarm UNANCHORED.
  4. A swarm reschedule: the holder changes; the row does not.

The tempting fix is a bound column plus a reconcile cron. Do not. A stored bound-ness flag is exactly the class of thing that goes stale and turns a guard green, the failure mode the occupancy guard’s own fail-closed comment is written against. Bound-ness is derived fresh at every decision point from four inputs that already exist: the rendered compose’s mount set, volume rows plus attachment, storage-provider link rows, and placement constraints parsed from the rendered compose. Only the rendered answer is cached, for display; no guard may read the cache.

What is continuous is small and cheap: the two operator commands above (1 and 2) must run the in-use join before performing a rebinding transition, and update (3) must re-derive and re-pin rather than only re-render. Those are three call sites, not a cron.


When a bound node dies, and no gate can refuse that

Section titled “When a bound node dies, and no gate can refuse that”

This is the case that most directly threatens the owner’s bar, and the one no refusal reaches. Five behaviours, in priority order.

1. Never silently reschedule, and this is already paid for at install

Section titled “1. Never silently reschedule, and this is already paid for at install”

For swarm, a BOUND deployment carries a placement constraint, so swarm’s own scheduler refuses to place it elsewhere. The task goes Pending: no suitable node. That is loud, correct, and non-destructive: the bytes are intact on a disk that is merely unreachable.

This is the connective tissue between the two sections and the strongest argument for enforcing at install: the install-time pin IS the dead-node guard. Everything below is about surfacing and recovery; this is the only part that prevents the loss, and it must be in place before the node dies, because nothing can be added afterwards.

2. Surface it as STRANDED, and name the node

Section titled “2. Surface it as STRANDED, and name the node”

Today node_poller stamps the node unreachable and its containers unknown, while the deployment row still reports an ordinary placement. There should be a first-class deployment state meaning “this deployment’s critical data is on a node that is not answering”: shc ls shows it, the HUD shows it, and it does not auto-resolve.

Distinguish it from ordinary unhealthiness, because the operator response is different: a stranded deployment must not be restarted, redeployed or scaled, and every one of those verbs should say so.

An unreachable node is not a dead node. The default posture is to wait, which is what SHC does today for deployments, and explicitly not what shc cluster repair does for node rows. Repair may sweep an unreachable node carrying no bound deployments; it must refuse one that carries them, and say which.

4. One explicit escape, which names the loss

Section titled “4. One explicit escape, which names the loss”

shc move <stack> --presumed-dead already is this verb, and it already behaves correctly: it classifies every mount, refuses X400619CLNDLM when any is local-only, treats an unreadable app template as lost rather than assuming safety (MountsProven), and requires --discard-local to proceed. The model’s contribution is not to build it but to make its classifier THE classifier, so the same reasoning is reachable from every path instead of one verb.

5. Restore from backup is the recovery, and it must be honest about what it cannot replay

Section titled “5. Restore from backup is the recovery, and it must be honest about what it cannot replay”

--discard-local currently means “abandon the local mounts”. Whether that costs four hours or everything depends on facts the operator cannot see: whether a backup ever captured that mount, how old it is, and whether a restore would actually replay it. Three real ways the answer is “no”:

  • a converter ran with no matching jobs.restore entry; apps/redis/backup.yaml documents this trap at length, and the restore “reports success and comes up with an empty keyspace”;
  • a host bind excluded by backup policy;
  • a swarm named volume from before ce/modules/backup/namedvolume.go landed, when a .bak of such a deployment shipped with no volume data at all.

Proposal: the binding model carries a recoverability fact per critical mount: last capture time and whether a restore path exists; and --discard-local prints it per mount before proceeding. That is the difference between an operator accepting a four-hour gap and an operator accepting total loss, and today the two are indistinguishable at the prompt.

This is the honest answer to “no gate can refuse a node dying”: the system cannot prevent it, so it must make the cost of every available response legible before the operator commits to one.


Both estates (promanager/hel1 and bitspur/hel1) are three nodes: one compose-runtime node and two swarm-runtime nodes, per promanager-ops/hetzner/hel1/infra/main.tf:139-140 and bitspur-ops/hetzner/hel1/infra/main.tf:144-145. Both configure storage.volumes.provider as hcloud, which is the block-volume backend and never fires because no catalogue app declares x-shc-volumes.

Can this be adopted without recreating anything? Yes

Section titled “Can this be adopted without recreating anything? Yes”

The model is pure derivation over facts already on disk. It adds no schema, moves no bytes, and re-renders nothing by itself. Adoption in waves 0 through 2 (see slices) touches zero deployments.

The behaviour changes are all in the refusal direction, on paths the estates can hit but do not exercise routinely:

  • shc cluster rm / node forget / terraform destroy of a node holding bound deployments begins refusing where it previously evicted and orphaned. This is the same path the addressing model’s slice 11 already proposes to gate; the two land together or slice 11 lands first with a count-based predicate and this model refines it.
  • shc cluster repair’s 24-hour auto-eviction begins refusing for such a node.

The one wave that changes a render is the swarm auto-pin. It re-renders swarm deployments to add a placement constraint, and neither estate has any, so on these two estates it is a no-op. See below.

Derived from the ops repositories and the code, without contacting either estate.

bitspur-ops/hetzner/hel1/deployments/corp.tf declares eight shc_deployment resources: postgres (as affine-db), affine, postgres (as huly-db), minio, huly, postgres (as keygen-db), keygen, and license-broker. None of them sets node, runtime or storage; all three attributes exist on the resource (terraform/provider/internal/provider/deployment_resource.go:205,217,301) and all three are absent from every block. promanager-ops’s deployments root declares no apps by design; the promanager application deploys from its own repository’s CI, which was not inspected.

Three code facts turn that into a classification:

  1. The deployment runtime defaults to compose, not to the node’s runtime: Runtime: RuntimeCompose in the model literal (ce/modules/app/service.go:444), overridden only when req.Runtime != "" (:487-489). So every one of the eight is a compose deployment, including any that landed on a swarm-runtime node.
  2. Compose DATA mounts become host binds under the deployment dir on the node that ran the install. Every one of these apps has critical DATA mounts: three Postgres instances, a MinIO object store, plus each application’s own state.
  3. deployment.storage is NULL, so the move classifier defaults the provider to "local" (clone_move_adapters.go:117-120), which is the correct answer here, arrived at by luck rather than by validation.

Verdict: on day one, every deployment on both estates classifies BOUND, each to exactly one node. Nothing is reclassified into a more dangerous state; nothing becomes newly unsafe. What the model adds is that the fact is now stated, and that removing the holder node stops being possible by accident.

Cell 6 is not currently occupied on either estate. No estate deployment sets --runtime swarm, so none is UNANCHORED. The trap is the product default for anyone who types --runtime swarm; it is not a live exposure on these two clusters. That distinction should inform urgency: this is urgent for the product, not for the estates.

And one finding that falls out of the same reading. Both estates’ deployment roots use the provider’s many-host SSH transport: hosts = ...cluster_ssh_hosts, every node tried in order until one connects (a deliberate 2026-07-27 fix). Combined with stampNode, which records whichever node handled the request, deployment placement on these estates is a function of which node answered SSH first. Two applies can place two deployments on two different nodes with no declared intent and no diff. Under this model that is directly visible: bound_to is derived from where the bytes actually are, so the accident becomes a stated fact instead of a hidden one. It is also an argument for the addressing model’s observation that ssh_hosts ordering is an optimisation and must not be built upon.

Zero impact on both estates today (no swarm deployments). For any cluster that does have one, the pin is added at the next render, which is an update or move, not a background sweep. A deployment already carrying its own deploy.placement.constraints is left alone: ApplyDeployPlacement respects an app’s self-pin (ce/modules/app/swarm_placement.go:58-63), and ApplyVolumePlacement deliberately overrides it for provisioned RWO. Which of those two precedents the binding pin follows is an owner ruling; see Q5.


It cannot make a deployment unbound. There is no shared data path for ordinary mounts (cells 2 and 7), so “move critical state onto a shared provider” is not an executable verb. Until --storage is either made real or retired, the only route from BOUND to UNBOUND is x-shc-volumes, which no catalogue app uses. The model can therefore refuse and report, but the owner’s escape hatch (“or becomes unbound”) does not exist yet.

It cannot see whether <StateHome> is itself on a share. An operator who genuinely mounts /var/lib/shc from CephFS gets no credit: nothing records or probes the deployment tree’s filesystem, and node.data_dir exists in the schema and is never written. SHC over-binds, which is the safe direction, but the one legitimate way to make compose mounts shared is invisible.

It cannot locate an already-running unpinned swarm task. The auto-pin prevents new ones; it does not retroactively find where an existing one’s bytes are. For those, the honest answer is UNANCHORED and a manual operator decision. Teaching SHC real swarm placement (docker service ps carries the node) is adjacent work this model would benefit from but does not require.

It does not fix driver_opts.device volumes pointing outside the deployment dir. Backup discovers them; the render never sees them, since they are a top-level declaration rather than a service mount source, so DetectMountType’s SYSTEM allowlist never fires. They are host paths in disguise and this model does not classify them. Called out because a locality hole is worse when a locality model exists.

It does not defend against provider drift on an existing volume row. EnsureVolume is get-or-create and does not update provider on an existing row, so a config flip can leave the row’s provider and the provider actually used at Bind divergent, and the stale value is what the move classifier reads.

It does not give an operator a way to declare “I want this unbound and I accept the consequences”. Deliberately. Until a shared data path exists, such a declaration would be a request SHC cannot honour, and accepting unsatisfiable requests silently is the bug this document is largely about.


Each slice is independently landable and independently provable. A slice is not done until its proofs run.

Slice 0: stop the bleeding, no model required

Section titled “Slice 0: stop the bleeding, no model required”

Four fixes that are correctness today and that the model would otherwise inherit. None needs a schema change or a new concept.

  1. The move and dead-move classifiers stop trusting deployment.storage. A provider that cannot be verified against a probed fact degrades to NotRebindable rather than being taken as gospel (ce/daemon/clone_move_adapters.go:117-120, ce/daemon/clone_dead_move_adapters.go:205-208). This alone closes the --storage nfs silent-loss chain by degrading it to backup and restore.
  2. shc cluster repair gains the deployment gate. It calls Remove(..., force=true) today with no gate at all (shc/modules/cluster/admin_router.go); at minimum it must apply the same X400017NODDEP check shc cluster rm does.
  3. shc storage nfs|cephfs disable runs the in-use join that DeleteProvider already implements, instead of bypassing it (ce/modules/storage/shared_manager.go:245-249).
  4. The storage documentation stops describing a system that does not exist. shc/docs/reference/cli/storage.md:8,16,233 and docs/architecture/modules/storage.md:161-167 claim provider-held deployment state, a tenant/environment default resolution, and a swarm refusal named X400044SWRLCL, none of which exists in either repo. Either fix the docs to describe HEAD, or land slice 4 and make them true; carrying a false safety claim is the worse of the three.

Proof: a unit test that an unverifiable nfs provider claim classifies NotRebindable; a repair run against a fixture node carrying a deployment refuses; a storage nfs disable with a deployment naming it refuses; a docs-lint pass over the corrected pages.

Slice 1: one classifier, one durability property

Section titled “Slice 1: one classifier, one durability property”

Collapse the six mount classifiers into core/volume, with backup, move, dead-move, HUD and the binding predicate as consumers. Add the durability: block to backup.yaml, three-source precedence, critical by default, converter-covered mounts critical. Resolve the absolute-host-bind disagreement between backup and move in backup’s favour (critical, node-local, binds). Declare durability across the 73 catalogue apps, starting with the 23 that already carry an exclude block.

Proof: a golden table over all 73 apps asserting every persistent mount’s durability and its source, so a catalogue change that alters a classification is a visible diff. Plus a fixture per structural rule (tmpfs, anonymous, socket, read-only bind, pseudo-root, RUNTIME name prefix) and a regression test that a converter-covered Postgres data directory reads critical.

Slice 2: the predicate, derived, and visible

Section titled “Slice 2: the predicate, derived, and visible”

Binding(D) → { state, bound_to, per-mount reasons } lifted out of the daemon move adapter into a shared, target-optional form: targetNode == "" means “tell me what this is bound to”. Then the two inspection verbs, shc <stack> binding and shc node bindings <node>, rendering the same reasons a refusal renders. Bound-ness appears as a column in shc ls.

Read-only throughout. Nothing refuses yet.

Proof: on a fixture cluster, shc node bindings names every deployment and the exact mounts that bind it; a deployment whose only critical mount is on a two-node shared provider reads UNBOUND; an unpinned swarm deployment with a critical DATA mount reads UNANCHORED. Run against a snapshot of both estates’ catalogue shapes, the eight bitspur deployments all read BOUND to one node.

Slice 3: install makes UNANCHORED unrepresentable

Section titled “Slice 3: install makes UNANCHORED unrepresentable”

The swarm auto-pin, reusing ApplyVolumePlacement’s mechanism and its replicas: 1 clamp. The derived-versus-operator pin distinction. The four refusals in the install table, including the shc scale guard, which has no volume check today at all.

Proof: a two-node swarm cluster where shc install <db-app> --runtime swarm with no flags produces a service carrying node.hostname==<installer> and replicas: 1; draining that node leaves the task Pending: no suitable node rather than starting empty on the peer and the peer’s volume is verified absent, not merely the task observed; shc scale to 2 refuses; shc unpin refuses on the derived pin and succeeds on an operator pin over shared storage. This is the proof that needs a real multi-node swarm harness (tests/hetzner or tests/proxmox), and it is the one proof in this document that cannot be faked with fixtures.

Slice 4: node removal splits data loss from availability

Section titled “Slice 4: node removal splits data loss from availability”

The four-way verdict table, applied to cluster rm, node forget, cluster repair, and (via the addressing model’s slice 11) the provider’s Delete. shc <stack> abandon --discard-local as the named escape.

Proof: removing a node holding a bound deployment refuses, names the deployment and the binding mounts, and is not --force-able; the same removal after shc move succeeds; removing the last node able to run an unbound deployment warns and proceeds under --force; a cluster containing one UNANCHORED deployment refuses every node removal with a message naming the uncertainty rather than a deployment.

stranded as a first-class deployment state, set when a bound deployment’s node goes unreachable. Restart / redeploy / scale refuse on a stranded deployment and say why. --presumed-dead’s classifier becomes the shared one from slice 1. Per-critical-mount recoverability (last capture and whether a restore path exists) printed before --discard-local proceeds.

Proof: a three-node fixture where killing the holder surfaces the deployment as stranded naming the node; the node returning clears it with no operator action; --discard-local prints, per mount, a capture age or “NOT CAPTURED”, and the “NOT CAPTURED” case is produced by a real converter with no matching restore job (the apps/redis/backup.yaml shape), not by a synthetic one.

The ruling is Q1, and Q1 is three rulings rather than one; see Q1 analysis. The REAL/RETIRED framing below is the original binary and is kept for the record; the proofs it names belong to rung 3, which the analysis recommends retiring. Under the analysis’s recommendation the REAL proof becomes rung 1’s (bytes verifiably under a probed store’s path) and the RETIRED proof becomes rung 3’s alone.

If REAL: --storage nfs|cephfs relocates MountsDir to /mnt/shc/<provider>/deployments/..., which gives the dormant storageapi.Provider interface a purpose, makes the storage documentation true, makes cells 2 and 7 reachable, and makes “become unbound” an executable verb for the first time. If RETIRED: the flag and the deployment.storage column both go, and the classifiers lose a fallback they should never have trusted.

Proof (REAL): a two-node cluster where a deployment installed --storage nfs has its bytes verifiably under /mnt/shc/nfs, reads UNBOUND, and shc move to the peer relocates with no data copy, with the negative leg proven too: the same install onto a node whose client mount failed refuses at install rather than recording a claim.

Proof (RETIRED): both estates plan clean with the attribute removed, and the classifiers’ provider input comes only from volume rows.

Probe the deployment directory’s filesystem at render and persist the result, so <StateHome> on a shared filesystem counts. The mechanism exists: ce/modules/storage/doctor.go:88 isMountPoint already parses /proc/mounts; it needs pointing at <deployDir>/mounts instead of at provider rows. cephfs / nfs / fuse.ceph imply a shared anchor; anything else node-local; a failed probe implies UNKNOWN, which binds.

Proof: a node with /var/lib/shc bind-mounted from an NFS export reads its compose deployments as UNBOUND; the same node with a local /var/lib/shc reads them BOUND; a probe failure reads UNANCHORED.

hasBinding → hasVolume; “bound swarm DATA mount” → “provisioned swarm DATA mount” throughout install_runner.go and the volume package. Purely mechanical, no behaviour change, and it should land before slice 2 publishes bound as an operator-facing word.

Proof: the rename is complete: no occurrence of bound in ce or shc means “has a block volume row”, asserted by a grep-shaped test in the style already used for wire-shape ratchets.


Product decisions, not engineering choices. Ordered by how much else depends on them.

Q1. --storage names three different questions. Which of the three is real? This question was first written as a binary (is --storage REAL or RETIRED?) and that binary is wrong. It collapses “WHICH NODE can serve these bytes” into “WHICH BACKEND ON THIS NODE holds them”, which are different questions with different answers. The owner’s correction, in one sentence: “You still might have multiple storage providers that are available for a specific node that you want to pick.” The three questions, separated by what each does to anchor:

rungthe questioneffect on anchorexists today?
1which of THIS node’s disks holds this deployment (an NVMe beside a bulk HDD, a ZFS pool beside ext4)stays { this node }; the deployment is still BOUND, you have merely chosen a diskregistry, CLI verb and provider implementation ship; the consumer was never written
2which cloud block plane provisions this deployment’s volumes (hcloud, EBS, …){ n : n.location == volume.location }, UNBOUND-IN-LOCATION; decision-table cells 3 and 8works: row-backed, fenced, RWO-clamped, declared via x-shc-volumes; the SELECTION surface does not
3a shared filesystem (nfs / cephfs) holding a deployment’s data directoryevery client node, truly UNBOUND; cells 2 and 7does not exist, and the schema cannot express it

Each rung takes its own ruling: BUILD, FINISH, KEEP or RETIRE. It is entirely coherent for one to be finished, one kept and one deleted, and a single verdict on “the flag” cannot express three different answers. What is not coherent is the middle state all three share today: an unvalidated claim that changes nothing on rungs 1 and 3, has one undocumented negative effect on rung 2, and until slice 0 was trusted by two classifiers as though it were a fact. Almost every other answer’s shape depends on this one. Analysis, evidence and a recommendation: Q1 analysis: the three-rung reading.

Q2. Is the DATA-LOSS refusal truly non---force-able? “I don’t want to ever be in a scenario where you can lose state” reads as yes. But an operator with a physically dead disk needs some way out, and --discard-local is the existing precedent for one that names what is being abandoned rather than waving a generic override. The recommendation is: non---force-able, with abandon --discard-local as the only door, and slice 5’s recoverability report printed before it opens. What --force means on the removal verbs TODAY (amnesia, not discard) and how that narrows this question: Node removal analysis.

Q3. Does durability belong in backup.yaml or in app.yaml? backup.yaml is where the criticality reasoning already lives, in good prose, written by app authors, in 23 of 73 apps; that is real continuity and the recommendation. The cost is that a data-safety property lives in a file named for a subsystem, which is a mild version of the overloading trap this document otherwise argues against. app.yaml reads as a first-class app property and is not backup-scoped, at the cost of splitting the reasoning across two files.

Q4. Should install AUTO-PIN, or REFUSE? The recommendation is auto-pin, because refusing makes stateful swarm impossible for all 73 catalogue apps while pinning is a behaviour change that costs nobody anything and closes the same hole. The counter-argument is that auto-pinning makes swarm silently non-HA for stateful workloads and an operator may reasonably feel SHC promised otherwise, which is why the recommendation is auto-pin plus bound-ness visible in shc ls, in the same slice. Refusing is the posture shc/docs/reference/cli/storage.md:233 already documents, so ruling “refuse” also makes an existing doc claim true.

Q5. When the binding pin meets an app’s own deploy.placement, who wins? Two precedents exist in one file: ApplyDeployPlacement respects an app’s self-pin, ApplyVolumePlacement overrides it. The binding pin protects data, which argues for override, but an app that pins itself usually has a reason. Recommendation: override, and report it, but this is a product call about whose intent is authoritative.

Q6. Is “unbound but nowhere left to run” a warning or a refusal? The model says warning: it is an availability choice the operator is entitled to make, and it is the only case where the gate softens. Confirming matters because getting it wrong in the strict direction makes the last node of a cluster unremovable, and in the loose direction makes a --force that means two different things again.

Q7. Should there be a rebuildable durability class between critical and ephemeral? Greenbone’s vt-data and huly’s elastic are both “losing this costs hours, not data”. Today they classify critical and therefore bind, which is safe and slightly over-strict. A third class could warn-but-allow. Recommendation: two classes now; a third is a promise to get the boundary right, and the boundary is exactly what nobody can agree on.

Q8. Should shc cluster repair’s 24-hour auto-eviction survive at all? It deletes deployment rows and node-local backup rows for any node quiet for a day, with force hardcoded and no gate. That reads as designed for phantom rows, not for live estates. Under this model it must at minimum refuse for bound deployments; the open question is whether a time-triggered destructive sweep belongs in the product at all. Both premises in that framing are false and the question needs re-asking: repair deletes no deployment row and no backup row, and its 24-hour leg is unreachable dead code because last_seen_at exists in no migration and no ent schema. What it actually does, and the hazard that replaces the stated one, are in Node removal analysis.


Analysis, not a decision. Q1 remains open and this document’s status line still reads “awaiting owner ruling”. Written 2026-08-02 from four independent evidence lanes and three judging lenses, re-verified first-hand against ce origin/main at 182e4b8 (the clean worktree /var/tmp/shc-s0/land-ce) and shc at 70fb6b45d. Every code claim below carries a file:line. Neither production estate was contacted by this analysis; live facts are attributed where they are second-hand.

rungverdictthe one reason
1: node-local storeFINISH, in three stages, with a probe as a hard preconditionThe need is real and SHC has no mechanism for it (not one, not a bad one) and the two workarounds an operator would reach for either silently drop the mount from every backup or vanish on reboot
2: block volume planeFINISHThe mechanism is sound and armed on both estates; the SELECTOR is negative-only and evaluated after the branch that needed it, so the day an estate has two volume-capable connections a declared x-shc-volumes stack silently host-binds and comes up healthy on the root disk
3: shared filesystem as a data pathRETIRE, scoped to the deployment-data-path claimIts unique buy is HA-for-stateful, permanently ruled out; its non-unique buy does not even arrive, because MOVE-5 forces every compose node relocation onto backup+restore regardless of mount class and all 24 estate deployments are compose

The panel split on rung 1 only, between BUILD and FINISH. Resolved as FINISH, for two reasons. First, the strongest BUILD argument (that SHC ships documentation promising shc install nextcloud --storage local-ssd) is stale: docs/reference/cli/storage.md was corrected at 76ffa6be4, hours before the panel read it, and now states plainly that “registering a provider does not move any deployment onto it”. The promise was withdrawn, which is the right call and which removes the deontic force of the argument. What remains for rung 1 is the owner’s stated need on its merits, which is enough, but is a thinner basis than BUILD implies. Second, BUILD reads as one authorisation for one piece of work, and rung 1 is not one piece of work: it is a cheap correctness fix that should land now, a mandatory probe, and a path change that should wait for a real second disk. Staging it is the honest verb.

The decomposition holds, with two amendments, and they lead

Section titled “The decomposition holds, with two amendments, and they lead”

The separability critic’s objection outranks the per-rung verdicts, so it goes first. It found two things, and I confirmed both.

Amendment 1: the ladder is not a complete carve. There is a fourth axis, it is live, and it is the one the binding model’s own escape hatch depends on. backup.repo selects where a deployment’s ARCHIVE lives. It is a config key, never a flag; it has its own anchor engine (DeleteNodeLocalBackupRows decides node-locality by string prefix against stateHome and deliberately KEEPS anything else, because “their data may well have survived the node, and deletion is the irreversible direction”, ce/modules/backup/ent_ops.go:214-223); and it defaults NODE-LOCAL: repo: null at ce/core/config/default.yaml:188, whose own comment instructs multi-node clusters to point it at “NFS: /mnt/shc-shared/backups (requires NFS storage provider)”. A grep of both ops repositories’ hetzner/hel1/infra/main.tf for backup returns nothing, so on both estates the archives sit on the same disk as the deployment they would restore. That is second-hand (from the ops repos, not from a box) and somebody should read it off a node before slice 5 is written, because if it holds, this document’s entire consolation (“the cost of under-binding is a restore”, --discard-local, slice 5’s recoverability report) is offering a copy that dies with the original. This is a larger live exposure than anything on the ladder, and it is not on the ladder.

Amendment 2: the rungs are separable for DELETION and for RULING, but not for CONSTRUCTION, and that converts to a precondition rather than a reversal. Separable for deletion, verified: the kind axis is a three-valued enum with a TOTAL classification map and an AST guard (providerTypeShared{local:false, nfs:true, cephfs:true}, ce/modules/storage/storageapi/types.go:38-68); only six non-test ce files import the shared-FS packages; rung 2 lives in a different table (volume / volume_attachment) reached by a different registry (volumeapi builders) and a different grammar (x-shc-volumes), and BuildVolumeBinder never queries store to choose a backend. Separable for ruling: the three arms of ClassifyMountDecision are already written and already disjoint, and the doctrinal exposure differs by the entire trilemma: rung 1 spends no corner, rung 2 spends none because the cloud substrate supplies the property, rung 3 spends both to buy back the one that was given up. Three different answers cannot come out of one verdict.

But not separable for construction, on four proofs:

  1. CreateDeploymentFolder is the same function, byte-for-byte modulo error arguments, in all three providers: providers/local/provider.go:82-96, providers/nfs/provider.go:71-85, providers/cephfs/provider.go:60-74, each deploymentPath → MkdirAll(0o750) → for _, sub := range []string{"mounts","files","app"}. Finishing rung 1’s data path hands rung 3 its data path.
  2. The registry cannot tell rung 1 from rung 3. store carries ONE cluster-wide field.Text("path") (ce/ent/schema/store.go:31); store_node carries only (store_id, node_id, created_at) with no per-node path override (ce/ent/schema/store_node.go:29-35); and shc storage link [name] [node-ids...] takes cobra.MinimumNArgs(2) (ce/modules/storage/commands/link.go:53-55) while Service.LinkNode checks only that the provider row exists and dedupes: no type check, no node-existence check, no probe (ce/modules/storage/service.go:497-514). So shc storage add bulk --path /mnt/bulk -n n1 && shc storage link bulk n2 n3 writes a local-type provider linked to three nodes at one path: the database shape of a shared filesystem, expressible with rung 1’s verbs after rung 3’s packages are deleted.
  3. Nothing probes the claim. routes.handleAdd builds CreateProviderInput and never sets capabilities (ce/modules/storage/router.go:179-186) while CreateProvider’s own contract two hundred lines away says the caller must detect them (service.go:193-195). Service.ProviderInstance (the only function in either repo that constructs a storageapi.Provider, and therefore the only route to local.Detect / IsAccessible) has zero callers in ce (grep -rn ProviderInstance --include='*.go' . returns the definition and its doc comment, nothing else) and two in shc, both unit tests. So all sixteen interface methods are unreachable, not two.
  4. The only validating storage-name surface reduces the whole namespace to VolumeRestoreOverride{Provider, Provision bool} (ce/daemon/backup_volume_capture.go:531-585): provision a block device, or don’t. A named rung-1 store has no field it could be returned in.

The resolution is a constraint on execution, not a reversal of the carve: a rung-1 FINISH must ship the probe that proves a named store is genuinely node-local and singleton (at storage add AND at storage link, on the target node) and must refuse a network filesystem type registered as local. Without it, anchor = { this node } is an operator-typed string with exactly the evidence quality slice 0 spent a night removing trust from, and rung 1 silently re-admits rung 3 through the back door. The pieces exist and none of them runs today: local.Detect, Provider.IsAccessible, isMountPoint parsing /proc/mounts (ce/modules/storage/doctor.go:87-107), and a transport that already POSTs mount work to a remote node’s internal API.

Rung 3’s unique buy (a stateful deployment surviving unplanned node loss without a restore) is HA-for-stateful, the corner the trilemma permanently gave up, and SHC would have to build the consistency domain itself; rung 2’s equivalent property is handed over free by the cloud substrate and rung 1 does not touch anchor at all. And rung 3’s non-unique buy never arrives: relocateBlocked := nodeChanging && !swarm sends every compose node relocation to StrategyBackupRestore regardless of mount class (ce/modules/clone/move_strategy.go:294-345, MOVE-5), and every deployment on both estates is compose, so a shared filesystem buys nothing on move or drain on the runtime that is actually in production.

One word meaning three things is what produced this question. Four axes now exist, so four nouns are needed, and no word may read naturally for two of them. Checked against this document’s own reserved set: bind (addressing, unchanged and owned by the node-addressing model), binding → provisioned (a mount carrying a volume row, slice 8), and bound / unbound / bound_to / anchor (this model, deployment scope only):

axisnounoperator surfacecollision check
rung 1store--store <name>, registered by shc storage add--store is unused across ce’s command flag registrations. Carries no multi-node implication, which is the property that matters. A store is a singleton anchor by construction, once probed
rung 2volumedeclared by the app in x-shc-volumes; backend selected by storage.volumes.provider (which needs a scope tier) and, if a per-deployment override is wanted, --volume-backend, never --storagealready the code’s word; slice 8’s hasBinding → hasVolume rename makes it unambiguous
rung 3sharereserved and unimplemented as a data path; a parse error in the --store slot, not a provider lookup. Legal only as a backup.repo targetnew word, no collision
fourth axisrepositorybackup.repoalready correctly named; the only change is that this document must stop assuming it is off-node

--storage is retired as an operator word. Not because it is worthless (it has exactly one live effect today, and any RETIRE ruling must pay for removing it (see the corrections)) but because a single argument position spellable by both local and nfs, told apart by a string nobody probes, is the mechanism that produced this entire question. The load-bearing rule is negative and should be written on the wall: a shared kind must never be a legal value in the rung-1 slot.

The same defect already recurs one layer down and should be fixed in the same breath: x-shc-volumes parses an unrecognised access token by falling back to RWO, commented “the parser stays permissive” (ce/core/volume/binding.go:130-147). So access: rwx (a request for shared storage) silently yields a single-writer block volume, with no warning and no lint. Two independent surfaces absorb a shared-storage request into a node-local answer without a word. Fix the slot, not the flag.

What this costs, including what the owner asked for and does not get

Section titled “What this costs, including what the owner asked for and does not get”

Plainly, in his own terms: the escape hatch does not arrive. He said “unless it’s moved or becomes unbound”. Under this recommendation “moved” works and “becomes unbound” does not, not for an ordinary compose deployment’s data directory, which is every deployment on both estates and every stateful app in the catalogue. Retiring rung 3 declines to build the one shape that would have made cells 2 and 7 reachable, so a BOUND deployment’s route out stays: shc move, which copies.

What he gets instead, and it should be stated as an obligation rather than a consolation:

  • Rung 2 becomes the real escape hatch, and it already works. A provisioned block volume detaches and reattaches with no copy, fenced by volume_attachment.lease_generation and RWO-clamped at install. Both estates already carry exactly one volume-capable hetzner connection, so the substrate is present and configuration-free. The gap is the catalogue: x-shc-volumes appears in zero of 73 apps. That is one compose extension block per stateful app, not a new consistency domain, and it is the follow-on that makes this retirement honest.
  • The ceiling gets written down instead of engineered around. move and drain stage through a LOCAL archive (MakeTempDir("shc_clone_") → <StateHome>/staging) before shipping it, and phase order is backup → stop → recover → uninstall, so source and target must each hold dataset plus archive simultaneously. On a single 610 GB disk that is a hard ceiling somewhere near 45% of the disk, and above it a deployment cannot be relocated at all. That is a real capability limit with a number attached and it belongs in the operator docs in plain words.
  • --discard-local stops being a coin flip, but only if slice 5’s recoverability report also reports whether the archive is on the same disk as the thing it would restore. See the fourth axis.

The rest of the bill:

  • Rung 1 stage 3 is pervasive but bounded. Twelve non-test inline re-derivations of filepath.Join(<x>Dir, "mounts") (daemon/clone_move_adapters.go:648, modules/app/install_runner.go:3023, modules/app/post_install_hooks.go:359, modules/adapter/commands/migrate_from.go:276, modules/backup/restore.go:384 and :471, modules/backup/service.go:1001, modules/backup/cli_runner.go:161, :388, :549, :1866, :2084), plus the two resolver families (core/paths.Paths.MountsRoot and app.DeploymentDir), plus six name-based invariants that encode “mounts is a child of the deployment dir” by string: install_runner.go:2081 (the keep-data survivor set), backup/archive.go:309 (separatelyArchivedDirs), doctor/diagnoses/database_orphaned_dirs.go:38, and three info.Name() == "mounts" walk tests at restore.go:1033, :1229, :1292. Plus shc’s hardcoded acme glob.
  • One ordering constraint that is not negotiable. PersistentMountsFromCompose classifies SHC data by string prefix (mountsPrefix := filepath.Join(deployDir, "mounts") + os.PathSeparator (ce/daemon/clone_move_adapters.go:648)) and anything absolute outside it falls into hostBinds, which the function’s own comment declares “classified ephemeral; SHC data never lives there by convention”. That site must be converted before any install can write outside the default root. On compose the failure is a placement lie rather than byte loss (MOVE-5 copies anyway, and captureHostBinds does tar absolute binds); on swarm it is StrategyEphemeralOnly and the bytes stay behind.
  • Two costs of RETIRE that must not be skipped. (i) refuseIfInUse joins BOTH entvolume.ProviderIn and entdeployment.StorageIn (ce/modules/storage/service.go:423-469) to gate shc storage rm, DisableNFS and DisableCephFS. Because the ent Volume CHECK forbids a shared-FS volume row, the deployment leg is the only live leg for nfs/cephfs. A rung-1 FINISH keeps that leg alive under a new flag name; if both retired, the guard would silently lose its subject. (ii) Slice 0 shipped two AST guards whose subject set is derived from providerTypeShared. Deleting ProviderTypeNFS / ProviderTypeCephFS makes them vacuous rather than red, the fifth “guard is green while the thing it guards is broken” of this session. They must be replaced with an assertion that the shared set is EMPTY and that no local provider is linked to more than one node.
  • Docs are doubled. docs/ and website/src/content/docs/ are independent copies with no sync target, so every page is edited twice.

R1a: validate, and finish the docs (land now; no path change, no migration). Install refuses a --store name with no store row, or a row not linked to the target node, reusing the join providerUsage already implements (ce/modules/storage/service.go:446-469). Today install stores the string verbatim (ce/modules/app/service.go:475-477; grep -ci storage ce/modules/app/install_refusal.go = 0) while shc move / shc recover refuse unknown names with X400865VOLSOV, two ends of one flag disagreeing about its domain, with only the destructive-adjacent end right. Docs: docs/reference/cli/storage.md is already correct as of 76ffa6be4, but docs/guide/operations/storage-providers.md:273-295 is still entirely fictional: its five-input algorithm names an app-declared requires.storage whitelist, a whitelist-checked per-stack --storage override, node affinity, a capability filter and a cluster default, and grep -rn 'requires.storage\|RequiresStorage' --include='*.go' over both repos returns zero. requires.storage also appears in docs/app-development/creating-apps.md, docs/guide/troubleshooting/README.md, and (as a live shell demo on the public site) website/landing/arch/where-it-lives/index.html:534-537 ($ shc install nextcloud --storage nfs → “the app’s requires.storage whitelist still has to allow it”) plus its landing-archive twin.

R1b: the probe (precondition for R1c, valuable alone). Run local.Detect + IsAccessible on the TARGET node at storage add and at storage link, over the existing remote transport. Persist the probed filesystem type. Refuse a network filesystem registered as local; refuse (or add a nullable per-node path column to store_node, the cheap append-only ADD COLUMN shape) a local provider linked to more than one node. Gives Service.ProviderInstance its first production caller and the sixteen dormant interface methods their first reachable path.

R1c: the resolver (defer until an estate has a second disk). One MountsRootFor(tenant, stack, env) reading the manifest’s store name → the provider row’s path. Convert clone_move_adapters.go:648 FIRST, then the eleven other inline joins, then the six string invariants. Design B (relocate mounts/ only, leaving compose.yaml / manifest.json / app/ under the state home) keeps GetStackFromDisk and the manifest findable and avoids the ~140-site whole-DeploymentDir variant. Follow-on: the scheduler’s per-node disk is statfs("/") (ce/subsystems/impl/scheduling_source.go:76, :104), so a relocated deployment would still be billed against the root filesystem.

R2a: make the block selector positive (~15 lines, one function). In ce/daemon/volume_binder.go:357-407, hoist the storageDefault check ABOVE the auto-select block. Today the default: arm of that switch warns X400327VOLAMB and returns nil at :398-400, before the flag is read at :404-407, so on a two-candidate estate the flag can never break the tie it sits next to, and the error’s own text tells the operator to edit storage.volumes.provider instead. If the name matches a live volume-capable connection or a registered block impl → select it; if it names a store or a share → decline, preserving today’s kill-switch exactly; if unrecognised → refuse LOUDLY rather than silently host-binding. Every recogniser already exists.

R2b: thread the resolved provider into the provision (two files). DaemonStorageOverrideResolver genuinely validates a provider name and refuses unknowns, then resolveProvisionedOwners passes it to ProvisionForRestore, which builds its binder as a.binderFn(ctx), wired at ce/daemon/services_backup_config.go:140-142 as d.BuildVolumeBinder(ctx, paths.Default.StateHome(), ""), and uses the provider argument only inside error strings (ce/daemon/backup_volume_capture.go:292-299). So shc move --storage aws-ebs-volume validates, succeeds, and re-provisions on the global backend. A validation chain terminating in a call that ignores its argument is worse than no validation.

R2c: two safety rails. (i) Service.Scale (ce/modules/app/service.go:859+) has no volume awareness at all, so a routine shc scale defeats the RWO pin plus replicas: 1 clamp the install stamped at install_runner.go:3153. (ii) A bound SWARM stack cannot be updated: install_async.go:522-533 deliberately resolves no binder (Bind is not idempotent), so shc update returns X400324VOLSWMNB whose registered template reads “no volume backend is configured”, which is false, and sends the operator to check a token that is fine.

R3: retire the shared filesystem as a DEPLOYMENT DATA PATH, scoped. Goes: the case "nfs","cephfs" arm of DaemonStorageOverrideResolver (ce/daemon/backup_volume_capture.go:529-542); the isSharedFSProvider arms of both move classifiers and the X503085SHFSMT warning; --store nfs|cephfs becomes a parse error at install and at move/recover; the public prose above. Does NOT automatically go: the ~6,250 non-test Go lines of node-side mount machinery (modules/storage/ceph 2230, providers/nfs 990, providers/cephfs 850, modules/storage/nfs 508, shared_manager.go 505, shared_manager_cephfs.go 331, commands/managed.go 499, swarm_label.go 103, daemon/cephfs_provisioner.go 102, daemon/services_ceph_peer.go 138), ten CLI verbs and ~10 pkg/apiclient methods. Whether those survive is a separate call keyed on backup.repo, because a shared ARCHIVE target is a share used the way a share is safe (one writer at a time, restic’s own locking, no live-writer semantics) and default.yaml:182-183 already documents it. Deleting the substrate to satisfy a doctrine about live data paths would break the documented archive path the doctrine never spoke to.

Free, and worth doing whichever way this rules. diagnoseStores does os.Stat(rec.Path) at ce/modules/storage/doctor.go:53 BEFORE isMountPoint at :67, and isMountPoint is the safe one, since it parses /proc/mounts. On a hung hard NFS mount shc doctor blocks inside the one check meant to notice the outage. One-line reorder.

None is required, in either direction. deployment.storage already exists, is already written, and is the field rung 1 needs; reusing it under the name --store costs zero migration. The ent field name stays storage; the concept is a store; one comment reconciles them.

Under RETIRE, the column must be RETIRED IN PLACE and never dropped. The house precedent is three lines below it in the same struct: launcher at ce/ent/schema/deployment.go:73-78: “RETIRED IN PLACE (pre-1.0) … No code reads or writes the column; it stays in the schema only because migrations are frozen (atlas.sum). Always NULL on new rows.” A naive drop is not merely expensive: SQLite has no DROP COLUMN for atlas, so it emits a table rebuild, and applyOne runs a whole migration file inside ONE BeginTx, where SQLite documents PRAGMA foreign_keys as a no-op. An evidence lane proved the consequence empirically against modernc sqlite: service / host_mapping / port_mapping / integration rows cascade-deleted, backup.deployment_id NULLed. Six tables hold FKs to deployment(id). Do not go near it.

The only additive change contemplated is R1b’s nullable per-node path on store_node, the one-line ADD COLUMN shape (migrations/20260730032524_node_cpu_cores.sql is the precedent), never a rebuild, never a flatten.

shc’s hand-written DDL mirrors need nothing. Both are partial subsets (shc/tests/unit/subsystems/impl/stack_reconcile_ports_test.go:39, shc/tests/unit/modules/doctor/diagnoses/database_test.go:27) and neither ever carried storage.

NO BACKCOMPAT is satisfied trivially. Three lanes independently read both estates read-only and agree: shc storage ls returns [] on both (the store table is empty; not even a local row), shc volume ls returns none, all 24 deployments carry storage IS NULL in both the DB and the on-disk sidecar, and all 24 are runtime: compose. Zero storage = attributes across all three ops repos; zero x-shc-volumes across all 73 catalogue apps; no shc_storage_provider terraform resource exists at all. Whichever way Q1 lands, it breaks zero live deployments. One forced exception survives regardless: shc/terraform/provider/internal/provider/state_upgrade.go holds the frozen decode-only v0 schema, and deploymentSchemaV0() must keep storage or every apply against pre-v1 state fails to decode. That is Terraform’s state-upgrade contract, not SHC’s.

What this unblocks, and how each downstream question changes shape

Section titled “What this unblocks, and how each downstream question changes shape”
  • Slices 1, 2, 3, 4: unblocked unchanged. The predicate, the classifier collapse, the auto-pin and the removal split were never waiting on --storage; they were waiting on Q1 only because Q1 decided whether an UNBOUND destination existed.
  • Slice 5 (the dead-node path): reshaped, and this is the sharpest change. The recoverability report gains a mandatory second leg: per-critical-mount capture age AND whether the archive is node-local. With rung 3 retired and backup.repo unset, --discard-local on a live estate today is not “four hours versus everything”; it is “everything”, and the prompt cannot say so.
  • Slice 6: becomes three sub-rulings instead of one. Its REAL/RETIRED proofs both need rewriting: the REAL proof was written for rung 3 (bytes verifiably under /mnt/shc/nfs, move with no data copy) and becomes rung 1’s (bytes verifiably under the store’s path, a same-node storage re-home, an install onto an unprobed store refused).
  • Slice 7 (the filesystem probe): repointed and promoted. It was written as a late, optional probe of <deployDir>/mounts at render, to give credit to an operator who mounts /var/lib/shc from CephFS. Under this recommendation it becomes R1b: a probe at storage add / storage link, on the target node, and a precondition of rung 1 rather than a nicety. Its shared-filesystem-implies-shared-anchor arm inverts: a cephfs fstype under a registered store becomes a REFUSAL, not credit.
  • Slice 8 (the vocabulary rename): grows by one word and should still land first. hasBinding → hasVolume as written, plus --storage → --store and the reserved share.
  • Q2 (non---force-able data-loss refusal): unchanged in principle, conditional in practice. The recommendation stands, but abandon --discard-local is only an honest door if the archive is off-node. backup.repo becomes a precondition of Q2’s answer, not a detail.
  • Q4 (auto-pin versus refuse): strengthened toward AUTO-PIN. The counter-argument was partly that “refuse” would make an existing doc claim true (storage.md:233, “Swarm-mode stacks require a shared provider”). That sentence has since been deleted as false, and rung 3’s retirement removes the destination a refusal would have sent the operator to. Refusing now means refusing stateful swarm with nowhere to go.
  • Q3, Q5, Q6, Q7: unchanged. None depends on which storage axis is real.
  • Q8 (cluster repair’s 24-hour sweep): worse than it read. It deletes node-local backup rows, and with backup.repo unset that is nearly all of them on both estates.
  • Not unblocked, and it should be a condition of ruling rung 3: the trilemma is cited by two architecture docs (docs/architecture/cluster/swarm-contention-map.md:493, swarm-quorum-and-health.md:605) as living in docs/guide/getting-started/concepts.md, and a grep of that file for trilemma|pick two|high availab|stateful|node loss|reschedul returns zero hits over all 328 lines. A ruling that is only ever cited and never stated will be re-litigated. Write the paragraph in the same commit.

Rung 1’s stage 3 rests on a demand claim nobody can establish. No node on either estate has a second disk (three lanes’ live lsblk: one 610.4 GB sda, sda1 ext4 at /, no spare device on any of six nodes). No ops repo sets a storage attribute. No catalogue app needs one. store is empty on both estates, which means the rung-1 registry may never have executed outside unit tests, and nobody grepped the bats/e2e suites to check. The affirmative case is the owner’s stated need plus a promise the docs have since withdrawn, not an observed operator hitting a wall.

What would overturn it: evidence that shc storage add has never been exercised anywhere outside unit tests, combined with the owner confirming he was describing a shape he wants rather than one he has. That collapses the ruling to R1a + R1b (validate, probe, document) with R1c indefinitely deferred, which is RETIRE-in-effect for the relocation half. Note the asymmetry: if the single-disk finding is WRONG, the case gets stronger, not weaker. R1a and R1b are worth landing under either outcome, which is the argument for staging.

A second, structural weakness, stated because two judges raised it independently: the rung-3 RETIRE leans on a doctrine that is not written where it is cited, and the distinction that spares rung 2 while killing rung 3 (the trilemma forgoes BUILDING a consistency domain, it does not forbid one the substrate hands you free) is this analysis’s reconstruction of intent, not a quoted ruling. If the owner’s actual line is narrower (“don’t build a distributed filesystem ourselves”), rung 3 still loses on MOVE-5 alone but loses as a COST argument rather than a doctrinal one. If it is broader (“no stateful failover, period”), rung 2’s location rebind is already over the line and its FINISH is wrong.

These came out of the evidence lanes and contradict facts asserted elsewhere tonight. They are load-bearing beyond Q1.

  1. --storage is NOT inert on rung 2. The claim that “every production call site of BuildVolumeBinder hardcodes an EMPTY storage” is wrong. Four DIRECT calls do (services_seams_stack.go:729, services_backup_config.go:141, volume_ops.go:430), but the mainline install goes through the seam: ResolveVolumeBinder(ctx, stateHome, req.Storage) at ce/modules/app/install_async.go:139, whose provider is d.BuildVolumeBinder registered at ce/daemon/services_extra.go:235-236. So --storage is today a live negative kill switch: if sd != "" && sd != providerName { return nil } (ce/daemon/volume_binder.go:404-407) declines the block binder and forces the host-bind path. It is the flag’s one real behaviour, it is undocumented as such, and retiring the flag silently re-enables block provisioning for anyone using it that way.
  2. There is no shc restore --storage flag. Exactly three registrations exist in either repo: shc install (ce/modules/app/commands/install.go:152), shc recover (ce/modules/app/commands/stack_lifecycle.go:641), and shc clone / shc move (ce/modules/clone/commands/main.go:300 via bindCloneFlags). DaemonStorageOverrideResolver’s false-premise problem is real, but its surface is move/recover, which matters because retiring the flag at install does not by itself retire that path.
  3. The dormancy is much wider than “BasePath() and CreateDeploymentFolder() have zero production callers”. Service.ProviderInstance is the only constructor of a storageapi.Provider in either repo and has zero callers in ce. Nothing in production ever holds a Provider, so all sixteen interface methods are unreachable, not two. storage.MountPath is a second, independent provider→path mapping with zero callers anywhere.
  4. docs/reference/cli/storage.md was corrected at shc 76ffa6be4 and now states that registering a provider moves nothing. Any argument resting on that page still promising --storage local-ssd is reading a superseded revision. But docs/guide/operations/storage-providers.md:273-295 and the public landing page still describe the fictional five-input selector, and requires.storage (its first input) appears in zero Go files in either repo. That is the same shipped-promise-with-no-implementation shape that taught two classifiers to trust a label.
  5. shc storage nfs|cephfs disable no longer bypasses the in-use guard. ce 182e4b8 added refuseIfInUse and placed it BEFORE the node teardown. An assertion to the contrary is describing a state that was true earlier the same night.
  6. Slice 0 is on ce origin/main, not in the working checkout. /home/admin/selfhosted-cloud/ce is 128 commits behind at 2815168 and still shows the pre-slice-0 classifier. Anyone re-verifying there will read the old case "nfs","cephfs": → Rebindable arm and reach a different conclusion. Use a fresh worktree off origin/main.
  7. node.default_store is not a dead COLUMN; it is a phantom. It appears in no ent schema, no ent/migrate/schema.go and no migration SQL, surviving only as a Go/wire orphan rendered by shc node info. docs/architecture/DRIFT-REGISTER.md:625 describes it as a column. It is also, notably, the exact shape a node-scoped store default would need, an axis this analysis priced nowhere, because the config scope ladder has Global/Tenant/Environment/Stack/Deployment and no NODE tier.
  8. Estate counts. Both estates run twelve deployments, not eight: the eight declared in corp.tf plus four master-tenant system apps (traefik, keycloak, postgres, openobserve) that no ops repo declares. Slice 2’s proof text (“the eight bitspur deployments”) will under-count. x-shc-scale appears in 48 distinct apps (63 occurrences), not “63 of them”; the x-shc-volumes = 0 of 73 figure is confirmed.

Node removal analysis: does force-remove back up, then delete?

Section titled “Node removal analysis: does force-remove back up, then delete?”

Analysis, not a decision. This document’s status line still reads “awaiting owner ruling”. Written 2026-08-02 from three evidence lanes and three judging lenses, re-verified first-hand against shc 17357d5fa and ce at the commit shc actually pins: 182e4b832582 (go.mod:8), not the local ce checkout, which is an ancestor. Every claim below carries a file:line. Neither production estate was contacted by this analysis; the handful of live facts are attributed as second-hand where they appear.

The owner’s question, put thinking aloud and explicitly uncertain:

“I would say if you force remove a node, it deletes the deployments too, but it backs them up first. Or not. I don’t know. I guess so. I don’t know.”

shc node forget gains the occupancy gate shc cluster rm already ships, and neither verb ever deletes a deployment row or takes a backup. --force keeps the meaning it already has everywhere else in SHC: “remove the node anyway and leave the work stranded”, and never acquires a second, destructive one.

What the operator types, end to end. Retiring a node that still answers: shc node drain <node>, then shc node forget <node> --yes. Removing a node that is gone: shc node forget <node> --yes, which refuses naming the stack count, then shc node forget <node> --yes --force, which removes the node and leaves its deployment rows stranded, after which shc move <stack> <stack> --node <peer> --presumed-dead recovers each one.

The one reason: force-remove does not delete deployments today, so the owner is not choosing between “delete” and “delete plus backup”; he is choosing whether to ADD deletion. rqliteNodeRemover.Remove (subsystems/rqlite_adapters.go:179-219, read end to end) evicts the raft member, writes UpdateState(nodeID, "removed"), may demote the engine to plain sqlite, and sets exactly two fields on its result: NodeID and Warnings. It touches no deployment row, stops no container, runs no docker node rm. Once the question is reframed that way, every consequence that makes deletion look necessary (a dangling DNS record, a stale Traefik backend, leaked quota, a stack frozen at running) turns out to be fixable by three predicates and a display change, none of which delete a row.

How the panel split, and how it resolved. All three lenses converged on four points: retain the rows, delete the false counters, fix the presumed-dead foreclosure first, and land the DNS predicate in the same change. They split on whether the verb should REFUSE when the node is reachable: the 3am-operator lens said no (a verb whose behaviour you must reason about at 3am is a bad verb), the other two said yes. Resolved by separating the two things “branch” was being asked to do: the destructive behaviour does not branch at all (nothing is ever destroyed, on any path), and the refusal branches on OCCUPANCY rather than on REACHABILITY. Occupancy is a raft-replicated DB fact that is equally available whether the node is alive, dead, or partitioned; reachability is a 2-second TCP dial that shares a failure domain with the partition it exists to detect. Branching on occupancy gives the honesty lens its refusal and the 3am lens its single predictable rule, and it costs no new classifier because X400017NODDEP already ships (shc/modules/cluster/admin_router.go:171-183).

This is also the shape the owner has already ruled for. docs/guide/operations/node-rotation.md:170-177 records it verbatim: “a destroy must refuse and name what is in the way; it must never auto-relocate. Relocation stays shc node drain”, together with the already-scoped mechanism on both sides: shc node forget gaining X400017NODDEP “with an explicit --force for the already-gone machine”, and the provider gaining shc_node.on_destroy ("refuse" by default, "force" per node). That second half matters, because “a refusal breaks tofu destroy” is the strongest objection to gating forget, and the answer to it is already designed. Backup-then-delete is auto-DESTRUCTION on a destroy path, which is further from that ruling than the auto-relocation it already forbids.

Does it branch on reachability? No, and that is deliberate

Section titled “Does it branch on reachability? No, and that is deliberate”

Nothing about what is destroyed depends on whether the node answers, because nothing is destroyed. What the verb SAYS depends on occupancy, which is knowable without a probe. Case by case:

situationwhat happens
Node dead, occupiedforget refuses with the stack count; --force removes the node and strands the rows. Recovery per stack is shc move … --presumed-dead, which requires the fix in the next section.
Node alive, occupiedIdentical refusal, and here the named remedy is real: shc node drain <node> genuinely relocates, then forget succeeds with no override at all. --force still works for the operator who means it, and strands rather than deletes.
Node partitioned but aliveIdentical, because the gate never asks. This is the case that decides the design: under a reachability branch, a partition fails both the durable unreachable stamp (roughly six seconds, a 2-second tick at modules/health/crons.go:75 against a default threshold of 3) and the fresh probe together, so the verb takes the dead branch against a node serving production traffic. Under an occupancy branch there is no misclassification to make.
Automatic 24-hour sweepThere is none; this is a correction, see below. shc cluster repair’s stale leg reads m.LastSeenAt, and last_seen_at exists in no migration and no ent schema. Only the unreachable leg can fire, and only when an operator runs the command.
tofu destroy of one nodeThe provider’s Delete runs shc node forget <name> --yes on the BOOTSTRAP over SSH (shc/terraform/provider/internal/provider/node_resource.go:1063) against a machine that is alive by construction, since the VM is destroyed strictly after (node_resource.go:1023-1026). It hits the refusal and must send on_destroy per the standing ruling. Worth flagging separately: runtime, nic and name carry RequiresReplace (node_resource.go:256-271), so editing runtime on a healthy production worker force-forgets a machine that is not being destroyed at all and then re-joins the same box.
tofu destroy of the whole estateEvery node is occupied and every removal needs the override, which is correct: a whole-estate teardown is exactly the case where the operator does mean it. This is also the one place where retaining rows is actively harmful, because the retained row is what keeps a dangling A record unreapable, which is why the DNS predicate below is not optional.

The “delete” half does not survive, and the reason is simpler than the code: SHC would be deleting the only description of the deployment it has left. Deleting a deployment row cascades away service, host_mapping, port_mapping, integration in both directions, and child deployments: six inbound foreign keys, all ON DELETE CASCADE (migrations/20260727022830_auto.sql:291,367,386,387,561,627). Exactly one inbound key is ON DELETE SET NULL: backup (:85). So the schema was deliberately built so the backup row outlives its deployment, but per lane 2’s live read, zero of the twenty-four production stacks across both estates has a backup, and bitspur’s shc backup ls returns literally “no backups”. host_mapping is the database’s only record of what hostnames a deployment published; its only other copy is the manifest sidecar on the disk being removed. Deleting the rows is the estate losing its own inventory at the exact moment it needs it to rebuild.

The “backs them up first” half does not survive either, and this is the part worth the owner’s attention, because it is not really about node removal at all. Two of the three reasons are addressed in the next section as a separate recommendation. The third is specific to this verb: a backup is only possible when the node answers, and the product has already ruled on the case where it does not. shc move --presumed-dead’s own header states it: “NO BACKUP LEG: the source is dead, so no pre-move backup can be taken” (modules/clone/move_presumed_dead.go:32-34). So “always back up first” cannot be an invariant; it degrades to “attempt, and refuse when you cannot”, and the population where it CAN succeed is precisely the population where SHC already has a better verb. For a live node the owner gets the backup-first relocation he is describing, done properly (backup, ship, bring up on a peer, then uninstall the source) and it is spelled shc node drain.

Put plainly: the instinct is right that a node should not be removed while its work is silently abandoned. It is wrong only in reaching for deletion and backup as the mechanism, when the mechanism is a refusal that names a verb that already exists.

Prerequisite: removal currently destroys the dead-node recovery verb

Section titled “Prerequisite: removal currently destroys the dead-node recovery verb”

This is a live bug, it is independent of any ruling, and it must land before the dead-node half of the recommendation works at all.

Remove writes cluster_state = "removed" (shc/subsystems/rqlite_adapters.go:199). PresumedDeadVerdict refuses unless the state is EXACTLY unreachable: if state != string(nodemod.ClusterStateUnreachable) at daemon/clone_dead_move_adapters.go:114-117. The lookup that feeds it, GetByName, carries no tombstone filter (modules/node/repository.go), unlike List, which does. So the tombstoned node resolves and the refusal fires:

presumed-dead refused, node {node} cluster_state is {state} not unreachable, if the node is up use a plain move modules/clone/errors.go:240-243

Net: shc node forget <dead node> permanently forecloses shc move <stack> <stack> --node <peer> --presumed-dead, the one path that rebinds surviving RWO volumes off a dead box, and it does so by telling the operator the node might be up, about a node the cluster has just tombstoned. The --presumed-dead flag’s own help text states the precondition that the removal destroys: “refused unless the node is registry-unreachable AND a fresh probe fails” (modules/clone/commands/main.go:270-271).

The fix is two lines: accept removed as satisfying GATE 1a. A node the cluster has tombstoned is strictly more dead than one merely stamped unreachable, and GATE 1b’s fresh probe (DeadNodeTCPProbe, 2-second budget, clone_dead_move_adapters.go:145-158) still guards the live case. Route touched: all three, since all three call Remove.

★ backup.repo defaults to the disk it would restore: a separate recommendation

Section titled “★ backup.repo defaults to the disk it would restore: a separate recommendation”

This is bigger than the node-removal question and should be ruled on independently of it. It is stated here as its own recommendation because it is the reason the owner’s backup instinct cannot be implemented, but it would remain true if node removal were never touched.

repo: null at core/config/default.yaml:188. The config layer does not resolve that default; the value is materialised at boot by the backup_mount subsystem, and resolveDefaultBackupRepo() reads the ENVIRONMENT, not the paths package:

stateHome := os.Getenv("SHC_STATE_HOME")
if stateHome == "" {
return "", ""
}
return filepath.Join(stateHome, "backups"), defaultBackupPassword()

subsystems/impl/backup_mount.go:416-425. And immediately below it:

func defaultBackupPassword() string {
if pw := os.Getenv("SHC_BACKUP_PASSWORD"); pw != "" {
return pw
}
return "shc-default-backup-password"
}

:431-437, in the AGPL public repo.

Per lanes 1 and 2’s live read (second-hand; not re-verified here), neither ops repo overrides backup.repo, SHC_STATE_HOME=/var/lib/shc comes from shc/shc.service:36, /var/lib/shc/backups is a real restic repo on both estates, and /var/lib/shc is the same filesystem as / on all six nodes. Two consequences, both independent of this ruling:

  1. The archives sit on the disk they exist to restore. On a single node that is a defensible default and the config comment says so. On a three-node estate it means the backup of every stack on compose01 is on compose01.
  2. They are encrypted with a password published in public source. Any archive in a default repo is readable by anyone who can read ce.

The recommendation: make an off-node backup.repo a first-class part of estate bring-up rather than an optional tuning step, and treat the hardcoded password as a defect in its own right. The remedy verb exists: shc config set backup.repo <value> (modules/config/commands/set.go:22). Until then, any feature that captures a backup as a safety net before a destructive operation is capturing it onto the thing being destroyed, which is the “success not earned” class: the operator is told there is a backup, and acts on it.

The exact strings and counters that must change

Section titled “The exact strings and counters that must change”

Every verb named below was checked against the live cobra tree, not from memory. shc cluster registers init, status, nodes, reconcile, logs, ping, cert, fabric (ce modules/cluster/commands/main.go:285-306) plus join, leave, repair, rm, mesh (shc modules/cluster/commands/commands_edition.go:34-40). There is no shc cluster forget and no shc cluster drain. shc node registers status, token, join, leave, forget, drain, undrain, swarm-tokens (modules/node/commands/fleet.go:27-34). shc move (ce modules/clone/commands/main.go:198), shc recover (ce modules/app/commands/stack_lifecycle.go:491) and shc config set (ce modules/config/commands/set.go:22) all exist.

Four never-assigned result fields, not one. git grep -nE "(DeploymentsEvicted|BackupsEvicted|SelfRemoval|MeshAddressReclaimed)" over shc returns only copy-outs and struct tags; the same grep at ce 182e4b832582 returns only declarations. Remove sets NodeID and Warnings and nothing else, so all four are structurally always zero.

whatwhereaction
SelfRemoval, MeshAddressReclaimed, DeploymentsEvicted, BackupsEvicted declaredce modules/cluster/service.go:2683-2689; ce modules/cluster/admin_router.go:77-80delete
copied out on cluster rmshc modules/cluster/admin_router.go:221-224delete
copied out on node forgetshc modules/cluster/lifecycle_router.go:364-365delete
rendered "%d deployments evicted" / "mesh IP … reclaimed" / "self"shc modules/cluster/commands/admin_fleet.go:279-287delete
rendered "forgot node %q (mesh ip %s reclaimed)"shc modules/node/commands/forget.go:86-89delete
wire typesshc pkg/apiclient/types_gen.go:995-996, 2270-2273; CLI result structs modules/cluster/commands/admin_fleet.go:48-50, modules/cluster/lifecycle_router.go:106-107, modules/node/commands/forget.go:19regenerate / delete

A correction on how this is described. Two of the three cases put to the panel framed this as an operator being told a false eviction count. That is not what ships: every render is guarded (if r.DeploymentsEvicted > 0 (admin_fleet.go:285), if r.MeshAddressReclaimed != "" (:282), if r.SelfRemoval (:279)), and the JSON fields are omitempty. No false count is ever printed. The shipped defect is one of omission, dead weight, and doc comments, which is why the fix is deletion rather than assignment. It still matters: a > 0 guard makes “no eviction happened” indistinguishable from “zero deployments”, so an operator infers a sweep that never runs.

Four doc comments that assert an evictor is called. None is. The only invocation of DeleteByNode in either repo is inside the /internal/unenroll executor at shc daemon/wiring/services_extra_formation.go:356, and it discards the count (if _, err :=), while /internal/unenroll itself has no client anywhere. DeleteNodeLocal has zero invocations at all.

  • ce modules/backup/ent_ops.go:208-212: “Called by the cluster force-removal path (node rm --force / node forget)”. False.
  • ce modules/app/deployment_repo.go:157-161: “Backs shc cluster rm <node> --force … The force path sweeps the deployments first”. False.
  • ce modules/cluster/service.go: the DeploymentEvictor and BackupEvictor interface docs, same claim.

A green test guarding the dead wire. shc subsystems/rqlite_cluster_seam_merge_test.go:159-162 asserts the evictors are wired, with a failure message describing a sweep that never happens. Deleting the claim means re-scoping that test, or the guard-is-green-while-broken class survives its own fix.

Operator-facing strings that name a verb that does not exist. shc cmd/shc-x9s/actions.go:398 renders shc cluster forget <node> -y, and cmd/shc-x9s/catalog.go:633 documents “evict the node from the cluster (shc cluster forget)”. The verb is shc node forget. The same block asserts at actions.go:393-396 that “the daemon’s forget route requires the confirmation flag … Sending yes:false would fail every time”, also false: handleClusterForget decodes ForgetRequest.Yes (lifecycle_router.go:96-99) and never reads it, so the confirmation the provider and the HUD send is decorative. If forget gains a gate, that confirmation has to become real server-side.

Two refusals whose text needs work. X400017NODDEP currently reads “node has stacks, rerun with force=true to override” (shc modules/cluster/admin_router.go:177-179); it names an API parameter, not anything an operator types; it should name shc node drain <node> and the CLI flag. And X409203CLSRMM, the sole-swarm-manager refusal, deliberately names no remedy, with an unusually careful comment explaining that SHC has none and that naming shc cluster promote was considered and rejected because it does not exist (ce modules/cluster/errors_extra.go:499-510). That reasoning is sound and should stand, with one narrow amendment: when the node’s state is already unreachable, a remedy DOES exist, because shc cluster repair runs only EnsureVoterRemovable and no swarm guard (shc modules/cluster/admin_router.go:85-101, read in full), so it reaps a dead sole manager that forget refuses. Naming it conditionally on that state keeps the discipline; naming it unconditionally would break it.

Schema: nothing. Migration: none. DDL mirror: untouched. This is the recommendation’s main advantage over a stored STRANDED state, and it is worth stating precisely, because the asymmetry is real and easy to miss. deployment.status carries a MATERIALIZED sixteen-value CHECK containing neither stranded nor paused (migrations/20260727022830_auto.sql:292), on the table targeted by six inbound foreign keys, so adding a value means a SQLite rebuild of the most FK-encumbered table in the schema, under append-only make migrate on two live estates, with the hand-written shc DDL mirror moving in lockstep. node.cluster_state has NO materialized CHECK (grep -c "cluster_state IN" over that migration returns 0) and already carries removed, written by the removal itself. Stranded-ness is therefore a JOIN predicate, not a column.

Route by route:

changecluster rmnode forgetcluster repair
Accept removed at presumed-dead GATE 1a✓✓✓
Delete the four counters and their renders✓✓n/a
Occupancy gate + --forceships alreadynewnot gated (correct: it reaps confirmed-dead members)
DNS keep-set predicate✓✓✓
shc ls stranded annotation✓✓✓
Fix the x9s wrong verbn/a✓n/a

Two predicates that are not optional. Retention is only defensible if the retained row stops lying, and both consequences are one WHERE clause each:

  1. DNS. ListIngressIntents is FROM deployment INNER JOIN host_mapping with no node-state predicate and no filter of any kind (modules/app/deployment_repo.go:328-350, SQL read verbatim), and it builds the DESIRED set for the orphan reaper, the Traefik file provider and the apps menu. A retained row keeps its hostname in DESIRED forever, so an A record aimed at a cloud IP the provider has already recycled is never released. Retention CAUSES that, so the predicate ships in the same change or the ruling is net-negative. The query already SELECTs deployment.node_id, so the fix is a join, not a subsystem.
  2. Status truth. repoRegistry.List calls repo.List(ctx, false) (modules/cluster/register.go:65-67) → ClusterStateNEQ("removed") (modules/node/repository.go:128-134), so the health poller can never reach a removed node and MarkContainersUnknown never fires for it. Rows on a force-removed HEALTHY node therefore read running forever. Cheapest correct form is display-only annotation from the node row.

And quota leaks either way, which is worth naming as a known cost rather than pretending retention is free: recordQuotaDelete runs only in Service.Uninstall, and the usage table has no FK to deployment, so retained rows stay counted against tenant quota, while deleting them via DeleteByNode (a bare Delete().Where(NodeID(...))) would leak the counters in the other direction.

Derived bound_to. Compatible, and this analysis strengthens the case for deriving rather than storing. Nothing here needs deployment.node_id to be correct, because nothing here computes a per-node deployment SET in order to destroy it; the occupancy gate only needs to know whether the count is non-zero, and X400017NODDEP already reports it that way. That matters because the set genuinely is not computable on swarm today: stampNode records the node that HANDLED THE INSTALL (modules/app/store.go:160-172), so any destructive verb keyed on it would be wrong in both directions. A recommendation that never keys destruction on that column is unaffected by the defect the binding model exists to fix, and can land before it.

STRANDED in slice 5. Affected in scope, not in direction. The state this analysis needs is exactly slice 5’s, and the recommendation is that it be built as a DERIVED predicate over node.cluster_state = 'removed' rather than a stored deployment.status value, for the CHECK-constraint asymmetry above. The counter-argument deserves recording, because it is the strongest one against this section: a derived predicate holds no state of its own, so a stranded row silently un-strands on a same-name, same-IP rejoin: resolveEnrollIdentity adopts the old node id when the joiner’s physical IP matches (shc modules/cluster/enroll_handler.go:751-776), consulting cluster_state in neither branch, and nothing verifies the disk was not reimaged in between. If that transition cannot be gated or at least made loud, the stored form earns its migration. Slice 5 should also note that its dead-node path does not function at all until the GATE 1a fix lands.

Q2: is the DATA-LOSS refusal --force-able? This analysis does not answer it, but it narrows what the answer has to cover, in two ways. First, --force on the removal verbs is currently AMNESIA, not discard: it bypasses an occupancy gate and evicts a raft member, and it destroys nothing. So ruling on Q2 is choosing what --force should mean, not amending what it means. Second, the recommendation above deliberately keeps the removal-path --force non-destructive precisely so that Q2’s ruling has somewhere clean to land: if the answer is “non---force-able with --discard-local as the only door”, that door already exists on shc move --presumed-dead (modules/clone/commands/main.go:272-273) and the removal path does not compete with it. If a discard on removal is ever wanted, the classifier DeleteNodeLocalBackupRows already contains (location is null, empty, under <stateHome>, or restic:<stateHome>/… (modules/backup/ent_ops.go:224-259)) can distinguish “every row here already has an off-node backup” from “this is the only copy”, which is the distinction Q2 turns on.

Q8: should cluster repair’s 24-hour auto-eviction survive? The premise needs correcting before the question can be answered; see below.

These contradict claims that are load-bearing elsewhere, including two in this document.

  1. This document’s Q8 says shc cluster repair “deletes deployment rows and node-local backup rows”. It deletes neither. Repair calls NodeRemover.Remove (shc modules/cluster/admin_router.go:101), which touches no deployment row and no backup row. DeleteByNode and DeleteNodeLocal have zero production callers in either repo. Q8’s engineering premise is false, though its product instinct (that a time-triggered destructive sweep is the wrong shape) is exactly why this analysis recommends against making it true.
  2. The 24-hour leg of repair is unreachable dead code. It reads m.LastSeenAt (admin_router.go:61), and last_seen_at appears in NO migration and in no ent schema: git grep -c last_seen_at over migrations/ at the pin returns nothing, and ent/schema/node.go has no such field. LastSeenAt survives only as a Go/wire projection whose own comment says no production constructor populates it (modules/cluster/types.go:90-96). The sole live selector is m.State == "unreachable". So Q8 should be re-asked as “should repair’s unreachable sweep survive”, and its real hazard is different from the one stated: the tidy path takes no confirmation at all (-y is read only inside the if force branch), is not dry-run by default, and fires on a roughly six-second signal.
  3. Repair is not unattended. No cron, no scheduled job, no CI caller; it is registered as a CLI verb and invoked by an operator. Descriptions of it as an automatic sweep, including in the brief this analysis answers, overstate it.
  4. The counters print nothing false. Every render is guarded and every JSON field is omitempty (admin_fleet.go:279-287). Framings of this as an operator being shown a fabricated eviction count (including in two of the three cases put to the panel) are wrong about the mechanism, though right that the fields must go.
  5. Line numbers drift between the local ce checkout and the pin. NodeRemovalResult’s fields are at modules/cluster/service.go:2683-2689 at 182e4b832582, not the 2474-2478 cited in two cases. Anyone verifying this section should use git show 182e4b832582:<path>; the local ce working tree is an ancestor and EnsureSwarmManagerRemovable in particular exists only at the pin.
  6. The backup.repo default does not flow through the config layer. One lane reported it as a config-resolved default; it is not. The config surface returns “key not found” on both estates while the repo plainly exists on disk, because the value is set at boot from os.Getenv by backup_mount (subsystems/impl/backup_mount.go:416-425). The conclusion is unaffected; the mechanism is observably different and matters to anyone trying to inspect or override it.

That a retained deployment row is worth enough to justify the drift it causes. The recommendation’s positive case is “keep the record, it is the only inventory the estate has left”, but the record may be insufficient for the thing it is being kept FOR, which is rebuilding the stack. The row carries app name and version, runtime, storage, providers, access and exposure; host_mapping carries the published hostnames. The install-time var_overrides and host specs do NOT live in either; they live in the manifest sidecar on the node’s disk (lane 2 traced a live example carrying an actual secret value, mode 0600). The backup row’s surviving manifest blob would carry them, but per that same lane no production stack on either estate has a backup. So if the retained row turns out to be diagnostics rather than a rebuild input, what this ruling preserves is worth less than it prices, and the DNS, quota and route drift it accepts gets relatively more expensive, which is the strongest form of the argument for deletion.

What would overturn it: take one production stack, delete nothing, and attempt a from-scratch reinstall using only the DB row plus host_mapping. If the result is not equivalent, the recommendation still holds on recoverability grounds (nothing was destroyed and the bytes remain on the disk while the VM lives), but its positive case collapses to “retention costs nothing”, and the honest headline becomes that the real asset is an off-node backup neither estate has, which no ruling about removal semantics substitutes for.

A second, cheaper check should run before implementation rather than after. The claim “retention cannot lose a byte” requires that nothing ELSE reacts destructively to a cluster_state = 'removed' stamp. A grep for that constant at the pin finds only the registry list filter and the certificate rotation path (which refuses to renew a decommissioned node’s identity (modules/cluster/certs_internal_router.go:208-222), a deliberate revocation and not a data action) plus the writers themselves. That is reassuring but not exhaustive: volume detach and fencing, fabric_pin and fabric_allocation (both FK-less, and the same-IP adoption branch reuses the old node id, which is exactly where a stale ordinal could collide), and the backup lease reaper were not each read in full. If any of them tears down a mount or reaps an archive on seeing the tombstone, the floor this recommendation stands on is not a floor.