Skip to content
SHC Docs

The node addressing model (proposal)

Status: open design. Slices 1 and 8 shipped on 2026-08-14 and slice 9 in part on 2026-08-15. Parts of slices 5 and 11 are built too (the occupied-node destroy refusal is in the Terraform provider), and the Nebula lighthouse work left in slice 5 went with the new line (SHC-1092). The rest is still a proposal. Ruling M10 (2026-08-14 models rulings) took the vocabulary rename and the address-family containers as ONE state upgrade, and they are on main: the management / internal / public containers replace host / private_ip / public_address on shc_node (state schema 1 → 2, with the v0 one-hop registered), modules/core and all fourteen substrates emit the typed nodes list, the four flat projections are REMOVED rather than aliased (M12), and every removed spelling is a plan-time tombstone rather than a silently-ignored attribute. The gate M10 set (a tofu plan showing zero changes on both live estates after the state upgrade) was run on 2026-08-14 against the real state of both hetzner/hel1/infra roots and printed No changes. on each. Slice 9’s demotion half is on main too (rulings M8 + M11): the vault cold start became a declared cluster capability with a replant verb that every removal route runs BEFORE it touches the holder, bootstrap stopped forcing replacement, and both plan-time identity refusals (the core module’s precondition and the provider’s refuseBootstrapIdentityMove) are deleted. What slice 9 still owes is listed in its own section: the module still models the seed as a SEPARATE RESOURCE, which is what keeps a live rotation destructive. Everything else below (slices 0, 2–7 and 10–12) is STILL A PROPOSAL, and slice 2 in particular carries ce-side preconditions that make it unsafe to attempt out of order. Read the slices for what each one still owes.

Written 2026-08-02, from four parallel audits of terraform/modules/*, terraform/provider, the ce daemon, and both live ops estates. Revised the same day with three constraints the owner added after those audits were briefed; see what the revision adds.

This document proposes one coherent way to describe a node’s addresses, used with the same words in terraform, the provider and the daemon. It supersedes the smaller cleanup already agreed with the owner (collapsing the four flat node_* output projections), because that cleanup is a consequence of the model rather than a change of its own.

  1. A shared vocabulary, with address family made explicit. “ip” is not a word this project gets to use: there is IPv4 and there is IPv6. Every role becomes a container holding both families rather than a family-blind scalar, and a glossary fixes one name per concept across all four layers. This revises the scalar shape the first draft proposed; see containers, not suffixes.
  2. Terraform stops caring which node ran shc init. The current surface conflates an infrastructure fact, a one-time cluster history and a runtime role, and the result is a wall: the init node cannot be removed. See genesis is not an identity.
  3. Node rotation and raft suffrage. Nodes must be addable and removable in any order, including the one that bootstrapped. See node lifecycle and raft suffrage and the runbook specification.

Points 2 and 3 are the same question wearing two hats: “who ran init” and “what is a node’s address” are both answered by refusing to let a create-time accident become a permanent identity.

For the fabric this model must serve, see network-fabric.md. For the bind/advertise primitive it consumes, see bind-advertise-split.md. For the cluster membership surfaces it touches, see cluster/overview.md.


The two layers do not share a vocabulary, and neither vocabulary names what it means.

Measured over terraform/ (excluding .terraform/ caches): public_ipv4 228, private_ip 161, nic_name 116, public_address 112, mesh_address 12 (all inside the provider’s shc_mesh plumbing, zero in modules/), internal_address 1 (a JSON tag on a read-only wire decode), fabric_cidr 0.

Measured over ce: public_address 125, internal_address 99, mesh_address 96, fabric_pin 15, bind_ip 10, private_ip 1.

The overlap is one word. That is the symptom. Three deeper facts are the disease:

1. Terraform’s names describe artifacts; the daemon’s describe roles. nodes[].public_ipv4 (modules/core/variables.tf:34) is not “the public IPv4”; it is the address terraform SSHes to. It has exactly two consumers: shc_node.host (modules/core/main.tf:260, :325) and local.cluster_ssh_hosts (:152-157). It is never sent to the daemon as an address. On modules/shc/proxmox/main.tf:140 and on modules/shc/openstack/main.tf:204 with floating IPs disabled, it holds an RFC1918 address. One field, two incompatible meanings, distinguished only by which substrate wrote it, and a name that forecloses IPv6.

2. The field that looks like an address is an assertion. nodes[].private_ip reaches shc_node.private_ip, whose only effect is a remote shell probe (provider/internal/provider/ssh_engine.go:584-602, consumed at node_resource.go:555 and :1401). No --private-ip flag exists; the value never reaches a daemon column. Its own schema doc says “changing it later does not re-run the gate” (node_resource.go:266). Both ops estates re-export it as node_private_ips believing it is the plane address.

3. The address that actually matters is unreachable from terraform. internal_address is the daemon’s only dialable node-row column (ce/modules/cluster/peer_seam.go:112-115 says so in those words). It carries the raft join-address list (modules/cluster/token_router.go:349), the WireGuard fabric peer endpoints (ce/subsystems/impl/fabric_reconcile.go:444-467), the nebula lighthouse physical host (modules/cluster/mesh_orchestrator_impl.go:181-186), the internal-API mTLS leaf’s only IP SAN (modules/cluster/enroll_handler.go:316-334), the LB backend set, the vault peer-unlock broadcast, and the LAN DNS rungs. Terraform cannot state it. Terraform can only name an interface (nic), from which the node derives an address by an IPv4-only, RFC1918-preferring heuristic (ce/core/nic/nic.go:98-140). And once written, the column can never be corrected: the upsert is internal_address = COALESCE(excluded.internal_address, node.internal_address) (ce/modules/node/repository.go:251) and the repository exposes UpdateNodeState, UpdateAvailability, UpdateMeshAddress, UpdatePublicAddress, UpdateCPUCores, UpdateSwarmIdentity, and no UpdateInternalAddress.

So the owner’s complaint is exact. The layers do not disagree by accident; they are describing different things with overlapping words, and the one thing that determines whether a cluster works at all has no name on the terraform side.


A node has a SET of addresses. Each element has a ROLE. A role is defined by WHO DIALS IT, not by what address class it holds.

Four roles. Nothing else is an address.

rolewho dials itportsprovenancenullable
managementterraform / the providerSSH (22) + derived daemon API (8000)declared by terraformno; a node with no management address is not manageable
internalpeer nodes4001 rqlite HTTP, 4002 raft, 4003 internal API, 4242/udp nebula underlay, 4243/udp fabric underlaydeclared by terraform, else derived on the node from nicno, but may be derived rather than stated
publicthe internet80/443 through system Traefik; the A/AAAA record targetdeclared by terraform, else detected from cloud metadatayes; null means “this node is not internet-reachable”
overlaycontainers, and the daemon’s own tunnelsnone fixed; it is an address inside a tunnelallocated by the cluster leaderyes; absent until the mesh/fabric is enabled

Three provenances, and the distinction is the whole design:

  • declared: terraform states it. Terraform is authoritative; drift is a diff.
  • derived: the node computes it locally from something terraform did state (an interface name). Terraform is not authoritative and cannot see the result today.
  • allocated: the raft leader hands it out from a cluster-scoped pool. Terraform must never state it, and must be able to read it.

And one rule that collapses most of the confusion:

No overlay has an endpoint of its own. Every SHC-owned overlay dials the internal address plus its own port.

Nebula’s underlay host is StripPort(internal_address) (mesh_orchestrator_impl.go:181-186). The fabric’s peer endpoint map is out[node.ID] = cluster.HostOnly(row.InternalAddress) (fabric_reconcile.go:444-467), whose own comment says it deliberately reuses the raft reachability story. So internal is not one of five addresses among equals; it is the load-bearing one, and every overlay is a layer on top of it, never an alternative to it.

Terraform declares three; the cluster reports four back.

# DECLARED: the per-node object modules take and emit
node = {
name = string
runtime = string
placement = object({ group = string, ordinal = number })
management_address = string # REQUIRED. terraform -> node.
internal_address = optional(string) # peer -> node. null => derive from `nic`.
nic = optional(string) # the interface `internal_address` is derived
# from when it is null. null => daemon auto-detect.
public_address = optional(string) # internet -> node. null => NOT internet-reachable.
}
# OBSERVED: added to the module `nodes` output, computed, never an input
observed_internal_address = string # what the daemon actually recorded
observed_public_address = string # declared-or-detected; makes metadata self-stamping visible
mesh_address = string # allocated; null when nebula is off
fabric_ip = string # allocated; null when the fabric is off

Four addresses in, one required. No ipv4 in any name. No "" sentinel; absence is null, always (the try(..., "") degradation in twelve substrate modules becomes try(..., null), and cluster_ssh_hosts filters on != null).

This is the actual deliverable. The rule for choosing names was: terraform stops inventing words and adopts the daemon’s, except for the one role the daemon does not have.

roleTHE NAMEterraform moduleproviderdaemon
managementmanagement_addressnodes[].management_address (was public_ipv4)shc_node.management_address (was host)not stored; the daemon never dials it
internalinternal_addressnodes[].internal_address (new; absorbs private_ip’s job)shc_node.internal_address (was private_ip)node.internal_address (unchanged)
publicpublic_addressnodes[].public_address (unchanged)shc_node.public_address (unchanged)node.public_address (unchanged)
overlay (nebula)mesh_addressnodes[].mesh_address (new, computed)shc_node.mesh_address (new, Computed)node.mesh_address (unchanged)
overlay (fabric)fabric_ipnodes[].fabric_ip (new, computed)shc_node.fabric_ip (new, Computed)fabric_allocation.address (unchanged)

Two of five role names are already right in the daemon and merely absent from terraform. One is right everywhere. One is read-only and only needs surfacing. Exactly one genuinely new word is introduced: management, because it names the one role the daemon has no opinion about: terraform’s own reach.

No daemon column is renamed. That is deliberate: a column rename is a migration on two live estates under the append-only migration doctrine, and it buys nothing the terraform-side rename does not already buy.

Superseded in part. The table above still fixes the ROLE names, which is the part that matters. The scalar SHAPE it implies (management_address = string) is revised by the next two sections: each role becomes a container holding one slot per address family.

private_ip and node_private_ips are the exact shape to eliminate, and not only because “private” is the wrong role word. “ip” is not a word this project gets to use. There is IPv4 and there is IPv6, they have different literal syntax, different DNS record types, different firewall CIDR arithmetic, and (today) different levels of support. A name that elides the difference cannot be checked, and a name that hard-codes one family has to be renamed the day the other arrives.

Two coherent answers exist. They are not equivalent.

(i) Family-suffixed scalars: management_ipv4, management_ipv6, internal_ipv4, internal_ipv6, public_ipv4, public_ipv6. Explicit, flat, terse in the single-stack case. But it doubles the field count immediately, it makes “this role, whichever family” unexpressible without a coalesce() at every reference, and (the deciding objection) it puts the family in the field name, so a role that gains a family gains a field, and every consumer’s for expression has to learn it.

(ii) A per-role container: management = { ipv4, ipv6, fqdn }. The ROLE is the container’s name and carries no family word at all; the FAMILY is the slot name and is always explicit. Adding a family is adding a slot to a type that already exists, not adding a top-level field to fourteen substrate modules and two estate roots.

Recommendation: (ii). Three reasons drawn from the topology matrix, not from taste:

  • A role’s cardinality is genuinely per-family, and the matrix already proves it. The public-only shape holds one address in three roles; the edge-NAT shape holds a public IPv4 and an RFC1918 IPv4 in two roles; a dual-stack Hetzner node today holds a public IPv4 and a public IPv6 (hetzner/main.tf:412 sets ipv6_enabled = true) and surfaces neither as a second field. Under (ii) that node stops lying the day someone fills in public.ipv6, with no rename anywhere.
  • The public role needs both families NOW, not later. ClassifyTarget already emits AAAA correctly (ce/modules/app/dns_target.go:129-137). The only reason SHC cannot publish an AAAA record for a node is that no terraform field can hold one. That is a real capability blocked by a naming decision, and (ii) unblocks it without touching the daemon column; public_address stores whichever member the DNS ladder asks for.
  • Families are not symmetric across roles, and the container makes that statable. A hostname is legal for management (the SSH engine resolves it; deriveEndpoint accepts it) and for public (the daemon column validates ip|hostname_rfc1123, ce/modules/cluster/types.go:142), and is not legal for internal (every producer builds that column from a literal address, and rowHostShared compares hosts textually). A flat scalar cannot say that; a container with a per-role slot table can.
# The address container. One shape, used for every role.
address = object({
ipv4 = optional(string)
ipv6 = optional(string)
fqdn = optional(string) # family-agnostic; resolves to either
})

Every role carries all three slots in its TYPE; that is what buys the “no second rename”. What differs per role is which slots carry a VALUE today. Three dispositions: honoured (works end to end), refused (the slot exists, a value in it fails the plan with a named code), meaningless (the slot is not part of that role’s grammar).

roleipv4ipv6fqdncardinality
managementhonouredrefused; splitHostPort truncates a bare v6 literal (node_resource.go:1632-1637)honouredat least one slot set
internalhonouredrefused; nic.Candidates drops every non-IPv4 (ce/core/nic/nic.go:98-101)meaningless; the column is compared textually by rowHostSharedat most one honoured
publichonoured (A)honoured (AAAA) the day a field exists to hold ithonoured (CNAME)all three may coexist
meshcomputedmeaninglessmeaninglessallocated from 10.42.0.0/16
fabriccomputedmeaningless; v6 supernets rejected (ce/modules/fabric/types.go:119-121)meaninglessallocated from 10.84.0.0/16

Refused means refused loudly. That is the whole point of writing the table down: today an IPv6 host silently becomes 2001:db8: and surfaces, much later, as a connection error nobody can trace back. A refusal that names the slot and the reason is strictly better than a capability that quietly is not one, and when IPv6 lands (slice 7), a refused cell becomes honoured and nothing renames.

The single-stack case stays readable, because a substrate module fills the container and the consumer never writes it by hand:

management = { ipv4 = hcloud_server.node.ipv4_address }
internal = { ipv4 = cidrhost(var.networking.subnet, offset) }
public = { ipv4 = hcloud_server.node.ipv4_address,
ipv6 = cidrhost(hcloud_server.node.ipv6_network, 1) }

One concept, one word, everywhere. This is the deliverable, not a summary of it. A concept that today has two names in two layers resolves to one; where a layer’s spelling still differs, the reason is stated and it is always the same reason (renaming a daemon column is a migration on two live estates, and buys nothing).

conceptTHE wordwhat it meansterraform moduleproviderdaemontoday’s names it replaces
cluster membership identitynamethe handle every verb takes; unique; immutablenodes[].nameshc_node.namenode.namenone
where terraform reaches the nodemanagementSSH target, and the base of the derived daemon API endpointnodes[].managementshc_node.managementnot storedpublic_ipv4, shc_node.host, node_public_ipv4s, cluster_ssh_hosts[].host
where peers reach the nodeinternalrqlite 4001, raft 4002, internal API 4003, nebula underlay 4242/udp, fabric underlay 4243/udp, the mTLS leaf’s IP SANnodes[].internalshc_node.internalnode.internal_address aprivate_ip, node_private_ips, bind_ip (partly)
where the internet reaches the nodepublicthe A/AAAA/CNAME target; 80/443 through system Traefiknodes[].publicshc_node.publicnode.public_address apublic_address (already right), topology.public_address b
the interface internal is derived fromnicin-guest interface NAME, never an addressnodes[].nicshc_node.nic--nic → sidecar nicnic_name (the module-scalar output, deleted)
where the daemon LISTENSbindnode-local only; re-resolved live every bootnot expressible cnot expressiblesidecar bind_ip, internal_api.bind, SHC_RQLITE_BIND_ADDRnone
nebula overlay addressmeshallocated by the leader; never an inputnodes[].mesh (computed)shc_node.mesh (Computed)node.mesh_address anone
WireGuard fabric addressfabricallocated by the fabric reconcile; never an inputnodes[].fabric (computed)shc_node.fabric (Computed)fabric_allocation.address afabric_pin d
the fabric POSTUREfabric (cluster scope)swarm | shc | none, plus its supernetbootstrap.fabric / .fabric_cidrshc_cluster.fabric / .fabric_cidrcluster_fabric.mode / .cidrnone
the node’s placement in the declared setplacement{group, ordinal}; ordering only, never identitynodes[].placementnot sentnot storedgroup + index as siblings
which node terraform PREFERS to run genesis onseedcreate-time only; inert on every later planseed_node_nameshc_node.seednot stored anywherebootstrap_node_name, shc_node.bootstrap
which node ACTUALLY ran genesis(no word, deliberately)not recorded by the daemon, and does not need to bedeleteddeletednothingbootstrap_node output
who leads raft RIGHT NOWleaderchanges without terraform; read, never declareddata.shc_cluster_status.leader_namesameStatus.LeaderIDcluster_ssh_hosts’ “most likely raft leader” claim
the address familyipv4 / ipv6never “ip”, never barecontainer slotscontainer slotswhichever member the column holdsip, ips, _ip, _ipv4 as suffixes

a The daemon column keeps its _address / _ip suffix. The suffix is not a second name; it marks “this layer holds exactly one member of the container”. Renaming the column is an append-only migration on two live estates and changes no behaviour.

b topology.public_address is a CLUSTER-scoped config row (dns.public_address) that collides by name with the NODE column. Under this glossary the cluster-scoped one becomes dns.entrypoint, because it is not a node’s public address; it is the cluster’s named entrypoint.

c bind stays node-local by design (bind-advertise-split). It is in the glossary so that nobody re-invents it under a third name.

d fabric_pin is not a node address and never was: it is one row per (deployment, service), the stable address a rendered compose service gets. It has no place in a node shape and is listed here only to stop the next reader from putting it there.

  • public_ipv4: wrong on two substrates, forecloses IPv6 by name.
  • private_ip: an assertion wearing an address’s name. Its job (the plane-NIC readiness gate) survives, attached to internal_address, where it is strictly better: today the gate checks a value that goes nowhere; tomorrow it checks the value the node will actually advertise.
  • group + index as sibling scalars: index is per-group and meaningless alone; (group, index) is the disambiguator, so it becomes one placement object. Neither is read by modules/core today.
  • The four flat projections node_names / node_public_ipv4s / node_private_ips / node_runtimes: one-line for expressions over nodes, coupled to it only by the prose “same order as nodes”, and already drifted (hostinger has a fifth, node_hostnames). Consumers write the for themselves against a self-describing object.
  • The "" sentinel in a required-string field.
  • local.all_nodes’ all_ prefix: the only other *nodes* local in fourteen modules is hostinger’s total_nodes. There is no some_nodes.

cluster_ssh_hosts survives and is not renamed away. It is not a projection: it re-orders bootstrap-first, drops address-less nodes, and folds in the SSH user (modules/core/main.tf:141-157). It is the fix for two live failures on 2026-07-27 and it stays.


Every shape is the same four roles with different membership. There are no special cases, only null sets.

shapemanagementinternalpublicoverlayderived cluster posture
public + private plane (hetzner hel1, both live estates)public IPv4RFC1918, derived from nic = enp7s0public IPv4allocated when oninternet = true
public only (hostinger, examples/hetzner-wan-nebula)public IPv4the same public IPv4the same public IPv4allocated when oninternet = true
LAN only (proxmox always; openstack with assign_floating_ips = false)LAN IPv4the same LAN IPv4, or a second NIC’snullallocated when oninternet = false
edge-NAT public (aws, gcp, azure, openstack-with-FIP)public IPv4 (1:1 NAT at the edge)RFC1918 VPC addresspublic IPv4allocated when oninternet = true
mixed / cross-substrateper nodemust be DECLARED on any NAT’d node: its edge address, not its NIC’sper nodeallocated when oninternet = true
+ nebulaunchangedunchanged; the overlay does not relocate the planeunchangedmesh_address allocated+ mesh
+ wireguard fabricunchangedunchangedunchangedfabric_ip allocated+ fabric
single nodeany of the abovesameoptionalnoneas above
declared but absent from statenullnullnullnullfiltered out of cluster_ssh_hosts

Three readings worth stating out loud:

  • Roles collapse; they do not vanish. On a public-only node all three declared roles hold the same string. That is not degeneracy; it is the model correctly reporting that one address serves three purposes. Today the same fact is expressed by putting an RFC1918 address in a field called public_ipv4 and hoping nobody reads the name.
  • public = null is the only signal that distinguishes LAN-only, and nothing in the type system or in any module inspects it today.
  • The mixed shape is the only one that requires a new capability, not just a new name: a NAT’d node must be able to declare an internal address it does not hold on any interface. Everything else in this table is expressible the day the rename lands.

”What is a node’s nebula IP if there’s only public?”

Section titled “”What is a node’s nebula IP if there’s only public?””

10.42.<a>.<b>: exactly what it would be on any other topology. The overlay address does not depend on the underlay at all.

It is allocated by the raft leader from a /16 pool at cluster mesh enable (shc/modules/nebula/service.go:240-272), persisted onto the node row after a successful install RPC (mesh_orchestrator_impl.go:404,433-442), and it is an allocation, not a derivation. Whether the node has one address or three, whether they are public or RFC1918, changes nothing about the nebula IP.

What the public-only case changes is not the overlay address but the overlay endpoint, and under this model that is not a separate concept at all, because of the rule above: the endpoint is internal_address + 4242/udp. On a public-only node, internal is the public address, so the endpoint is public and the mesh forms. On a NAT’d node, internal is RFC1918 and the mesh cannot form; the lighthouse publishes an unroutable physical host and every install RPC is dialled at an unroutable address.

That is the honest, uncomfortable answer, and it exposes the real bug the question was circling: the overlay cannot bootstrap the nodes it exists to bridge, because it derives its endpoints from the very reachability it is supposed to provide. examples/mixed-proxmox-hetzner works only because the Hetzner node happens to win the raft election. A lighthouse pin exists (shc cluster mesh lighthouse add, shc/modules/cluster/lighthouse_router.go, honoured at mesh_orchestrator_impl.go:198-212) and has no terraform surface whatsoever.

The model’s answer is therefore three-part, and the first two parts are where the value is:

  1. mesh_address / fabric_ip become computed, readable, per-node; the question becomes answerable by module.cluster.nodes[*].mesh_address instead of by reading EE source. Today mesh_address is reachable only through shc_mesh, a cluster singleton, keyed by name, and only after the mesh exists.
  2. internal_address becomes declarable, so a NAT’d node can state its edge address and the overlay’s endpoint becomes correct by construction. This is the surviving primitive of bind-advertise-split.md, consumed here.
  3. The lighthouse gets a terraform surface, so the mixed shape stops depending on an election.

”What if everything is only on LAN, with no internet at all?”

Section titled “”What if everything is only on LAN, with no internet at all?””

The set shrinks to {management, internal}, both usually holding the same address, and public = null becomes a first-class cluster posture rather than an accident.

Concretely, today, on proxmox:

management_address = 192.168.1.11 # what terraform SSHes to
internal_address = 192.168.1.11 # what peers dial (derived, nic="ens18")
public_address = null # nobody
mesh_address = 10.42.1.x # if the mesh is on

Everything on the cluster plane works. What breaks is everything downstream of public, and all three failures are currently silent:

  1. The cluster host record is never registered at all. RegisterSystemHostDNS hard-codes Exposure: "internet" (ce/modules/app/dns_lifecycle.go:305-311). The internet arm of the ladder skips the dns.public_address rung outright (dns_target.go:315-317, gated if !internet), then requires either a ready web-connection VIP or traefikPlacementPublicAddress, which returns "" for a private address (dns_target.go:465-469). It falls through to the fail-closed rung and warns X200980DNSDEF.
  2. The one knob terraform offers cannot reach it. topology.kind = "lan" + topology.public_address writes the dns.public_address config row (modules/core/main.tf:392-400), the exact key the internet arm refuses to read.
  3. ACME degrades silently. leEnabled = systemIngressHasPublicEndpoint(ctx) and only for internet intents (ce/modules/app/ingress_materialize.go:364-367); with no public endpoint the ingress quietly serves the internal SHC leaf. Terraform’s bootstrap.acme is accepted unconditionally and is then permanently inert, with no signal at either layer.

Per-deployment exposure = lan hosts do resolve, via rung 3 or the internal_address fallbacks. So a LAN-only estate today is not broken; it is quietly two-thirds configured, and the operator has no way to learn which third.

The model’s answer: derive a cluster-level internet posture from the node set (internet = anyof(nodes[*].public_address != null)) and make every consequence explicit rather than silent:

  • internet = false ⇒ RegisterSystemHostDNS registers the cluster host at LAN exposure, so it resolves;
  • internet = false + bootstrap.acme set ⇒ plan-time error, not an inert setting;
  • internet = false ⇒ dns.public_address is the primary rung, not a non-internet fallback;
  • internet = true on a node whose public_address is RFC1918 ⇒ plan-time error (today traefikPlacementPublicAddress silently returns "").

Whether LAN-only is a first-class topology with its own proof lane, or a degraded mode that merely stops lying, is an owner ruling; see Q1 below. The model supports either; only the amount of work differs.


The owner: “what if nodes are ripped out of the terraform and new nodes added and the node that did the original init fully gone. I would assume the terraform shouldn’t care about who ran init else it would paint us in a corner.”

He is right, and the corner is already built and load-bearing.

output "bootstrap_node" is described as “The shc init node: the FIRST node of the alphabetically-FIRST node_groups key” (terraform/modules/shc/hetzner/outputs.tf:6-13, identical in all fourteen substrates). That is a POSITIONAL value, recomputed on every plan, asserting a ONE-TIME HISTORICAL fact. Three estate roots feed bootstrap_node.public_ipv4 into dns.public_address, so the cluster’s published entrypoint silently follows the relabelling.

local.cluster_ssh_hosts inherits the same assumption, documented as “BOOTSTRAP FIRST (most likely raft leader)”, a claim about a runtime role that changes without terraform, encoded in a static list. The three roots are promanager-ops/hetzner/hel1/infra/main.tf:344, promanager-ops/hetzner/nbg1/infra/main.tf:136 and bitspur-ops/hetzner/hel1/infra/main.tf:324, each writing public_address: ${module.shc.bootstrap_node.public_ipv4}.

And the module and the provider each have a guard that turns the assumption into a wall:

  • modules/core/main.tf:324-329: a precondition on shc_node.bootstrap fails the PLAN when the pinned bootstrap_node_name leaves nodes. Its message says it outright: “SHC does not support rotating the bootstrap node through terraform.”
  • terraform/provider/internal/provider/bootstrap_identity_guard.go: refuseBootstrapIdentityMove fails the plan when the resource that is already the bootstrap is planned under a different name, or when bootstrap flips either way on an existing node.

Both guards are correct given the current model. They exist because the alternative was worse: name, nic and runtime all carry RequiresReplace, so a bootstrap that slid onto a joined node produced shc node forget <healthy voter> followed by shc init, which fails X400021ALRCLS after the eviction. Half-applied estate, no rollback.

So today: removing the init node from terraform is impossible, by construction, through both doors. That is not a bug in the guards. It is the guards correctly defending a model that made a create-time accident permanent.

Three things the current surface conflates

Section titled “Three things the current surface conflates”
what it iswho owns ithow it changesterraform’s job
infrastructure factthese nodes exist, at these addressesterraformon applydeclare it
cluster historyone node ran shc init, oncethe daemonnevernothing
runtime rolewho leads raft nowrafton its own, without terraformread it, never declare it

bootstrap_node, bootstrap_node_name and shc_node.bootstrap are all one field trying to be all three.

The daemon does not record cluster history, and does not need to

Section titled “The daemon does not record cluster history, and does not need to”

This is the finding that makes the whole thing tractable. Nothing in the daemon knows which node ran shc init. Verified:

  • ce/ent/schema/node.go:27-95: the node registry has no genesis column, no is_bootstrap, no founded_at.
  • ce/modules/node/commands/bootstrap_file.go:30-60: the node-local node_bootstrap.json sidecar carries runtime, node_name, node_id, nic, bind_ip, public_address, swarm identity, raft_tls, fabric, and no genesis flag. The init node’s sidecar is shape-identical to a joiner’s.
  • ce/modules/cluster/cluster_ca_store.go: the cluster CA’s public root lives in the REPLICATED certificates table (name="cluster") and its private key in the vault’s protected ca/ section. It does not live on the init node.
  • The nebula lighthouse defaults to the raft leader’s address (shc/modules/cluster/mesh_orchestrator_impl.go:181-186), not the init node’s.

The genesis asymmetry is entirely terraform’s own invention. One real artifact survives genesis on the init node, and it is a provisioning artifact rather than cluster history; see the vault exception.

Terraform should not export a bootstrap concept at all. The replacements:

# INPUT: create-time only, and inert on every subsequent plan.
seed_node_name = optional(string) # was bootstrap_node_name
# per node
seed = optional(bool) # was bootstrap; NOT RequiresReplace,
# NOT an identity, never read after Create
# OUTPUTS
nodes # the infrastructure fact, declared + observed
ssh_hosts # stable, arbitrary order; see below
# bootstrap_node : DELETED

Four rules make seed mean “one of them went first” instead of “this one is special forever”:

Rule S1: seed never forces replacement, and is never read after Create. Dropping a RequiresReplace plan modifier makes strictly fewer things replace, so it is free on both live estates. The provider’s ModifyPlan refusal of a bootstrap flip goes with it: under the new Create algorithm a flip is not destructive, so there is nothing to refuse.

Rule S2: Create is symmetric and adopt-first. Every shc_node, seed or not, runs the same decision at apply time:

probe the cluster over the many-host ssh transport
cluster exists AND lists this name healthy -> ADOPT (no cluster verb)
cluster exists -> JOIN
no cluster answers anywhere -> INIT

The JOIN path already does exactly this: node_join_recovery.go:65-115 catches X400021ALRCLS, cross-checks membership, and adopts a healthy member rather than wiping it. The INIT path does not: bootstrapSSH (node_resource.go:1384-1505) runs shc init unconditionally. Giving the init path the same recovery is the single change that unlocks rotation.

Rule S3: the create-time serializer is terraform’s DAG, not a node name. Something must go first on a from-empty apply or three nodes each genesis their own cluster. That is a concurrency detail and it stays one: the formation edge points at whichever node seed selected for this apply, and after Create nothing references it again. Note that SHC’s own row-create-is-authority guard (certificates.name UNIQUE, cluster_ca_store.go:24-31) prevents a forked CA within one cluster; it does not prevent two nodes forming two separate clusters. The DAG is the real serializer and the design should say so rather than imply the daemon defends it.

Rule S4: ssh_hosts stops claiming leader affinity. The provider already tries entries in order until one connects and is willing to serve the operation, skipping a follower’s not leader (ssh_hosts.go:526-607), and it remembers the winner in lastGoodSSHHost and reorders by it on the next call (orderBySSHCache:325-347). So terraform’s ordering is an optimisation on the first call of a session and irrelevant afterwards. It should emit a stable, arbitrary order (node name ascending), documented as arbitrary: stable so plans do not churn, arbitrary so nobody builds on it. The address-less filter and the folded-in SSH user stay: those are real semantics and they are the fix for two live failures on 2026-07-27.

And the one consumer that actually wanted something: three estate roots use bootstrap_node.public_ipv4 to fill dns.public_address. They never wanted the init node; they wanted a cluster entrypoint. That becomes entrypoint (see the glossary’s note b): derived from the node set, explicitly arbitrary and stable, and ultimately deleted when slice 4’s cluster posture lets the daemon’s own DNS ladder answer from node rows.

What a from-scratch apply does against an EXISTING live cluster

Section titled “What a from-scratch apply does against an EXISTING live cluster”

The harder half of the owner’s question, answered from the code. Four shapes: two refuse safely, and two fail silently, one of them by orphaning a live estate:

1. State intact, init node removed from nodes. → HARD WALL, no damage. Either seed_node_name/bootstrap_node_name is pinned and the module precondition fails the plan (modules/core/main.tf:324-329), or it is unpinned and defaults to nodes[0].name, which now names a different machine, and the provider’s refuseBootstrapIdentityMove fails the plan. Nothing applies. Nothing breaks. You simply cannot proceed. This is the corner, and it is the common case.

2. State lost, machines alive, init machine alive. → Apply FAILS, no damage. bootstrapSSH runs shc init, the daemon refuses X400021ALRCLS (ce/modules/cluster/errors.go:31-34), the apply errors out. Note the asymmetry: a join node in the same position converges by adopting; the seed node cannot, because the init path has no recoverAlreadyClustered.

3. ⚠ State lost, machines gone or rebuilt. → SECOND CLUSTER. Terraform reports SUCCESS. Terraform creates fresh VMs, shc init runs on a virgin machine and succeeds, the other fresh VMs join it, and the surviving original cluster (with all of its data, its vault, its raft log) is orphaned and invisible to the new state. Nothing errors. Nothing warns. This is the data-loss shape, and it is loud only because this document says so. It is exactly the shape the join path already defends against and the init path does not.

4. ⚠ Bootstrap VM replaced out-of-band, state intact. → Silent no-daemon node. hostRequiresReplaceIfJoin sets RequiresReplace = !bootstrap (node_resource.go:340-350), so a changed host on the seed node re-derives the endpoint in place and never re-runs anything. The resource happily points at a machine that has no shc package on it at all. Update (node_resource.go:765-781) rolls a package and repairs public_address; it does not check that a cluster exists there.

Rule S2 fixes 2, 3 and 4 in one stroke, because “probe first, then decide” turns all three into either an adopt or an honest refusal. It is therefore not a convenience; it is the fix for a data-loss shape that exists at HEAD.

The one real genesis asymmetry: the vault cold start

Section titled “The one real genesis asymmetry: the vault cold start”

There is exactly one thing the init node has that no other node does, and it is a provisioning artifact rather than cluster history:

writeVaultAutoUnlockDropIn (ssh_engine.go:918-960) persists the vault master password into a 0600 root-only systemd drop-in on the bootstrap node only. vault_autounlock_leader_only_test.go enforces the single call site structurally, and the posture is deliberate and correct: a joiner gets nothing, comes up locked, and pulls the unlock from an already-unlocked peer at runtime (ce/modules/vault/unlock_pull.go). The password never lands on a follower’s disk. Live-confirmed on bitspur/hel1 swarm01.

The consequence nobody has written down: the pull requires an already-unlocked peer. So if every node restarts at once (a power event, a rolling reboot that overlaps, a hypervisor maintenance window) and the init node is gone, no node can cold-start the vault and the whole control plane comes up X503906VLTLCK until an operator runs shc vault unlock -p by hand.

Rotating out the init node therefore silently removes the cluster’s only cold-start capability, and terraform has no way to move it.

The honest framing is that cold-start is a cluster capability (“at least one node can unlock without help”) and not a node identity. Two ways to express it, and this needs a ruling (Q9):

  • (a) Declare the capability. vault_password moves from shc_node.bootstrap to shc_cluster, and the module plants the drop-in on vault_cold_start_nodes nodes (default 1, chosen stably by name). Rotation then re-plants before it removes. This preserves “not every node” but changes a deliberately-pinned security posture, so it is the owner’s call, not an engineer’s.
  • (b) Remove the need. A quorum-derived unseal (the password reconstructible from a threshold of nodes) makes every node able to cold start and no node special. Strictly better, strictly larger, and out of scope for this proposal.

Doing neither is also a choice, but then the runbook must say, in capitals, that the init node is the cluster’s cold-start node and retiring it costs that capability.


The owner: “you have node a, b, and c, and when you terraform apply, it does the init on one of them, we don’t know which one. It should be opaque… and then the other two join, and then I should be able to join more and remove others, INCLUDING the one that did the init, whenever I want to. I should be able to rotate nodes in and out.”

Genesis opacity is answered above. This section is the other half: what the cluster does to itself when membership changes.

Node count and voter count are ALREADY decoupled

Section titled “Node count and voter count are ALREADY decoupled”

This is the answer to the owner’s worry that “we can never have an even number of voters, but I do want an even number of nodes”, and nothing in the docs says it, which is why the worry exists.

RaftTargetVoters = 3 (ce/modules/cluster/types.go:374). CalculateOptimalVoters(nodeCount, maxVoters) (ce/modules/cluster/voters.go:50-67) is an odd schedule: 0→0, 1→1, 2→1, 3–4→3, 5+→min(maxVoters, 5). ShouldJoinAsNonVoter(currentVoters, targetVoters) (voters.go:43-48) makes every node past the third join as a read-only non-voter, and the enroll handler applies it (shc/modules/cluster/enroll_handler.go:364-370), falling back to voter when the live count is unreadable.

So:

nodesvotersnon-votersquorumtolerates
11010
22 †020
33021
43121
53221
83521

† The 2-node cluster is the one place the shipped behaviour and the documented target disagree: CalculateOptimalVoters says 1, but suffrage is decided at JOIN time and the second node joins while currentVoters == 1 < 3, so it joins as a voter. voters.go:36-42 acknowledges this as a deliberate accepted transient: “the transient even set while GROWING toward the target… cannot be avoided without a runtime remove+rejoin rebalancer, which is out of scope by design”. Two voters means quorum 2 of 2 and zero fault tolerance, which is why the DR guidance is “three nodes or one”, and the model should say that where an operator will read it.

An even node count is fine. Four nodes is three voters plus one non-voter. This needs to be stated in the model and in the runbook, because the fear it dispels is real and the docs are silent.

Removing a VOTER: nothing promotes, and nothing can

Section titled “Removing a VOTER: nothing promotes, and nothing can”

This is the part nobody had verified. The answer is worse than “no promotion exists”: promotion is not implementable on the shipping engine.

  • Service.PromoteNode / DemoteNode exist (ce/modules/cluster/service.go:1621-1653) and are declared on the cluster-ops interface (interfaces.go:32-33).
  • They have zero callers: no HTTP route, no CLI verb, and no removal path invokes them. Grep across ce and shc finds only the declaration, the definition, and doc comments.
  • Both funnel into changeNodeRole, which ends at LeaderInfo.Assign(ctx, targetID, role).
  • rqliteLeaderInfo.Assign (shc/subsystems/rqlite_adapters.go:84-90) returns a typed error unconditionally: “rqlite decides voter/non-voter at join time; remove and rejoin to change a role.”

So the promote/demote verbs are dead code that would fail on any real cluster, and ShouldJoinAsNonVoter’s own comment names the gap: it is “a JOIN-TIME-ONLY policy… a runtime remove+rejoin rebalancer, which is out of scope by design”.

The concrete consequence. A 4-node cluster is 3 voters + 1 non-voter. Remove a voter:

  1. EnsureVoterRemovable (ce/modules/cluster/service.go:1715-1810) computes votersAfter = 2, requiredAfter = 2, and allows it.
  2. The eviction runs. The cluster is now 3 nodes: 2 voters, 1 non-voter.
  3. Quorum is 2 of 2. Fault tolerance is zero. Losing either voter wedges the cluster.
  4. The non-voter is never promoted. Nothing notices.
  5. The next voter removal is refused by the floor guard, correctly, but by then the cluster has already spent its fault tolerance silently.

Status already computes the signal: ExpectedVoters = CalculateOptimalVoters(healthy, RaftTargetVoters) and VoterDeficit = raftVoters < ExpectedVoters (service.go:1606-1607), surfaced by shc cluster status (commands/status.go:71-75) and (good news) already exposed to terraform as data.shc_cluster_status.voter_deficit / .expected_voters (cluster_status_data_source.go:36-38). The deficit is computed, rendered, and acted on by nobody.

What should happen. Since Assign cannot work, the only rebalance available is remove-and-rejoin of a NON-voter, which is cheap in principle (a raft membership change plus that node’s storage-engine restart; no app data moves) and always legal (removing a non-voter never trips the floor guard). Three levels, for the owner to pick between (Q10):

  • (a) Report only. The removal’s own output names the deficit and the fix. Zero new machinery; VoterDeficit stops being invisible.
  • (b) Refuse by default. A voter removal that would leave raftVoters < ExpectedVoters refuses unless the caller passes --rebalance or --accept-deficit. Symmetric with the deployment gate (below) and with the model’s overall “refuse, do not auto-move” posture.
  • (c) Rebalance. --rebalance performs remove+rejoin on one healthy non-voter BEFORE evicting the departing voter, so the voter set never transiently drops below target. This is the real fix and the only one that makes “rotate nodes in and out whenever I want” true.

Recommendation: (b) now, (c) as its own slice. (a) alone leaves the operator to discover a zero-fault-tolerance cluster from a status field.

Raft membership changes are themselves committed through raft. So a cluster that has lost quorum cannot vote to remove the dead voters that cost it quorum. Lose 2 of 3 voters and the survivor can do nothing: the eviction entry could never commit, and the guard refuses before it tries. Note which guard; the arithmetic here is subtle and worth stating, because the obvious reading is wrong. The configured count is unchanged by a crash (a dead peer still reports Role == voter), so votersAfter = 2 and requiredAfter = 2 and the first gate would pass. It is the SECOND, reachability-aware gate that refuses: reachableAfter = 1 < 2 → X409202CLSRMR (ce/modules/cluster/service.go:1798-1827). That gate exists precisely because the configured count cannot tell the safe 2→1 shrink from the wedge, and it FAILS CLOSED when no liveness signal is available at all.

shc cluster repair sweeps members that are unreachable or unseen for

24h and removes them while quorum still holds (shc/modules/cluster/admin_router.go:57-110); it runs the floor guard first and, on refusal, renders the escalation rather than a bare error.

shc cluster repair --force → Service.ForceLeader (ce/modules/cluster/service.go:2089-2196) is the destructive escape hatch: it stops the engine, rewrites raft membership on disk to a single-server configuration, and restarts. It refuses when a leader OTHER than itself is reachable (X403003LDRNLY), a genuine guard, and also the statement of its danger: it is a divergence machine. Against a live but partitioned peer that the local node merely cannot see, it produces two nodes each believing they are the cluster, and whichever one loses is rolled back to the winner’s log. --remove-other-nodes additionally evicts every other member, and previously-removed dead nodes cannot be “un-evicted”; their raft state is gone.

The rule the runbook must carry: force_leader is for a cluster that is already dead, never for one that is merely slow.

Refuse, do not auto-move, at the terraform boundary

Section titled “Refuse, do not auto-move, at the terraform boundary”

The owner: removing a node that still has app deployments “should fail… you would have to actually move those deployments… it shouldn’t just auto move things.”

There is a real tension here, and it resolves cleanly once the two verbs are separated.

shc node drain ALREADY relocates, and that is correct. (The verb is shc node drain, registered at CLI path node drain. The API method behind it is shc.cluster.drain, and the mismatch is a live trap: three separate pieces of work this week wrote shc cluster drain into operator-facing text by mirroring the method path instead of the command, and one reached the public generated error reference. There is no shc cluster drain.) shc/modules/cluster/drain.go flips availability to draining, lists running deployments, picks a target per row by runtime / lowest deployment count / availability=active, and asks a DeploymentMigrator to relocate (backup, cross-node ship, bring-up), then waits for the node’s deployment list to empty. Default budget 300s, and the file’s own header warns that “a many-stack drain may need a larger budget than the default”. Relocation is the verb the operator asked for. It should stay exactly as destructive as it is.

terraform destroy of an occupied node must REFUSE. And here is the defect: today it does not, and it does not even go through the route that would.

  • shc cluster rm HAS the gate: unless force=true, it lists deployments on the node and refuses with X400017NODDEP “node has stacks, rerun with force=true to override” (admin_router.go:171-183).
  • shc node forget has no deployment gate at all; its handler calls NodeRemover.Remove(ctx, m.ID, true) with force hardcoded (shc/modules/cluster/lifecycle_router.go:305), on the stated ground that forget must stay idempotent for reaping half-orphaned rows.
  • The provider’s Delete calls forget: buildNodeForgetArgv renders node forget <name> --yes (ssh_platform.go:586-597), and the HTTP fallback sends ForgetRequest{Yes: true} (node_resource.go:1042-1045).
  • rqliteNodeRemover.Remove (subsystems/rqlite_adapters.go:143-180) evicts the raft member and marks the row removed. It does not touch deployments at all, so the rows are not migrated and not deleted; they are left pointing at a node that no longer exists.

So terraform destroy -target=... on an occupied node evicts it from raft, orphans its deployment rows, and reports success. That is precisely what the owner says must not happen.

Is the refusal even reachable? Yes; the ordering already works. shc_node reads its host (and its ssh) from the server resource, so terraform’s dependency graph destroys the shc_node before the VM, and Delete’s own comment relies on this: it releases managed DNS and leaves raft “while the daemon on this node is still alive, which it is only until this resource returns” (node_resource.go:988-1017). The daemon is up and reachable at destroy time. The provider simply asks the wrong verb.

The proposal:

  1. The provider’s Delete stops force-forgetting. It calls the gated removal, and on X400017NODDEP fails the destroy with a diagnostic that names the deployments; the operator then runs shc node drain <node> explicitly, or passes an explicit escape.
  2. The escape is a per-resource on_destroy = "refuse" | "force" (default refuse), not a provider-wide flag, so a deliberate teardown of a whole estate can still say so once per node.
  3. shc node forget keeps force=true and keeps its reaping job; it is the verb for a node that is already gone, and gating it would break the one thing it exists for.
  4. drain gets no terraform surface. Relocation is an operator decision with a wall-clock budget and a real failure mode; expressing it as a side effect of terraform destroy is how “it shouldn’t just auto move things” gets violated by accident. Today the provider has no drain call at all; that is correct and should stay.

Graceful removal cannot run on a VM that no longer exists. The force path, and when each step is right:

situationverbwhy
node is up, has stacksshc node drain <node> then destroyrelocation is explicit; destroy then finds it empty
node is up, emptyterraform destroy (or shc node leave on it)gated removal succeeds
node is DOWN, cluster has quorumshc node forget <node>idempotent reap; the floor guard still applies and refuses if the reachable survivors could not reach their own quorum
several nodes DOWN, quorum holdsshc cluster repairsweeps unreachable / stale >24h, floor-guarded
quorum LOST, cluster is deadshc cluster repair --forcerewrites membership to single-server; divergence risk; see above
terraform state says a node exists and nothing doesterraform state rm then re-applyno cluster verb is issued either way

The row that does not exist today and should: “node is up, has stacks, and terraform is destroying it” currently lands in row 3’s behaviour without row 3’s justification.


docs/guide/operations/ has cluster-setup, upgrades, disaster-recovery, vault, storage and observability. It has nothing on rotating nodes in and out. Removal appears twice: cluster-setup.md:160-178 and disaster-recovery.md:161-211, both framed as failure recovery, neither as routine capacity management, and neither mentions the init node, the voter deficit, or terraform at all.

Writing it is not a deliverable of this proposal. Specifying it is. docs/guide/operations/node-rotation.md must contain:

  1. The three facts up front. Node count ≠ voter count (with the table above). An even node count is fine. Any node can be removed, including the one that ran shc init, once this model lands, and not before.
  2. Add a node. What suffrage it gets and why (ShouldJoinAsNonVoter), and that it will be a non-voter past the third, which is normal, not a degraded state.
  3. Remove a node, the happy path. Drain explicitly, verify the deployment list is empty, then destroy. Name the 300s default budget and how to raise it for a many-stack node.
  4. Remove a VOTER. Read shc cluster status before and after. What voter_deficit means. The remove-and-rejoin rebalance of a non-voter, step by step, until (c) above lands and automates it.
  5. Remove the node that ran shc init. The cold-start consequence (vault), what must be re-planted first, and how to verify it: the single most dangerous rotation and today the only one that is documented as impossible.
  6. The already-gone machine. The table above, verbatim.
  7. Quorum loss. The deadlock, why force_leader refuses when a leader is reachable, and the divergence risk when it does not.
  8. Terraform-driven rotation end to end. Edit nodes, plan, read the plan (what a legitimate rotation looks like versus what a destructive one looks like), apply, verify with shc cluster status and data.shc_cluster_status.
  9. What will refuse, and what to do about it. Every guard by code: X400017NODDEP, X409201CLSRMQ, X409202CLSRMR, X403003LDRNLY, X400021ALRCLS.

Proof obligation for the runbook itself: a lane that rotates every node of a three-node cluster out and back in, one at a time, including the seed, with the cluster serving throughout.


Every attribute whose change forces node replacement

Section titled “Every attribute whose change forces node replacement”

Replacement of an shc_node means shc node forget <name> followed by shc node join, evicting a live raft voter. bootstrap_identity_guard.go:3-33 documents the half-applied-estate incident this produced. The full list, at HEAD:

attributefile:linereplace triggerin the model?
shc_node.bootstrapnode_resource.go:165-174RequiresReplace (Bool)untouched
shc_node.namenode_resource.go:237-241RequiresReplaceuntouched
shc_node.nicnode_resource.go:242-246RequiresReplacename and meaning both preserved; see rule A
shc_node.runtimenode_resource.go:247-251RequiresReplaceuntouched
shc_node.hostnode_resource.go:176-191RequiresReplaceIf join node (hostRequiresReplaceIfJoin, :334-345); bootstrap re-derives in placerenamed to management_address; value identical, see rule B
shc_node.public_addressnode_resource.go:252-258none; in-place SSH repair (:805-838, :904-935)untouched
shc_node.private_ipnode_resource.go:259-267none, and inert after createrenamed to internal_address, must stay non-replacing; see rule C
shc_node.vault_passwordnode_resource.go:~280change REFUSED in placeuntouched

Can the model be reached without recreating nodes on the two production estates?

Section titled “Can the model be reached without recreating nodes on the two production estates?”

Yes, for the rename. No, for the internal_address declaration, unless the daemon gains a repair path first.

Three rules make the difference, and each is a hard gate on a slice:

Rule A: nic keeps its name AND its meaning. It is RequiresReplace and it is the only handle on internal_address today. In the model it means “the interface internal_address is derived from when internal_address is null”. Existing estates set nic and leave internal_address null, so behaviour is byte-identical, the value never changes, and there is no diff. Re-semanticing a RequiresReplace attribute is the single most expensive mistake available here: it replaces every node on promanager/hel1 (3x cx53) and bitspur/hel1 (3x cx53) simultaneously.

Rule B: every provider attribute rename ships a schema Version bump plus an UpgradeState that carries the old value across. The precedent mechanism exists and works: state_upgrade_ssh_host.go:82-130 derives the legacy model from the live schema (~30 lines, not eleven hand-copied schemas), and shc_node is already at Version: 1 for exactly this reason. host → management_address and private_ip → internal_address are both value-preserving renames; once the state is upgraded the plan is a no-op and RequiresReplaceIf sees no change. Skipping this is the 2026-07-27 failure repeated: both estates frozen at plan, and the file that documents it says the lesson in capitals: “CONFIG MIGRATION IS NOT STATE MIGRATION.”

Rule C: internal_address must never be RequiresReplace. If it is, the first estate that declares one replaces every node. Making it in-place-repairable requires a daemon change first, because today there is no path to correct the column at all:

  • add a targeted UpdateInternalAddress writer (the repository has five siblings and not this one);
  • add --internal-address to shc init and shc node join, threaded to the enroll physical_ip, with nic.Detect as the fallback when unset (today only the join path has an undocumented SHC_ADVERTISE_ADDR back door, and shc init has nothing);
  • re-issue the internal-API mTLS leaf on change; the leaf’s only IP SAN is host(internal_address) (enroll_handler.go:316-334), so without this a corrected address fails TLS verification and the repair deadlocks;
  • then the provider gets the same in-place SSH repair shape public_address already has.

Both ops repos pin a versioned module tarball (.../terraform-modules/0.0.14/shc-hetzner-0.0.14.tgz on promanager/hel1; 0.0.7 on the older lanes). So the module and the estate root must move in the same commit: bump the pin and rewrite the root’s nodes/outputs together, or the root references outputs the pinned tarball does not have.

Confirmed live consumers that the rename touches: node_public_ipv4s is re-exported by six estate roots, and is index-pinned at promanager-ops/openstack/us-west-or/infra/main.tf:166 and promanager-ops/azure/eastus/infra/main.tf:197, where module.shc.node_public_ipv4s[0] feeds dns.public_address, positional, and on the openstack lane it would write an RFC1918 address if floating IPs were ever disabled. Those two become module.shc.bootstrap_node.public_address in the same pass.

Neither estate pins bootstrap_node_name in its own root, so fixing the bootstrap_node output to resolve through the pin (rather than local.all_nodes[0]) changes no live value today; it is a free correctness fix, and it should land before anyone starts pinning.

One thing object-typed variables will not do for you: OpenTofu treats an unknown attribute on an object-typed variable as a warning, not an error. promanager-ops/hetzner/hel1/infra/main.tf:169 has been carrying firewall = { inter_node_open = false } (an attribute deleted in ab292948f and absent from the pinned 0.0.14 module) silently, for weeks, with tofu validate reporting Success!. Every renamed field therefore needs a same-named optional(...) tombstone whose validation block errors with the new spelling, deleted one release later. This is not backcompat; it is the house pattern already used for inter_node = "auto" (modules/shc/hetzner/variables.tf:73-77).

  • Rename only (slices 0–1, 3): zero node replacements, both estates, provided rules A and B hold.
  • internal_address as a declaration (slice 2): zero replacements if rule C holds, but rule C is gated on a ce change, so slice 2 cannot land before it. If it is attempted without rule C, the price is six raft evictions across two production clusters, serialized, each one the half-applied-estate class.
  • Everything downstream (slices 4–6): additive, computed, or cluster-scoped. No replacements.

Address-family containers (slice 8): zero replacements, one state upgrade. host → management, private_ip → internal, public_address → public are all scalar→object shape changes. The provider’s UpgradeState mechanism handles a shape change exactly as it handles a rename; state_upgrade_ssh_host.go:82-130 already migrated a scalar ssh.host into a hosts LIST, which is the same class of move and is the working precedent. RequiresReplaceIf on management compares the resolved value, so a migrated state plans as no change.

De-identifying genesis (slice 9): zero replacements, and it REMOVES replacement triggers. The changes are: drop RequiresReplace from bootstrap (strictly fewer replacements), rename bootstrap → seed (value-preserving, one UpgradeState), delete refuseBootstrapIdentityMove and the module precondition (both are plan refusals; deleting them cannot damage state), delete the bootstrap_node output (a same-commit fix in three estate roots), and add the adopt-first probe to bootstrapSSH (Create-path behaviour only; on both live estates the seed node already exists in state, so Create never runs again).

name, nic and runtime keep both their names and their RequiresReplace, the same Rule A discipline the addressing rename follows, for the same reason.

Net: the whole revision is reachable on promanager/hel1 and bitspur/hel1 with ZERO node replacements, and it reduces the number of attributes that can ever force one. The costs that remain are the two things that were always going to be expensive and are both gated on ce changes: internal as a declaration (rule C) and the vault cold-start capability (Q9).

Sequencing against the estates. The ops repos pin a versioned module tarball, so every slice that changes an output moves the module and the estate root in the same commit. Slice 9 additionally removes an output three roots read (bootstrap_node), so it is the one slice that cannot be landed module-first and rolled later.


Every design forecloses something. Named:

IPv6. The model’s names are family-agnostic (that is the point of killing public_ipv4), but the model does not deliver IPv6, and nobody should read it as doing so. The foreclosure is structural, in at least six independent places: nic.Candidates skips every address where ip.To4() == nil (ce/core/nic/nic.go:98-101); a v6-only host takes the warning branch at ce/modules/cluster/service.go:769-771, leaves bindIP == "", and writes a NULL internal_address; genesis builds host:port with fmt.Sprintf rather than net.JoinHostPort (service.go:850); three of five host-splitting helpers truncate a bare v6 literal at the last colon (cluster.StripPort, cluster.StripPortIfPresent, cluster.hostOnly), including the one the fabric endpoint map uses; the fabric CIDR validator rejects a v6 supernet (ce/modules/fabric/types.go:119-121); and the provider’s own splitHostPort silently turns 2001:db8::10 into host 2001:db8: port 10, verified by executing the helper. Substrate modules meanwhile provision IPv6 end to end (hetzner/main.tf:412, vultr/main.tf:512, ::/0 firewall rules on aws and linode) and surface it nowhere. This model makes IPv6 possible later without a second rename. It does not make IPv6 work.

Multi-NIC. One internal address per node. The daemon’s registry assumes it; rowHostShared (ce/modules/cluster/service.go:1874-1892) treats two rows sharing HostOnly(internal_address) as the address-aliased-zombie shape and refuses voter removal. A node with a separate storage plane and control plane is not expressible, nor is a per-role NIC. Only hetzner even has a per-group nic_name today (hetzner/main.tf:119), and every substrate emits a single scalar nic_name output that is a lie for exactly that one.

Multiple nodes behind one NAT. Forbidden by the same invariant. Three home nodes behind one edge IP cannot join a WAN cluster even with a declared internal_address, because they would share a host, and the model has no per-node port field to separate them. Accepting this as a permanent limit is an owner call (Q4).

A node whose address changes under it. The model makes drift visible (observed_internal_address) but not self-healing. Raft and the internal API both re-resolve --nic live on every boot (rqlite_engine.go:1255-1270, internal_api.go:195-215), while node_self_register.go:177 re-stamps the row from the frozen sidecar bind_ip. After a DHCP renew the node binds the new address and advertises the old one to every peer. Rule C fixes the terraform-declared case; it does not fix the case where reality moves and terraform does not know.

The management role has no daemon representation, so there is no drift detection on it and shc node update cannot repair it. If the management address goes stale, terraform is locked out. That is no worse than today, but the model does not fix it either.

The overlay still cannot bootstrap itself. Both overlays derive their endpoints from internal. An overlay can therefore never be the thing that makes an unreachable node reachable. Declaring internal is the escape hatch; the overlay is not.

Four address detectors, not one. core/nic/nic.go claims in its package comment to be “the single source of truth”, and is not: modules/node/commands.AdvertiseAddress (local_token.go:58-76), modules/app.AdvertiseAddress (dns_lifecycle.go:513-541) and modules/network.DefaultNICResolver (state.go:95-129) each have their own precedence, their own env var (SHC_ADVERTISE_ADDR, SHC_NIC), and two of them fall back to 127.0.0.1. The model does not collapse them; that is adjacent cleanup, and it is called out as slice 2’s companion because the loopback fallback on the join path produces a durable, unrepairable, loopback-SAN node.

Runtime voter rebalancing, on the shipping engine. Assign is unimplementable on rqlite (subsystems/rqlite_adapters.go:84-90), so “promote a non-voter” can only ever be remove-and-rejoin. That is a real operation with a real cost (the rejoining node’s storage engine restarts), and it can only run while quorum holds. A cluster that has already spent its fault tolerance cannot rebalance its way out.

A voter set larger than five. CalculateOptimalVoters caps at min(maxVoters, 5) and RaftTargetVoters is 3. A twenty-node cluster is three voters and seventeen non-voters. That is the correct raft answer and it is also a hard ceiling on control-plane fault tolerance that no amount of nodes buys past.

Two clusters formed by two concurrent applies. Nothing in the daemon prevents two nodes each running shc init and forming separate clusters. The row-create-is-authority guard on the cluster CA prevents a forked CA within one cluster; it says nothing about two clusters. Terraform’s DAG is the only serializer, so a hand-rolled root that wires no dependency edge between its nodes can still produce two clusters. The model makes this visible (seed is create-time-only and the edge is explicit) but does not defend against it.

Rotation while the cluster is degraded. Every guard in this model depends on being able to READ live raft membership: EnsureVoterRemovable queries LeaderInfo.Members, the deployment gate queries the deployment list, and the adopt-first probe queries membership over SSH. On a cluster that has lost quorum all of them fail, and rotation stops being available exactly when it is most wanted. force_leader is the answer and it is a different kind of operation with a different kind of risk.


Each slice is independently landable and independently provable. Proof obligations are explicit; a slice is not done until its proofs run.

Slice 0: the live bugs, no contract change

Section titled “Slice 0: the live bugs, no contract change”

Four fixes that are correctness today and that the model would otherwise inherit:

  1. bootstrap_node resolves through local.bootstrap_node_name instead of local.all_nodes[0], in all fourteen substrate modules, and gains public_address, which is what its three production consumers actually want. (hostinger’s docstring already claims it honours the pin; hostinger/outputs.tf:6-13 does not.)
  2. Each substrate’s cluster_ssh_hosts re-orders bootstrap-first the way modules/core’s does (core/main.tf:141-157), or its docstring stops claiming “BOOTSTRAP FIRST”. Both currently say it; only core does it.
  3. The plane-NIC readiness gate stops disagreeing with itself: it validates with net.ParseIP (accepts v6) and probes with ip -4 addr show (ssh_engine.go:597, :621), so a v6 value hangs the full five-minute timeout and then blames the image. And grep -qF is a substring match, so 10.0.0.1 is satisfied by an interface holding 10.0.0.10.
  4. sshNodeRow (ssh_platform.go:512-522) gains the fields apiclient.NodeListRow has and it lacks: public_address, swarm_node_id, swarm_role, cpu_cores. The comment at node_resource.go:1203 asserts the two wire shapes are field-identical and it is false. Harmless today because both are discarded; it becomes a live bug the instant slice 3 surfaces anything, because both live estates use the SSH transport.

Proof: unit tests for the ordering fix with a pinned non-first bootstrap; a substring-collision test for the gate; a reflective test asserting sshNodeRow and NodeListRow carry the same fields (the ratchet shape already used for the enroll request bodies).

LANDED 2026-08-14 (ruling M10), together with slice 8.

The rename, value-preserving, across modules/core + all fourteen substrates + the provider + both ops repo roots. nodes gets an explicit type (it is untyped in the substrate outputs today, which is what let fourteen modules drift). Tombstone validations on every removed field. The four flat projections deleted.

Proof: tofu plan against both estates’ state after the state upgrade shows no changes, not “no replacements”, no changes at all. That is the whole gate. Plus a fixture proving the tombstone converts OpenTofu’s “Object attribute is ignored” warning into a hard plan failure (the inter_node_open class).

Slice 2: internal_address becomes a declaration

Section titled “Slice 2: internal_address becomes a declaration”

ce side: --internal-address on init and join; UpdateInternalAddress in the node repository; internal-API leaf re-issue on change; collapse the four detectors onto core/nic; delete the 127.0.0.1 fallbacks on any path that produces a durable address. Provider side: internal_address as Optional, non-replacing, with the in-place SSH repair shape public_address already has.

Proof: a two-node cluster where the joiner declares an internal_address it does not hold on any interface, joins, replicates, and rotates its cert; then changes it and converges without a replace. A negative test that a join with no usable IPv4 and no declared address fails rather than silently recording 127.0.0.1:4003.

Slice 2 also gives X403092ENRSIP something real to validate; it is defined at shc/modules/cluster/errors_extra.go:275-288, tabled at :520, and has zero call sites. Wire it or delete it.

mesh_address, fabric_ip, observed_internal_address, observed_public_address as Computed on shc_node, propagated into the module nodes output. Zero replacement cost, zero state migration; Computed attributes never force replacement and never strand state that lacks them. Depends on slice 0’s fix 4.

Proof: a live estate read where an out-of-band shc node update --public-address appears as drift (today it is invisible forever), and where mesh_address is readable per node without instantiating shc_mesh.

Slice 4: the cluster posture, and the loud failures

Section titled “Slice 4: the cluster posture, and the loud failures”

Derive internet from the node set. Make ACME-on-a-LAN-only-cluster a plan-time error, make RegisterSystemHostDNS follow the posture instead of hard-coding internet exposure, and make an RFC1918 public_address a plan-time error instead of a silent "".

Proof: a from-empty LAN-only cluster (proxmox lane) where the cluster host record resolves (the thing that does not happen today) and where bootstrap.acme refuses at plan with a named error.

Correction, 2026-08-02. The first half of this slice landed while this document was being written (39055cde7, “the WireGuard fabric becomes lane-driveable”). shc_cluster.fabric / .fabric_cidr exist, bootstrap.fabric flows through modules/core onto the seed node’s shc init, and 4243/udp is in the substrate plane port sets gated on local.fabric_shc (hetzner/main.tf:82, :284). The audits’ “zero provider surface” reading was true at the commit they read and is no longer true. What follows is what REMAINS.

The lighthouse still has no terraform surface at all; grep -rn lighthouse terraform/provider/internal/provider/*.go terraform/modules/core/*.tf returns nothing. shc cluster mesh lighthouse add exists (shc/modules/cluster/lighthouse_router.go, honoured at mesh_orchestrator_impl.go:198-212) and is the only place in the whole daemon where an operator can pin a public address for a control-plane function. Without it the nebula mixed shape depends on which node wins the raft election, and under this model’s genesis opacity that is now explicitly a thing terraform must not assume.

Proof: a two-node fabric cluster stood up entirely from terraform, cross-node connection working, on a public plane (so the firewall rule is actually exercised), plus a mixed-substrate mesh whose lighthouse is pinned from terraform and survives a leader change.

Proven by plan: topology.kind = "lan" with no dns block emits zero resources; "wan" emits exactly one shc_mesh; topology.public_address is accepted on all three kinds and honoured on one, silently. It is not a network model; it is a three-way switch over which platform resources to create. Zero production estates use it, and their stated reason is recorded in the config: it models one all-or-nothing connection driving LB

  • DNS + LE together, and the estates want DNS without LB (promanager-ops/hetzner/hel1/infra/main.tf:254-262).

Split into independently-nullable dns / mesh / lb blocks, or delete it and bless the root-level-resources pattern both estates already use. Recommendation: split; deleting discards the shc_stack_ports.traefik wiring tests/hetzner depends on, and splitting is what would bring both production estates back onto the module.

Proof: both estates’ hand-composed root resources replaced by module blocks, tofu plan clean.

Scoped separately and gated on ruling Q3. Nothing in slices 0–6 forecloses it; nothing in them delivers it.

LANDED 2026-08-14 (ruling M10), together with slice 1, as the ruling required, one state upgrade rather than two.

Every role becomes {ipv4, ipv6, fqdn} per the slot table. One UpgradeState per changed provider attribute, following state_upgrade_ssh_host.go:82-130, which already migrated a scalar into a list and is the working precedent for a shape change. Slots that are accepted-but-not-honoured become named plan-time errors, replacing today’s silent truncation.

Lands with slice 1, not after it: renaming and reshaping in one state upgrade is one migration; doing them separately is two, and the second one has to migrate the first one’s output.

Proof: tofu plan against both estates’ state after the upgrade shows no changes at all, the same gate slice 1 carries. Plus a fixture per row of the slot table: an internal.fqdn refuses at plan; a management.ipv6 refuses at plan with the truncation named; a public.ipv6 publishes an AAAA record end to end.

PARTLY LANDED, 2026-08-15. On main:

  • the vault cold start is a DECLARED CLUSTER CAPABILITY: vault_config’s coldstart_holder row, the shc vault coldstart replant verb, and EnsureColdStartRelocated run by every removal route (shc cluster rm, shc node forget (the verb tofu destroy calls), and the cluster repair sweep) BEFORE their first mutation, or X409906VLTCSH. Route coverage is derived from chi’s route table, so a removal route added tomorrow is swept tomorrow. This was M8’s stated prerequisite;
  • bootstrap no longer forces replacement and is never read after Create;
  • both plan-time identity refusals are DELETED (the core module’s bootstrap_node_name-vanished precondition and the provider’s refuseBootstrapIdentityMove), with the daemon’s own named refusals (X409026SECCLS, X409029CLSGEN, X409906VLTCSH, X400017NODDEP, the M9 voter-deficit gate) named as the replacement protection;
  • shc_node.bootstrap’s Delete no longer issues the whole-cluster DNS release when the cluster SURVIVES the removal. That release deletes every record the estate owns, and it was unreachable-and-therefore-safe only while the two identity guards existed; deleting them made the first rotation an operator performs take a live estate off the internet.

What slice 9 still owes, in descending order of damage; under M11 each is a defect, not a corner to document:

  1. shc_node.bootstrap is a SEPARATE RESOURCE from shc_node.workers, and name still forces replacement. So a rotation does not remove one node; it MIGRATES a live node between two resource addresses: shc_node.workers["swarm01"] is destroyed (a real shc node forget of a healthy member) while shc_node.bootstrap is replaced ONTO swarm01. The plan is clean and the apply is destructive. The fix is the uniform shape this document has argued for throughout: ONE for_each shc_node keyed by node name, with seed = (name == local.seed_name) as a plain in-place attribute, and it needs static moved blocks in each estate root, which is a state upgrade of the same size as M10’s.
  2. local.cluster_ref anchors the whole estate on the seed’s outputs (endpoint / admin_token / ca_pem from shc_node.bootstrap[0]). Every worker, config, connection, deployment and DNS record rides it, so re-creating the seed resource re-derives the control-plane reference for the entire graph. ssh = local.platform_ssh is already node-set-derived and fine; the other three are the genesis anchor.
  3. output "bootstrap_node" survives in modules/core and all fourteen substrates, a POSITIONAL value (local.ssh_ordered_nodes[0]) asserting a one-time historical fact, recomputed every plan. Slice 9 deletes it in favour of entrypoint; the estates consume it today through the pinned 0.0.16 module tarball, so the deletion and their move are one program, not a live hazard.
  4. cluster_ssh_hosts is still emitted BOOTSTRAP FIRST and still claims leader affinity (“most likely raft leader”) in modules/core and the substrate docstrings. Rule S4: stable name-ascending, documented as arbitrary. Note the coupling to (3): the hetzner substrate’s bootstrap_node reads local.ssh_ordered_nodes[0], so the reorder and the output deletion have to move together or the output starts naming the alphabetically-first node.
  5. shc-role = bootstrap|worker cloud labels in all fourteen substrates. A create-time accident stamped onto the infrastructure as a permanent label; nothing reads it, and after any rotation it lies.
  6. bootstrapSSH still REFUSES rather than adopts. X409026SECCLS closes the data-loss shape (a from-scratch apply genesising a second cluster), which is the important half, but the join path CONVERGES in the same position (node_join_recovery.go) and the init path does not. Rule S2’s adopt-first probe is still owed.
  7. bootstrap is still spelled bootstrap, not seed, in the provider schema, the module inputs and the estate roots.

The rest of this section is the original proposal text.

bootstrap → seed, RequiresReplace dropped, bootstrap_node output deleted (with the three estate roots moved to entrypoint in the same commit), both plan-time identity refusals deleted, and (the load-bearing change) bootstrapSSH gains the adopt-first probe that node_join_recovery.go already implements for the join path. ssh_hosts re-emitted in stable name order with the leader-affinity claim removed from its docstring.

Depends on nothing in slices 1–8. It could land first, and there is an argument that it should: it closes a data-loss shape (from-scratch apply genesising a second cluster) that exists at HEAD, and it is the only slice whose absence makes the estates un-rotatable.

Proof: a three-node cluster where the seed node is removed from nodes, a fourth is added, tofu apply succeeds, the cluster keeps its identity and its data, and shc cluster status shows the expected membership. Plus a negative test: an apply with no state against a live cluster ADOPTS rather than genesising a second one, the shape that today succeeds while orphaning an estate.

Slice 10: suffrage becomes visible, then correct

Section titled “Slice 10: suffrage becomes visible, then correct”

Three steps, each independently landable:

  1. Visible. Wire VoterDeficit into the removal path’s output and into the shc cluster status summary line an operator actually reads. It is already computed (ce/modules/cluster/service.go:1606-1607) and already exposed to terraform (data.shc_cluster_status.voter_deficit), and acted on by nobody.
  2. Refusing. A voter removal that would leave raftVoters < ExpectedVoters refuses unless the caller passes --rebalance or --accept-deficit.
  3. Correct. --rebalance performs remove-and-rejoin on one healthy non-voter BEFORE evicting the departing voter. This is the only rebalance the shipping engine permits; Assign is unimplementable on rqlite (subsystems/rqlite_adapters.go:84-90).

Same slice, cheap and overdue: delete or wire PromoteNode / DemoteNode. They are declared, defined, unreachable, and would fail on any real cluster.

Proof: a four-node cluster (3 voters + 1 non-voter) where removing a voter leaves three voters, not two, verified from shc cluster status, not from the removal’s own return value.

Slice 11: destroy refuses, drain relocates

Section titled “Slice 11: destroy refuses, drain relocates”

The provider’s Delete stops calling shc node forget --yes for a node that is up. It calls the gated removal and, on X400017NODDEP, fails the destroy naming the deployments. on_destroy = "refuse" | "force" per resource, defaulting to refuse. shc node forget keeps force=true and keeps its reaping job. drain gains no terraform surface.

Proof: terraform destroy -target on a node carrying a running stack FAILS with the stack named; the same destroy after an explicit shc node drain succeeds; and shc node forget on an already-dead node still reaps it.

docs/guide/operations/node-rotation.md, to the specification above. It is the last slice because it documents behaviour slices 9–11 create, and the first thing an operator will touch.

Proof: the lane in the runbook spec: rotate every node of a three-node cluster out and back in, one at a time, including the seed, with the cluster serving throughout.


These are product decisions, not engineering choices.

Q1. Is a LAN-only estate a first-class topology, or a degraded mode? Today it is neither: it works on the cluster plane and silently half-configures DNS and TLS. First-class means slice 4 plus a proof lane that stays green (a from-empty proxmox cluster whose cluster host resolves and whose internal-CA posture is stated, not discovered). Degraded-mode means slice 4 shrinks to “fail loudly and document it”. The difference is roughly one lane’s worth of ongoing acceptance cost.

Q2. Does terraform become AUTHORITATIVE for internal_address (slice 2), or does it stay an assertion? Staying an assertion is free and preserves the exact ambiguity this document exists to remove; “what is this node’s address?” can only be answered by a declaration, not by a hint plus a probe. It also leaves the mixed/multi-cloud shape permanently unsolvable, since a NAT’d node can never declare an address it does not hold. Recommendation: declare.

Q3. Is IPv6 in scope, and at what depth? Three coherent postures: (a) name-only: this proposal’s rename, so a future IPv6 program needs no second rename; (b) fail-loud: additionally reject bare v6 at plan with a named error, since the current behaviour is a silent wrong endpoint; (c) full support, which means nic.Detect returning a family-tagged result, one host-splitter, a v6-capable fabric CIDR, and reworking cidrhost() pinning plus ${addr}/32 firewall sources across five substrate modules. (a) and (b) are cheap; (c) is a separate program. Recommendation: (b) now, (c) as its own proposal.

Q4. Is “one node per internal address host” a permanent product limit? It forbids multiple nodes behind one NAT, the obvious home-lab and branch-office shape. Lifting it means separating node identity from node address in the registry (rowHostShared and the raft member↔row bridge both key on the host today), which is a real data-model change.

Q5. Is public-only (raft, rqlite and the internal API bound to a public IP, firewalled to peer /32s) a SUPPORTED shape or a documented hazard? Four substrates allow it and examples/hetzner-wan-nebula ships it. Raft (4002) and the internal API (4003) are mTLS from genesis (RaftTLS: true, ce/modules/cluster/service.go:817-820). rqlite’s HTTP API on 4001 was not established by any audit and is the deciding fact; it should be checked before this is ruled on.

Q6. Does topology survive as a split, or is it deleted? See slice 6. Zero production use today; the estates’ reason for turning it off is recorded and is fixable by splitting.

Q7. Per-group nic_name exists only on hetzner. Universalise it (and delete the scalar nic_name output that misreports it), or delete the per-group override so all fourteen substrates match? Universalising is the only shape compatible with a heterogeneous cluster.

Q8. Family containers or family-suffixed scalars? The recommendation is containers, and the deciding argument is that a container makes dual-stack expressible without a second rename, which is the reason the owner raised address family in the first place. The cost is a nested object where three substrate modules currently write a bare string, and a slot table that has to be maintained honestly as support arrives. Suffixed scalars are flatter and cheaper today and are the shape that has to be renamed later.

Q9. Is the vault cold start a node identity or a cluster capability? Today it is a node identity by accident: the master password is planted on the genesis node alone, deliberately and with a structural test enforcing it, and rotating that node out silently costs the cluster its ability to cold-start after a whole-estate restart. Options (a) declare the capability and plant on N nodes (which changes a security posture that was pinned on purpose, and is therefore the owner’s call), or (b) make every node able to cold-start via a quorum-derived unseal, which is strictly better and a separate program. Doing neither is legitimate, but then rotation of the seed node is a documented capability loss and the runbook must say so in capitals.

Q10. On removing a voter: report, refuse, or rebalance? Today the cluster silently goes 3 voters → 2, quorum 2 of 2, zero fault tolerance, and the non-voter that should have taken the slot is never promoted, because promotion is unimplementable on rqlite and the only rebalance is remove-and-rejoin. Report is free. Refuse is symmetric with every other guard in this model. Rebalance is the only option that makes “rotate nodes in and out whenever I want” actually true, and it is real orchestration with a real cost.

Q11. Does terraform destroy of an occupied node refuse? The recommendation is yes, per the owner’s own instruction, and note that saying yes means the provider stops using shc node forget, which is the verb it uses today and which hardcodes force. The alternative reading is that a destroy is already an explicit operator act and a second confirmation is friction. It is a product decision about what terraform destroy MEANS, so it is listed here rather than decided.

Q12. Should terraform be able to express suffrage at all? A voter = optional(bool) per node would let an operator say “these three are the control plane, the rest are capacity” instead of letting join order decide. It is expressible only as remove-and-rejoin, it would make an apply able to restart a node’s storage engine, and it hands operators a loaded gun (an even voter set is one typo away). The alternative is that suffrage stays entirely the daemon’s business and terraform only READS it through data.shc_cluster_status.

Q13. Should slice 9 land FIRST, ahead of the whole addressing rename? It depends on nothing else, it closes a data-loss shape that exists at HEAD (a from-scratch apply genesising a second cluster while the original keeps running), and until it lands neither production estate can rotate a node out at all. The argument against is that it touches the same resources slice 1 renames, so landing them separately means two state upgrades instead of one.

Q14. Does entrypoint survive, or does the daemon own it? Three estate roots feed a node’s public address into dns.public_address positionally today. The short answer is entrypoint, an explicitly arbitrary, stable derivation from the node set. The right answer is that the daemon’s DNS ladder should read node rows and terraform should write nothing, which is slice 4’s cluster posture. Shipping entrypoint as an interim means shipping a name that is intended to be deleted.