The node addressing model (proposal)
Status: open design. Slices 1 and 8 shipped on 2026-08-14 and slice 9 in part on 2026-08-15. Parts of slices 5 and 11 are built too (the occupied-node destroy refusal is in the Terraform provider), and the Nebula lighthouse work left in slice 5 went with the new line (SHC-1092). The rest is still a proposal. Ruling M10 (2026-08-14 models rulings) took the vocabulary rename and the address-family containers as ONE state upgrade, and they are on
main: themanagement/internal/publiccontainers replacehost/private_ip/public_addressonshc_node(state schema 1 → 2, with the v0 one-hop registered),modules/coreand all fourteen substrates emit the typednodeslist, the four flat projections are REMOVED rather than aliased (M12), and every removed spelling is a plan-time tombstone rather than a silently-ignored attribute. The gate M10 set (atofu planshowing zero changes on both live estates after the state upgrade) was run on 2026-08-14 against the real state of bothhetzner/hel1/infraroots and printedNo changes.on each. Slice 9’s demotion half is onmaintoo (rulings M8 + M11): the vault cold start became a declared cluster capability with a replant verb that every removal route runs BEFORE it touches the holder,bootstrapstopped forcing replacement, and both plan-time identity refusals (the core module’s precondition and the provider’srefuseBootstrapIdentityMove) are deleted. What slice 9 still owes is listed in its own section: the module still models the seed as a SEPARATE RESOURCE, which is what keeps a live rotation destructive. Everything else below (slices 0, 2–7 and 10–12) is STILL A PROPOSAL, and slice 2 in particular carries ce-side preconditions that make it unsafe to attempt out of order. Read the slices for what each one still owes.
Written 2026-08-02, from four parallel audits of terraform/modules/*,
terraform/provider, the ce daemon, and both live ops estates.
Revised the same day with three constraints the owner added after
those audits were briefed; see what the revision adds.
This document proposes one coherent way to describe a node’s addresses,
used with the same words in terraform, the provider and the daemon. It
supersedes the smaller cleanup already agreed with the owner (collapsing
the four flat node_* output projections), because that cleanup is a
consequence of the model rather than a change of its own.
What the revision adds
Section titled “What the revision adds”- A shared vocabulary, with address family made explicit. “ip” is not a word this project gets to use: there is IPv4 and there is IPv6. Every role becomes a container holding both families rather than a family-blind scalar, and a glossary fixes one name per concept across all four layers. This revises the scalar shape the first draft proposed; see containers, not suffixes.
- Terraform stops caring which node ran
shc init. The current surface conflates an infrastructure fact, a one-time cluster history and a runtime role, and the result is a wall: the init node cannot be removed. See genesis is not an identity. - Node rotation and raft suffrage. Nodes must be addable and removable in any order, including the one that bootstrapped. See node lifecycle and raft suffrage and the runbook specification.
Points 2 and 3 are the same question wearing two hats: “who ran init” and “what is a node’s address” are both answered by refusing to let a create-time accident become a permanent identity.
For the fabric this model must serve, see network-fabric.md. For the bind/advertise primitive it consumes, see bind-advertise-split.md. For the cluster membership surfaces it touches, see cluster/overview.md.
Problem
Section titled “Problem”The two layers do not share a vocabulary, and neither vocabulary names what it means.
Measured over terraform/ (excluding .terraform/ caches):
public_ipv4 228, private_ip 161, nic_name 116, public_address 112,
mesh_address 12 (all inside the provider’s shc_mesh plumbing, zero in
modules/), internal_address 1 (a JSON tag on a read-only wire decode),
fabric_cidr 0.
Measured over ce: public_address 125, internal_address 99, mesh_address
96, fabric_pin 15, bind_ip 10, private_ip 1.
The overlap is one word. That is the symptom. Three deeper facts are the disease:
1. Terraform’s names describe artifacts; the daemon’s describe roles.
nodes[].public_ipv4 (modules/core/variables.tf:34) is not “the public
IPv4”; it is the address terraform SSHes to. It has exactly two
consumers: shc_node.host (modules/core/main.tf:260, :325) and
local.cluster_ssh_hosts (:152-157). It is never sent to the daemon as
an address. On modules/shc/proxmox/main.tf:140 and on
modules/shc/openstack/main.tf:204 with floating IPs disabled, it holds
an RFC1918 address. One field, two incompatible meanings, distinguished
only by which substrate wrote it, and a name that forecloses IPv6.
2. The field that looks like an address is an assertion.
nodes[].private_ip reaches shc_node.private_ip, whose only effect is
a remote shell probe (provider/internal/provider/ssh_engine.go:584-602,
consumed at node_resource.go:555 and :1401). No --private-ip flag
exists; the value never reaches a daemon column. Its own schema doc says
“changing it later does not re-run the gate”
(node_resource.go:266). Both ops estates re-export it as
node_private_ips believing it is the plane address.
3. The address that actually matters is unreachable from terraform.
internal_address is the daemon’s only dialable node-row column
(ce/modules/cluster/peer_seam.go:112-115 says so in those words). It
carries the raft join-address list (modules/cluster/token_router.go:349),
the WireGuard fabric peer endpoints
(ce/subsystems/impl/fabric_reconcile.go:444-467), the nebula lighthouse
physical host (modules/cluster/mesh_orchestrator_impl.go:181-186), the
internal-API mTLS leaf’s only IP SAN
(modules/cluster/enroll_handler.go:316-334), the LB backend set, the
vault peer-unlock broadcast, and the LAN DNS rungs. Terraform cannot
state it. Terraform can only name an interface (nic), from which the
node derives an address by an IPv4-only, RFC1918-preferring heuristic
(ce/core/nic/nic.go:98-140). And once written, the column can never be
corrected: the upsert is
internal_address = COALESCE(excluded.internal_address, node.internal_address)
(ce/modules/node/repository.go:251) and the repository exposes
UpdateNodeState, UpdateAvailability, UpdateMeshAddress,
UpdatePublicAddress, UpdateCPUCores, UpdateSwarmIdentity, and no
UpdateInternalAddress.
So the owner’s complaint is exact. The layers do not disagree by accident; they are describing different things with overlapping words, and the one thing that determines whether a cluster works at all has no name on the terraform side.
The model
Section titled “The model”A node has a SET of addresses. Each element has a ROLE. A role is defined by WHO DIALS IT, not by what address class it holds.
Four roles. Nothing else is an address.
| role | who dials it | ports | provenance | nullable |
|---|---|---|---|---|
management | terraform / the provider | SSH (22) + derived daemon API (8000) | declared by terraform | no; a node with no management address is not manageable |
internal | peer nodes | 4001 rqlite HTTP, 4002 raft, 4003 internal API, 4242/udp nebula underlay, 4243/udp fabric underlay | declared by terraform, else derived on the node from nic | no, but may be derived rather than stated |
public | the internet | 80/443 through system Traefik; the A/AAAA record target | declared by terraform, else detected from cloud metadata | yes; null means “this node is not internet-reachable” |
overlay | containers, and the daemon’s own tunnels | none fixed; it is an address inside a tunnel | allocated by the cluster leader | yes; absent until the mesh/fabric is enabled |
Three provenances, and the distinction is the whole design:
- declared: terraform states it. Terraform is authoritative; drift is a diff.
- derived: the node computes it locally from something terraform did state (an interface name). Terraform is not authoritative and cannot see the result today.
- allocated: the raft leader hands it out from a cluster-scoped pool. Terraform must never state it, and must be able to read it.
And one rule that collapses most of the confusion:
No overlay has an endpoint of its own. Every SHC-owned overlay dials the
internaladdress plus its own port.
Nebula’s underlay host is StripPort(internal_address)
(mesh_orchestrator_impl.go:181-186). The fabric’s peer endpoint map is
out[node.ID] = cluster.HostOnly(row.InternalAddress)
(fabric_reconcile.go:444-467), whose own comment says it deliberately
reuses the raft reachability story. So internal is not one of five
addresses among equals; it is the load-bearing one, and every overlay is
a layer on top of it, never an alternative to it.
The declared set and the observed set
Section titled “The declared set and the observed set”Terraform declares three; the cluster reports four back.
# DECLARED: the per-node object modules take and emitnode = { name = string runtime = string placement = object({ group = string, ordinal = number })
management_address = string # REQUIRED. terraform -> node. internal_address = optional(string) # peer -> node. null => derive from `nic`. nic = optional(string) # the interface `internal_address` is derived # from when it is null. null => daemon auto-detect. public_address = optional(string) # internet -> node. null => NOT internet-reachable.}# OBSERVED: added to the module `nodes` output, computed, never an inputobserved_internal_address = string # what the daemon actually recordedobserved_public_address = string # declared-or-detected; makes metadata self-stamping visiblemesh_address = string # allocated; null when nebula is offfabric_ip = string # allocated; null when the fabric is offFour addresses in, one required. No ipv4 in any name. No "" sentinel;
absence is null, always (the try(..., "") degradation in twelve
substrate modules becomes try(..., null), and cluster_ssh_hosts filters
on != null).
One name per role, in all three layers
Section titled “One name per role, in all three layers”This is the actual deliverable. The rule for choosing names was: terraform stops inventing words and adopts the daemon’s, except for the one role the daemon does not have.
| role | THE NAME | terraform module | provider | daemon |
|---|---|---|---|---|
| management | management_address | nodes[].management_address (was public_ipv4) | shc_node.management_address (was host) | not stored; the daemon never dials it |
| internal | internal_address | nodes[].internal_address (new; absorbs private_ip’s job) | shc_node.internal_address (was private_ip) | node.internal_address (unchanged) |
| public | public_address | nodes[].public_address (unchanged) | shc_node.public_address (unchanged) | node.public_address (unchanged) |
| overlay (nebula) | mesh_address | nodes[].mesh_address (new, computed) | shc_node.mesh_address (new, Computed) | node.mesh_address (unchanged) |
| overlay (fabric) | fabric_ip | nodes[].fabric_ip (new, computed) | shc_node.fabric_ip (new, Computed) | fabric_allocation.address (unchanged) |
Two of five role names are already right in the daemon and merely absent
from terraform. One is right everywhere. One is read-only and only needs
surfacing. Exactly one genuinely new word is introduced: management,
because it names the one role the daemon has no opinion about: terraform’s
own reach.
No daemon column is renamed. That is deliberate: a column rename is a migration on two live estates under the append-only migration doctrine, and it buys nothing the terraform-side rename does not already buy.
Superseded in part. The table above still fixes the ROLE names, which is the part that matters. The scalar SHAPE it implies (
management_address = string) is revised by the next two sections: each role becomes a container holding one slot per address family.
Address family: containers, not suffixes
Section titled “Address family: containers, not suffixes”private_ip and node_private_ips are the exact shape to eliminate, and
not only because “private” is the wrong role word. “ip” is not a word
this project gets to use. There is IPv4 and there is IPv6, they have
different literal syntax, different DNS record types, different firewall
CIDR arithmetic, and (today) different levels of support. A name that
elides the difference cannot be checked, and a name that hard-codes one
family has to be renamed the day the other arrives.
Two coherent answers exist. They are not equivalent.
(i) Family-suffixed scalars: management_ipv4, management_ipv6,
internal_ipv4, internal_ipv6, public_ipv4, public_ipv6.
Explicit, flat, terse in the single-stack case. But it doubles the field
count immediately, it makes “this role, whichever family” unexpressible
without a coalesce() at every reference, and (the deciding objection)
it puts the family in the field name, so a role that gains a family
gains a field, and every consumer’s for expression has to learn it.
(ii) A per-role container: management = { ipv4, ipv6, fqdn }.
The ROLE is the container’s name and carries no family word at all; the
FAMILY is the slot name and is always explicit. Adding a family is adding
a slot to a type that already exists, not adding a top-level field to
fourteen substrate modules and two estate roots.
Recommendation: (ii). Three reasons drawn from the topology matrix, not from taste:
- A role’s cardinality is genuinely per-family, and the matrix already
proves it. The public-only shape holds one address in three roles; the
edge-NAT shape holds a public IPv4 and an RFC1918 IPv4 in two roles; a
dual-stack Hetzner node today holds a public IPv4 and a public IPv6
(
hetzner/main.tf:412setsipv6_enabled = true) and surfaces neither as a second field. Under (ii) that node stops lying the day someone fills inpublic.ipv6, with no rename anywhere. - The public role needs both families NOW, not later.
ClassifyTargetalready emits AAAA correctly (ce/modules/app/dns_target.go:129-137). The only reason SHC cannot publish an AAAA record for a node is that no terraform field can hold one. That is a real capability blocked by a naming decision, and (ii) unblocks it without touching the daemon column;public_addressstores whichever member the DNS ladder asks for. - Families are not symmetric across roles, and the container makes that
statable. A hostname is legal for
management(the SSH engine resolves it;deriveEndpointaccepts it) and forpublic(the daemon column validatesip|hostname_rfc1123,ce/modules/cluster/types.go:142), and is not legal forinternal(every producer builds that column from a literal address, androwHostSharedcompares hosts textually). A flat scalar cannot say that; a container with a per-role slot table can.
# The address container. One shape, used for every role.address = object({ ipv4 = optional(string) ipv6 = optional(string) fqdn = optional(string) # family-agnostic; resolves to either})Every role carries all three slots in its TYPE; that is what buys the “no second rename”. What differs per role is which slots carry a VALUE today. Three dispositions: honoured (works end to end), refused (the slot exists, a value in it fails the plan with a named code), meaningless (the slot is not part of that role’s grammar).
| role | ipv4 | ipv6 | fqdn | cardinality |
|---|---|---|---|---|
management | honoured | refused; splitHostPort truncates a bare v6 literal (node_resource.go:1632-1637) | honoured | at least one slot set |
internal | honoured | refused; nic.Candidates drops every non-IPv4 (ce/core/nic/nic.go:98-101) | meaningless; the column is compared textually by rowHostShared | at most one honoured |
public | honoured (A) | honoured (AAAA) the day a field exists to hold it | honoured (CNAME) | all three may coexist |
mesh | computed | meaningless | meaningless | allocated from 10.42.0.0/16 |
fabric | computed | meaningless; v6 supernets rejected (ce/modules/fabric/types.go:119-121) | meaningless | allocated from 10.84.0.0/16 |
Refused means refused loudly. That is the whole point of writing the
table down: today an IPv6 host silently becomes 2001:db8: and
surfaces, much later, as a connection error nobody can trace back. A
refusal that names the slot and the reason is strictly better than a
capability that quietly is not one, and when IPv6 lands (slice 7), a
refused cell becomes honoured and nothing renames.
The single-stack case stays readable, because a substrate module fills the container and the consumer never writes it by hand:
management = { ipv4 = hcloud_server.node.ipv4_address }internal = { ipv4 = cidrhost(var.networking.subnet, offset) }public = { ipv4 = hcloud_server.node.ipv4_address, ipv6 = cidrhost(hcloud_server.node.ipv6_network, 1) }The glossary
Section titled “The glossary”One concept, one word, everywhere. This is the deliverable, not a summary of it. A concept that today has two names in two layers resolves to one; where a layer’s spelling still differs, the reason is stated and it is always the same reason (renaming a daemon column is a migration on two live estates, and buys nothing).
| concept | THE word | what it means | terraform module | provider | daemon | today’s names it replaces |
|---|---|---|---|---|---|---|
| cluster membership identity | name | the handle every verb takes; unique; immutable | nodes[].name | shc_node.name | node.name | none |
| where terraform reaches the node | management | SSH target, and the base of the derived daemon API endpoint | nodes[].management | shc_node.management | not stored | public_ipv4, shc_node.host, node_public_ipv4s, cluster_ssh_hosts[].host |
| where peers reach the node | internal | rqlite 4001, raft 4002, internal API 4003, nebula underlay 4242/udp, fabric underlay 4243/udp, the mTLS leaf’s IP SAN | nodes[].internal | shc_node.internal | node.internal_address a | private_ip, node_private_ips, bind_ip (partly) |
| where the internet reaches the node | public | the A/AAAA/CNAME target; 80/443 through system Traefik | nodes[].public | shc_node.public | node.public_address a | public_address (already right), topology.public_address b |
the interface internal is derived from | nic | in-guest interface NAME, never an address | nodes[].nic | shc_node.nic | --nic → sidecar nic | nic_name (the module-scalar output, deleted) |
| where the daemon LISTENS | bind | node-local only; re-resolved live every boot | not expressible c | not expressible | sidecar bind_ip, internal_api.bind, SHC_RQLITE_BIND_ADDR | none |
| nebula overlay address | mesh | allocated by the leader; never an input | nodes[].mesh (computed) | shc_node.mesh (Computed) | node.mesh_address a | none |
| WireGuard fabric address | fabric | allocated by the fabric reconcile; never an input | nodes[].fabric (computed) | shc_node.fabric (Computed) | fabric_allocation.address a | fabric_pin d |
| the fabric POSTURE | fabric (cluster scope) | swarm | shc | none, plus its supernet | bootstrap.fabric / .fabric_cidr | shc_cluster.fabric / .fabric_cidr | cluster_fabric.mode / .cidr | none |
| the node’s placement in the declared set | placement | {group, ordinal}; ordering only, never identity | nodes[].placement | not sent | not stored | group + index as siblings |
| which node terraform PREFERS to run genesis on | seed | create-time only; inert on every later plan | seed_node_name | shc_node.seed | not stored anywhere | bootstrap_node_name, shc_node.bootstrap |
| which node ACTUALLY ran genesis | (no word, deliberately) | not recorded by the daemon, and does not need to be | deleted | deleted | nothing | bootstrap_node output |
| who leads raft RIGHT NOW | leader | changes without terraform; read, never declared | data.shc_cluster_status.leader_name | same | Status.LeaderID | cluster_ssh_hosts’ “most likely raft leader” claim |
| the address family | ipv4 / ipv6 | never “ip”, never bare | container slots | container slots | whichever member the column holds | ip, ips, _ip, _ipv4 as suffixes |
a The daemon column keeps its _address / _ip suffix. The
suffix is not a second name; it marks “this layer holds exactly one
member of the container”. Renaming the column is an append-only migration
on two live estates and changes no behaviour.
b topology.public_address is a CLUSTER-scoped config row
(dns.public_address) that collides by name with the NODE column. Under
this glossary the cluster-scoped one becomes dns.entrypoint, because it
is not a node’s public address; it is the cluster’s named entrypoint.
c bind stays node-local by design (bind-advertise-split). It
is in the glossary so that nobody re-invents it under a third name.
d fabric_pin is not a node address and never was: it is
one row per (deployment, service), the stable address a rendered compose
service gets. It has no place in a node shape and is listed here only to
stop the next reader from putting it there.
What disappears
Section titled “What disappears”public_ipv4: wrong on two substrates, forecloses IPv6 by name.private_ip: an assertion wearing an address’s name. Its job (the plane-NIC readiness gate) survives, attached tointernal_address, where it is strictly better: today the gate checks a value that goes nowhere; tomorrow it checks the value the node will actually advertise.group+indexas sibling scalars:indexis per-group and meaningless alone;(group, index)is the disambiguator, so it becomes oneplacementobject. Neither is read bymodules/coretoday.- The four flat projections
node_names/node_public_ipv4s/node_private_ips/node_runtimes: one-lineforexpressions overnodes, coupled to it only by the prose “same order asnodes”, and already drifted (hostinger has a fifth,node_hostnames). Consumers write theforthemselves against a self-describing object. - The
""sentinel in a required-string field. local.all_nodes’all_prefix: the only other*nodes*local in fourteen modules is hostinger’stotal_nodes. There is nosome_nodes.
cluster_ssh_hosts survives and is not renamed away. It is not a
projection: it re-orders bootstrap-first, drops address-less nodes, and
folds in the SSH user (modules/core/main.tf:141-157). It is the fix for
two live failures on 2026-07-27 and it stays.
The topology matrix, instantiated
Section titled “The topology matrix, instantiated”Every shape is the same four roles with different membership. There are no special cases, only null sets.
| shape | management | internal | public | overlay | derived cluster posture |
|---|---|---|---|---|---|
| public + private plane (hetzner hel1, both live estates) | public IPv4 | RFC1918, derived from nic = enp7s0 | public IPv4 | allocated when on | internet = true |
public only (hostinger, examples/hetzner-wan-nebula) | public IPv4 | the same public IPv4 | the same public IPv4 | allocated when on | internet = true |
LAN only (proxmox always; openstack with assign_floating_ips = false) | LAN IPv4 | the same LAN IPv4, or a second NIC’s | null | allocated when on | internet = false |
| edge-NAT public (aws, gcp, azure, openstack-with-FIP) | public IPv4 (1:1 NAT at the edge) | RFC1918 VPC address | public IPv4 | allocated when on | internet = true |
| mixed / cross-substrate | per node | must be DECLARED on any NAT’d node: its edge address, not its NIC’s | per node | allocated when on | internet = true |
| + nebula | unchanged | unchanged; the overlay does not relocate the plane | unchanged | mesh_address allocated | + mesh |
| + wireguard fabric | unchanged | unchanged | unchanged | fabric_ip allocated | + fabric |
| single node | any of the above | same | optional | none | as above |
| declared but absent from state | null | null | null | null | filtered out of cluster_ssh_hosts |
Three readings worth stating out loud:
- Roles collapse; they do not vanish. On a public-only node all three
declared roles hold the same string. That is not degeneracy; it is the
model correctly reporting that one address serves three purposes. Today
the same fact is expressed by putting an RFC1918 address in a field
called
public_ipv4and hoping nobody reads the name. public = nullis the only signal that distinguishes LAN-only, and nothing in the type system or in any module inspects it today.- The mixed shape is the only one that requires a new capability, not
just a new name: a NAT’d node must be able to declare an
internaladdress it does not hold on any interface. Everything else in this table is expressible the day the rename lands.
The owner’s two questions, answered
Section titled “The owner’s two questions, answered””What is a node’s nebula IP if there’s only public?”
Section titled “”What is a node’s nebula IP if there’s only public?””10.42.<a>.<b>: exactly what it would be on any other topology. The
overlay address does not depend on the underlay at all.
It is allocated by the raft leader from a /16 pool at cluster mesh enable (shc/modules/nebula/service.go:240-272), persisted onto the node
row after a successful install RPC
(mesh_orchestrator_impl.go:404,433-442), and it is an allocation, not
a derivation. Whether the node has one address or three, whether they are
public or RFC1918, changes nothing about the nebula IP.
What the public-only case changes is not the overlay address but the
overlay endpoint, and under this model that is not a separate concept
at all, because of the rule above: the endpoint is
internal_address + 4242/udp. On a public-only node, internal is
the public address, so the endpoint is public and the mesh forms. On a
NAT’d node, internal is RFC1918 and the mesh cannot form; the
lighthouse publishes an unroutable physical host and every install RPC is
dialled at an unroutable address.
That is the honest, uncomfortable answer, and it exposes the real bug the
question was circling: the overlay cannot bootstrap the nodes it exists
to bridge, because it derives its endpoints from the very reachability
it is supposed to provide. examples/mixed-proxmox-hetzner works only
because the Hetzner node happens to win the raft election. A lighthouse
pin exists (shc cluster mesh lighthouse add,
shc/modules/cluster/lighthouse_router.go, honoured at
mesh_orchestrator_impl.go:198-212) and has no terraform surface
whatsoever.
The model’s answer is therefore three-part, and the first two parts are where the value is:
mesh_address/fabric_ipbecome computed, readable, per-node; the question becomes answerable bymodule.cluster.nodes[*].mesh_addressinstead of by reading EE source. Todaymesh_addressis reachable only throughshc_mesh, a cluster singleton, keyed by name, and only after the mesh exists.internal_addressbecomes declarable, so a NAT’d node can state its edge address and the overlay’s endpoint becomes correct by construction. This is the surviving primitive of bind-advertise-split.md, consumed here.- The lighthouse gets a terraform surface, so the mixed shape stops depending on an election.
”What if everything is only on LAN, with no internet at all?”
Section titled “”What if everything is only on LAN, with no internet at all?””The set shrinks to {management, internal}, both usually holding the
same address, and public = null becomes a first-class cluster posture
rather than an accident.
Concretely, today, on proxmox:
management_address = 192.168.1.11 # what terraform SSHes tointernal_address = 192.168.1.11 # what peers dial (derived, nic="ens18")public_address = null # nobodymesh_address = 10.42.1.x # if the mesh is onEverything on the cluster plane works. What breaks is everything
downstream of public, and all three failures are currently silent:
- The cluster host record is never registered at all.
RegisterSystemHostDNShard-codesExposure: "internet"(ce/modules/app/dns_lifecycle.go:305-311). The internet arm of the ladder skips thedns.public_addressrung outright (dns_target.go:315-317, gatedif !internet), then requires either a ready web-connection VIP ortraefikPlacementPublicAddress, which returns""for a private address (dns_target.go:465-469). It falls through to the fail-closed rung and warnsX200980DNSDEF. - The one knob terraform offers cannot reach it.
topology.kind = "lan"+topology.public_addresswrites thedns.public_addressconfig row (modules/core/main.tf:392-400), the exact key the internet arm refuses to read. - ACME degrades silently.
leEnabled = systemIngressHasPublicEndpoint(ctx)and only for internet intents (ce/modules/app/ingress_materialize.go:364-367); with no public endpoint the ingress quietly serves the internal SHC leaf. Terraform’sbootstrap.acmeis accepted unconditionally and is then permanently inert, with no signal at either layer.
Per-deployment exposure = lan hosts do resolve, via rung 3 or the
internal_address fallbacks. So a LAN-only estate today is not broken;
it is quietly two-thirds configured, and the operator has no way to
learn which third.
The model’s answer: derive a cluster-level internet posture from the
node set (internet = anyof(nodes[*].public_address != null)) and make
every consequence explicit rather than silent:
internet = false⇒RegisterSystemHostDNSregisters the cluster host at LAN exposure, so it resolves;internet = false+bootstrap.acmeset ⇒ plan-time error, not an inert setting;internet = false⇒dns.public_addressis the primary rung, not a non-internet fallback;internet = trueon a node whosepublic_addressis RFC1918 ⇒ plan-time error (todaytraefikPlacementPublicAddresssilently returns"").
Whether LAN-only is a first-class topology with its own proof lane, or a degraded mode that merely stops lying, is an owner ruling; see Q1 below. The model supports either; only the amount of work differs.
Genesis is not an identity
Section titled “Genesis is not an identity”The owner: “what if nodes are ripped out of the terraform and new nodes added and the node that did the original init fully gone. I would assume the terraform shouldn’t care about who ran init else it would paint us in a corner.”
He is right, and the corner is already built and load-bearing.
The corner, in the module’s own words
Section titled “The corner, in the module’s own words”output "bootstrap_node" is described as “The shc init node: the FIRST
node of the alphabetically-FIRST node_groups key”
(terraform/modules/shc/hetzner/outputs.tf:6-13, identical in all
fourteen substrates). That is a POSITIONAL value, recomputed on every
plan, asserting a ONE-TIME HISTORICAL fact. Three estate roots feed
bootstrap_node.public_ipv4 into dns.public_address, so the cluster’s
published entrypoint silently follows the relabelling.
local.cluster_ssh_hosts inherits the same assumption, documented as
“BOOTSTRAP FIRST (most likely raft leader)”, a claim about a runtime
role that changes without terraform, encoded in a static list. The three
roots are promanager-ops/hetzner/hel1/infra/main.tf:344,
promanager-ops/hetzner/nbg1/infra/main.tf:136 and
bitspur-ops/hetzner/hel1/infra/main.tf:324, each writing
public_address: ${module.shc.bootstrap_node.public_ipv4}.
And the module and the provider each have a guard that turns the assumption into a wall:
modules/core/main.tf:324-329: apreconditiononshc_node.bootstrapfails the PLAN when the pinnedbootstrap_node_nameleavesnodes. Its message says it outright: “SHC does not support rotating the bootstrap node through terraform.”terraform/provider/internal/provider/bootstrap_identity_guard.go:refuseBootstrapIdentityMovefails the plan when the resource that is already the bootstrap is planned under a differentname, or whenbootstrapflips either way on an existing node.
Both guards are correct given the current model. They exist because the
alternative was worse: name, nic and runtime all carry
RequiresReplace, so a bootstrap that slid onto a joined node produced
shc node forget <healthy voter> followed by shc init, which fails
X400021ALRCLS after the eviction. Half-applied estate, no rollback.
So today: removing the init node from terraform is impossible, by construction, through both doors. That is not a bug in the guards. It is the guards correctly defending a model that made a create-time accident permanent.
Three things the current surface conflates
Section titled “Three things the current surface conflates”| what it is | who owns it | how it changes | terraform’s job | |
|---|---|---|---|---|
| infrastructure fact | these nodes exist, at these addresses | terraform | on apply | declare it |
| cluster history | one node ran shc init, once | the daemon | never | nothing |
| runtime role | who leads raft now | raft | on its own, without terraform | read it, never declare it |
bootstrap_node, bootstrap_node_name and shc_node.bootstrap are all
one field trying to be all three.
The daemon does not record cluster history, and does not need to
Section titled “The daemon does not record cluster history, and does not need to”This is the finding that makes the whole thing tractable. Nothing in the
daemon knows which node ran shc init. Verified:
ce/ent/schema/node.go:27-95: the node registry has no genesis column, nois_bootstrap, nofounded_at.ce/modules/node/commands/bootstrap_file.go:30-60: the node-localnode_bootstrap.jsonsidecar carriesruntime,node_name,node_id,nic,bind_ip,public_address, swarm identity,raft_tls,fabric, and no genesis flag. The init node’s sidecar is shape-identical to a joiner’s.ce/modules/cluster/cluster_ca_store.go: the cluster CA’s public root lives in the REPLICATEDcertificatestable (name="cluster") and its private key in the vault’s protectedca/section. It does not live on the init node.- The nebula lighthouse defaults to the raft leader’s address
(
shc/modules/cluster/mesh_orchestrator_impl.go:181-186), not the init node’s.
The genesis asymmetry is entirely terraform’s own invention. One real artifact survives genesis on the init node, and it is a provisioning artifact rather than cluster history; see the vault exception.
The model: seed, not bootstrap
Section titled “The model: seed, not bootstrap”Terraform should not export a bootstrap concept at all. The replacements:
# INPUT: create-time only, and inert on every subsequent plan.seed_node_name = optional(string) # was bootstrap_node_name
# per nodeseed = optional(bool) # was bootstrap; NOT RequiresReplace, # NOT an identity, never read after Create# OUTPUTSnodes # the infrastructure fact, declared + observedssh_hosts # stable, arbitrary order; see below# bootstrap_node : DELETEDFour rules make seed mean “one of them went first” instead of “this one
is special forever”:
Rule S1: seed never forces replacement, and is never read after
Create. Dropping a RequiresReplace plan modifier makes strictly fewer
things replace, so it is free on both live estates. The provider’s
ModifyPlan refusal of a bootstrap flip goes with it: under the new
Create algorithm a flip is not destructive, so there is nothing to refuse.
Rule S2: Create is symmetric and adopt-first. Every shc_node,
seed or not, runs the same decision at apply time:
probe the cluster over the many-host ssh transport cluster exists AND lists this name healthy -> ADOPT (no cluster verb) cluster exists -> JOIN no cluster answers anywhere -> INITThe JOIN path already does exactly this:
node_join_recovery.go:65-115 catches X400021ALRCLS, cross-checks
membership, and adopts a healthy member rather than wiping it. The INIT
path does not: bootstrapSSH (node_resource.go:1384-1505) runs
shc init unconditionally. Giving the init path the same recovery is the
single change that unlocks rotation.
Rule S3: the create-time serializer is terraform’s DAG, not a node
name. Something must go first on a from-empty apply or three nodes each
genesis their own cluster. That is a concurrency detail and it stays one:
the formation edge points at whichever node seed selected for this
apply, and after Create nothing references it again. Note that SHC’s
own row-create-is-authority guard (certificates.name UNIQUE,
cluster_ca_store.go:24-31) prevents a forked CA within one cluster;
it does not prevent two nodes forming two separate clusters. The DAG
is the real serializer and the design should say so rather than imply the
daemon defends it.
Rule S4: ssh_hosts stops claiming leader affinity. The provider
already tries entries in order until one connects and is willing to
serve the operation, skipping a follower’s not leader
(ssh_hosts.go:526-607), and it remembers the winner in
lastGoodSSHHost and reorders by it on the next call
(orderBySSHCache:325-347). So terraform’s ordering is an optimisation on
the first call of a session and irrelevant afterwards. It should emit a
stable, arbitrary order (node name ascending), documented as
arbitrary: stable so plans do not churn, arbitrary so nobody builds on
it. The address-less filter and the folded-in SSH user stay: those are
real semantics and they are the fix for two live failures on 2026-07-27.
And the one consumer that actually wanted something: three estate
roots use bootstrap_node.public_ipv4 to fill dns.public_address. They
never wanted the init node; they wanted a cluster entrypoint. That
becomes entrypoint (see the glossary’s note b): derived from
the node set, explicitly arbitrary and stable, and ultimately deleted when
slice 4’s cluster posture lets the daemon’s own DNS ladder answer from
node rows.
What a from-scratch apply does against an EXISTING live cluster
Section titled “What a from-scratch apply does against an EXISTING live cluster”The harder half of the owner’s question, answered from the code. Four shapes: two refuse safely, and two fail silently, one of them by orphaning a live estate:
1. State intact, init node removed from nodes. → HARD WALL, no
damage. Either seed_node_name/bootstrap_node_name is pinned and the
module precondition fails the plan (modules/core/main.tf:324-329), or it
is unpinned and defaults to nodes[0].name, which now names a different
machine, and the provider’s refuseBootstrapIdentityMove fails the plan.
Nothing applies. Nothing breaks. You simply cannot proceed. This is
the corner, and it is the common case.
2. State lost, machines alive, init machine alive. → Apply FAILS, no
damage. bootstrapSSH runs shc init, the daemon refuses
X400021ALRCLS (ce/modules/cluster/errors.go:31-34), the apply errors
out. Note the asymmetry: a join node in the same position converges by
adopting; the seed node cannot, because the init path has no
recoverAlreadyClustered.
3. ⚠ State lost, machines gone or rebuilt. → SECOND CLUSTER. Terraform
reports SUCCESS. Terraform creates fresh VMs, shc init runs on a
virgin machine and succeeds, the other fresh VMs join it, and the
surviving original cluster (with all of its data, its vault, its raft
log) is orphaned and invisible to the new state. Nothing errors. Nothing
warns. This is the data-loss shape, and it is loud only because this
document says so. It is exactly the shape the join path already defends
against and the init path does not.
4. ⚠ Bootstrap VM replaced out-of-band, state intact. → Silent
no-daemon node. hostRequiresReplaceIfJoin sets
RequiresReplace = !bootstrap (node_resource.go:340-350), so a changed
host on the seed node re-derives the endpoint in place and never
re-runs anything. The resource happily points at a machine that has no
shc package on it at all. Update (node_resource.go:765-781) rolls a
package and repairs public_address; it does not check that a cluster
exists there.
Rule S2 fixes 2, 3 and 4 in one stroke, because “probe first, then decide” turns all three into either an adopt or an honest refusal. It is therefore not a convenience; it is the fix for a data-loss shape that exists at HEAD.
The one real genesis asymmetry: the vault cold start
Section titled “The one real genesis asymmetry: the vault cold start”There is exactly one thing the init node has that no other node does, and it is a provisioning artifact rather than cluster history:
writeVaultAutoUnlockDropIn (ssh_engine.go:918-960) persists the vault
master password into a 0600 root-only systemd drop-in on the bootstrap
node only. vault_autounlock_leader_only_test.go enforces the single
call site structurally, and the posture is deliberate and correct: a
joiner gets nothing, comes up locked, and pulls the unlock from an
already-unlocked peer at runtime (ce/modules/vault/unlock_pull.go). The
password never lands on a follower’s disk. Live-confirmed on
bitspur/hel1 swarm01.
The consequence nobody has written down: the pull requires an
already-unlocked peer. So if every node restarts at once (a power
event, a rolling reboot that overlaps, a hypervisor maintenance window)
and the init node is gone, no node can cold-start the vault and the
whole control plane comes up X503906VLTLCK until an operator runs
shc vault unlock -p by hand.
Rotating out the init node therefore silently removes the cluster’s only cold-start capability, and terraform has no way to move it.
The honest framing is that cold-start is a cluster capability (“at least one node can unlock without help”) and not a node identity. Two ways to express it, and this needs a ruling (Q9):
- (a) Declare the capability.
vault_passwordmoves fromshc_node.bootstraptoshc_cluster, and the module plants the drop-in onvault_cold_start_nodesnodes (default 1, chosen stably by name). Rotation then re-plants before it removes. This preserves “not every node” but changes a deliberately-pinned security posture, so it is the owner’s call, not an engineer’s. - (b) Remove the need. A quorum-derived unseal (the password reconstructible from a threshold of nodes) makes every node able to cold start and no node special. Strictly better, strictly larger, and out of scope for this proposal.
Doing neither is also a choice, but then the runbook must say, in capitals, that the init node is the cluster’s cold-start node and retiring it costs that capability.
Node lifecycle and raft suffrage
Section titled “Node lifecycle and raft suffrage”The owner: “you have node a, b, and c, and when you terraform apply, it does the init on one of them, we don’t know which one. It should be opaque… and then the other two join, and then I should be able to join more and remove others, INCLUDING the one that did the init, whenever I want to. I should be able to rotate nodes in and out.”
Genesis opacity is answered above. This section is the other half: what the cluster does to itself when membership changes.
Node count and voter count are ALREADY decoupled
Section titled “Node count and voter count are ALREADY decoupled”This is the answer to the owner’s worry that “we can never have an even number of voters, but I do want an even number of nodes”, and nothing in the docs says it, which is why the worry exists.
RaftTargetVoters = 3 (ce/modules/cluster/types.go:374).
CalculateOptimalVoters(nodeCount, maxVoters)
(ce/modules/cluster/voters.go:50-67) is an odd schedule: 0→0, 1→1,
2→1, 3–4→3, 5+→min(maxVoters, 5).
ShouldJoinAsNonVoter(currentVoters, targetVoters)
(voters.go:43-48) makes every node past the third join as a read-only
non-voter, and the enroll handler applies it
(shc/modules/cluster/enroll_handler.go:364-370), falling back to voter
when the live count is unreadable.
So:
| nodes | voters | non-voters | quorum | tolerates |
|---|---|---|---|---|
| 1 | 1 | 0 | 1 | 0 |
| 2 | 2 † | 0 | 2 | 0 |
| 3 | 3 | 0 | 2 | 1 |
| 4 | 3 | 1 | 2 | 1 |
| 5 | 3 | 2 | 2 | 1 |
| 8 | 3 | 5 | 2 | 1 |
† The 2-node cluster is the one place the shipped behaviour and
the documented target disagree: CalculateOptimalVoters says 1, but
suffrage is decided at JOIN time and the second node joins while
currentVoters == 1 < 3, so it joins as a voter. voters.go:36-42
acknowledges this as a deliberate accepted transient: “the transient
even set while GROWING toward the target… cannot be avoided without a
runtime remove+rejoin rebalancer, which is out of scope by design”.
Two voters means
quorum 2 of 2 and zero fault tolerance, which is why the DR guidance is
“three nodes or one”, and the model should say that where an operator will
read it.
An even node count is fine. Four nodes is three voters plus one non-voter. This needs to be stated in the model and in the runbook, because the fear it dispels is real and the docs are silent.
Removing a VOTER: nothing promotes, and nothing can
Section titled “Removing a VOTER: nothing promotes, and nothing can”This is the part nobody had verified. The answer is worse than “no promotion exists”: promotion is not implementable on the shipping engine.
Service.PromoteNode/DemoteNodeexist (ce/modules/cluster/service.go:1621-1653) and are declared on the cluster-ops interface (interfaces.go:32-33).- They have zero callers: no HTTP route, no CLI verb, and no removal path invokes them. Grep across ce and shc finds only the declaration, the definition, and doc comments.
- Both funnel into
changeNodeRole, which ends atLeaderInfo.Assign(ctx, targetID, role). rqliteLeaderInfo.Assign(shc/subsystems/rqlite_adapters.go:84-90) returns a typed error unconditionally: “rqlite decides voter/non-voter at join time; remove and rejoin to change a role.”
So the promote/demote verbs are dead code that would fail on any real
cluster, and ShouldJoinAsNonVoter’s own comment names the gap: it is
“a JOIN-TIME-ONLY policy… a runtime remove+rejoin rebalancer, which is
out of scope by design”.
The concrete consequence. A 4-node cluster is 3 voters + 1 non-voter. Remove a voter:
EnsureVoterRemovable(ce/modules/cluster/service.go:1715-1810) computesvotersAfter = 2,requiredAfter = 2, and allows it.- The eviction runs. The cluster is now 3 nodes: 2 voters, 1 non-voter.
- Quorum is 2 of 2. Fault tolerance is zero. Losing either voter wedges the cluster.
- The non-voter is never promoted. Nothing notices.
- The next voter removal is refused by the floor guard, correctly, but by then the cluster has already spent its fault tolerance silently.
Status already computes the signal:
ExpectedVoters = CalculateOptimalVoters(healthy, RaftTargetVoters) and
VoterDeficit = raftVoters < ExpectedVoters (service.go:1606-1607),
surfaced by shc cluster status (commands/status.go:71-75) and (good
news) already exposed to terraform as
data.shc_cluster_status.voter_deficit / .expected_voters
(cluster_status_data_source.go:36-38). The deficit is computed,
rendered, and acted on by nobody.
What should happen. Since Assign cannot work, the only rebalance
available is remove-and-rejoin of a NON-voter, which is cheap in principle
(a raft membership change plus that node’s storage-engine restart; no app
data moves) and always legal (removing a non-voter never trips the floor
guard). Three levels, for the owner to pick between (Q10):
- (a) Report only. The removal’s own output names the deficit and the
fix. Zero new machinery;
VoterDeficitstops being invisible. - (b) Refuse by default. A voter removal that would leave
raftVoters < ExpectedVotersrefuses unless the caller passes--rebalanceor--accept-deficit. Symmetric with the deployment gate (below) and with the model’s overall “refuse, do not auto-move” posture. - (c) Rebalance.
--rebalanceperforms remove+rejoin on one healthy non-voter BEFORE evicting the departing voter, so the voter set never transiently drops below target. This is the real fix and the only one that makes “rotate nodes in and out whenever I want” true.
Recommendation: (b) now, (c) as its own slice. (a) alone leaves the operator to discover a zero-fault-tolerance cluster from a status field.
The quorum-loss deadlock, stated plainly
Section titled “The quorum-loss deadlock, stated plainly”Raft membership changes are themselves committed through raft. So a
cluster that has lost quorum cannot vote to remove the dead voters that
cost it quorum. Lose 2 of 3 voters and the survivor can do nothing: the
eviction entry could never commit, and the guard refuses before it tries.
Note which guard; the arithmetic here is subtle and worth stating,
because the obvious reading is wrong. The configured count is unchanged by
a crash (a dead peer still reports Role == voter), so votersAfter = 2
and requiredAfter = 2 and the first gate would pass. It is the SECOND,
reachability-aware gate that refuses: reachableAfter = 1 < 2 →
X409202CLSRMR (ce/modules/cluster/service.go:1798-1827). That gate
exists precisely because the configured count cannot tell the safe 2→1
shrink from the wedge, and it FAILS CLOSED when no liveness signal is
available at all.
shc cluster repair sweeps members that are unreachable or unseen for
24h and removes them while quorum still holds (
shc/modules/cluster/admin_router.go:57-110); it runs the floor guard first and, on refusal, renders the escalation rather than a bare error.
shc cluster repair --force → Service.ForceLeader
(ce/modules/cluster/service.go:2089-2196) is the destructive escape
hatch: it stops the engine, rewrites raft membership on disk to a
single-server configuration, and restarts. It refuses when a leader OTHER
than itself is reachable (X403003LDRNLY), a genuine guard, and also the
statement of its danger: it is a divergence machine. Against a live
but partitioned peer that the local node merely cannot see, it produces
two nodes each believing they are the cluster, and whichever one loses is
rolled back to the winner’s log. --remove-other-nodes additionally
evicts every other member, and previously-removed dead nodes cannot be
“un-evicted”; their raft state is gone.
The rule the runbook must carry: force_leader is for a cluster that is
already dead, never for one that is merely slow.
Refuse, do not auto-move, at the terraform boundary
Section titled “Refuse, do not auto-move, at the terraform boundary”The owner: removing a node that still has app deployments “should fail… you would have to actually move those deployments… it shouldn’t just auto move things.”
There is a real tension here, and it resolves cleanly once the two verbs are separated.
shc node drain ALREADY relocates, and that is correct. (The verb is
shc node drain, registered at CLI path node drain. The API method behind
it is shc.cluster.drain, and the mismatch is a live trap: three separate
pieces of work this week wrote shc cluster drain into operator-facing text
by mirroring the method path instead of the command, and one reached the
public generated error reference. There is no shc cluster drain.)
shc/modules/cluster/drain.go flips availability to draining, lists
running deployments, picks a target per row by runtime / lowest deployment
count / availability=active, and asks a DeploymentMigrator to relocate
(backup, cross-node ship, bring-up), then waits for the node’s deployment
list to empty. Default budget 300s, and the file’s own header warns that
“a many-stack drain may need a larger budget than the default”.
Relocation is the verb the operator asked for. It should stay exactly as
destructive as it is.
terraform destroy of an occupied node must REFUSE. And here is the
defect: today it does not, and it does not even go through the route that
would.
shc cluster rmHAS the gate: unlessforce=true, it lists deployments on the node and refuses withX400017NODDEP“node has stacks, rerun with force=true to override” (admin_router.go:171-183).shc node forgethas no deployment gate at all; its handler callsNodeRemover.Remove(ctx, m.ID, true)with force hardcoded (shc/modules/cluster/lifecycle_router.go:305), on the stated ground that forget must stay idempotent for reaping half-orphaned rows.- The provider’s
Deletecalls forget:buildNodeForgetArgvrendersnode forget <name> --yes(ssh_platform.go:586-597), and the HTTP fallback sendsForgetRequest{Yes: true}(node_resource.go:1042-1045). rqliteNodeRemover.Remove(subsystems/rqlite_adapters.go:143-180) evicts the raft member and marks the rowremoved. It does not touch deployments at all, so the rows are not migrated and not deleted; they are left pointing at a node that no longer exists.
So terraform destroy -target=... on an occupied node evicts it from
raft, orphans its deployment rows, and reports success. That is precisely
what the owner says must not happen.
Is the refusal even reachable? Yes; the ordering already works.
shc_node reads its host (and its ssh) from the server resource, so
terraform’s dependency graph destroys the shc_node before the VM,
and Delete’s own comment relies on this: it releases managed DNS and
leaves raft “while the daemon on this node is still alive, which it is
only until this resource returns” (node_resource.go:988-1017). The
daemon is up and reachable at destroy time. The provider simply asks the
wrong verb.
The proposal:
- The provider’s
Deletestops force-forgetting. It calls the gated removal, and onX400017NODDEPfails the destroy with a diagnostic that names the deployments; the operator then runsshc node drain <node>explicitly, or passes an explicit escape. - The escape is a per-resource
on_destroy = "refuse" | "force"(defaultrefuse), not a provider-wide flag, so a deliberate teardown of a whole estate can still say so once per node. shc node forgetkeeps force=true and keeps its reaping job; it is the verb for a node that is already gone, and gating it would break the one thing it exists for.draingets no terraform surface. Relocation is an operator decision with a wall-clock budget and a real failure mode; expressing it as a side effect ofterraform destroyis how “it shouldn’t just auto move things” gets violated by accident. Today the provider has no drain call at all; that is correct and should stay.
The already-gone machine
Section titled “The already-gone machine”Graceful removal cannot run on a VM that no longer exists. The force path, and when each step is right:
| situation | verb | why |
|---|---|---|
| node is up, has stacks | shc node drain <node> then destroy | relocation is explicit; destroy then finds it empty |
| node is up, empty | terraform destroy (or shc node leave on it) | gated removal succeeds |
| node is DOWN, cluster has quorum | shc node forget <node> | idempotent reap; the floor guard still applies and refuses if the reachable survivors could not reach their own quorum |
| several nodes DOWN, quorum holds | shc cluster repair | sweeps unreachable / stale >24h, floor-guarded |
| quorum LOST, cluster is dead | shc cluster repair --force | rewrites membership to single-server; divergence risk; see above |
| terraform state says a node exists and nothing does | terraform state rm then re-apply | no cluster verb is issued either way |
The row that does not exist today and should: “node is up, has stacks, and terraform is destroying it” currently lands in row 3’s behaviour without row 3’s justification.
The node-rotation runbook
Section titled “The node-rotation runbook”docs/guide/operations/ has cluster-setup, upgrades, disaster-recovery,
vault, storage and observability. It has nothing on rotating nodes in
and out. Removal appears twice: cluster-setup.md:160-178 and
disaster-recovery.md:161-211, both framed as failure recovery, neither
as routine capacity management, and neither mentions the init node, the
voter deficit, or terraform at all.
Writing it is not a deliverable of this proposal. Specifying it is.
docs/guide/operations/node-rotation.md must contain:
- The three facts up front. Node count ≠ voter count (with the table
above). An even node count is fine. Any node can be removed, including
the one that ran
shc init, once this model lands, and not before. - Add a node. What suffrage it gets and why (
ShouldJoinAsNonVoter), and that it will be a non-voter past the third, which is normal, not a degraded state. - Remove a node, the happy path. Drain explicitly, verify the deployment list is empty, then destroy. Name the 300s default budget and how to raise it for a many-stack node.
- Remove a VOTER. Read
shc cluster statusbefore and after. Whatvoter_deficitmeans. The remove-and-rejoin rebalance of a non-voter, step by step, until (c) above lands and automates it. - Remove the node that ran
shc init. The cold-start consequence (vault), what must be re-planted first, and how to verify it: the single most dangerous rotation and today the only one that is documented as impossible. - The already-gone machine. The table above, verbatim.
- Quorum loss. The deadlock, why
force_leaderrefuses when a leader is reachable, and the divergence risk when it does not. - Terraform-driven rotation end to end. Edit
nodes, plan, read the plan (what a legitimate rotation looks like versus what a destructive one looks like), apply, verify withshc cluster statusanddata.shc_cluster_status. - What will refuse, and what to do about it. Every guard by code:
X400017NODDEP,X409201CLSRMQ,X409202CLSRMR,X403003LDRNLY,X400021ALRCLS.
Proof obligation for the runbook itself: a lane that rotates every node of a three-node cluster out and back in, one at a time, including the seed, with the cluster serving throughout.
Migration
Section titled “Migration”Every attribute whose change forces node replacement
Section titled “Every attribute whose change forces node replacement”Replacement of an shc_node means shc node forget <name> followed by
shc node join, evicting a live raft voter. bootstrap_identity_guard.go:3-33
documents the half-applied-estate incident this produced. The full list, at
HEAD:
| attribute | file:line | replace trigger | in the model? |
|---|---|---|---|
shc_node.bootstrap | node_resource.go:165-174 | RequiresReplace (Bool) | untouched |
shc_node.name | node_resource.go:237-241 | RequiresReplace | untouched |
shc_node.nic | node_resource.go:242-246 | RequiresReplace | name and meaning both preserved; see rule A |
shc_node.runtime | node_resource.go:247-251 | RequiresReplace | untouched |
shc_node.host | node_resource.go:176-191 | RequiresReplaceIf join node (hostRequiresReplaceIfJoin, :334-345); bootstrap re-derives in place | renamed to management_address; value identical, see rule B |
shc_node.public_address | node_resource.go:252-258 | none; in-place SSH repair (:805-838, :904-935) | untouched |
shc_node.private_ip | node_resource.go:259-267 | none, and inert after create | renamed to internal_address, must stay non-replacing; see rule C |
shc_node.vault_password | node_resource.go:~280 | change REFUSED in place | untouched |
Can the model be reached without recreating nodes on the two production estates?
Section titled “Can the model be reached without recreating nodes on the two production estates?”Yes, for the rename. No, for the internal_address declaration,
unless the daemon gains a repair path first.
Three rules make the difference, and each is a hard gate on a slice:
Rule A: nic keeps its name AND its meaning. It is RequiresReplace
and it is the only handle on internal_address today. In the model it
means “the interface internal_address is derived from when
internal_address is null”. Existing estates set nic and leave
internal_address null, so behaviour is byte-identical, the value never
changes, and there is no diff. Re-semanticing a RequiresReplace
attribute is the single most expensive mistake available here: it replaces
every node on promanager/hel1 (3x cx53) and bitspur/hel1 (3x cx53)
simultaneously.
Rule B: every provider attribute rename ships a schema Version bump
plus an UpgradeState that carries the old value across. The precedent
mechanism exists and works: state_upgrade_ssh_host.go:82-130 derives the
legacy model from the live schema (~30 lines, not eleven hand-copied
schemas), and shc_node is already at Version: 1 for exactly this
reason. host → management_address and private_ip →
internal_address are both value-preserving renames; once the state is
upgraded the plan is a no-op and RequiresReplaceIf sees no change.
Skipping this is the 2026-07-27 failure repeated: both estates frozen
at plan, and the file that documents it says the lesson in capitals:
“CONFIG MIGRATION IS NOT STATE MIGRATION.”
Rule C: internal_address must never be RequiresReplace. If it is,
the first estate that declares one replaces every node. Making it
in-place-repairable requires a daemon change first, because today there is
no path to correct the column at all:
- add a targeted
UpdateInternalAddresswriter (the repository has five siblings and not this one); - add
--internal-addresstoshc initandshc node join, threaded to the enrollphysical_ip, withnic.Detectas the fallback when unset (today only the join path has an undocumentedSHC_ADVERTISE_ADDRback door, andshc inithas nothing); - re-issue the internal-API mTLS leaf on change; the leaf’s only IP SAN
is
host(internal_address)(enroll_handler.go:316-334), so without this a corrected address fails TLS verification and the repair deadlocks; - then the provider gets the same in-place SSH repair shape
public_addressalready has.
The tarball coupling
Section titled “The tarball coupling”Both ops repos pin a versioned module tarball
(.../terraform-modules/0.0.14/shc-hetzner-0.0.14.tgz on
promanager/hel1; 0.0.7 on the older lanes). So the module and the estate
root must move in the same commit: bump the pin and rewrite the root’s
nodes/outputs together, or the root references outputs the pinned
tarball does not have.
Confirmed live consumers that the rename touches: node_public_ipv4s is
re-exported by six estate roots, and is index-pinned at
promanager-ops/openstack/us-west-or/infra/main.tf:166 and
promanager-ops/azure/eastus/infra/main.tf:197, where
module.shc.node_public_ipv4s[0] feeds dns.public_address, positional,
and on the openstack lane it would write an RFC1918 address if floating
IPs were ever disabled. Those two become
module.shc.bootstrap_node.public_address in the same pass.
Neither estate pins bootstrap_node_name in its own root, so fixing the
bootstrap_node output to resolve through the pin (rather than
local.all_nodes[0]) changes no live value today; it is a free
correctness fix, and it should land before anyone starts pinning.
One thing object-typed variables will not do for you: OpenTofu treats
an unknown attribute on an object-typed variable as a warning, not an
error. promanager-ops/hetzner/hel1/infra/main.tf:169 has been carrying
firewall = { inter_node_open = false } (an attribute deleted in
ab292948f and absent from the pinned 0.0.14 module) silently, for
weeks, with tofu validate reporting Success!. Every renamed field
therefore needs a same-named optional(...) tombstone whose validation
block errors with the new spelling, deleted one release later. This is not
backcompat; it is the house pattern already used for inter_node = "auto"
(modules/shc/hetzner/variables.tf:73-77).
The price, stated plainly
Section titled “The price, stated plainly”- Rename only (slices 0–1, 3): zero node replacements, both estates, provided rules A and B hold.
internal_addressas a declaration (slice 2): zero replacements if rule C holds, but rule C is gated on a ce change, so slice 2 cannot land before it. If it is attempted without rule C, the price is six raft evictions across two production clusters, serialized, each one the half-applied-estate class.- Everything downstream (slices 4–6): additive, computed, or cluster-scoped. No replacements.
What the revision adds to the price
Section titled “What the revision adds to the price”Address-family containers (slice 8): zero replacements, one state
upgrade. host → management, private_ip → internal,
public_address → public are all scalar→object shape changes. The
provider’s UpgradeState mechanism handles a shape change exactly as it
handles a rename; state_upgrade_ssh_host.go:82-130 already migrated a
scalar ssh.host into a hosts LIST, which is the same class of move and
is the working precedent. RequiresReplaceIf on management compares the
resolved value, so a migrated state plans as no change.
De-identifying genesis (slice 9): zero replacements, and it REMOVES
replacement triggers. The changes are: drop RequiresReplace from
bootstrap (strictly fewer replacements), rename bootstrap → seed
(value-preserving, one UpgradeState), delete
refuseBootstrapIdentityMove and the module precondition (both are plan
refusals; deleting them cannot damage state), delete the bootstrap_node
output (a same-commit fix in three estate roots), and add the adopt-first
probe to bootstrapSSH (Create-path behaviour only; on both live estates
the seed node already exists in state, so Create never runs again).
name, nic and runtime keep both their names and their
RequiresReplace, the same Rule A discipline the addressing rename
follows, for the same reason.
Net: the whole revision is reachable on promanager/hel1 and
bitspur/hel1 with ZERO node replacements, and it reduces the number of
attributes that can ever force one. The costs that remain are the two
things that were always going to be expensive and are both gated on ce
changes: internal as a declaration (rule C) and the vault cold-start
capability (Q9).
Sequencing against the estates. The ops repos pin a versioned module
tarball, so every slice that changes an output moves the module and the
estate root in the same commit. Slice 9 additionally removes an output
three roots read (bootstrap_node), so it is the one slice that cannot be
landed module-first and rolled later.
What the model cannot do
Section titled “What the model cannot do”Every design forecloses something. Named:
IPv6. The model’s names are family-agnostic (that is the point of
killing public_ipv4), but the model does not deliver IPv6, and
nobody should read it as doing so. The foreclosure is structural, in at
least six independent places: nic.Candidates skips every address where
ip.To4() == nil (ce/core/nic/nic.go:98-101); a v6-only host takes the
warning branch at ce/modules/cluster/service.go:769-771, leaves
bindIP == "", and writes a NULL internal_address; genesis builds
host:port with fmt.Sprintf rather than net.JoinHostPort
(service.go:850); three of five host-splitting helpers truncate a bare
v6 literal at the last colon (cluster.StripPort,
cluster.StripPortIfPresent, cluster.hostOnly), including the one the
fabric endpoint map uses; the fabric CIDR validator rejects a v6 supernet
(ce/modules/fabric/types.go:119-121); and the provider’s own
splitHostPort silently turns 2001:db8::10 into host 2001:db8: port
10, verified by executing the helper. Substrate modules meanwhile
provision IPv6 end to end (hetzner/main.tf:412, vultr/main.tf:512,
::/0 firewall rules on aws and linode) and surface it nowhere. This
model makes IPv6 possible later without a second rename. It does not make
IPv6 work.
Multi-NIC. One internal address per node. The daemon’s registry
assumes it; rowHostShared (ce/modules/cluster/service.go:1874-1892)
treats two rows sharing HostOnly(internal_address) as the
address-aliased-zombie shape and refuses voter removal. A node with a
separate storage plane and control plane is not expressible, nor is a
per-role NIC. Only hetzner even has a per-group nic_name today
(hetzner/main.tf:119), and every substrate emits a single scalar
nic_name output that is a lie for exactly that one.
Multiple nodes behind one NAT. Forbidden by the same invariant. Three
home nodes behind one edge IP cannot join a WAN cluster even with a
declared internal_address, because they would share a host, and the
model has no per-node port field to separate them. Accepting this as a
permanent limit is an owner call (Q4).
A node whose address changes under it. The model makes drift
visible (observed_internal_address) but not self-healing. Raft and the
internal API both re-resolve --nic live on every boot
(rqlite_engine.go:1255-1270, internal_api.go:195-215), while
node_self_register.go:177 re-stamps the row from the frozen sidecar
bind_ip. After a DHCP renew the node binds the new address and
advertises the old one to every peer. Rule C fixes the terraform-declared
case; it does not fix the case where reality moves and terraform does not
know.
The management role has no daemon representation, so there is no
drift detection on it and shc node update cannot repair it. If the
management address goes stale, terraform is locked out. That is no worse
than today, but the model does not fix it either.
The overlay still cannot bootstrap itself. Both overlays derive their
endpoints from internal. An overlay can therefore never be the thing
that makes an unreachable node reachable. Declaring internal is the
escape hatch; the overlay is not.
Four address detectors, not one. core/nic/nic.go claims in its
package comment to be “the single source of truth”, and is not:
modules/node/commands.AdvertiseAddress (local_token.go:58-76),
modules/app.AdvertiseAddress (dns_lifecycle.go:513-541) and
modules/network.DefaultNICResolver (state.go:95-129) each have their
own precedence, their own env var (SHC_ADVERTISE_ADDR, SHC_NIC), and
two of them fall back to 127.0.0.1. The model does not collapse them;
that is adjacent cleanup, and it is called out as slice 2’s companion
because the loopback fallback on the join path produces a durable,
unrepairable, loopback-SAN node.
Runtime voter rebalancing, on the shipping engine. Assign is
unimplementable on rqlite (subsystems/rqlite_adapters.go:84-90), so
“promote a non-voter” can only ever be remove-and-rejoin. That is a real
operation with a real cost (the rejoining node’s storage engine
restarts), and it can only run while quorum holds. A cluster that has
already spent its fault tolerance cannot rebalance its way out.
A voter set larger than five. CalculateOptimalVoters caps at
min(maxVoters, 5) and RaftTargetVoters is 3. A twenty-node cluster is
three voters and seventeen non-voters. That is the correct raft answer and
it is also a hard ceiling on control-plane fault tolerance that no amount
of nodes buys past.
Two clusters formed by two concurrent applies. Nothing in the daemon
prevents two nodes each running shc init and forming separate clusters.
The row-create-is-authority guard on the cluster CA prevents a forked CA
within one cluster; it says nothing about two clusters. Terraform’s DAG
is the only serializer, so a hand-rolled root that wires no dependency
edge between its nodes can still produce two clusters. The model makes
this visible (seed is create-time-only and the edge is explicit) but
does not defend against it.
Rotation while the cluster is degraded. Every guard in this model
depends on being able to READ live raft membership: EnsureVoterRemovable
queries LeaderInfo.Members, the deployment gate queries the deployment
list, and the adopt-first probe queries membership over SSH. On a cluster
that has lost quorum all of them fail, and rotation stops being available
exactly when it is most wanted. force_leader is the answer and it is a
different kind of operation with a different kind of risk.
Slices
Section titled “Slices”Each slice is independently landable and independently provable. Proof obligations are explicit; a slice is not done until its proofs run.
Slice 0: the live bugs, no contract change
Section titled “Slice 0: the live bugs, no contract change”Four fixes that are correctness today and that the model would otherwise inherit:
bootstrap_noderesolves throughlocal.bootstrap_node_nameinstead oflocal.all_nodes[0], in all fourteen substrate modules, and gainspublic_address, which is what its three production consumers actually want. (hostinger’s docstring already claims it honours the pin;hostinger/outputs.tf:6-13does not.)- Each substrate’s
cluster_ssh_hostsre-orders bootstrap-first the waymodules/core’s does (core/main.tf:141-157), or its docstring stops claiming “BOOTSTRAP FIRST”. Both currently say it; only core does it. - The plane-NIC readiness gate stops disagreeing with itself: it
validates with
net.ParseIP(accepts v6) and probes withip -4 addr show(ssh_engine.go:597,:621), so a v6 value hangs the full five-minute timeout and then blames the image. Andgrep -qFis a substring match, so10.0.0.1is satisfied by an interface holding10.0.0.10. sshNodeRow(ssh_platform.go:512-522) gains the fieldsapiclient.NodeListRowhas and it lacks:public_address,swarm_node_id,swarm_role,cpu_cores. The comment atnode_resource.go:1203asserts the two wire shapes are field-identical and it is false. Harmless today because both are discarded; it becomes a live bug the instant slice 3 surfaces anything, because both live estates use the SSH transport.
Proof: unit tests for the ordering fix with a pinned non-first
bootstrap; a substring-collision test for the gate; a reflective test
asserting sshNodeRow and NodeListRow carry the same fields (the ratchet
shape already used for the enroll request bodies).
Slice 1: the vocabulary
Section titled “Slice 1: the vocabulary”LANDED 2026-08-14 (ruling M10), together with slice 8.
The rename, value-preserving, across modules/core + all fourteen
substrates + the provider + both ops repo roots. nodes gets an explicit
type (it is untyped in the substrate outputs today, which is what let
fourteen modules drift). Tombstone validations on every removed field. The
four flat projections deleted.
Proof: tofu plan against both estates’ state after the state upgrade
shows no changes, not “no replacements”, no changes at all. That is
the whole gate. Plus a fixture proving the tombstone converts OpenTofu’s
“Object attribute is ignored” warning into a hard plan failure (the
inter_node_open class).
Slice 2: internal_address becomes a declaration
Section titled “Slice 2: internal_address becomes a declaration”ce side: --internal-address on init and join; UpdateInternalAddress in
the node repository; internal-API leaf re-issue on change; collapse the
four detectors onto core/nic; delete the 127.0.0.1 fallbacks on any
path that produces a durable address. Provider side: internal_address as
Optional, non-replacing, with the in-place SSH repair shape
public_address already has.
Proof: a two-node cluster where the joiner declares an
internal_address it does not hold on any interface, joins, replicates,
and rotates its cert; then changes it and converges without a replace.
A negative test that a join with no usable IPv4 and no declared address
fails rather than silently recording 127.0.0.1:4003.
Slice 2 also gives X403092ENRSIP something real to validate; it is
defined at shc/modules/cluster/errors_extra.go:275-288, tabled at :520,
and has zero call sites. Wire it or delete it.
Slice 3: the observed set
Section titled “Slice 3: the observed set”mesh_address, fabric_ip, observed_internal_address,
observed_public_address as Computed on shc_node, propagated into the
module nodes output. Zero replacement cost, zero state migration;
Computed attributes never force replacement and never strand state that
lacks them. Depends on slice 0’s fix 4.
Proof: a live estate read where an out-of-band shc node update --public-address appears as drift (today it is invisible forever), and
where mesh_address is readable per node without instantiating shc_mesh.
Slice 4: the cluster posture, and the loud failures
Section titled “Slice 4: the cluster posture, and the loud failures”Derive internet from the node set. Make ACME-on-a-LAN-only-cluster a
plan-time error, make RegisterSystemHostDNS follow the posture instead
of hard-coding internet exposure, and make an RFC1918 public_address a
plan-time error instead of a silent "".
Proof: a from-empty LAN-only cluster (proxmox lane) where the cluster
host record resolves (the thing that does not happen today) and
where bootstrap.acme refuses at plan with a named error.
Slice 5: the fabric becomes declarable
Section titled “Slice 5: the fabric becomes declarable”Correction, 2026-08-02. The first half of this slice landed while this document was being written (
39055cde7, “the WireGuard fabric becomes lane-driveable”).shc_cluster.fabric/.fabric_cidrexist,bootstrap.fabricflows throughmodules/coreonto the seed node’sshc init, and4243/udpis in the substrate plane port sets gated onlocal.fabric_shc(hetzner/main.tf:82,:284). The audits’ “zero provider surface” reading was true at the commit they read and is no longer true. What follows is what REMAINS.
The lighthouse still has no terraform surface at all;
grep -rn lighthouse terraform/provider/internal/provider/*.go terraform/modules/core/*.tf returns nothing. shc cluster mesh lighthouse add exists (shc/modules/cluster/lighthouse_router.go, honoured at
mesh_orchestrator_impl.go:198-212) and is the only place in the whole
daemon where an operator can pin a public address for a control-plane
function. Without it the nebula mixed shape depends on which node wins the
raft election, and under this model’s genesis
opacity that is now explicitly a thing
terraform must not assume.
Proof: a two-node fabric cluster stood up entirely from terraform, cross-node connection working, on a public plane (so the firewall rule is actually exercised), plus a mixed-substrate mesh whose lighthouse is pinned from terraform and survives a leader change.
Slice 6: topology
Section titled “Slice 6: topology”Proven by plan: topology.kind = "lan" with no dns block emits zero
resources; "wan" emits exactly one shc_mesh; topology.public_address
is accepted on all three kinds and honoured on one, silently. It is not a
network model; it is a three-way switch over which platform resources to
create. Zero production estates use it, and their stated reason is
recorded in the config: it models one all-or-nothing connection driving LB
- DNS + LE together, and the estates want DNS without LB
(
promanager-ops/hetzner/hel1/infra/main.tf:254-262).
Split into independently-nullable dns / mesh / lb blocks, or delete
it and bless the root-level-resources pattern both estates already use.
Recommendation: split; deleting discards the shc_stack_ports.traefik
wiring tests/hetzner depends on, and splitting is what would bring both
production estates back onto the module.
Proof: both estates’ hand-composed root resources replaced by module
blocks, tofu plan clean.
Slice 7: IPv6
Section titled “Slice 7: IPv6”Scoped separately and gated on ruling Q3. Nothing in slices 0–6 forecloses it; nothing in them delivers it.
Slice 8: address-family containers
Section titled “Slice 8: address-family containers”LANDED 2026-08-14 (ruling M10), together with slice 1, as the ruling required, one state upgrade rather than two.
Every role becomes {ipv4, ipv6, fqdn} per the
slot table. One UpgradeState
per changed provider attribute, following
state_upgrade_ssh_host.go:82-130, which already migrated a scalar into a
list and is the working precedent for a shape change. Slots that are
accepted-but-not-honoured become named plan-time errors, replacing
today’s silent truncation.
Lands with slice 1, not after it: renaming and reshaping in one state upgrade is one migration; doing them separately is two, and the second one has to migrate the first one’s output.
Proof: tofu plan against both estates’ state after the upgrade shows
no changes at all, the same gate slice 1 carries. Plus a fixture per
row of the slot table: an internal.fqdn refuses at plan; a
management.ipv6 refuses at plan with the truncation named; a
public.ipv6 publishes an AAAA record end to end.
Slice 9: genesis stops being an identity
Section titled “Slice 9: genesis stops being an identity”PARTLY LANDED, 2026-08-15. On main:
- the vault cold start is a DECLARED CLUSTER CAPABILITY:
vault_config’scoldstart_holderrow, theshc vault coldstart replantverb, andEnsureColdStartRelocatedrun by every removal route (shc cluster rm,shc node forget(the verbtofu destroycalls), and thecluster repairsweep) BEFORE their first mutation, orX409906VLTCSH. Route coverage is derived from chi’s route table, so a removal route added tomorrow is swept tomorrow. This was M8’s stated prerequisite; bootstrapno longer forces replacement and is never read after Create;- both plan-time identity refusals are DELETED (the core module’s
bootstrap_node_name-vanished precondition and the provider’srefuseBootstrapIdentityMove), with the daemon’s own named refusals (X409026SECCLS,X409029CLSGEN,X409906VLTCSH,X400017NODDEP, the M9 voter-deficit gate) named as the replacement protection; shc_node.bootstrap’s Delete no longer issues the whole-cluster DNS release when the cluster SURVIVES the removal. That release deletes every record the estate owns, and it was unreachable-and-therefore-safe only while the two identity guards existed; deleting them made the first rotation an operator performs take a live estate off the internet.
What slice 9 still owes, in descending order of damage; under M11 each is a defect, not a corner to document:
shc_node.bootstrapis a SEPARATE RESOURCE fromshc_node.workers, andnamestill forces replacement. So a rotation does not remove one node; it MIGRATES a live node between two resource addresses:shc_node.workers["swarm01"]is destroyed (a realshc node forgetof a healthy member) whileshc_node.bootstrapis replaced ONTO swarm01. The plan is clean and the apply is destructive. The fix is the uniform shape this document has argued for throughout: ONEfor_eachshc_nodekeyed by node name, withseed = (name == local.seed_name)as a plain in-place attribute, and it needs staticmovedblocks in each estate root, which is a state upgrade of the same size as M10’s.local.cluster_refanchors the whole estate on the seed’s outputs (endpoint/admin_token/ca_pemfromshc_node.bootstrap[0]). Every worker, config, connection, deployment and DNS record rides it, so re-creating the seed resource re-derives the control-plane reference for the entire graph.ssh = local.platform_sshis already node-set-derived and fine; the other three are the genesis anchor.output "bootstrap_node"survives inmodules/coreand all fourteen substrates, a POSITIONAL value (local.ssh_ordered_nodes[0]) asserting a one-time historical fact, recomputed every plan. Slice 9 deletes it in favour ofentrypoint; the estates consume it today through the pinned0.0.16module tarball, so the deletion and their move are one program, not a live hazard.cluster_ssh_hostsis still emitted BOOTSTRAP FIRST and still claims leader affinity (“most likely raft leader”) inmodules/coreand the substrate docstrings. Rule S4: stable name-ascending, documented as arbitrary. Note the coupling to (3): the hetzner substrate’sbootstrap_nodereadslocal.ssh_ordered_nodes[0], so the reorder and the output deletion have to move together or the output starts naming the alphabetically-first node.shc-role = bootstrap|workercloud labels in all fourteen substrates. A create-time accident stamped onto the infrastructure as a permanent label; nothing reads it, and after any rotation it lies.bootstrapSSHstill REFUSES rather than adopts.X409026SECCLScloses the data-loss shape (a from-scratch apply genesising a second cluster), which is the important half, but the join path CONVERGES in the same position (node_join_recovery.go) and the init path does not. Rule S2’s adopt-first probe is still owed.bootstrapis still spelledbootstrap, notseed, in the provider schema, the module inputs and the estate roots.
The rest of this section is the original proposal text.
bootstrap → seed, RequiresReplace dropped, bootstrap_node output
deleted (with the three estate roots moved to entrypoint in the same
commit), both plan-time identity refusals deleted, and (the load-bearing
change) bootstrapSSH gains the adopt-first probe that
node_join_recovery.go already implements for the join path.
ssh_hosts re-emitted in stable name order with the leader-affinity claim
removed from its docstring.
Depends on nothing in slices 1–8. It could land first, and there is an argument that it should: it closes a data-loss shape (from-scratch apply genesising a second cluster) that exists at HEAD, and it is the only slice whose absence makes the estates un-rotatable.
Proof: a three-node cluster where the seed node is removed from
nodes, a fourth is added, tofu apply succeeds, the cluster keeps its
identity and its data, and shc cluster status shows the expected
membership. Plus a negative test: an apply with no state against a
live cluster ADOPTS rather than genesising a second one, the shape that
today succeeds while orphaning an estate.
Slice 10: suffrage becomes visible, then correct
Section titled “Slice 10: suffrage becomes visible, then correct”Three steps, each independently landable:
- Visible. Wire
VoterDeficitinto the removal path’s output and into theshc cluster statussummary line an operator actually reads. It is already computed (ce/modules/cluster/service.go:1606-1607) and already exposed to terraform (data.shc_cluster_status.voter_deficit), and acted on by nobody. - Refusing. A voter removal that would leave
raftVoters < ExpectedVotersrefuses unless the caller passes--rebalanceor--accept-deficit. - Correct.
--rebalanceperforms remove-and-rejoin on one healthy non-voter BEFORE evicting the departing voter. This is the only rebalance the shipping engine permits;Assignis unimplementable on rqlite (subsystems/rqlite_adapters.go:84-90).
Same slice, cheap and overdue: delete or wire PromoteNode /
DemoteNode. They are declared, defined, unreachable, and would fail on
any real cluster.
Proof: a four-node cluster (3 voters + 1 non-voter) where removing a
voter leaves three voters, not two, verified from shc cluster status,
not from the removal’s own return value.
Slice 11: destroy refuses, drain relocates
Section titled “Slice 11: destroy refuses, drain relocates”The provider’s Delete stops calling shc node forget --yes for a node
that is up. It calls the gated removal and, on X400017NODDEP, fails the
destroy naming the deployments. on_destroy = "refuse" | "force" per
resource, defaulting to refuse. shc node forget keeps force=true and
keeps its reaping job. drain gains no terraform surface.
Proof: terraform destroy -target on a node carrying a running stack
FAILS with the stack named; the same destroy after an explicit
shc node drain succeeds; and shc node forget on an already-dead
node still reaps it.
Slice 12: the runbook
Section titled “Slice 12: the runbook”docs/guide/operations/node-rotation.md, to the specification
above. It is the last slice because it
documents behaviour slices 9–11 create, and the first thing an operator
will touch.
Proof: the lane in the runbook spec: rotate every node of a three-node cluster out and back in, one at a time, including the seed, with the cluster serving throughout.
Open questions: owner rulings
Section titled “Open questions: owner rulings”These are product decisions, not engineering choices.
Q1. Is a LAN-only estate a first-class topology, or a degraded mode? Today it is neither: it works on the cluster plane and silently half-configures DNS and TLS. First-class means slice 4 plus a proof lane that stays green (a from-empty proxmox cluster whose cluster host resolves and whose internal-CA posture is stated, not discovered). Degraded-mode means slice 4 shrinks to “fail loudly and document it”. The difference is roughly one lane’s worth of ongoing acceptance cost.
Q2. Does terraform become AUTHORITATIVE for internal_address (slice
2), or does it stay an assertion? Staying an assertion is free and
preserves the exact ambiguity this document exists to remove; “what is
this node’s address?” can only be answered by a declaration, not by a hint
plus a probe. It also leaves the mixed/multi-cloud shape permanently
unsolvable, since a NAT’d node can never declare an address it does not
hold. Recommendation: declare.
Q3. Is IPv6 in scope, and at what depth? Three coherent postures:
(a) name-only: this proposal’s rename, so a future IPv6 program needs no
second rename; (b) fail-loud: additionally reject bare v6 at plan with a
named error, since the current behaviour is a silent wrong endpoint;
(c) full support, which means nic.Detect returning a family-tagged
result, one host-splitter, a v6-capable fabric CIDR, and reworking
cidrhost() pinning plus ${addr}/32 firewall sources across five
substrate modules. (a) and (b) are cheap; (c) is a separate program.
Recommendation: (b) now, (c) as its own proposal.
Q4. Is “one node per internal address host” a permanent product
limit? It forbids multiple nodes behind one NAT, the obvious home-lab
and branch-office shape. Lifting it means separating node identity from
node address in the registry (rowHostShared and the raft member↔row
bridge both key on the host today), which is a real data-model change.
Q5. Is public-only (raft, rqlite and the internal API bound to a public
IP, firewalled to peer /32s) a SUPPORTED shape or a documented
hazard? Four substrates allow it and examples/hetzner-wan-nebula ships
it. Raft (4002) and the internal API (4003) are mTLS from genesis
(RaftTLS: true, ce/modules/cluster/service.go:817-820). rqlite’s
HTTP API on 4001 was not established by any audit and is the deciding
fact; it should be checked before this is ruled on.
Q6. Does topology survive as a split, or is it deleted? See slice 6.
Zero production use today; the estates’ reason for turning it off is
recorded and is fixable by splitting.
Q7. Per-group nic_name exists only on hetzner. Universalise it (and
delete the scalar nic_name output that misreports it), or delete the
per-group override so all fourteen substrates match? Universalising is the
only shape compatible with a heterogeneous cluster.
Q8. Family containers or family-suffixed scalars? The recommendation is containers, and the deciding argument is that a container makes dual-stack expressible without a second rename, which is the reason the owner raised address family in the first place. The cost is a nested object where three substrate modules currently write a bare string, and a slot table that has to be maintained honestly as support arrives. Suffixed scalars are flatter and cheaper today and are the shape that has to be renamed later.
Q9. Is the vault cold start a node identity or a cluster capability? Today it is a node identity by accident: the master password is planted on the genesis node alone, deliberately and with a structural test enforcing it, and rotating that node out silently costs the cluster its ability to cold-start after a whole-estate restart. Options (a) declare the capability and plant on N nodes (which changes a security posture that was pinned on purpose, and is therefore the owner’s call), or (b) make every node able to cold-start via a quorum-derived unseal, which is strictly better and a separate program. Doing neither is legitimate, but then rotation of the seed node is a documented capability loss and the runbook must say so in capitals.
Q10. On removing a voter: report, refuse, or rebalance? Today the cluster silently goes 3 voters → 2, quorum 2 of 2, zero fault tolerance, and the non-voter that should have taken the slot is never promoted, because promotion is unimplementable on rqlite and the only rebalance is remove-and-rejoin. Report is free. Refuse is symmetric with every other guard in this model. Rebalance is the only option that makes “rotate nodes in and out whenever I want” actually true, and it is real orchestration with a real cost.
Q11. Does terraform destroy of an occupied node refuse? The
recommendation is
yes, per the owner’s own instruction, and note that saying yes means the
provider stops using shc node forget, which is the verb it uses today
and which hardcodes force. The alternative reading is that a destroy is
already an explicit operator act and a second confirmation is friction. It
is a product decision about what terraform destroy MEANS, so it is
listed here rather than decided.
Q12. Should terraform be able to express suffrage at all? A
voter = optional(bool) per node would let an operator say “these three
are the control plane, the rest are capacity” instead of letting join
order decide. It is expressible only as remove-and-rejoin, it would make
an apply able to restart a node’s storage engine, and it hands operators a
loaded gun (an even voter set is one typo away). The alternative is that
suffrage stays entirely the daemon’s business and terraform only READS it
through data.shc_cluster_status.
Q13. Should slice 9 land FIRST, ahead of the whole addressing rename? It depends on nothing else, it closes a data-loss shape that exists at HEAD (a from-scratch apply genesising a second cluster while the original keeps running), and until it lands neither production estate can rotate a node out at all. The argument against is that it touches the same resources slice 1 renames, so landing them separately means two state upgrades instead of one.
Q14. Does entrypoint survive, or does the daemon own it? Three
estate roots feed a node’s public address into dns.public_address
positionally today. The short answer is entrypoint, an explicitly
arbitrary, stable derivation from the node set. The right answer is that
the daemon’s DNS ladder should read node rows and terraform should write
nothing, which is slice 4’s cluster posture. Shipping entrypoint as an
interim means shipping a name that is intended to be deleted.
References
Section titled “References”- network-fabric.md: the
fabric this model must express; the source of the
--fabric/--fabric-cidrsurface slice 5 wires up - bind-advertise-split.md: the surviving primitive slice 2 consumes: “where peers dial me” decoupled from “where I listen”
- cross-cloud-convergence.md: the plane-NIC bring-up finding class the readiness gate exists to catch
- cluster/overview.md: membership, the internal mTLS mesh, and the enroll flow the model’s addresses ride
- core/networking.md: scope networks and aliases, which this model does not touch
- 113-removal-quorum-fix-v2.md: the quorum-floor guard the suffrage section builds on, and the wedge it closed
- operations/cluster-setup.md and operations/disaster-recovery.md: the two places node removal is documented today, both as failure recovery; the runbook slice 12 specifies is the missing third
- shc-node-per-node.md: the
symmetric-node refactor this proposal finishes by removing the last
asymmetry (
bootstrap)