Symmetric node model: an shcnode for every node, shccluster becomes pure config (2026-07-15)

STATUS: IMPLEMENTED (2026-07-17), with flag-family renames. The symmetric model landed: terraform/provider/internal/provider/node_resource.go declares “EVERY node in a cluster is an shc_node; exactly one carries bootstrap = true”, with the genesis inputs on both shc_node and the pure-config shc_cluster. Two input families were renamed by F8/F9 between this design and the landed shape — read the mapping tables below with these substitutions:

• traefik_le {email, staging} → the acme nested block (named providers + default), forwarded as shc init --acme-provider / --acme-directory / --acme-email (the --traefik-le* flags are deleted; init_test.go pins them absent). See the acme-providers proposal.
• system_apps (string, --system-apps) → the modules map (per-app overrides; the key set is the install selection), forwarded as shc init --modules-config.

Today the cluster/node resource split is ASYMMETRIC. shc_cluster owns the bootstrap node (nodes[0]) AND carries the cluster’s genesis config; shc_node is only the join nodes. So an N-server cluster is 1 shc_cluster + (N-1) shc_node. The bootstrap node is special-cased twice over — once as “the config lives here” and once as “the SSH-init happens here” — which is why cluster_resource.go re-implements a whole parallel copy of the SSH engine handoff (bootstrapSSH, adopt) that node_resource.go already has (joinSSH, waitJoined).

TARGET: symmetric. There is an shc_node for EVERY node (all N). Exactly one is flagged bootstrap = true and runs shc init; the rest run shc node join. shc_cluster becomes PURE CONFIG — the declarative home for the cluster’s genesis parameters (host_domain, package default, vault_password, system_apps, nebula, traefik_le). It no longer owns a node, does no SSH, and emits no endpoint.

This is a clean schema break (no back-compat, per the estate doctrine — docs/proposals/ house rule; no_backcompat_no_direct_dns). Existing deployed clusters migrate their STATE by hand (§5) because moved{} cannot cross resource types.

1. Some node MUST init first. The bootstrap node runs shc init, which mints the cluster CA + genesis (vault seal, endpoint, join token, admin surface). Join nodes run shc node join <token> and CONSUME endpoint/ca_pem/token from the bootstrap. This data dependency is irreducible. Grounding: ssh_engine.go:693 buildInitArgv vs :747 buildJoinArgv; internal/modules/cluster/service.go (CA mint + genesis at lifecycle priority 7, only on the init node).

2. shc init genesis flags are init-time-only. --host / --host_domain, --system-apps, --nebula, --vault-password-file, --traefik-le* are all consumed once, at genesis, on the init node (buildInitArgv, ssh_engine.go:693). There is NO daemon route to re-key the vault, change the installed system-app set, or flip nebula after init — today’s clusterResource.Update already REFUSES a vault_password change (cluster_resource.go:531) and treats host_domain/system_apps/nebula as “persist-only, does not re-run init”. This is the crux of the cycle resolution (§2): these fields are genesis inputs, not post-init config.

3. Podman clusters are single-node only. service.go:629-631 warns “podman clusters are single-node only” when the detected runtime is ModePodman. A podman bootstrap can have no join nodes.

4. Runtime is orthogonal to control-plane leadership. The bootstrap node may be any runtime (compose/swarm/podman) — buildInitArgv passes --runtime straight through. The only coupling is the podman single-node rule (3). Bootstrap-eligibility is NOT otherwise runtime-dependent.

5. shc init mints no HTTP-usable admin credential. Unchanged from today (cluster_resource.go:62-73, ssh_engine.go:82-97): the daemon’s only remote-auth scheme is a Keycloak OIDC bearer with no non-interactive bootstrap client. admin_token on the init node echoes the provider block’s token (null when none). Not a regression — a relocation.

node_resource.go:71-89. Attributes: cluster (opt nested override), host (req, replace), package (opt {deb_url, version}), ssh (opt {user, private_key(sens), port, host_key}), adopt (opt bool), name (req, replace), nic (opt, replace), runtime (opt, replace), public_address (opt), private_ip (opt). Computed: node_id, healthy, id.

The join surface is retained verbatim; the schema GAINS a bootstrap flag, the genesis-config inputs the init node needs, and the cluster-cred outputs the init node emits. All genesis inputs and all init-only outputs are meaningful only when bootstrap = true (null/ignored otherwise; plan-time validated — see §2).

Genesis inputs (nebula/host_domain/system_apps/vault_password/traefik_le) are declared directly on shc_node so a direct provider user can init a cluster without ever declaring shc_cluster. In the modules/core path the module wires them from shc_cluster.main.* (§3), so shc_cluster stays the single declarative home.

cluster_resource.go:81-95. Inputs: package (req), nebula, host_domain, system_apps, vault_password(sens), traefik_le, bootstrap (req nested: host/ssh/adopt/nic/name/runtime/public_address/private_ip). Computed: id, endpoint, ca_pem, admin_token(sens), ready, bootstrap_node_id.

Everything node-shaped is removed: the whole bootstrap {} block, endpoint, ca_pem, admin_token, ready, bootstrap_node_id. What remains is the genesis-config record.

shc_cluster emits nothing load-bearing. Its computed surface is just a stable id. Nothing downstream references an shc_cluster output — that is what keeps the DAG acyclic (§2).

• Create: exactly today’s shc_cluster bootstrap path, moved onto shc_node and reusing the existing engine. adopt = true ⇒ poll /api/health until ready, fetch CA (cluster_resource.go:646 adopt). ssh {} ⇒ node-prepared gate → deb install → registry pre-auth → shc init over SSH with the genesis flags (ssh_engine.go:799 runClusterInitOverSSH, buildInitArgv) → readiness poll. Then EMIT endpoint, ca_pem, admin_token (echo), ready, node_id, and mint join_token via shc node token (ssh_platform.go:333 mintNodeTokenOverSSH) on its own socket.
• Read: refresh ready + ca_pem + node_id via a single health/CA probe (cluster_resource.go:478 Read, verbatim). A transient outage marks ready=false, never removes the resource.
• Update: package change ⇒ in-place roll over SSH (upgrade.go). public_address change ⇒ shc node update over SSH. host change ⇒ re-derive endpoint in place (VM IP reshuffle). vault_password change ⇒ REFUSED loud (cluster_resource.go:531 — the vault stays sealed with the genesis password; persisting a new one would make state lie about DR).
• Delete: state-only (cluster_resource.go:633). Tearing down the VM is the substrate module’s job; the node self-leaves raft when its server dies.

Byte-for-byte today’s shc_node (node_resource.go:231 Create → joinSSH/adopt; :392 waitJoined; :548 Delete = shc node forget with the quorum/daemon-down tolerations; node_join_recovery.go X400021ALRCLS force-rejoin). Emits node_id/healthy/id; the bootstrap-only computed attrs are null.

• Create: validate (package.deb_url present), set id, write state. No SSH, no node, no endpoint — a declarative settings record.
• Read: no-op (nothing external to reconcile).
• Update: vault_password change ⇒ REFUSED (mirrors the init node — the seal is immutable). All other genesis fields ⇒ persist to state and print a warning that they are genesis-only (they will not re-init a live cluster; a real re-genesis is a bootstrap-node replace).
• Delete: state-only.

The question the refactor forces: does shc_cluster take the endpoint as an INPUT and apply config after init, or does the bootstrap shc_node apply that config at init and shc_cluster stay thin? Pick A.

Design A (recommended). shc_cluster holds the genesis config as pure inputs. The bootstrap shc_node REFERENCES those inputs and applies ALL of them at shc init. shc_cluster performs no SSH, takes no endpoint, and emits no endpoint. The dependency DAG:

shc_cluster.main ──(package default: every node; genesis config: bootstrap node)──▶ shc_node[bootstrap] ──(endpoint/ca_pem/admin_token/join_token/ssh)──▶ shc_node[join…]
        └──(package default)──────────────────────────────────────────────────────▶ shc_node[join…]

• shc_cluster has NO inputs from any node and NO load-bearing outputs → it is a pure source leaf. No edge points back into it.
• shc_node[bootstrap] depends on shc_cluster (genesis config) and emits the creds. shc_node[join] depends on shc_node[bootstrap].
• No cycle. The dreaded cycle (shc_cluster → bootstrap node → shc_cluster) never forms because shc_cluster consumes nothing from the bootstrap node and exposes no endpoint for anyone to consume back.

Why not Design B (shc_cluster applies post-init config against the endpoint). Every field the user assigned to shc_cluster — host_domain, system_apps, nebula, vault_password, package — is an init-time genesis parameter with NO post-init apply path (physics 2; today’s Update already refuses/ignores them). So a Design-B shc_cluster would have nothing legitimate to apply post-init, yet would need an shc_cluster → bootstrap node edge (to read the endpoint) for no benefit — and the genuinely post-init-mutable cluster config (dns.provider, acme.email, connections, mesh, LB ports) already lives in shc_config/shc_connection/shc_mesh/ shc_stack_ports, which reference the bootstrap node’s endpoint directly. B adds an edge and a temptation to “apply” un-appliable config; A does not.

join_token staleness note. The bootstrap node’s join_token is a minted shc node token (default TTL 1h). It is emitted for the cloud-init/adopt consumers (data.shc_cloudinit, data.shc_enrollment_token). The modules/core SSH path does NOT consume it — the SSH join mints a FRESH token per node over the bootstrap socket at join time (node_resource.go:287 mintNodeTokenOverSSH), which is correct under long applies. This split is documented so a stale stored token is never load-bearing. Open decision for sign-off: emit join_token at all (per the refactor brief) vs. leave enrollment entirely to data.shc_enrollment_token (a data source that re-reads each plan). Recommendation: emit it best-effort, document the staleness, keep the SSH path minting fresh.

terraform/modules/core/variables.tf and the module "core" call sites are UNCHANGED (var.nodes / var.ssh / var.bootstrap / var.topology keep their shapes — confirmed against hetzner/main.tf:309-324 and mixed-proxmox-hetzner/main.tf’s direct module "core" call). Only main.tf’s bootstrap layer and two lines of outputs.tf change.

core/outputs.tf: cluster_endpoint / cluster_ca_pem now read the bootstrap node’s outputs instead of shc_cluster.main[0]:

output "cluster_endpoint" {
  value = local.bootstrap_set ? shc_node.nodes[local.bootstrap_node.name].endpoint : null
}
output "cluster_ca_pem" {
  value = local.bootstrap_set ? shc_node.nodes[local.bootstrap_node.name].ca_pem : null
}

core/main.tf LAYER bootstrap (replaces cluster_resource.go-era shc_cluster.main + shc_node.workers):

locals {
  bootstrap_set  = try(var.bootstrap.host, null) != null || try(var.bootstrap.package.deb_url, null) != null
  bootstrap_node = var.nodes[0]
  # ssh material resolution unchanged (var.ssh.private_key | file(private_key_path))
  ssh_private_key = var.ssh == null ? null : (
    var.ssh.private_key != null ? var.ssh.private_key : file(pathexpand(var.ssh.private_key_path))
  )
  # platformSSH = the BOOTSTRAP node's admin-CLI SSH, wired onto every join
  # node's cluster.ssh + every topology resource (UNCHANGED shape).
  platform_ssh = local.bootstrap_set && var.ssh != null ? {
    host        = local.bootstrap_node.public_ipv4
    user        = var.ssh.user
    private_key = local.ssh_private_key
    port        = var.ssh.port
    host_key    = var.ssh.host_key
  } : null

  # cluster_ref now points at the BOOTSTRAP NODE's outputs, not shc_cluster.
  cluster_ref = local.bootstrap_set ? {
    endpoint = shc_node.nodes[local.bootstrap_node.name].endpoint
    token    = shc_node.nodes[local.bootstrap_node.name].admin_token
    ca_pem   = shc_node.nodes[local.bootstrap_node.name].ca_pem
    ssh      = local.platform_ssh
  } : null
}

# Podman single-node guard (physics 3): a podman node cannot have peers.
check "podman_single_node" {
  assert {
    condition     = !local.bootstrap_set || local.bootstrap_node.runtime != "podman" || length(var.nodes) == 1
    error_message = "podman clusters are single-node only (internal/modules/cluster/service.go:629) — a podman bootstrap node cannot have join nodes."
  }
}

# PURE-CONFIG cluster: the declarative genesis home. No node, no ssh.
resource "shc_cluster" "main" {
  count = local.bootstrap_set ? 1 : 0

  package        = var.bootstrap.package
  host_domain    = var.bootstrap.host != "" ? var.bootstrap.host : null
  nebula         = var.bootstrap.nebula
  system_apps    = var.bootstrap.system_apps
  vault_password = sensitive(var.bootstrap.vault_password)
}

# EVERY node — bootstrap flag on nodes[0].
resource "shc_node" "nodes" {
  for_each = local.bootstrap_set ? { for n in var.nodes : n.name => n } : {}

  bootstrap = each.value.name == local.bootstrap_node.name

  # Join nodes reference the bootstrap node's creds; the bootstrap node omits
  # cluster (it is the origin). cluster.ssh (platform_ssh) is what the join
  # node's token-mint + membership-poll ride (ssh_platform.go).
  cluster = each.value.name == local.bootstrap_node.name ? null : local.cluster_ref

  # Genesis config on the bootstrap node ONLY, sourced from shc_cluster.main.
  host_domain    = each.value.name == local.bootstrap_node.name ? shc_cluster.main[0].host_domain : null
  system_apps    = each.value.name == local.bootstrap_node.name ? shc_cluster.main[0].system_apps : null
  nebula         = each.value.name == local.bootstrap_node.name ? shc_cluster.main[0].nebula : null
  vault_password = each.value.name == local.bootstrap_node.name ? sensitive(shc_cluster.main[0].vault_password) : null

  host           = each.value.public_ipv4
  adopt          = false
  name           = each.value.name
  nic            = each.value.nic
  runtime        = each.value.runtime
  public_address = try(each.value.public_address, null)
  private_ip     = try(each.value.private_ip, null)

  package = shc_cluster.main[0].package  # cluster-wide deb default, all nodes
  ssh = {
    user        = var.ssh.user
    private_key = local.ssh_private_key
    port        = var.ssh.port
    host_key    = var.ssh.host_key
  }
}

The LAYER topology block (main.tf:154-306) is unchanged EXCEPT depends_on = [shc_node.workers] becomes depends_on = [shc_node.nodes] on every topology resource, and local.cluster_ref already resolves to the bootstrap node’s outputs — so shc_connection/shc_config/shc_mesh/ shc_stack_ports need no other edit.

• Single node (bootstrap only), N=1. for_each over var.nodes yields one node with bootstrap = true; no join nodes. shc_cluster.main pure config. The all_nodes list from the front doors already supports N=1 (nodes validation: length > 0, core/variables.tf:41). Works unchanged.
• Podman (always single-node). The check "podman_single_node" block fails the plan if a podman bootstrap has peers. A podman bootstrap node inits fine (buildInitArgv passes --runtime podman; service.go:650 handles podman NIC-optional). Runtime stays orthogonal otherwise (physics 4).
• adopt path. bootstrap = true, adopt = true ⇒ the node self-inits via its cloud-init data.shc_cloudinit (role=cluster) and the resource polls /api/health, fetches CA, emits creds (today’s clusterResource.adopt, relocated). Join nodes adopt = true poll membership. The adopt path is the one exercised by the existing acc tests and by provider/examples/{hetzner,aws} — it stays first-class.

An existing old-model cluster in state is shc_cluster.main (owns node-1 + config) + shc_node.workers["…"] (joins). moved{} cannot lift shc_cluster.main → shc_node (type change), so this is manual state surgery. Both Deletes are state-only and both shc_node matchers are identity-first, so nothing below touches the live cluster.

# 0. Snapshot state first.
terraform state pull > pre-migrate.tfstate

# 1. Drop the OLD bootstrap-owning shc_cluster from state (Delete is
#    state-only — the real node/cluster is untouched). Its schema is about
#    to change to pure-config, so the old state must go, not be refreshed.
terraform state rm 'module.core.shc_cluster.main[0]'

# 2. Rewrite config to the new model (new pure-config shc_cluster +
#    shc_node.nodes for-each ALL nodes). Add same-type moved{} for the join
#    nodes whose address changes shc_node.workers -> shc_node.nodes:
#      moved { from = module.core.shc_node.workers  to = module.core.shc_node.nodes }
#    (same type — ALLOWED. Terraform re-keys every join instance in place.)

# 3. IMPORT the bootstrap node into the new symmetric shc_node (adopt
#    semantics — Read reconciles membership, never re-inits):
terraform import 'module.core.shc_node.nodes["node-1"]' node-1

# 4. Verify: NO destroy/replace of any node, NO re-init, NO forget.
terraform plan     # expect: shc_cluster.main create (no-op record); nodes in-place/no-op

After step 3 the imported bootstrap node has name/id seeded; the first plan/apply derives endpoint from config host and refreshes ca_pem/ready via Read. admin_token re-echoes the provider token; join_token re-mints best-effort. The new pure-config shc_cluster.main is created fresh (no-op Create). Estates driving modules/core through a front door do this ONCE per state (the addresses are module.core.…).

terraform destroy the platform layer (or state rm all shc_* and let the VMs stay) then re-apply on the new model from empty. Simpler, but re-runs shc init (new CA, new genesis) — only for throwaway clusters.

No provider UpgradeState upgrader is written. A within-type v0→v1 upgrader cannot turn a node-owning shc_cluster into a pure-config one (the node is leaving the resource entirely), and the cross-type move is inexpressible in schema versioning — hence the documented surgery.

Provider Go tests:

• cluster_resource_test.go — rewrite. Drop TestAccClusterResource_Adopt, _SSHBootstrap, _BootstrapHostUpdate, _MissingBootstrapMechanism, _PEMWhitespaceNoOpPlan (all assert the bootstrap-owning shape). Replace with a pure-config test: create shc_cluster with package + genesis fields, assert id, assert vault_password update is refused, assert NO endpoint/ca_pem/bootstrap attrs exist.
• node_resource_test.go — extend. Keep _Adopt, _SSHJoin, _MissingJoinMechanism, _PEMWhitespaceNoOpPlan (join path). ADD TestAccNodeResource_Bootstrap*: the init path (adopt + SSH), asserting the emitted endpoint/ca_pem/admin_token/join_token/ready and that the shc init argv (not node join) hits the fake SSH server. Move the _SSHBootstrap init-argv assertions here from cluster_resource_test.go.
• provider_schema_test.go:30 — update the offline schema assertions for the new shc_node attribute set (bootstrap + genesis inputs + init-only outputs) and the trimmed shc_cluster.
• mock_test.go / bootstrap_data_sources_test.go — check for shc_cluster endpoint/adopt fixtures; repoint to shc_node[bootstrap].

Module + acceptance:

• terraform/modules/core/main.tf (rewrite §3), outputs.tf (2 lines).
• tests/hetzner/terraform/* — config unchanged (goes through the front door); it validates the new core wiring end-to-end on a live 4-node LB cluster. Re-run make test/hetzner as the standing gate.

Example configs (direct-resource roots only):

• terraform/provider/examples/hetzner/main.tf and .../aws/main.tf — rewrite. resource "shc_cluster" "main" loses its bootstrap {} block and becomes pure config; a new resource "shc_node" "bootstrap" (bootstrap = true, host = node[0]) is added; every cluster = { endpoint = shc_cluster.main.endpoint, token = shc_cluster.main.admin_token, ca_pem = shc_cluster.main.ca_pem } reference (≈12 per file) repoints to shc_node.bootstrap.{endpoint,admin_token,ca_pem}. Same for the join data.shc_cloudinit cluster block.
• terraform/provider/examples/provider/provider.tf — doc comment mentions cluster = shc_cluster.main; update to shc_node.bootstrap.
• terraform/examples/* (per-cloud + mixed) — no change: they consume the front-door / module "core" public interface, which is unchanged.

Docs:

• terraform/modules/core/README.md and each terraform/modules/shc/*/README.md that narrates “shc_cluster on nodes[0] / shc_node join nodes[1:]” — reword to “shc_node per node, bootstrap flag on nodes[0]; shc_cluster pure config”.
• Provider resource docs (generated from the schema MarkdownDescriptions) — regenerate.

• What breaks for direct provider consumers. The shc_cluster.bootstrap {} block is gone; shc_cluster.{endpoint,ca_pem,admin_token,ready,bootstrap_node_id} outputs are gone. Anything referencing shc_cluster.main.endpoint (etc.) must repoint to shc_node.<bootstrap>.endpoint. The bootstrap node is now a first-class shc_node with bootstrap = true. State needs the §5 surgery.
• What does NOT break. The front-door module public interface is unchanged — module "shc/hetzner" (and every peer cloud) and the direct module "core" call in mixed-proxmox-hetzner keep the same node_groups / nodes / ssh / bootstrap / topology variables and the same cluster_endpoint / cluster_ca_pem outputs. Estates driving through a front door do not churn their configs at all — only their STATE migrates once (§5), and only if they manage the platform in the same root. The join shc_node’s day-2 surface (node join, forget, in-place package roll, public-address repair, X400021ALRCLS recovery) is byte-for-byte preserved.

Recommended implementation order (each step green before the next):

1. Provider Go first, TDD. (a) Add bootstrap + genesis inputs + init-only outputs to shc_node, port bootstrapSSH/adopt into the Create bootstrap branch, write the bootstrap-node acc tests. (b) Strip shc_cluster to pure config, rewrite its tests. (c) Update provider_schema_test.go. Ship the provider as a sha-tarball.
2. Core module against the new provider: rewrite main.tf bootstrap layer + outputs.tf, add the podman check and the join-node moved{}. Validate with terraform validate + a plan on tests/hetzner/terraform.
3. Examples + docs: repoint the two direct-resource roots, reword READMEs, regenerate provider docs.
4. Live gate: make test/hetzner (4-node LB cluster from empty) — proves the symmetric model end-to-end, including the podman guard on a single-node variant.
5. Migration doc: land §5 in the module README; dry-run Path 1 against a throwaway old-model state before publishing.

1. Emit join_token at all? (§2 staleness note.) Recommend: yes, best-effort + documented; SSH path keeps minting fresh.
2. host on the bootstrap node — replace or in-place? Recommend: in-place re-derive (RequiresReplaceIf(!bootstrap)) to preserve today’s shc_cluster VM-IP-reshuffle behavior; alternative is uniform RequiresReplace (simpler, but a bootstrap IP change would re-genesis the whole cluster).
3. Keep shc_cluster at all for direct provider users? It is optional in Design A (the bootstrap shc_node can carry genesis config itself). Keep it as the declarative single-config home the module references — but a direct user could skip it. Confirm we want the (small) input duplication.