Live host reconfig — changing the cluster host without re-genesis
Status: design. No product code in this change; every claim below is grounded in file:line references verified against the current tree.
The cluster host (shc init --host, a.k.a. SHC_HOST / the terraform
host_domain) is today a genesis-only setting: it is consumed once at shc init, persisted into a node-LOCAL file on the bootstrap node, and there is no
operator-facing way to change it on a live cluster short of a full teardown.
The terraform provider says so in as many words: “host_domain / modules /
nebula are consumed once, at shc init, on the bootstrap node … A real
re-genesis is a bootstrap shc_node replace”
(terraform/provider/internal/provider/cluster_resource.go:356-366).
This doc establishes (1) the inventory of genesis-only settings and why each is pinned, (2) the complete consumption graph of the cluster host — which consumers already self-heal, which are frozen, and which mint state that dangles — and (3) a staged mechanism for changing the host on a live cluster, built almost entirely out of convergence machinery the codebase already has.
1. Genesis-only settings — the inventory
Section titled “1. Genesis-only settings — the inventory”Traced from the shc init flag set (internal/modules/cluster/commands/
init.go:457-469) through InitRequest (commands/main.go:91) → the Init hook
(internal/modules/cluster/register.go:209-227) → Service.Init
(internal/modules/cluster/service.go:617) → the persisted NodeBootstrap
(service.go:739-765):
| Setting | Where pinned | Why genesis-only |
|---|---|---|
Cluster host (--host) | node_bootstrap.json host field (service.go:526, :750), written ONLY on the genesis node | The subject of this doc — see §2/§3. No mutator exists: shc init hides itself once clustered (init.go:158-160), the re-run resume path never re-reads --host (init.go:220-297 — resume skips h.Init entirely), and no scoped-config key covers it. |
| Vault master password | vault KEK at genesis | Hard-refused in-place even at the terraform layer: “the cluster’s system vault stays sealed with the password shc init used at genesis” (cluster_resource.go:344-355). Changing it is key-rotation work, out of scope here. |
Container runtime (--runtime) | node_bootstrap.json runtime (service.go:507) | Every deploy path branches on it (e.g. upstreamName, internal/modules/app/inject_traefik.go:293-299); no repair verb; changing it means redeploying every stack — re-genesis by another name. |
Fabric mode + CIDR (--fabric, --fabric-cidr) | node_bootstrap.json fabric/fabric_cidr (service.go:556-565) | Same LOCAL file-sourced posture rationale as RaftTLS: the enroll handler must read it before any cluster-store row exists. The flag itself warns “renumbering later re-renders the whole cluster” (init.go:468). |
| Raft-TLS posture | node_bootstrap.json raft_tls (service.go:532-555) | Read before the storage engine starts, so it cannot live in the DB; flipping is documented as “a deliberate, coordinated, all-nodes-then-restart operation” (service.go:548-554). |
| Node identity (node.id) | <state>/node.id (register.go:231-245) | The raft/registry identity; changing it is a leave/rejoin, already covered by the node lifecycle. |
Deliberate contrasts — settings that LOOK genesis-shaped but are already live-mutable, and that this proposal uses as precedent:
--public-address: repaired in place byshc node update --public-address— “update a node’s public metadata in place (no reinstall, no rejoin)”, including a localnode_bootstrap.jsonrewrite (internal/modules/node/commands/update.go:22-33). This is the verb shape §4 extends.- ACME provider/email: pure scoped config (
acme.providers.*,acme.default), seeded by init but changeable any time (docs/proposals/2026-07-16-acme-providers.md). - System-app selection: post-genesis it is
shc install/uninstall <app> -t master(init.go:233-235); the terraform “genesis” warning overstates it. - The system Keycloak’s DEDICATED host (
system.apps.keycloak.host): fully live-changeable replicated config with a self-healing reconcile and a loud coded warn (§3.2). This is the crown precedent: the platform already performs a live OIDC-issuer move for one host class — just not for the cluster host itself.
2. How the cluster host is stored — and why that is the real pin
Section titled “2. How the cluster host is stored — and why that is the real pin”The genesis value flows --host → InitRequest.Host (init.go:374) →
InitConfig.Host (register.go:223) → NodeBootstrap.Host → the genesis
node’s node_bootstrap.json (service.go:750). From there, EVERY consumer
reads the process env: seedSHCHostFromBootstrap publishes the persisted
value into SHC_HOST once at daemon boot, before the composition root reads
it (internal/daemon/shc_host_seed.go:36-60, called at daemon.go:161-167).
Three structural consequences, each independently pinning the host:
- Node-local, not replicated. Joining nodes never receive the host: the
joiner-side bootstrap struct has NO host field at all
(internal/modules/node/commands/local_swarm.go:104-121) and the join write
populates none (local_join.go:257-268). On a multi-node cluster only the
genesis node (or a node with hand-set systemd env) knows the host.
RegisterSystemHostDNScarries a whole workaround paragraph for exactly this: “this leaderOnly reconcile runs on the elected background-work node, which is frequently a swarm-JOINED node whose node_bootstrap.json carries nohostfield, so seedSHCHostFromBootstrap never sets SHC_HOST there” (internal/modules/app/dns_lifecycle.go:181-190). Worse: the ingress sweep is gated on the TRAEFIK-host node, not the genesis node (internal/subsystems/impl/ingress_materialize.go:8-16), so on swarm the keycloak basePath reconcile can run on a node where the host reads "" — desired basePath "" vs deployed “/auth” is exactly the drift shape that triggers a recreate-to-root (keycloak_basepath_reconcile.go:320-335). The node-local host is not merely inconvenient; it is a live hazard. - Process-env transport. ~10 call sites read
os.Getenv("SHC_HOST")directly (inject_traefik.go:234, system_apps.go:292/:335/:481, services.go:1379, stack_reconcile_otel.go:156, …), with the bare-host normalization duplicated three times (systemIngressDomaininject_traefik.go:233-239,bareSHCHostservices.go:1376-1390, the otel seam’sHost()stack_reconcile_otel.go:153-165). An env var can only change with a daemon restart, so even a hand-edit ofnode_bootstrap.jsonneeds a restart per node to take effect. - No mutator.
shc initis hidden once clustered and its resume path ignores--host;--forcere-consumes it only by wiping local cluster state first (init.go:224-230). There is nocluster.hostconfig key (the only host-shaped keys are per-app:system.apps.<name>.host, register.go:113). Terraform’s Update warns and does nothing (cluster_resource.go:356-366).
3. The consumption graph — what the host actually feeds
Section titled “3. The consumption graph — what the host actually feeds”3.1 Consumers that already re-derive per tick / per call (self-healing)
Section titled “3.1 Consumers that already re-derive per tick / per call (self-healing)”If the effective SHC_HOST value changes (today: file edit + restart), these converge with no further work:
- Traefik system routes: the ingress sweep on the Traefik-host node
(MaterializeIngress, ingress_materialize.go:298; 60s ticker + immediate
first sweep, :59-64) rewrites the keycloak route
Host(SHC_HOST) && PathPrefix(/auth)(inject_traefik.go:1522-1560), the openobserve/otelroute (:1656+), the dex/dexroute (:1672+), and the API routeHost(host) && (PathPrefix(/api) || /internal/ws)(writeTraefikShcAPIRoute, :1400-1481, called from ingress_materialize.go:399). - Certs for the host: the SHC-CA leafs served for those routes are minted
against the CURRENT host on every route write (
serveSHCLeafinject_traefik.go:739,serveSHCAPILeaf:1490-1520). With an ACME provider configured, Traefik requests a real certificate for whatever host the router names (:1408-1412) — new host, new HTTP-01 order, automatically. Certs are NOT genesis-pinned. - Keycloak issuer pin (KC_HOSTNAME): the compose renders
KC_HOSTNAME: {{ publicUrl }}{{ basePath }}(apps/keycloak/compose.yaml:67) from the seededpublicUrlvar, whose single source issystemKeycloakPublicURL()=system.apps.keycloak.hostoverride else SHC_HOST (system_apps.go:305-311, :349-356). The ingress sweep carries a HOST-axis drift detector:SystemKeycloakHostnamePinvs the deployed KC_HOSTNAME (ingress_materialize.go:401-424; keycloak_basepath_reconcile.go:276-297, :337-358). A change on an already-pinned keycloak “is a live ISSUER move — it converges (recreate) with ONE loud coded warn, never a hard gate” — X200900KCISUR: “live SSO sessions invalidate and consumers must re-render, reconcile is converging, this is not a hard failure” (internal/modules/app/errors.go:463-469). The recreate restates the full fixed var set so the new publicUrl actually renders (SystemKeycloakInstallVariables, keycloak_basepath_reconcile.go:87). NOTE the quoted contract’s second clause: the recreate heals only the PROVIDER half; “consumers must re-render” has no machinery behind it at all — that consumer half is the largest dangle in the tree (§3.3, first bullet). - Auth service / token validation: the OIDC issuer used for strict
issuer-match validation is derived lazily per call —
SetURLResolverreadsapp.SystemKeycloakHost()“read LAZILY so a keycloak-host override converges without a daemon restart” (services.go:1247-1257), same for the base path (:1263-1265);PublicURL(auth/service.go:280) feedsexpectedIssuer(:482). Foreign/old issuers fail closed (auth/service.go:383, :529) — which is the session-invalidation semantics, not a dangle. - Keycloak backchannel dial: the node-aware dialer’s rewrite key is a
closure over
app.SystemKeycloakHost()(services.go:1221-1222, keycloak_dialer.go:70), so dial-rewrites follow the live value. - Forward-auth: the sidecar resolves realm+issuer per request through
GET /api/auth/oidc/realm-for-host(auth/router.go:122, :193-200), and the login client’s redirectUris are the bare"*"wildcard — “Keycloak matches it for ANY gated host, so no host needs re-provisioning” (auth/forward_auth_client.go:97-111). Nothing host-shaped is baked in. - Managed DNS for the host:
SystemIngressFingerprintembeds the cluster host (dns_target.go:530-541); the proxy-reconcile watcher re-firesRegisterSystemHostDNSon any fingerprint change — “SHC_HOST is a fingerprint input, so a host change re-registers too” (proxy_reconcile.go:301-313; dns_lifecycle.go:176-243). The record for the NEW host appears automatically (fail-closed target ladder included). - OTel export: the managed-provider reconcile drift-checks every
persisted key each pass, so
otel.providers.managed.endpoint = https://<host>converges to a changed host (internal/modules/health/reconcile/actions.go:371-449). - HUD/host rows, capabilities: derived per call (services.go:1346-1374).
3.2 Consumers frozen at daemon boot (need a restart today)
Section titled “3.2 Consumers frozen at daemon boot (need a restart today)”- The keycloak transport’s SNI/rewrite host:
keycloakTransportDeps. shcHostis a plain string captured once inwireAuth(services.go:1172, :1235-1241; keycloak_transport.go:52-56). The dialer reads lazily but the transport’s host does not. - The env seed itself:
seedSHCHostFromBootstrapruns once per boot and never overwrites a set env (shc_host_seed.go:37-39); everyos.Getenv("SHC_HOST")reader is therefore boot-frozen per process. - The boot-time OTel exporter default (daemon.go:598+) — though the reconcile in §3.1 re-attaches/re-persists on later passes.
3.3 Minted state that does NOT self-heal (the true dangles)
Section titled “3.3 Minted state that does NOT self-heal (the true dangles)”- Every Keycloak-integrated tenant stack’s baked issuer — the largest
class: any consumer wired through the
ssoplug getsvars.keycloak.issuerinjected at RENDER time aspublicUrl + basePath + /realms/<realm>(injectDerivedKeycloakIssuer, internal/modules/app/integration_render.go:1096-1122, call site :434-438), and for the shared system keycloak thatpublicUrlIS the cluster host (systemKeycloakPublicURL, system_apps.go:349-355, seeded into the sso.socket.yamlpublicUrlfield). The derived issuer lands in the consumer’s rendered compose env and stays there: tenant stacks have NO reconcile axis (the ingress sweep reconciles only the system keycloak/openobserve recreates), and these stacks typically carry no--host, so even a host-bearing enumeration (HostedDeployments) misses them entirely. The affected set is the injector’s own inventory — kutt, peertube, defectdojo, blockscout, postiz, minio, gitlab, dependency-track, librechat, vaultwarden, netbird, penpot, paperless, highlight, matrix, … (integration_render.go:403-433) — plus the gitea/wordpress/huly job-level equivalents that build the same publicUrl+basePath issuer inside their integration jobs. The failure is HARD, with live precedent recorded at the injector: strict OIDC discovery validates the provider’sissuerclaim against the configured value, and matrix’s MAS hard-refused to start on exactly this mismatch (integration_render.go:1077-1084). So after a host flip the system keycloak recreates onto the new KC_HOSTNAME within one sweep (§3.1) while every SSO consumer still presents the OLD-host issuer — discovery/issuer-match breaks for ALL of them until each stack re-renders. This is X200900KCISUR’s “consumers must re-render” clause made concrete: the code names the obligation; nothing discharges it. It IS dischargeable with existing machinery: the manifest sidecar persists every integration edge (persistedState.Integrations, install_runner.go:1839; plug + provider stack/tenant/env, :1933-1948), so the class is enumerable by the same manifest walkHostedDeploymentsuses (proxy_select.go:103-118), and the replay contract (a bare update re-renders against the provider’s LIVE socket data) already produces the new issuer once keycloak itself has moved — see Phase B and W5. - OpenObserve’s dex SSO stanza:
O2_DEX_BASE_URL/O2_DEX_REDIRECT_URL/O2_CALLBACK_URLall render fromvars.sso.publicUrl(apps/openobserve/compose.yaml:55-57), seeded from SHC_HOST at install (system_apps.go:193-208) and restated on recreate (SystemOpenObserveInstallVariables, openobserve_basepath_reconcile.go:89-114) — but the openobserve reconcile has only TWO drift axes, basePath and SSO-enabled (ingress_materialize.go:438-465; ReconcileOpenObserveBasePath openobserve_basepath_reconcile.go:317). A host-only change (basePath unchanged, SSO still on) is INVISIBLE to it: the deployed dex keeps issuing redirects to the OLD host until a manualshc update. The keycloak reconcile got a host axis; the openobserve one never did. (Same class: the compose’sextra_hosts: <host>:host-gatewaypin, apps/openobserve/compose.yaml:115, needs the same recreate to move.) - The OLD host’s DNS record:
RegisterSystemHostDNSonly ever SYNCS the current host (dns_lifecycle.go:219-226);DeleteRecordForHostis called solely from per-deployment rollback/uninstall (dns_lifecycle.go:92, :120) and the manual DNS API (dns/router.go:112). After a host change the old A/AAAA record (plus its ownership TXT) points at the cluster forever. - Live SSO sessions / minted tokens: old-issuer tokens fail strict issuer-match — bounded, accepted, and already the documented semantics of the keycloak-host precedent (X200900KCISUR, errors.go:455-469).
- Operator-side references: remote CLI logins target
https://<SHC_HOST>(“the daemon API is served only under the cluster’s own host”, auth/commands/pin.go:63, :157-160); terraform recordshost_domainas the resource id (cluster_resource.go:281-285). Both are re-point-and-re-login, not converge. - Tenant app hosts: deliberately NOT derived from the cluster host — the
per-app ingress is a no-op without an explicit
--host(inject_traefik.go:301-306), and system apps DROP a supplied--host(system_apps.go:103-128). If an operator chose subdomains of the old domain, each stack needs its ownshc update --hostpass; per-deployment DNS re-registration on fingerprint change replays the SAME hosts (proxy_reconcile.go:302-316), it does not rename them.
3.4 Explicit non-dangles (traced and refuted)
Section titled “3.4 Explicit non-dangles (traced and refuted)”- Vault paths: keyed
shc/{system|tenants/<t>}/…by scope, never by host (internal/modules/vault/paths.go:11-32). A host change touches no secret path. - Node enroll material: the internal-API leafs SAN the node’s IPs plus
nodeName/nodeID— not the cluster host (internal/modules/cluster/enroll_handler.go:292-293). Enroll, raft membership, join tokens, and the whole cluster plane are IP/node-identity addressed (see docs/proposals/bind-advertise-split.md for that plane’s own story) and are untouched by an ingress-host change. - Cluster/global CA: host-agnostic; the served leafs (which DO name the host) are re-minted per sweep (§3.1).
- Keycloak realm rows / clients: realms are named by tenant; the issuer host lives only in the KC_HOSTNAME pin + derived URLs (§3.1); redirect URIs are wildcarded (§3.1). Nothing in the realm store persists the old host — but this clears only the PROVIDER side of the SSO plane. It is NOT an all-clear for SSO as a whole: the CONSUMER side — every integrated stack’s rendered issuer — is the first §3.3 dangle.
4. Proposal — staged live host change
Section titled “4. Proposal — staged live host change”The shape follows from §3: almost everything already converges off ONE value; the work is to make that value replicated + mutable, close the three real dangle classes (the SSO-consumer baked issuer, the openobserve host axis, the old DNS record), and sequence the flip so the outage window is only the issuer move the platform already knows how to survive.
Step 1 — replicate the intent: cluster.host scoped config
Section titled “Step 1 — replicate the intent: cluster.host scoped config”Introduce a GLOBAL scoped-config key cluster.host (registered alongside the
system.apps.* keys, register.go:113 precedent). Read precedence for the ONE
shared resolver (today’s systemIngressDomain, inject_traefik.go:233,
promoted to the single reader; the bareSHCHost and otel-seam duplicates
delete):
cluster.host (replicated) > SHC_HOST env > node_bootstrap.json hostwith a write-once boot migration whose gate is deliberately NOT
leadership: ANY node whose local node_bootstrap.json carries a host
writes it through the config service iff cluster.host is still unset —
a CAS-on-unset (create-if-absent) write. Leadership cannot be the gate,
by §2’s own first consequence: the elected leader is frequently a
swarm-JOINED node whose bootstrap file has no host field at all
(local_swarm.go:104-121, the exact hazard dns_lifecycle.go:181-190 works
around), so a leader-gated “if the local bootstrap carries a host” check
evaluates false on the leader and never runs on the genesis node — and
cluster.host never populates on precisely the multi-node topology §2
is about. With the any-carrier gate, concurrent writers are idempotent
(first-writer-wins under CAS; two nodes carrying DIFFERENT hosts —
hand-edited files — lose the race loudly with a coded warn, never an
overwrite). The migration sources the bootstrap FILE, not the env: a
hand-set SHC_HOST systemd env is node-local dev/e2e state and must not
replicate itself cluster-wide (note seedSHCHostFromBootstrap lets env
shadow the file, shc_host_seed.go:37-39 — the migration deliberately
does not). Env stays as a dev/e2e override only. One edge stays manual:
if the only host-carrying bootstrap (the genesis node) is permanently
gone before the migration ever ran, no automatic source exists — the
Step-2 verb (or a bare shc config set cluster.host) IS the migration
then. This kills the §2 defect class: every node — including a Traefik
host that is not the genesis node — observes the same host, read live
per sweep, no restart. (The keycloak dedicated host already proves the
pattern: it is “read from REPLICATED scoped config … available on every
node regardless of SHC_HOST”, dns_lifecycle.go:184-188.)
Step 2 — the verb: shc cluster host set <new-host>
Section titled “Step 2 — the verb: shc cluster host set <new-host>”Mirror shc node update’s in-place-repair shape (node/commands/update.go:
22-33): a daemon method that (a) preflights, (b) records
cluster.host.previous = <old> + writes cluster.host = <new>, (c) rewrites
the local bootstrap’s host field on the genesis node so a later boot-seed
agrees, (d) drives the SSO-consumer re-render pass (Phase B below), and
(e) prints the convergence contract (what invalidates, what to wait for),
enumerating BOTH affected deployment classes — the host-bearing set
(HostedDeployments, proxy_select.go:77-79) whose renames stay
operator-owned, AND the keycloak-integrated set (§3.3 first bullet, off
the persisted integration edges) the verb re-renders itself. Enumerating
only host-bearing stacks would miss the SSO class entirely: those stacks
typically carry no --host. A bare shc config set cluster.host must ALSO converge —
the explicit config write IS the gate, exactly the X200900KCISUR doctrine
(errors.go:460-462) — the verb only adds preflight + ceremony.
Preflight (warn-don’t-refuse, same soft posture as the rest of ingress): new host resolves in public DNS OR managed DNS covers its zone; the target ladder yields a public endpoint (else the registration will defer, dns_lifecycle.go:202-208); vault unlocked (the keycloak/openobserve recreates render vault-backed secrets; locked ⇒ X200958KCBPRC retry loop, errors.go:432-437).
Step 3 — make-before-break, then one atomic issuer flip
Section titled “Step 3 — make-before-break, then one atomic issuer flip”- Phase A (additive, before the flip): with
cluster.host.nextstaged (or simply by the verb doing this synchronously before the flip): register managed DNS for the NEW host through the existingSyncRecordsForHostpath, and widen the system route writers to accept a host SET so/auth,/otel,/dex,/apianswer on BOTH hosts, each with its own leaf/ACME cert (the writers are already per-host parameterized single-host functions; the leaf mint is per-route, inject_traefik.go:739). Nothing user-visible changes: the issuer is still the old host. - Phase B (the flip): write
cluster.host. The §3.1 machinery converges within one sweep interval (60s, ingress_materialize.go:64): routes and leafs re-key, KC_HOSTNAME recreate fires with the loud issuer-move warn, auth resolvers and the backchannel follow, otel re-persists, the new-host DNS record re-syncs (already present from Phase A — a no-op). Live SSO sessions invalidate — bounded, loud, precedented. THEN the SSO-consumer re-render pass (the §3.3 first-bullet class): for every deployment whose manifest carries anssointegration edge to a cluster-host-derived provider, replay a bare update — the render re-reads the provider’s LIVE socket data, so the injected issuer comes out as the new host (integration_render.go:434-438), and the job-level issuer builders (gitea/wordpress/huly) re-run with the same fresh publicUrl. Ordered AFTER the keycloak recreate converges, so re-provisioning jobs and the freshly served discovery document agree. The fan-out is verb-driven for ceremony/progress, but the same pass hangs off the cluster.host change in the converge sweep (W5) so a bareshc config set cluster.hostconverges too — the config write stays the gate, per Step 2. Idempotent and re-runnable: the enumeration is a durable manifest walk, and a replay against an already-new issuer is the byte-identical no-op the replay contract guarantees. - Phase C (grace + retire): keep the old host’s routers as 308 redirects
to the new host for a grace window (new router class in the system route
writers), then delete the old host’s managed record via the existing
DeleteRecordForHost(dns/service.go:533-550, ownership-TXT guarded) keyed offcluster.host.previous, clear that key, and sweep the old-host served leafs (the served-leaf ledger already tracks them for renewal, ingress_materialize.go:30-33 — retire = remove instead of renew).
Why no dual-ISSUER window: KC_HOSTNAME is single-valued (apps/keycloak/compose.yaml:67) and the auth service strict-matches one expected issuer per realm, failing closed on any other (auth/service.go:461-483, :529). A true dual-issuer window would mean either un-pinning KC_HOSTNAME (per-request issuers — the headless posture the pin exists to eliminate, compose.yaml:53-66) or teaching every validation path a two-issuer set — the exact “two seams covering one credential” shape that bites. The keycloak-dedicated-host precedent already accepted the bounded invalidation instead; the cluster host inherits that decision.
Rollback
Section titled “Rollback”shc cluster host set <old> — the machinery is direction-agnostic (every
converger compares desired vs deployed, none remembers history). During the
Phase-C grace the old DNS record still exists, so rollback is symmetric flip
plus a second session invalidation. After retire, rollback is just a fresh
forward migration to the old name. The one asymmetry: cluster.host.previous
must be overwritten, not chained — exactly one retire obligation outstanding
at a time; a second flip during an unexpired grace first executes (or
cancels) the pending retire.
Failure modes
Section titled “Failure modes”- DNS lag: flip before the new record propagates ⇒ browsers can’t reach the new issuer while old-host sessions are already invalidated. Phase A + “wait for TTL” guidance is the mitigation; the verb should surface the new record’s observed resolvability in its output.
- ACME issuance fails for the new host (rate limit, unpropagated DNS): with an ACME provider configured the router requests the real resolver and no SHC leaf shadows it (inject_traefik.go:1461-1468) ⇒ Traefik serves its default cert until issuance succeeds — reachable but browser-warned. Preflight warns; nothing goes dark.
- Vault locked at flip: recreates fail, coded warns fire (X200958KCBPRC / X200959OOBPRC), routes/DNS still converge; keycloak keeps serving on the OLD issuer pin until unlock ⇒ mixed state where the routes answer on the new host but the issuer is old. Acceptable-but-ugly; the preflight check exists to avoid entering it knowingly.
- Mid-flip crash / leader change:
cluster.hostis a replicated row; every converger is an idempotent desired-vs-deployed compare inside restartable sweeps. No step is load-bearing in memory. - Stale env overrides: a node with hand-set
SHC_HOSTsystemd env would shadow the new value under today’s precedence — which is why Step 1 puts replicated config ABOVE env, inverting the current seed logic (shc_host_seed.go:37-39). The verb warns if any node’s env disagrees. - Old-domain tenant apps: their HOSTS are unchanged and unbroken (their
hosts are their own, §3.3) — but if they were subdomains of the old
domain the operator owns the per-stack
shc update --hostrenames. The verb’s output should enumerate them (HostedDeploymentsalready exists, proxy_reconcile.go:86-87). Their SSO is a different story: host-bearing or not, every keycloak-integrated stack is in the §3.3 re-render class and is converged by the Phase-B pass, not left to the operator. - A consumer stack down/failing during the re-render pass: its replay fails with a per-stack coded warn; the pass is re-runnable (durable manifest enumeration, idempotent replays), and until that stack re-renders its SSO stays broken on the old issuer — exactly the X200900KCISUR invalidation, scoped to that one stack, never a gate on the rest of the flip.
Work items (subsystems needing code)
Section titled “Work items (subsystems needing code)”- W1 —
cluster.hostkey + single resolver + boot migration; delete thebareSHCHost(services.go:1376-1390) and otel-seam (stack_reconcile_otel.go:153-165) duplicates; env demoted below config. - W2 —
shc cluster host setverb + daemon method + preflight; terraformshc_cluster.Updatecalls it instead of warning (cluster_resource.go:356-366). - W3 — un-freeze the keycloak transport host:
keycloakTransportDeps.shcHoststring → func (keycloak_transport.go:52-56, wiring services.go:1235-1241). - W4 — openobserve HOST drift axis: compare the deployed dex origin
(rendered
O2_DEX_BASE_URL) againstsystemOpenObservePublicURL(), mirroring the keycloak DesiredHost/DeployedHost pair (openobserve_basepath_reconcile.go:265-282, ingress_materialize.go:438-465). - W5 — SSO-consumer re-render: an
SSOIntegratedDeployments()sibling ofHostedDeployments(same manifest walk, proxy_select.go:103-118, filtering persisted integration edges with plugsso, install_runner.go:1839/:1933-1948, resolved against a cluster-host-derived provider); Phase-B fan-out of bare updates over that set, driven by the verb AND by the converge sweep on a cluster.host change, per-stack coded warn on failure, re-runnable. - W6 — system route writers take a host set (Phase A dual-serve) + the Phase-C 308 redirect router class.
- W7 — DNS retire:
cluster.host.previousbookkeeping + grace-expiryDeleteRecordForHostsweep + old-leaf retirement from the served-leaf ledger. - W8 — a coded event for the cluster-host flip itself (the cluster-host twin of X200900KCISUR), emitted by the verb and by the converge sweep.
- W9 — delete the node-local-host workarounds once W1 lands: the RegisterSystemHostDNS empty-host carve-out (dns_lifecycle.go:181-190) and the traefik-on-worker recreate-to-root hazard (§2) both die structurally.
Test plan
Section titled “Test plan”Unit: resolver precedence (config > env > bootstrap); the boot migration —
CAS-on-unset write fires on a host-carrying node that is NOT leader, is a
strict no-op when cluster.host is already set, and never sources a
hand-set env (bootstrap file only); SSOIntegratedDeployments selects
exactly the manifests carrying an sso edge (host-bearing or not) and
skips non-sso integrations; keycloak transport host-func swap; openobserve
host-axis drift matrix (host-only change ⇒ recreate; steady state ⇒ strict
no-op — clone of the keycloak reconcile’s churn guard,
keycloak_basepath_reconcile.go:337-347); dual-host route writer emits both
routers + both leafs; retire sweep deletes exactly the previous host’s
record/leafs and is a no-op with no cluster.host.previous.
Hermetic e2e (local, no cluster): init with host A, install an
sso-integrated consumer (no --host) → flip to host B via the verb →
assert within one sweep: dynamic-dir routers name B, KC_HOSTNAME
re-rendered to B (the existing DeployedKeycloakHost reader), the
consumer’s re-rendered compose env carries the B issuer
(https://B/auth/realms/<realm>), otel managed endpoint row = https://B,
old-host routers now 308. The existing services_keycloak_host_test.go
patterns cover the issuer-derivation side.
Live acceptance (proxmox multi-node, swarm): genesis on node-1 with host A,
Traefik placed on node-2 (the §2 hazard shape), raft leadership parked on a
joined node so the boot migration proves its non-leader gate, flip to B:
assert no recreate-to-root ever fires, SSO login round-trips on B — for the
system keycloak AND for an sso-integrated tenant app with no --host
(post-re-render OIDC discovery passes issuer-match on B), old sessions 401,
A serves 308s through the grace, A’s record deleted after it.
Non-goals
Section titled “Non-goals”- Renaming tenant-app hosts (operator-owned; the verb only enumerates them). Their SSO issuers are NOT a non-goal: those are platform-injected (§3.3) and re-rendered by the Phase-B pass (W5).
- A steady-state multi-host / alias ingress model — the dual-serve window is a transition tool, not a feature.
- Dual-issuer token acceptance in the auth service (rejected above).
- Changing the other genesis-pinned settings (§1): vault password, runtime, fabric, raft-TLS posture each need their own design; this doc only establishes the inventory.
- The cluster-plane (raft/enroll) addressing — host-agnostic today (§3.4) and owned by docs/proposals/bind-advertise-split.md.