The SHC scheduler (design)
Status: open design (SHC-1058). Most of this is built on next: the assignment store and its epoch (ce
modules/scheduler), the leader’s engine loop and its passes (shc registers the failover and rollout passes), grace, node return, global mode and rollouts. Failover moves a lost node’s stateful replicas and single writers too: shc’s failover pass wires ce clone’s dead-source move, the configured volume backend’s STONITH agent read per node, and the watchdog fact each node records on its row (SHC-1150, 3.9, 3.10, S16); unit specs prove it and the node-loss rig run is still to come. The cutover gate (SHC-1165) has not passed. §4.2’s JSON, §4.3’s reasons and §6’s columns differ from the code (fenceapi.ExplainDoc, ceent/schema), which is the reference, and §5.1’sshc fenceverbs becameshc label,shc constraintandshc selector. Its seventeen open questions were decided by agent on Clay’s standing delegation, 2026-09-30, reversible (section 9). It answers the question list on SHC-1058 and builds on the ruled fences design,2026-09-29-multi-region-fences.md(SHC-1050), cited below as “SHC-1050 §n”. It contradicts none of it: where SHC-1050 handed a question to SHC-1058, this document answers it, and where this document refines a SHC-1050 surface (theshc fenceverbs), it says so.
Written against shc corp/main 7f484505c and ce corp/main 6e194b44b.
ce paths are in the ce repo; every other path is in shc. Where a comment and
the code disagree, the code wins and the comment is listed in
section 10.
Clay’s architecture rulings of 2026-09-30 and 2026-10-01 are folded in (SHC-1198) and listed in section 11. Where one supersedes a line below, the line is replaced and dated. The open questions live in one table in the new-line simplification §9.
The owner’s rulings this design is bound by (SHC-1050 decisions 16 to 28, and the SHC-1058 thread of 2026-09-29):
- Swarm is retired. SHC is the only scheduler. Nodes run a compose runtime
as a dumb executor:
docker,podman,incusorapple, all four the architecture now (2026-09-30, 2026-10-01). - Nebula addresses nodes today and the
shcfabric carries container traffic; the target, ruled 2026-09-30, is two kernel WireGuard tunnels,mesh0for nodes andshc0for containers, never double wrapped, landing after the cutover (2026-10-01). - A fence is a named node set and means placement only. The old volume attachment fencing is renamed lease (SHC-1050 §6.1). This document uses the new names throughout: “attachment lease”, “lease generation”, “lease loss”.
- One placement engine, with explain, used by install, move, drain, failover, scaling and volume provisioning.
- 3 or 5 voters in the
control-planefence; since 2026-09-30 the count is automatic and odd, 1, 3 or 5, with automatic promotion and demotion (new-line simplification §4.9). - A resolver on every node and a read-only cluster config directory on every node, materialised from the control plane.
- One compose project per node a deployment touches (a per-node projection), one entry per replica (2026-10-01); a Traefik on every node fed through the file provider.
- No backwards compatibility. bitspur moves with a one-off script outside the product (SHC-1061). promanager is not touched.
- Rules come from the resource (hard), the integration (socket default, plug override), the deployment, and the scope ladder. No free-floating policies.
- A move works on a dead app as long as its data is reachable and a target satisfies the rules. Unreachable data waits for its node, or is removed by force as orphaned.
- A node that reboots starts nothing it lost ownership of.
- Automatic failover reuses the dead-source move (
--presumed-dead) and waits for the old node to stop itself first (decision 25). - Every Swarm compose key keeps its effect or is refused at render. None is silently dropped. A key with a native form is translated with a warning that names it (2026-09-30).
Vocabulary used below:
| word | meaning |
|---|---|
| app | the authored thing in the catalog (apps/<name>/), never “package” (2026-10-01) |
| deployment | one app for one tenant in one environment; the running unit and its own network; never “stack” for the running thing, which names only the axis (2026-09-30) |
| service | one authored compose service inside a deployment: template, desired replicas, healthcheck, DNS name |
| engine | the one placement engine. Runs on the raft leader. Writes assignments. |
| replica | one running copy of one service of one deployment: (deployment, service, ordinal) |
| assignment | the row that binds a replica to a node, with a monotonic generation |
| node agent | the part of shcd on every node that runs that node’s assignments |
| projection | the compose project the node agent runs for one deployment on one node: one entry per replica, named <service>-r<ordinal> |
| clusterfs | the read-only cluster config directory, /var/lib/shc/clusterfs |
| node lease | a per-node liveness lease renewed by a quorum write (new) |
| attachment lease | today’s per-volume self-fence lease, renamed by SHC-1050 §6.1 |
1. End state
Section titled “1. End state”Checks a stranger can run on a three-node estate with no Swarm:
shc install wikiwith no--nodelands on the node the engine picked, andshc fence explain -s wikiprints why, down to the scorer’s terms.shc install wikion an estate with no node that fits printspending, not a local install, and explain names the nodes and what each lacked.- Power a node off. Its stateless deployments run elsewhere within about 95 s
(35 s lease + 60 s grace). Its reattachable stateful deployments run
elsewhere within about 5 min 35 s, on their own volumes, and never while
the old node could still write (3.10). Its local-disk deployments show
down, stay assigned to it, and start when it returns, unless an operator fails one over by restoring its backup elsewhere (3.9). - Power the node back on. It starts nothing until it has confirmed ownership with the control plane, removes the copies that moved, and starts the ones it still owns. Nothing moves back on its own.
shc update -s wiki -e prod --scale web=3on a stateless service puts one replica on each node, and Traefik on every node load-balances across the three. Kill one node: its replica leaves every Traefik and resolver answer within the lease TTL and is replaced after the grace.shc update -s wiki -e prodwithdeploy.update_config: {parallelism: 1, order: start-first}replaces one replica at a time, and a replica that never reports healthy stops the rollout (failure_action: pause) or reverses it (rollback).shc node drain n2moves each deployment inside its eligible set, keeps--hadeployments serving throughout, and leaves the per-node platform set (Traefik, forward-auth, resolver) running until the node is empty.- Every compose key Swarm honoured either works under SHC or is refused at
render by name. A catalog sweep renders all 106 apps under
apps/<tier>/<app>/compose.yamlwith zero silent drops. shc fence ls,shc fence show,shc fence explainand the:fences,:rulesand:placementHUD views answer the same questions from the same data.
2. Architecture
Section titled “2. Architecture”2.1 What exists today, in one table
Section titled “2.1 What exists today, in one table”| piece | today | where |
|---|---|---|
| desired state | the replicated deployment row; its node is recorded_node_id (where the bytes are) and pinned_node_id (the operator’s --node) | ce ent/schema/deployment.go:104,131 |
| who places | Swarm, for the swarm runtime; for compose/podman, the EE scorer through the install dispatcher, which falls back to “the node that answered” when it has no opinion | ce modules/app/placement_seam.go, modules/app/install_dispatch.go, daemon/services_install_placement.go:48; modules/placement/advisor.go:80-137 |
| scorer | deterministic, integer, spread-weighted; returns every candidate’s terms; cannot refuse | modules/placement/placement.go:27-50,175,221-240 |
| capacity gates | EnforceNodeCapacity, EnforceSwarmPoolCapacity, EnforceLargestReplicaFitsNode exist and no production code calls them | modules/quota/enforcement.go:168,200,246 |
| node liveness | the leader polls every node every 2 s and stamps cluster_state=unreachable after 3 failed polls | ce modules/health/crons.go:81,194, modules/health/node_poller.go:396-397,876-930 |
| node intent | availability = active, draining, paused (cordon); drain adds drained, partially_drained | ce modules/node/types.go:73-83, modules/cluster/service.go:2970-2973, modules/cluster/availability.go:46-60; modules/cluster/drain.go:121 |
| self-heal | Docker’s restart: unless-stopped, stamped on every compose and podman service that declares no restart:; the leader-only reconcile cron runs compose up -d for its own node’s deployments and logs X200924RSTRMT for everyone else’s | ce modules/app/restart_default.go:26, modules/health/reconcile/reconcile.go:252-271 |
| single writer | per-volume attachment lease renewed every 5 s, TTL 15 s; after 2 failed renews the node stops the owner’s containers, lazily unmounts and flushes | ce subsystems/impl/volume_lease.go:68-82 |
| failover | EE ticker, off by default, rebinds a volume (never the workload) on lease expiry after three consecutive qualifying 15 s ticks, or on a STONITH confirm. The constants describe a 45 s dwell; the code counts ticks, so the real floor is about 30 s after the first qualifying tick | subsystems/volume_failover.go:36-58,80,89,465-500; daemon/wiring/volume_rebind_ops.go:42-60 |
| dead-source move | shc move --presumed-dead rebuilds a deployment on a target from replicated state; needs unreachable and a failing probe | ce modules/clone/move_presumed_dead.go:13-51, modules/clone/types.go:129-135 |
| replicas | deploy.replicas and --scale feed compose, so every replica lands on the one node the project runs on | ce modules/app/deploy_overrides.go:28-83, modules/app/services_writer.go:181-189 |
| rolling update | compose up -d --wait recreates changed services together; a failed converge restarts what it stopped; no image rollback | ce modules/app/compose_driver.go:79-120, modules/app/converge_restore.go, modules/app/converge_journal.go |
| drain | EE, journalled, per-deployment relocation, --ha read as a disruption budget, pin gate | modules/cluster/drain.go:300,361,1201-1340, drain_disruption.go, drain_pin_gate.go |
| ingress | one system Traefik on one node, Docker-label provider plus a file provider the daemon writes and SIGHUPs | ce addons/traefik/compose.yaml:624-651, subsystems/impl/ingress_materialize.go, modules/app/ingress_materialize.go:88,324 |
| leader | rqlite raft; LeaderHub polls the leader id every 5 s and fans OnLeader out; CE has no elector | ce subsystems/impl/leader_hub.go:246-257, subsystems/impl/leader_elector.go |
2.2 The loop
Section titled “2.2 The loop” replicated control plane (rqlite) deployments, rules, fences, assignments, node leases, revision ▲ writes (API, quorum) │ reads │ ▼ ┌─────┴──────────── leader ───────────────────────┐ │ engine: filter (fences) → fit (capacity) │ │ → spread/apart → score → bind │ │ writes assignment(gen+1) + placement_event │ │ render: Jinja once per replica → spec + digest │ └─────────────────────────────────────────────────┘ │ every node: materialiser (DB → clusterfs, atomic; copies, never renders) ▼ /var/lib/shc/clusterfs (read-only mounts) ── node agent ── executor assignments, projections, routes, runs its projection, (docker, podman, certs, resolver zones restarts crashes, incus, apple) reports observed stateThree parts in the loop, two of them new in kind (the two node-side workers that keep the mesh and the firewall are listed after them):
- The engine runs only on the leader, like every other leader-only
subsystem (
LeaderOnly: true, cesubsystems/lifecycle/subsystem.go:29-45). It holds no state of its own between ticks. Its input is the replicated tables; its output is assignment rows. rqlite forwards any member’s write to the current leader (core/db/rqlitedriver/discovery.go:44-56), so a deposed engine’s write would still commit. Every bind is therefore fenced by an engine epoch: a new engine incrementsscheduler.epochwhen it takes leadership, and each bind statement is guarded byepoch = <mine>as well as a compare-and-swap on the assignment generation. A deposed engine’s binds match no row. It runs on events (install, scale, update, node condition change, fence edit) and on a 5 s safety tick. The leader also renders, once (ruled 2026-10-01): at install or update it evaluates Jinja for every replica, the deployment context plusreplica.ordinal,replica.locationandreplica.node, and stores each replica’s spec in rqlite with a digest (deployment_spec, section 6). Nothing else ever renders, so a daemon upgrade changes no projection until an explicit update, and a new leader holds every stored spec through raft before it serves. - The node agent runs on every node. It reads its own assignments from
clusterfs, renders nothing, and runs the projection shcd materialised for
it. It owns container restarts and the boot gate (3.9) and the
readiness report (3.13). Today’s per-node container recorder
(ce
subsystems/impl/container_recorder_node.go) is its observation half. - The materialiser runs on every node and writes clusterfs from the
control plane (3.24). Today’s ingress sweep, which writes route and cert
files into a node-local Traefik directory and SIGHUPs Traefik
(ce
modules/app/ingress_materialize.go:324,modules/app/traefik_reload.go), is its first consumer and moves onto it. It copies stored specs into projections and derives the Traefik and resolver sections from readiness rows; it never renders and never enriches (3.24).
This is the Kubernetes split (scheduler binds, kubelet runs) with SHC’s fences in place of labels and a file tree in place of the watch API. It is not Swarm’s: Swarm’s managers push tasks to workers over a dispatcher stream, and a worker that loses its manager keeps running tasks with no ownership check at restart.
shcd on a node is four workers (2026-10-01). None of them touches a packet; the data plane is the kernel.
| worker | reads | writes or does |
|---|---|---|
| materialiser | SHC’s rqlite | this node’s clusterfs revision: stored specs become projections, readiness becomes the Traefik and resolver sections. It copies; it never renders |
| node agent | its clusterfs | runs the four checks below, hands projections to the executor, restarts containers, reads health, reports readiness and the revision it applied to rqlite |
| mesh coordinator | the peer table in rqlite, cached locally for boot | keeps mesh0 and shc0 configured over netlink: peers, keys, grants, the best endpoint per peer, a per-route MTU from its probes. Control plane only, no packet path |
| policy applier | the compiled per-node ruleset in its revision | loads it with one atomic nft -f, verifies the loaded version hash, re-applies on drift. On a Mac the same ruleset comes out as pf |
Four checks before the agent runs anything (ruled 2026-10-01), each with a concrete test:
- The revision is whole. The materialiser built it aside, verified the
store did not move while it read, swapped each section’s
currentwith a rename, and wroteREVISIONlast (3.24). - The entry’s hash matches the stored spec digest on its assignment (deployment, service, ordinal, generation, digest). A mismatch does not run and reports why; that catches a corrupted disk, a materialiser bug and a hand-edited file alike. The revision carries a digest of its own.
- The replica is still this node’s at the current generation and
engine epoch, with the lease held for anything exclusive; a file saying
“run
pg-r1here” is never enough on its own. - Its cross-node dependencies are ready. A
depends_onon a service running elsewhere holds the start until that replica reports ready.
Correct here means intact and still owned, not newest. A node acting on a revision a second old is fine; checks 2 and 3 make sure it can only run what is its own, and the next materialise removes what a newer revision took away.
2.3 Objects and states
Section titled “2.3 Objects and states”Replica and assignment. The engine places replicas, not deployments. A
deployment with one service and no replicas: has one replica, and the
default implicit node colocation of a deployment’s services (SHC-1050 §5.1) puts
every one of its replicas on one node, so the common case is today’s shape.
assignment state machine:
pending ──bind──▶ assigned ──agent starts──▶ starting ──ready──▶ ready ▲ │ │ │ │ node lost past its grace, movable unready ◀────────────┘ │ ▼ └──────────── rescheduling ──bind(gen+1)──▶ assigned (new node)
node lost past its grace, unmovable ──▶ down (stays on its node; starts on return) data unreachable, operator forces ──▶ orphaned ──▶ removedNode conditions. Two columns exist, cluster_state (membership and
health, ce ent/schema/node.go:46-60) and availability (operator intent,
:66). The engine reads one derived condition, shown in shc node ls
and the HUD:
| condition | when | new replicas | running replicas |
|---|---|---|---|
| ready | node lease fresh, cluster_state healthy | yes | kept |
| pressured | node lease fresh, cluster_state degraded with a resource reason (disk, inode, memory) | no | kept |
| cordoned | availability=paused | no | kept |
| draining, drained | availability | no | moved by drain; global replicas stop last |
| rebooting | announced shutdown or maintenance window, with an expected_back_by | no | held, nothing rescheduled before the deadline |
| lost | node lease expired (TTL 35 s) | no | removed from every endpoint list; single writers there have been told to stop (3.10); each deployment’s reschedule clock runs from the expiry |
| down | lost past every grace that applies to its work (3.8), or a STONITH agent confirmed power off | no | the movable ones already rescheduled; the pinned ones shown down |
The condition gates new replicas only. A running replica leaves a node
through drain, rescheduling, a fence edit (refused while it would strand
one, 5.1), or its deployment’s removal, never because the node was cordoned
or pressured. That is Kubernetes’ NoSchedule behaviour, and it is what keeps
the per-node platform set running on a cordoned node (3.17).
The candidate query admits degraded today
(modules/placement/advisor.go:62-67); under this design a degraded node
with a pressure reason takes no new work, as a Kubernetes node with a
pressure taint does.
3. The questions
Section titled “3. The questions”Each part says what Kubernetes and Swarm do, what SHC does, what exists, and what is missing.
3.1 Desired state and reconciliation
Section titled “3.1 Desired state and reconciliation”Kubernetes stores desired objects and every controller converges actual toward them. Swarm does the same for services and tasks inside its raft.
SHC. The desired state is the deployment row, its rules, and its assignments. The engine converges assignments toward the deployment (how many replicas, where). Each node agent converges containers toward its assignments. Neither needs the other to be up: assignments survive a leader change, and a node agent keeps running what it was last told.
| exists | missing |
|---|---|
the deployment row as desired state; the leader-only reconcile cron runs compose up -d every minute for broken deployments on the leader only and logs X200924RSTRMT for the rest (ce modules/health/reconcile/reconcile.go:252-271, modules/health/reconcile/errors.go:105-128); the leader-only container reconciler reaps orphan rows every 60 s (ce subsystems/impl/container_reconciler.go); a boot pass settles rows parked mid-operation (ce subsystems/impl/parked_stack_reconcile.go) | the assignment table; a per-node reconciler, so a broken deployment on a non-leader node is repaired by its own node; the engine loop |
3.2 Filtering and scoring
Section titled “3.2 Filtering and scoring”Kubernetes filters nodes with predicates, scores the survivors with weighted plugins, and reports “0/5 nodes are available” with a reason per node. Swarm filters by constraints, resources, platform and ports, then spreads by task count.
SHC. Four stages, in order, for each replica:
- Eligible set from fences, exactly SHC-1050 §5.1: absolute rules on
every scope, resource reach, compiled colocation, and the node condition
ready. - Fit: reservation against free capacity (3.3). A hard filter, not a score.
- Spread and apart: hard spread and anti-affinity rules (3.4, 3.6) remove nodes; soft ones become score terms.
- Score with today’s scorer, unchanged in contract: it ranks and never
refuses. Its
Decisionkeeps every term for explain.
A replica with no survivor is pending, with the reason per node that
removed it (section 4). An empty eligible set from rules alone is a
refusal at admission, before the QUEUED row, as SHC-1050 §3.7 rules.
Pending is for “nothing fits now”; refused is for “nothing can ever fit
these rules”.
| exists | missing |
|---|---|
the scorer with memory, cpu, count, tenant and app terms, deterministic tie-break, unknown axes abstaining (modules/placement/placement.go:175,240-300); candidate filter on condition and availability (advisor.go:62-67); memory headroom filter that falls back to all candidates when every node is full (advisor.go:107-131); the keep-off-voters filter (advisor.go:244) | fence filtering (SHC-1050 phase 3); fit as a filter that yields pending instead of “no opinion”; per-node reasons; spread as a filter |
3.3 Requests, limits and capacity
Section titled “3.3 Requests, limits and capacity”Kubernetes schedules on requests, enforces limits in the kubelet, and
subtracts system-reserved from allocatable. Swarm schedules on
reservations and ignores limits for placement.
SHC. Placement uses the reservation; compose applies the limit.
Both come from today’s quota chain: one quota per service, limit = quota ×
deploy.limit_ratio (default 1.5), reservation = quota ×
deploy.reservation_ratio (default 1.0) (ce modules/app/resource_quota.go:13-14,200,
modules/app/resource_reservations.go:125). A node’s allocatable is its
total minus a system reserve for shcd, the runtime and the per-node platform
set. A node’s committed figure stays derived, never materialised, but it is
derived from assignments instead of from the deployment anchor. Today’s
ledger charges every service’s full desired replica count to the node the
deployment is recorded on (modules/quota/commitment.go:120-132), which is
right only while every replica sits on that node. Under the engine:
- a node is charged, per assigned replica, that replica’s reservation;
- a
pendingreplica is charged to its tenant’s quota (3.22), never to a node; - a surge copy during a
start-firstupdate or drain is charged on its node for as long as it exists, so a surge that does not fit waits; - a global replica is charged on every node it runs on.
The engine is the only writer of assignments and binds one replica at a
time, and each bind is one statement batch that re-reads the target’s
committed figure, checks fit, and writes the assignment under the epoch
guard (2.2). Two binds cannot both admit against the same free memory, the
race usage once lost to eight concurrent charges (the header of
modules/quota/commitment.go).
Unknown capacity stays unknown: a node whose memory_bytes is NULL is kept
as a candidate and flagged by explain, never read as zero, which is today’s
rule (ce ent/schema/node.go:115 comment, modules/quota/enforcement.go
header).
| exists | missing |
|---|---|
per-node cpu_cores and memory_bytes (ce ent/schema/node.go:91,115); the quota chain and reservations derived from limits (ce modules/app/resource_quota.go, resource_reservations.go); the commitment ledger, per deployment anchor (modules/quota/commitment.go); the node gates (modules/quota/enforcement.go:168), uncalled | a caller: fit as stage 2 of the engine; a system reserve per node (config scheduler.system_reserved, default 1 GiB + 5 % of memory, 500 m CPU); the commitment derived from assignments, surge and global replicas included; the fit check inside the bind |
3.4 Affinity and anti-affinity as fence rules
Section titled “3.4 Affinity and anti-affinity as fence rules”Kubernetes has node affinity (required and preferred), pod affinity and pod anti-affinity over a topology key. Swarm has placement constraints and spread preferences.
SHC. Every one of these is a rule on an object, in the SHC-1050
vocabulary. SHC-1050 defined two forms; this design adds four. Five of the
six narrow and never widen (SHC-1050 §3.3); admit is the toleration of 3.5
and only lifts a reserved fence out of the base set:
| form | spelling | Kubernetes analogue | judged on |
|---|---|---|---|
| stay in | fence=<kind>:<name> (SHC-1050) | required node affinity | node membership |
| stay out | avoid=<kind>:<name> | required node anti-affinity (NotIn) | node membership |
| colocate | colocate=<plug>:<kind|none> (SHC-1050) | pod affinity | the partner’s eligible set, compiled to one derived fence (SHC-1050 §5.2) |
| apart | apart=<deployment>[/<service>]:<kind> | required pod anti-affinity | the partner’s current assignments |
| spread | spread=<kind>[,skew=<n>][,max=<n>][,hard] | topology spread constraint | this deployment’s own replicas (3.6) |
| admit | admit=<kind>:<name> | toleration | reserved fences only (3.5) |
Ruled 2026-10-01: operators never type spread, apart or a region. The
engine derives both from the --scale shape (3.6) and from stateful: true; they stay as its internal vocabulary and in explain. An operator
states a rule with one flag per form, each taking <kind>:<name>:
--fence, --avoid, --colocate and --admit, plus --location <name>
for --fence location:<name> (SHC-1237, ruled 2026-10-02; --rule is gone).
apart is judged on assignments, not eligible sets, because “never share a
location” cannot be compiled to one fence ahead of time; it is symmetric, so
moving either end re-checks it. Preferred (soft) node affinity is not a rule:
the scorer’s tenant and app spread terms already are the soft form, and
SHC-1050’s external-socket preference (§3.5) is the one scored location
preference.
| exists | missing |
|---|---|
Swarm constraints rendered by SHC (node.hostname==, voter excludes, RWO pins: ce modules/app/swarm_placement.go:33,90,171); drain’s constraint matcher (modules/cluster/drain.go:1201) | the rule store (SHC-1050 phase 2.4); the avoid, apart, spread and admit forms; apart evaluation in the engine |
3.5 Taints and tolerations: reserved fences
Section titled “3.5 Taints and tolerations: reserved fences”Kubernetes taints a node so that only pods that tolerate it land there
(NoSchedule), prefers not to (PreferNoSchedule), or evicts (NoExecute).
SHC. This maps cleanly onto one fence attribute, so it lands; the words
“taint” and “toleration” do not. A fence can be reserved
(shc fence set <fence> --reserved). A node in a reserved fence is outside
every deployment’s base candidate set unless one of the deployment’s rules,
at any rung, names that fence with fence= (enter it and stay in it) or
admit= (may enter it, may also run elsewhere). admit= is the toleration;
fence= is a toleration plus node affinity, as a Kubernetes pod that both
tolerates a taint and selects the tainted nodes.
NoScheduleis--reserved.PreferNoSchedulehas no SHC form: a preference that nothing enforces is the free-floating policy the owner ruled out. An operator who wants it writesavoid=on the scopes that should stay off.NoExecuteis drain.
Reservation is part of the base candidate set, like the node condition, not
a rule, so naming a reserved fence at the deployment rung does not loosen an
estate rule (SHC-1050 §3.3 stays true). The keep-workloads-off-voters filter
(modules/placement/advisor.go:244) becomes shc fence set control-plane --reserved. A system-priority deployment (3.7) is admitted to every
reserved fence without a rule; that is what lets the per-node platform set
run on every node, reserved or not (3.17).
| exists | missing |
|---|---|
filterVoterTier, the only taint-like filter (advisor.go:244) | the reserved fence attribute, its base-set rule, admit=, the system-priority admission |
3.6 Replicas and topology spread
Section titled “3.6 Replicas and topology spread”Kubernetes spreads by topology spread constraints (max skew, per topology
key, DoNotSchedule or ScheduleAnyway) and gives StatefulSet pods a stable
name and their own volume claim. Swarm spreads replicas by node and by
placement.preferences.
SHC. Replicas across nodes, per the owner’s 2026-09-29 answer:
deploy.replicas: N(or--scale <service>=Nonshc installorshc update) asks for N replicas of a service. The engine places each one independently inside the service’s eligible set. There is noshc scaleverb (ruled 2026-10-01).- Scale is one flag with three shapes (ruled 2026-10-01), persisted as
a deployment override and capped by the app’s
x-shc-scale.max:--scale web=3is three replicas, one per node where possible, anywhere the deployment is allowed;--scale pg=eu:2,us:1is exact counts per location, with the total, the allowed locations and the spread by location implied;--scale proxy=per:locationis one in every value of that fence, following the estate as it grows. Fence kinds arelocationandprovider; “region” is a name someone gives a location. Nohomekey and nofailoverkey: a primary stays where it was last promoted, nothing fails back on its own, and which side may promote automatically is quorum arithmetic on the counts (eu:2,us:1recovers on its own when us is lost and needs a manual promote when eu is; three locations make it automatic everywhere). A deployment is addressed with-s/--stack,-e/--environmentand the tenant flag orshc ctx, never atenant/environment/stackpath. Lands in SHC-1145 and SHC-1146. - One compose entry per replica, always (ruled 2026-10-01). The
projection carries one entry per replica on that node, named
<service>-r<ordinal>, each a single container. Compose’s ownreplicas:is never used: it cannot give each copy its own pin, environment or volume, and the runtimes disagree about it. The bare service name is a local alias only when every replica of the service is on this node; otherwise it falls through to the resolver (3.15). Each service numbers its replicas from 0. One copy on one node is the same code path as nine across three locations. - Spread. For kind K, the domains are the K fences that meet the eligible
set. Skew is the largest replica count in a domain minus the smallest.
spread=K,skew=1keeps skew at or under 1: as a score term by default, as a filter with,hard.max=<n>caps the replicas in any one K domain and is always hard; it is the only way to say “one per node”, because a skew of 1 still allows three replicas on two nodes as 2 + 1. A node in no K fence is not a candidate under a hard K spread or a cap, and explain names it. A replicated service with no spread rule gets an implicit softspread=node,skew=1. - Members. A service declaring
x-shc-scale: {stateful: true}gets StatefulSet semantics: replica<service>-r<ordinal>, its own volume per ordinal, its own pinned fabric address per ordinal, ordinals started in order, and an implicitspread=node,max=1: one ordinal per node, so a 3-ordinal service needs three eligible nodes. Replication between them stays the app’s job. Ruled 2026-10-01:stateful: truereplaces this design’sidentity: stable, with no alias; since every ordinal sticks anyway (above), the flag means distinct members: own volume, own name, one per node, started in order. The key is gone since: distinct members are a{% members %}group (groups proposal), whose members prefer separate nodes through an implicit softspread=node,skew=1counted over the group. A group whose block saysseparate="required"takesspread=node,max=1over the group instead: no node takes two members, on install, update, failover, a move or a rebalance, and a member with no node of its own is pending with reasonseparate. An install that asks for more members than the eligible nodes is refused, not left pending (X409184FNCSEP). - Volumes. A replicated service may mount RWX storage (a shared
filesystem). An RWO volume with
replicas > 1and nostateful: trueis refused at render, which keeps today’s rule that RWO is single writer (cemodules/app/swarm_placement.go:150-170). No replicated volumes in the first cut (ruled 2026-10-01): local volumes, restic, failover by restore; LINSTOR comes later as a per-site capability. - Ceilings.
x-shc-scale.maxstays the one ceiling every replica change consults (cemodules/app/replica_ceiling.go,modules/app/deploy_overrides.go:28-50).
| exists | missing |
|---|---|
deploy.replicas and --scale, all on one node (ce modules/app/services_writer.go:181-189, deploy_overrides.go:80-83); the x-shc-scale schema with max, backup, restore (ce modules/app/scale_extension.go); 97 services in 73 catalog apps declare x-shc-scale.max: 1; one catalog service uses replicas (apps/standalone/pulp/compose.yaml:322) | per-replica assignments and one entry per replica; the three --scale shapes; spread; stateful: true with per-ordinal volumes and pins; the pin table keyed by ordinal (today (deployment, service), ce modules/fabric/pins.go:17-19,86-101) |
3.7 Priority and preemption
Section titled “3.7 Priority and preemption”Kubernetes evicts lower-priority pods to fit a pending higher-priority one. Swarm has neither.
SHC. Priority, no preemption. A deployment carries priority: system
(the builtins), high, normal (default) or low. Beyond admitting
system to reserved fences (3.5), it only orders work:
- after a node goes down, rescheduling runs in priority order, then by
today’s system-first order (ce
modules/health/reconcile/system_first.go); - drain moves in priority order;
- a pending queue is retried in priority order.
Preemption is refused because an SHC move is a stateful cutover (backup,
ship, restore, health gate; 38 to 51 s measured per relocation,
modules/cluster/drain_disruption.go:14-16), not a pod restart. Evicting a
database to fit a web app is never the right trade on these estates. Explain
prints what would fit if something moved, as a hint only.
| exists | missing |
|---|---|
| system-first reconcile order | the priority column and its three uses |
3.8 Node health: heartbeat, lease, grace; dead, restarting or powered off
Section titled “3.8 Node health: heartbeat, lease, grace; dead, restarting or powered off”Kubernetes: each kubelet renews a Lease every 10 s; the node goes Unknown
after node-monitor-grace-period (50 s by default); pods get the default 300 s
not-ready/unreachable toleration before eviction. Swarm: workers
heartbeat the dispatcher every 5 s; a node is down after missing a few,
and its tasks are rescheduled at once.
SHC. Two signals, two jobs, and a grace per class of work.
- The node lease (new) is the liveness truth. Every node’s shcd renews
one
node_leaserow every 5 s with a quorum write, TTL 35 s, the same cadence as today’s attachment lease (cesubsystems/impl/volume_lease.go:68-82). A node that cannot commit its renew knows it is cut off; a leader that sees the lease expire knows the node is cut off or dead. Neither can tell which, which is why the self-stop exists (3.10). - The poller keeps its job: health detail (disk, inode, memory, certs)
and the fast
unreachablestamp (3 failed polls on a 2 s tick, cemodules/health/node_poller.go:396-397,modules/health/crons.go:81). It no longer decides liveness for the scheduler: a poll can fail on a path the node’s raft traffic does not use.
Dead, restarting, powered off.
| what happened | what SHC sees | what the engine does |
|---|---|---|
announced reboot (shc node maintenance, or shcd stopping during a system shutdown) | the node writes rebooting with expected_back_by before it goes | holds everything until the deadline (default scheduler.reboot_window, 5 min); reschedules nothing |
| unannounced crash or reboot | lease expires; the node is lost | reschedules each deployment by movability at lease expiry + its grace |
| powered off, confirmed | a STONITH agent reads the power state back as off (the Hetzner agent does this today, ce modules/storage/providers/hcloud/stonith.go) | down at once; the grace is skipped because the old node cannot write |
| alive but partitioned | lease expires at the leader; the node itself fails its renews | the node starts stopping its single writers after two failed renews (3.10); the engine treats it as lost and runs the same clocks |
Grace. One clock per deployment, started at the node’s lease expiry.
When it runs out the deployment is rescheduled at once; there is no second
wait. The node’s condition turns down when the longest clock on it has run
out, or at once on a STONITH confirm.
| class | default | config key | why |
|---|---|---|---|
| stateless (no persistent data) | 60 s | scheduler.grace.stateless | a replacement is a compose up elsewhere; other replicas usually serve meanwhile |
| stateful, reattachable | 5 min | scheduler.grace.stateful | a move is a rebind or a backup+restore, and most reboots finish inside 5 min; Kubernetes’ default toleration is the same 300 s. Never shorter than the single-writer stop window of 3.10, which the engine enforces |
The keys resolve through the scope cascade (core/scope), so a deployment
can shorten its own grace. They are settings, not placement rules, so the
owner’s no-free-floating-policies ruling does not reach them.
| exists | missing |
|---|---|
poller and unreachable stamp; joining/healthy/degraded/unreachable/leaving/removed/lease_pending (ce ent/schema/node.go:60); attachment lease; STONITH seam with the Hetzner agent; containers marked unknown once per death episode (ce modules/health/node_poller.go:144-160) | the node lease; the rebooting state and expected_back_by; lost and down conditions; grace keys; an announced-shutdown hook in shcd; shc node maintenance |
3.9 Rescheduling, and a node that comes back
Section titled “3.9 Rescheduling, and a node that comes back”Kubernetes recreates a Deployment’s pods elsewhere after the toleration; StatefulSet pods are never recreated until the old pod is confirmed gone. Swarm reschedules every task of a down node at once and, when the node returns, shuts down the stale tasks it finds there.
SHC. When a node’s lease expires, each deployment on it is handled by its movability, which the engine derives and explain prints. “Clock” is the deployment’s grace from 3.8, started at the lease expiry:
| movability | when | when its clock runs out |
|---|---|---|
| free | no persistent mounts | new assignments inside the eligible set; nothing moves, because there is nothing to carry |
| reattach | its data is on RWO block volumes whose reach holds another eligible node, or on a shared filesystem | moved by the dead-source move (--presumed-dead flow), which rebinds the surviving volumes behind a lease-generation bump; the stateful clock is never shorter than the single-writer stop window (3.10, decision 25) |
| pinned | a local volume or host bind, a fence=node:<n> rule, a volume whose reach is that one node, or inputs the control plane does not hold (below) | stays assigned, shown down, starts when the node returns |
| far | a volume whose reach holds no node of the control-plane fence (decision 24) | stays assigned; taken over only on a STONITH confirm or an operator’s --presumed-dead |
Failover by restore (2026-10-01). With no replicated volumes in the
first cut, a deployment on local volumes fails over by restore: its latest
verified restic backup restored onto an eligible node through
restore-as-move (shc recover, 3.25), losing the writes since that backup.
pinned means the engine never does that on its own, because the data
sits on the lost node and the owner’s rule is that unreachable data waits
for its node; it does not mean the deployment cannot be recovered. Explain
prints a pinned deployment’s last backup, its age and the restore command,
and an operator starts it.
The dead-source move stays the one mover. The engine calls the same
PresumedDeadProvider the CLI path uses (ce modules/clone/types.go:1145-1149),
with the target it chose, instead of building a second flow. Its two gates
stay: the node is lost past the clock (the node lease replaces the
unreachable stamp as gate 1’s first half) and a fresh probe fails; local
mounts are refused, so a pinned deployment never reaches it. Two things it
lacks today:
- A replica operand. The provider plans and installs a whole deployment
and repoints its node (ce
modules/clone/move_presumed_dead.go:135-160). That is right for the default shape, every service on one node. A stateless replica of a split deployment needs no move at all, only a new assignment. A stable-identity ordinal on its own volume needs the move scoped to that ordinal, so its healthy siblings’ volumes are not rebound. - Lossless inputs. Today’s reconstruction states that it loses published
--portmappings and install-time--varoverrides (cedaemon/clone_dead_move_adapters.go:342-345). An unattended move must not silently come back on app defaults. The new line stores the full effective specification in the replicated store (6); until a deployment’s inputs are all there, the engine classes itpinnedand explain says which input is missing.
No flip-flop. Today an interrupted move is reverted by default, and one
that already tore its source down is refused for unattended recovery; only a
rename finishes forward (ce modules/clone/move_journal.go:598-612). An
interrupted dead-source move leaves the rebound volumes on the target and
asks for a retry (ce modules/clone/move_presumed_dead.go:321). So the
engine records the chosen target on the assignment before it starts the
move, and a new engine after a leader change retries the same move to the
same target until it lands. A node that returns mid-move is not a reason to
stop: its lease generation is already stale. When a node returns:
- Its movable deployments already live elsewhere. The node agent finds their
containers without a current assignment and removes them (
compose downof that projection, volumes kept forshc reclaim). - Its pinned and far deployments are still assigned to it and start after the boot gate below.
- Its freed capacity takes new work. Nothing moves back unless an operator moves it or runs a rebalance (3.19).
The boot gate (the owner’s safety requirement). No SHC-managed container may be started by the runtime’s own restart policy at boot, because the node cannot yet know what it still owns. So:
- The runtime must see no restart policy from either source. Compose lets
deploy.restart_policyoverride the top-levelrestart:(cemodules/app/restart_effective.go:1-30), so the render stampsrestart: "no"and removesdeploy.restart_policyfrom the runtime’s document, keeping today’s rule that strips the Swarm stanza. The authored policy from both keys, resolved as X200932RSTDIA resolves a conflict today (cemodules/app/errors_reboot.go:12-31), is recorded as labels for the node agent. This reversesApplyRestartDefault, which suppliesunless-stoppedto every compose and podman service that declares norestart:(cemodules/app/restart_default.go:26-47). - The node agent owns crash restarts: on a container
dieevent (cesubsystems/impl/docker_events.goalready streams them) it applies the recorded policy, and only while clusterfs still assigns the replica to this node and, for a single writer, its lease is live. Compose’srestart:values keep their meaning, with Docker’s own backoff (100 ms doubling to 1 min, reset after 10 s up).deploy.restart_policykeeps its fields:condition(none,on-failure,any),delayas the first backoff step,max_attemptsas the retry cap, andwindowas the time a restart must stay up to count as a success. - On Podman,
podman-restart.servicestays disabled; nothing in either repo enables it today. - At boot the node agent starts nothing, the per-node platform set
included, until it has (1) re-read its assignments from the leader with a
linearizable read and re-materialised clusterfs, (2) acquired its node
lease, (3) re-acquired each assigned RWO volume’s attachment lease at the
current lease generation. This is decision 23 read literally: the platform
set holds no data, but whether this node is still a member and still
assigned the platform set is exactly what it cannot know yet. A bumped
generation means another node took the volume over, so the local copy is
removed, not started.
shc node statuson that node says what it is waiting for. Every start after the gate still passes the four checks of 2.2.
This also retires two workarounds that exist because the runtime’s restart
policy runs unsupervised: the operator step “run docker compose down on
the returned node” (ce modules/clone/move_presumed_dead.go:43-51) and the
reason compose stop in a backup needs a journal to come back
(ce modules/backup/stop_journal.go:1-30), which becomes a node-agent
restart.
| exists | missing |
|---|---|
the dead-source move and its gates; the move journal; boot hydration that skips rows homed elsewhere (ce modules/clone/move_presumed_dead.go:43-45); Docker events on every node; the rule that strips deploy.restart_policy under compose (ce modules/app/restart_effective.go) | movability derivation; engine-driven dead-source moves with roll-forward; a replica-scoped dead-source move; the replicated full specification; stale-copy removal; the boot gate; node-agent restarts; restart: "no" at render |
3.10 At most one writer
Section titled “3.10 At most one writer”Kubernetes StatefulSets wait for the old pod to be deleted, which a partitioned kubelet never confirms, so an operator force-deletes or a fencing agent confirms. Swarm has no guarantee: a partitioned worker keeps its task while the manager starts another.
SHC. Three layers. The first two are today’s code, extended; the third is new, because the first two cannot prove a stop on their own.
- The node stops itself. The self-fence loop, renamed lease loss, starts
stopping every container of a deployment holding an RWO attachment lease
after two failed renews (ce
subsystems/impl/volume_lease.go:14-22,68-82). The stops are sequential, bounded at 10 s per container, and a failure only logs (cesubsystems/impl/volume_lease_node.go:134-177). This design extends the same stop to every single-writer service: one with an RWO volume, or one whosex-shc-scale.maxis 1, the data-safety ceiling 97 catalog services already declare. A single writer on host binds has no attachment lease to lose, so the node lease stands in: failing to renew it twice stops it. The attachment leases stay the authority for volumes; the node lease is the authority for liveness and for volume-less single writers. Decision 24’s far-location exemption still applies. - A watchdog outside shcd. The loop above runs inside shcd, so a stopped, hung or suspended shcd renews nothing and stops nothing while its containers keep writing. A small systemd unit on every node, separate from shcd, reads the lease expiry shcd writes to a local file after each committed renew. When that expiry passes, it stops the node’s single-writer containers itself, escalating to a kill, and records what it did. It needs no network and no control plane.
- The engine waits for the stop, and prefers proof. A replacement for a
single writer is bound on a STONITH confirm, or only after the old node’s
lease has been expired for longer than TTL + grace + the stop budget
(10 s times the node’s single-writer containers, plus the watchdog’s
escalation), measured as elapsed time from the expiry, not as a count of
ticks. Today’s compile-time assertion checks only dwell > TTL + grace
(
subsystems/volume_failover.go:108), and the dwell is three counted ticks rather than an elapsed interval (:465-500); both are fixed as part of this. The attachment lease generation bump is what refuses a stale remount if the old node was only slow (cemodules/clone/move_presumed_dead.go:36-41).
Lease margins (ruled 2026-10-01, the SHC-1182 fixes): the holder stops well before anyone else may take over, on a monotonic clock, so a paused VM that wakes past expiry has already lost; a lease lifetime is longer than a leader election, so an election stops nothing; a new leader honours every existing lease for one full lifetime. Proven on the rig before the cutover.
What remains is one case no timer closes: a node frozen whole (a suspended VM) resumes after the takeover with its containers mid-write, before its watchdog notices. Only a STONITH confirm excludes that. So a STONITH agent is used wherever the provider has one (Hetzner today), and a lease-expiry takeover is allowed only where it does not, and only with the watchdog installed (decision S16).
Stateless replicated services get no such wait; two copies of a stateless web process for a few seconds is harmless.
| exists | missing |
|---|---|
| the lease-loss loop and its timings; the failover dwell and its compile-time assertion; lease-generation-gated mounts; the Hetzner STONITH agent | self-stop for single writers without a volume; the watchdog; an elapsed-time wait that includes the stop budget |
3.11 High availability: stateless and stateful
Section titled “3.11 High availability: stateless and stateful”| stateless | stateful, single writer | stateful, clustered | |
|---|---|---|---|
| shape | N replicas, soft spread by node | one replica | N replicas with stateful: true, one per node |
| node loss | the survivors serve at once (endpoints drop the lost replica within the lease TTL); a replacement 60 s after the lease expired | down for the grace plus the move (5 min + rebind), never while the old node could still write (3.10) | the app’s own replication fails over; SHC replaces the lost ordinal on its own volume only if the volume is reattachable, else waits |
| update | rolling, start-first allowed | stop-first only | rolling by ordinal, stop-first |
| drain | surge on the target, then stop on the source | move; --ha refuses a gap (today’s budget) | one ordinal at a time |
“SHC has no HA” is true today and written into the scorer’s comment
(modules/placement/placement.go:160); this table is what replaces it.
3.12 Partitions, leader election, split brain, controller HA
Section titled “3.12 Partitions, leader election, split brain, controller HA”Leader election is raft’s (rqlite). The LeaderHub polls the leader id
every 5 s and fans OnLeader out (ce subsystems/impl/leader_hub.go:246-257);
the engine is one more leader-only subsystem. On a leader change the new
engine re-derives everything from the tables; there is no in-memory queue to
lose. The pending queue is a view over assignment.state = pending.
Split brain. Voters live in the control-plane fence (SHC-1050 §3.9);
non-voters cannot be elected; a minority cannot commit. A deposed engine on
the majority side could still reach the leader, because rqlite forwards
writes, so every bind carries the engine epoch (2.2): the new engine’s
epoch bump makes the old engine’s binds match no row. Epoch and generation
together mean one engine binds at a time and never on a stale read of the
assignment.
A partitioned node cannot renew its lease, stops its single writers
(except decision 24’s far volumes), and keeps serving stateless work from its
last clusterfs; its routes are frozen until the heal (SHC-1050 §5.8).
That the minority side stops its stateful work is intentional (2026-10-01):
the price of never running two primaries, and it means a two-node estate is
not highly available for stateful work. shc doctor warns when stateful
work runs with fewer than three voters; a witness node with no workloads
is the cheap third.
Controller HA is therefore the control plane’s HA: with 3 voters, one loss keeps the engine running after one election plus one LeaderHub poll.
| exists | missing |
|---|---|
| raft, the LeaderHub, leader-only subsystems, the control-plane fence design | the engine epoch and the epoch-guarded bind; the LeaderHub poll lowered to 1 s once the engine rides it (decision S12) |
3.13 Health checks: startup, readiness, liveness
Section titled “3.13 Health checks: startup, readiness, liveness”Kubernetes has three probes. Docker has one healthcheck with a
start_period; plain Docker never restarts an unhealthy container, Swarm
replaces it.
SHC maps the one healthcheck onto all three:
- startup is
start_period: failures inside it do not count. - readiness is
healthy: a replica joins Traefik backends and resolver answers only when healthy, and leaves them when unhealthy. A service with no healthcheck is ready once it is running (ruled 2026-10-01; this replaces the earlier rule that it waited out the updatemonitorperiod). 93 of the 106 catalog apps declare a healthcheck. - liveness keeps Swarm’s effect: a container unhealthy for
x-shc-health.restart_after(default 5 min,0disables) is restarted by the node agent. The default is longer than Swarm’s immediate replacement on purpose: the one measured case on these estates was a container unhealthy for 13.8 hours that nothing restarted (cesubsystems/impl/container_unhealth_watch.go:1-12), and a database crash-looping on a slow recovery is the failure a short timer invites.
Ruled 2026-10-01: the healthcheck is defined once, in compose; run by the
runtime (on apple, which has none, the node agent runs the test command
with container exec on the declared interval); read by the node agent;
recorded per replica and summed per service. Readiness flips are dampened
(several failures to go unready, several successes to come back), coalesced
to at most one revision a second, and Traefik and the resolver reload only
when their section’s digest changed; a replica that flaps repeatedly is held
out with a visible condition (3.26). Ruled 2026-10-18: a replica flaps once
its readiness flips 4 times in 10 minutes. It is held out only while
another replica of its service that does not flap is ready, so the only
ready replica of a service keeps serving, and it comes back after 5 minutes
of ready reads, an unready read starting them again. Held out or not,
shc doctor names it (deployments.replicas-flapping, condition
deployment.ReplicaFlapping) and its service’s health reads flapping.
| exists | missing |
|---|---|
compose up --wait gating install on health, podman emulated (ce modules/app/compose_driver.go:79-120); the unhealth watcher, which reports and does nothing else (ce subsystems/impl/container_unhealth_watch.go, modules/app/unhealth.go); readiness dampened, coalesced to a revision a second and reloaded on a changed section digest, and a flapping replica held out (ce modules/app/readiness_damper.go, modules/app/holdout, modules/clusterfs/materialiser.go) | readiness feeding endpoints; the liveness restart; x-shc-health |
3.14 Rolling updates and rollback
Section titled “3.14 Rolling updates and rollback”Kubernetes: maxSurge, maxUnavailable, readiness-gated, kubectl rollout undo. Swarm: update_config (parallelism 1, delay 0 s, failure_action: pause, monitor 5 s, max_failure_ratio: 0, order stop-first by default)
and rollback_config.
SHC honours update_config and rollback_config with Swarm’s defaults.
An update is a durable rollout row: deployment, from-spec, to-spec, the
batch in progress, state. The engine drives it replica by replica: move a
batch of parallelism assignments to the new spec, wait for each to be
ready and to stay ready for monitor, wait delay, next batch. A leader
change resumes the rollout from its row. A rollout that cannot finish
because a node is down stays pending, which is correct; hanging is not
(2026-10-01). The command returns after a timeout saying which replicas
moved and which node has not reported, and the rollout resumes on its own
when the node returns or rolls back on request.
order: start-firstcreates the new replica as a second incarnation of the same(deployment, service, ordinal)beside the old one, charged on its node (3.3), and retires the old incarnation once the new one is ready. It is refused at render for a single writer.- A replica that never becomes ready fails after its healthcheck’s
start_periodplusmonitor, bounded bytimeouts.compose_converge(900 s today, cecore/timeouts/timeouts.go:343). - On failure past
max_failure_ratio,pausestops and reports,rollbackre-applies the from-spec to the batches already done, andcontinuecarries on. shc update <deployment> --rollback[=<digest>]is the rollback on request. It makes a kept spec current again, the one the current spec replaced or the one the digest names, and writes the deployment row’s settings back from that spec in the same transaction. A rollout still running or paused is turned around underrollback_config; after a finished one, a new rollout takes the replicas to the kept spec underupdate_config. It puts the spec back, not data: volumes keep what the newer spec wrote, and the plan says so.shc backup restoreis the data path. A bare rollback with nowhere to go, a deployment that keeps one spec included, is refusedX404547DSPRBGnaming--rollback=<digest>.
The from-spec has to be kept, and today nothing keeps it: the converge
journal’s before-image is the list of containers that were running, not the
specification that produced them (ce modules/app/converge_journal.go:90-114),
and a converge overwrites the rendered artifacts
(ce modules/app/converge_restore.go:40-50). The new line stores each
deployment’s effective specification by digest in the replicated store and
keeps the last two, which is also what makes the dead-source move lossless
(3.9). A single-replica deployment on one node, the common case, updates
exactly as today: one batch, stop-first.
| exists | missing |
|---|---|
compose up -d --wait over the whole project; restart of containers a failed converge stopped (ce modules/app/converge_restore.go); the container before-image journal; settings rollback for an abandoned deploy (ce modules/app/service.go:927); update_config and rollback_config are accepted keys (ce modules/app/package_trust_compose.go:293-296) and used by no catalog app | the rollout row; batching by replica; incarnations for start-first; the monitor window; retained specifications and rollback to them |
3.15 Service discovery when a replica moves
Section titled “3.15 Service discovery when a replica moves”Kubernetes updates EndpointSlices; clients reach a stable ClusterIP. Swarm
gives each service a VIP on the overlay, or DNS round-robin with
endpoint_mode: dnsrr.
SHC. The per-node resolver (SHC-1050 decision 26) answers a service name with the fabric addresses of that service’s ready replicas, across nodes, from clusterfs. A reschedule changes the answer as soon as the new assignment is ready and materialised; a lost node’s replicas drop out within the lease TTL. Answers carry a 5 s TTL.
Names: the resolver serves the names the fabric bakes into extra_hosts
today (ce modules/fabric/pins.go:17-19), so nothing an app references
changes. A projection sets dns to the node resolver and dns_search to
the deployment’s zone and the environment’s zone
(<stack>.<environment>.<tenant>.shc.internal,
<environment>.<tenant>.shc.internal), so a bare web and a plug’s
proxy.postgres both resolve, and a name the compose network does not hold
falls through the runtime’s embedded resolver to the node resolver. Compose
adds a service’s own name as a network alias whatever the authored aliases
say, so every entry is named <service>-r<ordinal> (always, ruled
2026-10-01; S11 said only when replicas span nodes) and the bare name is a
local alias only when every replica of the service is on this node. The
local alias never hides remote replicas. Only the SHC resolver filters by
readiness; the local alias does not, so a local copy that is unready still
answers on it.
The resolver itself (ruled 2026-09-30): a few hundred lines of Go on
miekg/dns, one per node, not CoreDNS. It answers
<service>.<stack>.<environment>.<tenant>.shc.internal with the ready
replicas’ pins, nearest first, TTL 5 s; <node>.nodes.shc.internal with a
node’s mesh address; a routed hostname with the ready edges of the app’s
location, empty where the location has none (ce modules/resolver, zone
shc.internal). Everything else is forwarded to the host’s nameservers
from the same section. It never dials shcd and keeps answering from its
last revision through a partition. Wiring it into containers (dns,
dns_search) per runtime belongs to the executor.
A stable virtual IP per service (a per-node L4 proxy) is the later option
the owner named. Until it exists, the resolver’s shape is dnsrr.
| exists | missing |
|---|---|
pinned fabric addresses per service; render-time extra_hosts | the resolver (SHC-1050 phase 8.4); readiness-filtered answers; dns_search in projections |
3.16 Ingress without the routing mesh
Section titled “3.16 Ingress without the routing mesh”Swarm publishes a port on every node and routes it to a task anywhere
(mode: ingress). Kubernetes uses a Service plus an ingress controller.
SHC. HTTP and TLS go through a Traefik on every node (SHC-1050 §3.13),
whose routes and backends come from clusterfs: for each host, the fabric
addresses of the ready replicas, preferring the Traefik’s own location. The
engine is the source of the backend list; readiness and drain remove a
backend before its container stops. Traefik’s own loadBalancer.healthCheck
stays on as a second guard for the seconds a node’s copy lags. A Traefik per
node works now because it reads only clusterfs files that are identical on
every node, its backends are pins any node reaches, and certificates are
issued once by the daemon-side ACME source and delivered everywhere; the old
singleton existed because Traefik read local Docker labels and the routing
mesh carried 80 and 443 to it (2026-10-01; new-line simplification §3.2).
Published ports (10 catalog apps carry ports: blocks: mailcow, gitea’s
ssh, netbird’s signal and relay, kubo, ethereum, keycloak’s ldap, the
traefik edge). Under Swarm a published port, short form included, rides
the routing mesh and answers on every node. No SHC mechanism forwards one
port from every node, so reading the short form as host publication would
drop that reachability without saying so, which decision 21 forbids.
Ruled 2026-10-01, superseding the three bullets this section carried
(long-form mode: host honoured as host publication, short form and
mode: ingress refused, the five-app sweep of SHC-1081): an app never
publishes a host port by itself. ports: and expose: in compose both
declare only the container ports a service listens on; a ports: entry is
treated as expose: and its host side is not honoured. To keep decision
21 (nothing dropped in silence), the render warns when it discards an
authored host side and names --port; that warning is this design’s
reading, not part of the ruling. Publication is a
deployment decision: --host binds a service to a hostname through Traefik
as a route file; --port publishes on the node, or through a load-balancer
connection with conn@, and is the only thing that renders a ports:
entry (long form mode: host) on the replica’s compose entry, with the edge
and DNS for the name following the placement. Tenant reach is by policy.
Not ruled: where an app’s default mappings live, and --port on a service
with more than one replica (new-line simplification §9).
| exists | missing |
|---|---|
one Traefik, Docker-label provider plus file provider (ce addons/traefik/compose.yaml:624-651); the fabric’s per-node Ingress class (ce modules/fabric/policy/model.go:117-147); --host and --port on the deployment (ce modules/app/port_mappings.go) | Traefik on every node from clusterfs (SHC-1050 phase 8.1); backend lists from assignments; ports: read as a declaration with its warning; --port rendering the mode: host entry on the replica |
3.17 Global deployments
Section titled “3.17 Global deployments”Kubernetes DaemonSets and Swarm mode: global run one copy per eligible node
and follow nodes in and out.
SHC. deploy.mode: global is one replica per node of the eligible set.
Creating and keeping are separate questions, as they are for every replica
(2.3). The engine creates a global replica on a node that is in the
rule-derived set and ready. It removes one only when the node leaves
the rule-derived set (a fence edit, a node removed) or is drained. Cordon,
pressure and a lost lease do not remove it, so cordoning a node never takes
its ingress or resolver away. Drain does not move a global replica: it stops
last, when the node is empty, as kubectl drain --ignore-daemonsets leaves
DaemonSet pods.
The per-node platform set (Traefik, forward-auth, resolver; decision 26
keeps the builtin set at four apps) is a global deployment in the master
tenant with priority: system, admitted to every reserved fence (3.5). It
waits at boot like everything else (3.9). The other builtins (keycloak,
openobserve, postgres) are ordinary deployments; keycloak and postgres
belong in the control-plane fence’s location (SHC-1050 §3.15).
global-job is a global service whose replicas each run to one clean exit,
a node that joins later included, and replicated-job the same on its
deploy.replicas replicas. Neither is a job (3.21).
| exists | missing |
|---|---|
a single Traefik host (ce modules/app/ingress_materialize.go:88); Swarm global mode, which no catalog app uses | global mode in the engine; the platform set as a global deployment |
3.18 Drain, maintenance, disruption budgets
Section titled “3.18 Drain, maintenance, disruption budgets”Kubernetes: cordon, kubectl drain with PodDisruptionBudgets. Swarm: node
availability drain evicts every task at once.
SHC. Today’s EE drain keeps its shape (journal, pin gate, --ha as the
disruption budget, serial moves, resume) and changes its source of truth:
- targets come from the engine, per replica, inside each eligible set, not
from
docker service inspect(SHC-1050 phase 3.5); - a stateless replicated service drains with surge: the new replica is ready on the target before the old one stops;
- replicated services get an implicit budget of
max_unavailable: 1, declarable asx-shc-disruption.max_unavailable;--hakeeps its meaning for single-replica deployments; - global replicas stay until the node is empty.
Maintenance. shc node maintenance <node> --window 30m [--drain] cordons
the node and announces it rebooting with expected_back_by = now + window,
so the engine reschedules nothing from it inside the window. With --drain
it drains first. shc node maintenance <node> --end lifts it; the node’s
return (3.9) ends it too.
| exists | missing |
|---|---|
drain, undrain, cordon, the budget, the pin gate, the journal (modules/cluster/drain.go, drain_disruption.go, drain_pin_gate.go, drain_journal.go, cordon.go) | engine targets; surge drain; max_unavailable; shc node maintenance |
3.19 Rebalancing
Section titled “3.19 Rebalancing”Kubernetes has none by default (the descheduler is an add-on). Swarm never rebalances running tasks.
SHC. None automatic, as the rebalancer v1 ruling set it
(docs/decisions/2026-08-11-rebalancer-v1-foundations.md §0) and the owner
repeated on SHC-1058: nothing moves back except by operator request.
shc rebalance --plan prints the moves that would reduce skew and free the
most constrained nodes, each with its explain; --apply runs them through
the ordinary move. A rebalance rule attached to a scope is a later option
(decision S10).
The one exception: group members move apart (Clay, 2026-10-01). The
leader’s separation pass (members.separate, shc modules/separation)
makes the first moves a rebalance would make, one member off each node
that holds more than one member of a group, without an operator.
- What arms it. A node joins, a replica leaves a ready node, or a node
comes back after a failover moved a replica off it. Nothing else does.
The pass’s first read is a baseline, so a new leader, an shcd restart
and a daemon upgrade arm nothing. A group already sharing a node stays
put and reads “not redundant” until one of those events or an
operator’s
shc cluster rebalance. - What holds a move. The cluster must have been settled 10 minutes (no node joined, left or changed readiness), the freeze must be open and the move limiter must have room. No rollout may be under way, and every other member of the group must be ready. A member whose durable data stops it while it copies moves on its own only up to 20 GiB. Bigger copies need the operator’s rebalance, which shows the plan and its downtime.
- Pacing. One separation at a time across the cluster, announced 15
minutes ahead in
shc statusandshc doctor(“db-2 moves to n3 at T”). A group waits 10 minutes after a move, and a failed move backs off an hour. - Leader change. It drops the armed pass and the announced move. The next event re-arms it, and doctor’s “not redundant” line covers the gap.
3.20 Node-pressure eviction
Section titled “3.20 Node-pressure eviction”The kubelet evicts pods under memory, disk or PID pressure. Swarm does not.
SHC. No eviction. A node under pressure becomes pressured (2.3’s
table) and takes no new work; the kernel’s OOM killer enforces memory limits,
and the disk crons keep pruning (ce modules/health/crons.go:198-200,
CronDiskAutoPrune, CronImagePrune). Evicting by priority would be a
preemption by another name (3.7).
3.21 Jobs and cron
Section titled “3.21 Jobs and cron”Today. The leader’s schedule dispatcher enqueues task rows
(ce subsystems/impl/schedule_dispatcher.go); workers claim them with a
compare-and-swap that honours affinity_node_id
(ce modules/task/service.go:434-447, ent/schema/task.go:78); a stale
claim is released after 60 s (ce modules/task/worker.go:77-78); a
container job runs docker compose run --rm on the claiming node
(ce modules/job/context_container.go). Backups run on the node that holds
the volume.
SHC. A job is placed by the engine like a replica and gets the chosen
node as its affinity_node_id. A job of a deployment colocates with the
deployment’s node by default, since that is where its volumes are. A task
whose affinity names a node that has been lost past the stateless grace is
re-placed instead of waiting forever (today it waits). Swarm’s replicated-job
and global-job run as replicas, not jobs (3.17). Schedules stay keyed by scope, not node
(SHC-1050 §3.16), so a reschedule changes nothing about when they fire.
| exists | missing |
|---|---|
| dispatcher, task affinity, stale-claim recovery, container context | engine placement for jobs; re-placement off a down node; the two job modes |
3.22 Tenant quotas
Section titled “3.22 Tenant quotas”Today. Caps per stack, environment and tenant, enforced before the
runtime moves, charged on render as quota × replicas
(ce modules/app/quota_seam.go:41-67,117, modules/quota/).
SHC. Unchanged in shape. The charge is the desired replica count, charged
at admission, so a pending replica is charged; a global deployment is charged
per node it runs on. A replica refused by a tenant cap is refused, not
pending (reason quota).
3.23 Swarm compose keys: honoured or refused
Section titled “3.23 Swarm compose keys: honoured or refused”The rule (owner, 2026-09-29): a key that had an effect under Swarm keeps it
or is refused at render, by name. Ruled 2026-09-30 (new-line simplification
Q1): a key with a native form is translated and the render warns, naming
the native form; a key with none is refused. Counts are over the 106 apps
at apps/<tier>/<app>/compose.yaml.
| key | catalog use | under SHC |
|---|---|---|
deploy.replicas | 1 app | replicas across nodes (3.6) |
deploy.mode: replicated / global | 0 | engine placement / one per eligible node (3.17) |
deploy.mode: replicated-job / global-job | 0 | replicas that each run to one clean exit: deploy.replicas of them / one per eligible node (3.17) |
deploy.placement.constraints | 0 | rules: node.hostname==X → fence=node:X, node.labels.K==V → fence=K:V, != → avoid=, each with a warning naming the native form; anything else refused (SHC-1050 §3.7). node.role==manager is refused too: there are no managers, and the control-plane fence holds non-voters as well as voters (SHC-1050 §3.9), so fence=control-plane would not mean the same thing |
deploy.placement.preferences (spread: node.labels.K) | 0 | spread=K (soft), with a warning naming it |
deploy.placement.max_replicas_per_node | 0 | spread=node,max=<n> (3.6), with a warning naming it |
deploy.resources.limits | most apps | compose applies them |
deploy.resources.reservations | 4 apps (onlyoffice, openarchiver, pulp, stirling-pdf) | capacity (3.3); where absent, derived from the limit as today (ce modules/app/resource_reservations.go) |
deploy.resources.reservations.devices | jellyfin, ollama (as runtime: nvidia or commented out) | an NVIDIA GPU (capabilities: [gpu], driver nvidia or none) is the device fence: the replica needs a node whose capabilities record gpu-nvidia, and incus and podman get it as a CDI device; any other device is refused (NODEFACTS) |
deploy.resources.reservations.generic_resources | 0 | refused: no fence for it |
deploy.restart_policy | 0 | the node agent’s restart policy, with condition, delay, max_attempts and window (3.9); removed from the runtime’s document so it cannot override restart: "no"; restart: still governs when both are set (ce X200932RSTDIA, modules/app/errors_reboot.go:12-31) |
deploy.update_config, rollback_config | 0 | rolling updates (3.14) |
deploy.endpoint_mode: dnsrr | 0 | honoured: the resolver’s shape |
deploy.endpoint_mode: vip | 0 | refused until the per-node L4 VIP exists |
deploy.labels | 0 | container labels; traefik.* ones become file routes like labels |
top-level configs | 12 apps | staged onto the chosen node; a content change rolls the service through the SHC-1040 per-service digest (ce modules/app/mount_digest.go) |
top-level secrets | 0 | rendered from the vault into a RAM-only mount at start, never an on-disk bind |
networks with driver: overlay | 0 | the fabric, with a warning naming the native form |
ports, any form | 10 apps | declares the container ports only, as expose: does, and the host side is not honoured (ruled 2026-10-01); the render warns, naming --port (3.16) |
ports long form mode: host | 0 today | the same declaration; publication comes only from --port on the deployment, which renders this form on the replica’s entry (3.16) |
healthcheck | 93 apps | startup, readiness, liveness (3.13) |
depends_on with conditions | 50 apps | inside one projection, compose honours it as today. Across nodes the node agent holds the consumer’s start until the provider’s assignment reaches the state the condition names: service_started running, service_healthy ready, service_completed_successfully a one-shot that exited 0 |
What goes away: the Swarm loader reconciliation (ce modules/app/swarm_compat.go,
its four constructs exist only because docker stack deploy uses a narrower
loader), the file-bind lift into Swarm configs
(ce modules/app/swarm_file_configs.go), and the node pin that made
deployments with rendered files immobile (ce modules/app/swarm_mobility.go). The rule
that strips deploy.restart_policy so compose cannot prefer it
(ce modules/app/restart_effective.go) stays: the boot gate depends on it.
3.24 The cluster config directory
Section titled “3.24 The cluster config directory”The owner’s pmxcfs-like design (SHC-1058, 2026-09-29): a materialiser in shcd, not FUSE. Who writes it (2026-10-01): every node’s own materialiser, from rqlite; not the leader and not a shared filesystem. On the rig home01 and home02 carry the same revision digest, written at different times into differently numbered local directories.
/var/lib/shc/clusterfs/ REVISION control-plane revision, its digest and write time; written last, once every section is in; the node agent's completion marker node/ current -> rev-000184213 swapped atomically after a full write rev-000184213/ REVISION every section carries its own copy assignments.json this node's replicas: deployment, service, ordinal, generation, spec digest, state projections/<tenant>/<environment>/<stack>/compose.yaml traefik/ the Traefik container mounts only this current -> rev-000184213 rev-000184213/ dynamic/*.yaml every route, backends = ready replicas certs/ issued once, from the cert broker (0700) resolver/ the resolver mounts only this current -> rev-000184213 rev-000184213/zones/*.json names → ready replica fabric addresses, node names → mesh addresses auth/ forward-auth mounts only this current -> rev-000184213 rev-000184213/ realm per host, gating declarationsThe first layout was one rev-<n>/ tree holding every section behind one
current. Superseded 2026-10-01 by a revision directory per section (Mounts,
below).
- Writes go only through the API into the replicated store. Nothing writes into clusterfs but the materialiser.
- Render, enrich, materialise (ruled 2026-10-01). Render is Jinja over
the app’s files and the deployment’s settings, once, on the leader, at
install or update, with a per-replica
replicaobject (ordinal,location,node); the result is a per-replica spec stored in rqlite with a digest. Compose${VAR}is not used for per-replica values; it resolves once per file. Enrichment is the part of that render that adds what the author did not write: the pin,dnsanddns_search, labels, theSHC_*variables, CA trust, telemetry, the volume suffix. Materialising is this node writing the stored specs and the derived Traefik and resolver sections into its revision directory. Nodes never render or enrich, so a daemon upgrade changes no projection until an explicit update; a spec that uses onlyreplica.ordinalmoves as an assignment change with no re-render, and one that usesreplica.nodeorreplica.locationis re-rendered by the leader for the new placement. clusterfs is never an input: nothing that decides or renders reads it. - Consistency. The materialiser reads the revision counter, reads what
it needs, and reads the counter again; a changed counter discards the
build and starts over, so one revision directory never mixes two states.
Each section’s directory is written whole and fsynced, then that
section’s
currentis swapped with a rename, andREVISIONis written last. A reader of a section sees one revision of it or the next, never half of one. The last three revisions of each section are kept. - Freshness. A revision counter row bumps with every write that changes
what a node would materialise. The materialiser polls it every second and
is kicked on the event bus, the shape today’s ingress sweep already uses
(ce
subsystems/impl/ingress_materialize.go:37,Kick()on an exposure change). The poll and the kick are only how the materialiser notices a new revision; events themselves are durable, aneventrow broadcast to every peer with retry and recovery on start (cemodules/events). Each section carriescurrent/REVISIONwith its revision, digest and write time, so a container that mounts only its section can say how stale it is; the node agent reads the rootREVISIONcompletion marker;shc node statusprints both. - Mounts. A bind mount of a
currentlink would pin the revision it resolved at container start. This design first had every consumer mount the stable parent,/var/lib/shc/clusterfs, read-only, and read throughcurrent/inside the container, the relative symlink resolving at each access. Superseded 2026-10-01: a platform container mounts only its own section, read-only (the resolverresolver/, Traefiktraefik/, forward-authauth/), never the whole tree, so each section gets its own revision directory and its owncurrentand the mount is that section’s stable parent; a bind mount still never pins a stale revision. Traefik’s file provider points atcurrent/dynamicinside its section and gets a SIGHUP after each swap, as the daemon already sends after its writes today (ceaddons/traefik/compose.yaml:634-651,modules/app/traefik_reload.go); the resolver reloads on the same signal, and either reload fires only when that section’s digest changed (3.13). - Small files only. App payloads keep the staging path.
- Secrets never land here in plaintext. Projections carry
ref+vault://references; the node agent resolves them at start into the RAM-only mount of 3.23. Today’s renderedcompose.yamlinterpolates plaintext secrets (cemodules/app/secret_file.go, the list of credential-bearing artifacts); the projection must not. Traefik’s private keys are the one exception, readable only by root and Traefik. Ruled 2026-10-01 (SHC-1159, 3.26): the node agent caches only values resolved from external vault providers (aws, 1password, hashicorp and the rest) for the replicas this node currently runs, in a root-only per-node file and never in rqlite; each value is sealed with the vault’s existing envelope scheme (a fresh AES-256-GCM DEK wrapped by the in-memory Argon2id KEK from unlock), so a node that cannot unlock its vault cannot read its cache, and an entry drops when its replica leaves the node. Local-provider secrets are not cached: every node already holds them, encrypted, in its rqlite copy. This narrows the cache stated earlier the same day, which covered every resolved secret. - Partitions. A cut-off node keeps its last revision and keeps serving from it (SHC-1050 §5.8).
3.25 Backups of a stopped app
Section titled “3.25 Backups of a stopped app”The owner asked that every app’s backup run against a stopped or crashed
app, because failover and restore-as-move depend on it. Today every database
SHC ships quiesces itself in a pre_backup hook run inside the running
container (ce modules/backup/hook_bracket.go:1-20), which a crashed app
cannot answer.
SHC. A backup of a deployment that is not running skips the hook bracket
and captures its volumes as a crash-consistent file archive, and the archive
records consistency: crash. Every database app recovers from a crash
image on start (postgres and mysql replay their logs). A catalog test runs
backup and restore against each database app stopped, and an app that
cannot be restored from a crash image declares it, which makes it pinned
for failover and makes explain say why.
3.26 Six gotchas, with verdicts
Section titled “3.26 Six gotchas, with verdicts”Raised 2026-10-01 while walking the loop of 2.2; each carries Clay’s verdict and what the design does about it.
| gotcha | verdict | what the design does |
|---|---|---|
| Lease timing. A paused VM wakes believing it holds an expired lease; an election longer than a lease stops every stateful replica for nothing | fix (SHC-1182) | the holder stops well before anyone may take over, on a monotonic clock; lease lifetime exceeds election time; a new leader honours existing leases for one lifetime. A frozen VM that resumes after a takeover is excluded only by a STONITH confirm; without one, the residual risk of 3.10 remains, which is why a lease-expiry takeover needs the watchdog (S16). Proven on the rig before the cutover |
| A daemon upgrade restarts everything. If nodes rendered, a new shcd rendering one label differently would recreate every container on every node | fixed by render once | the leader renders at install or update and stores each spec with a digest; nodes write stored specs; nothing changes until an explicit update (2.2, 3.24) |
| The minority side stops its stateful work. A cut-off node cannot renew and stops its databases although it is fine | intentional | the price of never two primaries. shc doctor warns when stateful work runs with fewer than three voters; a witness node is the cheap third (3.12) |
| A flapping healthcheck churns the estate. Every readiness flip is a revision every node writes and may reload on | dampen and coalesce | several failures to go unready and several successes to return; at most one revision a second; Traefik and the resolver reload only on a changed section digest; a repeat flapper is held out visibly (3.13) |
| Secrets at restart during a partition. Projections carry vault references the agent resolves at start; a cut-off node cannot resolve an external-provider ref to restart a container | ruled on (2026-10-01, SHC-1159): external providers only | a root-only per-node file of the values resolved from external vault providers for current replicas, never in rqlite, each sealed with the vault’s envelope scheme (fresh AES-256-GCM DEK wrapped by the Argon2id KEK from unlock), dropped when a replica leaves. Local-provider secrets need no cache: every node holds them encrypted in its rqlite copy. The trade is external values resting on disk, sealed, and a rotation unseen until the provider is reachable again (3.24) |
Waiting for acknowledgement forever. A node down mid-update would hang shc update | the rollout is a stored, resumable object | the command returns after a timeout with which replicas moved and which node has not reported; the rollout resumes when the node returns or rolls back on request (3.14) |
4. Explain
Section titled “4. Explain”shc fence explain (SHC-1050 §3.17) is the scheduler’s one explain. This
design adds the replica table, the pending and refused reasons, the
capacity step, spread, the failover plan and the history.
4.1 Text
Section titled “4.1 Text”A placed deployment:
$ shc fence explain -s nextcloud -t acme -e proddeployment acme/nextcloud/prod docker priority normal movability reattachstatus placed 1/1 ready
replicas SERVICE REPLICA NODE GEN STATE READY SINCE app 0 gor-02 4 ready yes 2026-10-02 14:03
eligible gor-01 gor-02 resource volume nc-data stay in storage:ceph-ogden gor-01 gor-02 gor-03 integration postgres -> acme/postgres/prod share location (socket default) location:gorillaservers/ogden tenant acme stay in site:us-west (2026-10-02) gor-01 gor-02 hil-01 deployment acme/nextcloud/prod no rules condition ready gor-01 gor-02 capacity needs 2.0 GiB, 500m fits gor-01 gor-02move group acme/nextcloud/prod, acme/postgres/prod (location)
scored gor-02 812/1000 memory 930 cpu 700 count 640 tenant 900 app 1000 gor-01 640/1000 memory 610 cpu 690 count 500 tenant 900 app 1000failover 5m0s after its node's lease expires: dead-source move inside storage:ceph-ogden (rebind)
history 2026-10-02 14:02 rescheduled app/0 gor-03 -> gor-02 5m0s after gor-03's node lease expired (13:57)A pending one names every node and what removed it, the Kubernetes “0/5 nodes are available” line with the fence that did it:
status pending since 14:05 (3m) app/0 0/5 nodes fit 2 outside site:ogden tenant acme hil-01 hil-02 1 cordoned gor-03 2 insufficient memory: needs 4.0 GiB gor-01 (1.2 GiB free) gor-02 (0.8 GiB free) nothing is retried until capacity or rules changeA refusal prints the SHC-1050 lines up to the first fence that emptied the set, then the commands that would satisfy it.
4.2 JSON
Section titled “4.2 JSON”-o json is the document the HUD :placement view and the API return:
{ "deployment": {"tenant": "acme", "stack": "nextcloud", "environment": "prod", "runtime": "docker", "priority": "normal", "movability": "reattach"}, "status": {"state": "placed", "since": "2026-10-02T14:03:10Z", "ready": 1, "desired": 1, "reason": null}, "replicas": [{"service": "app", "ordinal": 0, "node": "gor-02", "generation": 4, "state": "ready", "ready": true, "since": "2026-10-02T14:03:10Z"}], "eligible": { "nodes": ["gor-01", "gor-02"], "steps": [{"source": "resource", "object": "volume nc-data", "rule": "fence=storage:ceph-ogden", "scope": null, "fence": "storage:ceph-ogden", "kept": ["gor-01", "gor-02", "gor-03"], "removed": ["hil-01"], "loosened": null}], "emptied_by": null }, "move_group": {"members": ["acme/nextcloud/prod", "acme/postgres/prod"], "derived": [{"kind": "location", "fence": "location:gorillaservers/ogden", "from": "acme/nextcloud/prod -> acme/postgres/prod"}]}, "spread": [{"service": "app", "kind": "node", "skew": 1, "hard": false, "domains": {"node:gor-01": 0, "node:gor-02": 1}}], "score": {"weights": "spread", "candidates": [{"node": "gor-02", "score": 812, "runtime_match": true, "terms": [{"name": "memory_headroom", "normalised": 930, "weight": 40, "abstained": false, "reason": ""}]}]}, "pending": [], "failover": {"movability": "reattach", "after_lease_expiry": "5m0s", "path": "presumed-dead rebind"}, "history": [{"at": "2026-10-02T14:02:41Z", "verb": "rescheduled", "service": "app", "ordinal": 0, "from": "gor-03", "to": "gor-02", "reason": "node_down", "detail": "5m0s after gor-03's node lease expired (13:57)"}]}score.candidates[].terms is today’s placement.Term, unchanged
(modules/placement/placement.go:188-200).
4.3 Reasons
Section titled “4.3 Reasons”One vocabulary for pending[].reason, status.reason and
history[].reason, so a script can match on it:
outside_fence, avoided_fence, reserved_fence, volume_reach,
colocation, apart, spread, max_per_node, separate, condition (with the node’s
condition), insufficient_cpu, insufficient_memory, port_in_use,
architecture, runtime, quota, node_down, node_returned,
operator_move, drain, rebalance, update.
capacity_unknown is not a reason to remove a node; it is a flag on a kept
one.
5. Surfaces
Section titled “5. Surfaces”5.1 shc fence
Section titled “5.1 shc fence”Today there is no shc fence command; SHC-1050 §3.18 listed its verbs. This
is their exact surface, built by SHC-1079 with the HUD views of 5.3. Every verb takes the global -o tty|text|json|yaml
and --fields. Fence ids are <kind>:<name>; <node> is a node name or id,
as every shc node verb takes it.
shc fence ls [--kind <kind>] [--origin derived|resource|system|custom] [--node <node>] [--all]
Lists fences. Without --all it hides the node: singletons and resource
fences, which number one per node and one per volume. --node lists the
fences a node is in, which includes them.
FENCE KIND ORIGIN NODES READY RESERVED RULES DEPLOYMENTScontrol-plane control-plane system 3 3 no 0 2location:hetzner/hel1 location derived 3 3 no 1 14location:unknown location derived 1 1 no 0 0provider:hetzner provider derived 3 3 no 0 0site:ogden site custom 2 2 no 3 6-v adds MEMBERS, BACKUP-REPO, DESCRIPTION, CREATED. Warnings print
under the table: an unlocated node on a multi-site estate (SHC-1050 §3.1),
a control-plane fence with fewer than three ready nodes or no spare
(SHC-1050 §3.9). JSON rows carry id, kind, name, origin,
description, reserved, backup_repo, members[] {node, source, condition}, rule_count, deployment_count, created_at, warnings[].
shc fence show <fence>
The fence’s attributes, then three tables: MEMBERS (node, condition,
availability, source: derived, terraform, cli or join), RULES naming it
(scope, object, form, set by, when, reason), and DEPLOYMENTS it shapes
(deployment, the rule that brings the fence in, where it runs).
shc fence create <kind>:<name> [--node <node>]... [--description <text>] [--reserved] [--dry-run]
Creates a custom fence. kind matches [a-z][a-z0-9-]{0,31}, name
[a-z0-9][a-z0-9.-]{0,62}. Refused by name: a built-in kind (node,
provider, location, control-plane), a resource kind (volume,
storage, lb, ip, endpoint), an id that exists, an unknown node.
--reserved refuses when a deployment already on a listed node does not
name the fence, naming each one. Prints the created fence as show does.
shc fence rm <fence> [-y] [--dry-run]
Deletes a custom fence. Refuses a non-custom origin, and refuses while any
rule names the fence, listing each rule with the shc rule rm that removes
it. Confirms unless -y.
shc fence add-node <fence> <node>... [--dry-run]
shc fence remove-node <fence> <node>... [--dry-run]
Membership of a custom fence, or of control-plane. Both are idempotent.
Either refuses, naming the deployment and the rule, when a deployment running
on the node would leave its eligible set: removing the node from a fence the
deployment is held in, or adding it to a reserved fence the deployment does
not name. The fix is to move or drain first; there is no --force, because
a fence edit that strands a running deployment is the action at a distance
fences exist to prevent. remove-node on control-plane refuses a node that
holds a voter slot and warns below three members.
shc fence set <fence> [--description <text>] [--reserved|--no-reserved] [--backup-repo <url>|--no-backup-repo]
Attributes. --reserved applies to custom, location, provider and
control-plane fences and refuses as create --reserved does.
--backup-repo applies to location fences only (SHC-1050 §3.16).
shc fence explain -s <stack> [-t <tenant>] [-e <environment>] [--service <service>] [--history <n>] [--what-if-node <node>] [--what-if-rule <rule>]...
Section 4. The deployment is addressed the way every deployment verb
addresses it, by the scope flags -s, -e and -t (or shc ctx), never
a tenant/environment/stack path (2026-10-01); the header prints the
tenant/stack/environment triple. --what-if-node explains one node’s
verdict as if it were the only candidate; --what-if-rule evaluates the set
with extra rules added and changes nothing. --history bounds the history
rows (default 10).
Authority. Reads are open to any authenticated operator; explain
follows the deployment’s tenant authorisation, so a tenant admin can explain
their own deployments and sees fence ids but not other tenants’ rules.
Fence writes need platform:manage (SHC-1050 decision 27).
Where it lives. The fence store, its reconciler, the CLI and the explain renderer are in ce, because a single-node CE estate has built-in fences and explain answers there too. The multi-node engine loop, rescheduling and failover are EE in shc, where the scorer, drain and volume failover already are.
5.2 Other command changes
Section titled “5.2 Other command changes”| command | change |
|---|---|
shc install, update, clone | --fence, --avoid, --colocate and --admit take <kind>:<name>, and --location <name> is --fence location:<name>; --rule is gone (SHC-1237, 2026-10-02). spread and apart are engine-derived (3.4, 2026-10-01). There is no move or bind verb. --scale on install and update takes the three shapes of 3.6 |
shc rule add|ls|rm | SHC-1050 §3.18, with the new forms |
shc scale | goes (ruled 2026-10-01); --scale on shc install and shc update places the replicas across nodes and prints the explain replica table |
shc node ls | CONDITION column (2.3) |
shc node status | on the node: boot gate state, clusterfs revision and age |
shc node maintenance <node> --window <d> [--drain], --end | 3.18 |
shc rebalance --plan|--apply | 3.19 |
shc cluster status | engine state: pending replicas, nodes lost or down, grace clocks running |
5.3 HUD
Section titled “5.3 HUD”Today the HUD has no fence type: shc hud is x9s with the provider in
pkg/x9sprov (cli/hud.go), and every type is a resource returned by
catalogResources() (pkg/x9sprov/catalog.go:276-318). Three types are
added, each in its own type_<name>.go with its test, so the shared files
change by one registration line each (the house rule for parallel HUD
lanes). Each follows the rules of docs/design/hud-types.md §1: every verb
is a CLI verb first, every key comes from a declared action or relation, and
the confirm dialog prints the exact shc command.
:fences (alias fence). Platform-wide (Scoped: false), like :nodes.
Lists what shc fence ls --all lists.
- Narrow:
FENCE(name role) ·KIND·ORIGIN·NODES·READY·RESERVED·RULES·DEPLOYMENTS. - Wide:
MEMBERS·BACKUP-REPO·DESCRIPTION·AGE(age role). - Relations (
v):nmember nodes (Drill: Enter),rrules naming it,ddeployments it shapes. - Keys: Enter →
:nodesfiltered to members. ctrl-n create (shc fence create), a form with kind, name, description, reserved, and one toggle per node, the shape:storage’s node form already uses (pkg/x9sprov/type_storage_nodes.go:100-115).eedit: the same form prefilled; the confirm prints oneshc fence setplus oneadd-nodeorremove-nodeper toggled node. ctrl-d delete (shc fence rm <fence> -y), Danger, offered only whereORIGINiscustom(the action’sWhen). dis theshc fence show -o jsondocument.
:rules (alias rule). Scoped to the active tenant; estate-rung rules
show in every tenant with SCOPE global, read-only for a tenant admin.
- Narrow:
SCOPE·TARGET(the scope key) ·FORM·RULE(the rule as its flag without the dashes,fence storage:fast) ·LOOSENS·SET-BY·AGE. - Wide:
REASON·FENCE·ID. - Relations:
fthe fence it names,dthe deployments it binds. - Keys: ctrl-n add (
shc rule add --fence <kind>:<name>with the scope flags; a deployment-rung rule printsshc update -s <stack> --fence <kind>:<name>); ctrl-d remove (shc rule rm <id>). ctrl-j onFENCE→:fences.
:placement (hidden, drill-only, read-only). The explain view.
- Reached by
vpon a:deploymentsrow (pis free in that picker:pkg/x9sprov/catalog.go:401-414usess c b h j v k g i n e a w r). - Rows are explain’s lines:
STEP(replica, rule, condition, capacity, spread, score, failover, history) ·SOURCE·OBJECT·RULE·FENCE·NODE·LEFT(nodes remaining) ·VERDICT(kept, removed, chosen, pending, refused). dis theshc fence explain -o jsondocument. ctrl-j followsFENCEto:fencesandNODEto:nodes.
Existing types gain: :nodes a v f relation (the fences the node is
in; f is free, the picker uses d v t w, catalog.go:1183-1186) and a
CONDITION column; :deployments a wide PLACEMENT column (placed,
pending, refused, down, moving) and the v p relation; :tenants a
v p relation to :rules at that tenant (the picker uses
d u q k g c r, catalog.go:1573-1579).
No global key changes: every key above is already in the verb table with
the same meaning (docs/design/hud-types.md §2), so the collision test
holds. hud-types.md gains §3.43 to §3.45; testdata/hud_commands.txt is
regenerated after checking each new printed line is a real command
(TestEveryHUDExplainParses), and each new daemon call names its tenant the
way the route reads it (SHC-973).
5.4 API
Section titled “5.4 API”Routes in the existing /api/method/shc.<module>.<verb> shape:
shc.fence.list, get, create, delete, add-node, remove-node,
set, explain; shc.rule.list, add, delete. Fence routes are
platform-wide and read no tenant; shc.fence.explain and the rule routes
read the tenant the deployment or scope belongs to. The openapi document
regenerates.
5.5 Terraform
Section titled “5.5 Terraform”SHC-1050 §3.18 (shc_fence, shc_rule, rules on shc_deployment,
location, provider, fences on nodes), plus reserved and
backup_repo on shc_fence.
6. Data model
Section titled “6. Data model”All in the new line’s single initial migration (SHC-1050 §6.3), beside SHC-1050 §5.6:
assignment(deployment_id, service, ordinal, incarnation, node_id, generation, state, spec_digest, target_node_id, previous_node_id, reason, updated_at), unique on(deployment_id, service, ordinal, incarnation). Two incarnations of one ordinal exist only during astart-firstupdate or a surge drain (3.14, 3.18).recorded_node_idkeeps its SHC-1050 §3.7 meaning, where the bytes are: the node holding the deployment directory and its persistent volumes. For the default shape, every service on one node, that is also where every replica runs. For a split deployment it is the node of its stateful services, and each replica’s node is on its assignment. The readers that assume every container sits onrecorded_node_id(logs, exec, deployment inspect, the container recorder’s per-deployment status) read assignments instead; backup capture already goes to the node holding each volume.deployment_spec(deployment_id, digest, spec, created_at): the full effective specification (app digest, vars with secrets as vault references,--hostand--portmappings, integrations, posture), the last two kept per deployment. Rollback (3.14) and the lossless dead-source move (3.9) read it. It is also the home of the per-replica rendered specs and their digests, which the leader writes at render and the materialiser copies (render once, 2026-10-01).rollout(deployment_id, from_digest, to_digest, batch, state, updated_at).scheduler(epoch, holder_node_id, since): one row, bumped by each new engine (2.2).node_lease(node_id, boot_id, leased_until, renewed_at).node:cluster_stategainsrebooting; newexpected_back_by,reserved_cpu_millicores,reserved_memory_bytes.lostanddownare derived fromnode_leaseand the grace clocks, never stored.deployment:priority.placement_event(deployment_id, service, ordinal, at, verb, from_node, to_node, reason, detail), bounded to the last 100 per deployment.revision(id, value, updated_at): one row, bumped by every write a node would materialise.fence.reserved(bool) and theavoid,apart,spreadandadmitforms inplacement_rule.form.- Compose extensions:
x-shc-scale.stateful(wasidentity; renamed 2026-10-01, no alias),x-shc-health.restart_after,x-shc-disruption.max_unavailable.
7. Before Swarm leaves an estate
Section titled “7. Before Swarm leaves an estate”An estate may drop Swarm when all of these hold on it, each proven on a rig first:
- Nebula on every node as the addressing plane (SHC-1050 phase 1). The two-tunnel mesh (SHC-1201) lands after the cutover (ruled 2026-10-01).
- Fabric slices 2 and 3 and Traefik on the fabric (SHC-1050 7.1 to 7.3).
- The engine places every install through the dispatcher inside fences (SHC-1050 phase 3; this design’s B phase).
- Node leases, conditions, the boot gate and node-agent restarts (A phase).
- Rescheduling by movability and the node-return path (C phase).
- Per-node projections and the resolver answer every cross-node name the estate’s deployments use (B3, SHC-1050 8.4).
- The per-node platform set on every node and an edge per location (SHC-1050 8.1, 8.3).
- The render honours or refuses every Swarm key, and a dry-run render of every deployment installed on the estate refuses nothing (E phase).
- Every deployment’s backup restores from a stopped app on a scratch node (3.25).
- Drain works with no
docker service inspectread (SHC-1050 3.5). - Every deployment’s full specification is in the replicated store, so no automatic move comes back on app defaults (3.9).
- The single-writer watchdog runs on every node, and every node whose
provider has a STONITH agent has it configured (3.10). It stops single
writers on docker, podman, incus and apple, and on macOS a launchd job
(
org.selfhostedcloud.shc-watchdog) runs it. - Every capability in the feature parity inventory works on the rig (new-line simplification §5). Deleting Swarm removes a mechanism, never a capability.
With the next branch these thirteen conditions are the cutover gate for
next becoming main (SHC-1165), not a gate on deleting Swarm code, which
the new line does first.
bitspur moves by SHC-1061’s path B (fresh cluster, app by app; SHC-1050 §6.3):
- Dry run of the one-off script: inventory (
shc ls --all-tenants -o json), every deployment rendered under the new line with refusals listed, move groups in order, every DNS change. Stop on any refusal. - Verified backups of every deployment and of the control plane; stop on any failed verify.
- Stand up the new-line cluster as fresh VMs on both Gorilla boxes (pve02, pve03) under Proxmox, docker runtime, with terraform: Nebula, the control-plane fence, site fences. Each app’s data arrives by restoring its restic backup, never by copying volumes off compose01, and compose01 is never upgraded in place (ruled 2026-10-01; this replaces the earlier hel1 target). After the cutover pve02 and pve03 become bare Incus hosts.
- Restore the master tenant first (postgres, keycloak, openobserve), then tenants in move-group order, app by app with both estates live, each one: a fresh backup right before the move, restore, health gate, DNS re-point, old copy left running until the new one passes.
- Point the edge (pfSense HAProxy) at the new Traefiks.
- Soak: compose01 stays untouched as the rollback for two weeks after the last app moves (ruled 2026-10-01), switching back being a DNS change plus starting the old deployment; then retire it, keep its disks until the owner signs off, and delete the script.
promanager is not touched. It stays on its current line until the owner decides otherwise.
8. Phased build order
Section titled “8. Phased build order”The order below is superseded by the delete-first order in the new-line simplification (SHC-1092), which merges B1 into 3.2, B2 into 2.3, inverts B5 and SHC-1050 8.1, and moves G to the front. The steps’ content stands.
One numbered step is one ticket and one MR. The letters are this design’s phases; the numbers in brackets are SHC-1050’s.
A. Node truth. Depends on SHC-1050 phase 0.
- A1
node_leaseand its renew loop on every node; node conditions;CONDITIONinshc node ls. - A2 The clusterfs materialiser: revision row, consistent builds, atomic revision directories per section, each platform container mounting only its own section (2026-10-01); the ingress sweep moves onto it.
- A3
assignmentrows with generations and incarnations; theschedulerepoch and the epoch-guarded bind; the dispatcher executes assignments for single-replica installs; the readers ofrecorded_node_idthat mean “where the containers are” move to assignments. - A4 Node-agent restarts and the boot gate:
restart: "no"and nodeploy.restart_policyin the runtime’s document, the restart supervisor on Docker events, podman-restart kept off, the stop journal and converge restore reworked onto it. - A5
deployment_spec: the full effective specification replicated, the last two kept. - A6 The single-writer watchdog, a systemd unit outside shcd. It stops
single writers on docker, podman, incus and apple, and on macOS a launchd
job (
org.selfhostedcloud.shc-watchdog) runs it.
B. The engine. Depends on A3 and SHC-1050 steps 3.1 to 3.5. SHC-1050
step 3.6 keeps turning deploy.placement.constraints into rules; its
replicas and mode: global half moves to B4 and B5 here, so neither design
waits on the other.
- B1 Fit as a filter: the capacity gates called, the system reserve,
pendinginstead of “no opinion”. - B2 Explain v2: replica table, reasons,
placement_event, JSON (section 4). - B3 Per-node projections with
dns_search(needs SHC-1050 8.4). - B4 Replicas across nodes, one compose entry per replica, the three
--scaleshapes,spread,stateful: true, per-ordinal volumes and pins. - B5 Global mode; the per-node platform set as a global deployment (needs SHC-1050 8.1).
- B6 Reserved fences and the
avoid,apartandadmitforms; the system-priority admission. - B7 Priority.
C. Failure. Depends on A and B1.
- C1 Movability derivation; grace clocks from lease expiry; stateless replicas re-assigned when their clock runs out.
- C2 Reattachable failover through the dead-source move, with roll-forward after a leader change and a replica-scoped variant, after the single-writer stop window measured as elapsed time (decision 25, 3.10); single-writer self-stop without a volume. Depends on A5 and A6.
- C3 Node return: stale-copy removal, pinned restart, no flip-flop.
- C4
rebooting: the announced-shutdown hook,shc node maintenance, the STONITH-confirmed shortcut. - C5 Jobs off a lost node.
D. Rollout. Depends on B.
- D1 Readiness feeding endpoints; the liveness restart.
- D2 Rolling updates: the rollout row, incarnations for
start-first, rollback to the retained specification. Depends on A5. - D3 Drain on the engine: per replica, surge,
max_unavailable, global replicas last. - D4
shc rebalance --plan|--apply.
E. Swarm keys. Depends on B4 and B5.
- E1 The render translates or refuses every key in 3.23, with the Q1
warning naming the native form;
ports:read as a declaration and--portas the one publication (3.16; the SHC-1081 sweep tomode: hostis withdrawn, 2026-10-01); a catalog sweep test renders all 106 apps with no silent drop. - E2 Configs staged per node; secrets from the vault into RAM-only mounts.
- E3 The stopped-app backup path and its catalog test (3.25).
F. Surfaces.
- F1
shc fenceand the HUD fence views (SHC-1079): the CLI verbs of 5.1 exactly, their API routes, and the:fences,:rulesand:placementHUD types of 5.3. It builds on SHC-1050 2.1 and 2.2 (the fence tables, the reconciler, terraform) and takes over the CLI halves of those steps and of 2.3 (explain rendering), so the verbs land once.:placementrenders the explain JSON of B2 once B2 lands, and SHC-1050’s shorter explain before that. - F2
shc cluster statusengine summary.
G. Swarm out. SHC-1050 7.5 removes the Swarm runtime once section 7 holds; SHC-1061 moves bitspur.
9. Decisions
Section titled “9. Decisions”The seventeen questions this design left open were decided by agent on Clay’s standing delegation, 2026-09-30, reversible, the way SHC-1050 decisions 24 to 28 were. Each took the recommended option. The rejected options are kept so a reversal knows what it is choosing between. They are numbered S1 to S17 so they cannot be mistaken for SHC-1050’s decisions 1 to 28, which they build on and do not change.
-
S1. shcd restarts crashed containers; the runtime’s restart policy is off. The render stamps
restart: "no", removesdeploy.restart_policyfrom the runtime’s document, and the node agent applies the authored policy (3.9). Cost: while shcd is down a crash waits for it; systemd restarts shcd in seconds. Rejected: keepingunless-stoppedand rewriting container config beforedockerdstarts, which depends on Docker’s private on-disk format and has no Podman equivalent; keeping the runtime’s policy as is, which starts containers at boot before ownership is known and breaks decision 23. -
S2. Grace: 60 s for stateless work, 5 min for reattachable stateful work, from lease expiry, as scoped config a deployment may shorten, never below the single-writer stop window (3.8). Rejected: one grace for everything, which either strands stateless apps for minutes or moves stateful data on every reboot; Swarm’s immediate rescheduling, which moves a database for a two-minute reboot; Kubernetes’ flat 300 s.
-
S3. A single writer is a service with an RWO volume or
x-shc-scale.max: 1, the ceiling 97 catalog services already declare as a data-safety directive (3.10). Rejected: RWO volumes only, which misses single writers on host binds; a new explicit key, which is a catalog sweep for a fact the catalog already states. -
S4. A far node that reboots during a WAN cut waits for the heal. Its far-location volumes may keep running through a cut (decision 24) but are not started after a reboot without a confirmed ownership. An operator break-glass verb (
shc node start-local) comes only if a real outage asks for it. Rejected: starting far-location apps from the last clusterfs at boot, which breaks decision 23; a lease authority per site, which decision 24 already rejected. -
S5. No preemption. Priority orders rescheduling, drain and the pending queue (3.7); its one other effect is that
systempriority is admitted to every reserved fence (3.5), which is how the per-node platform set reaches every node. Rejected: Kubernetes-style priority preemption, which would evict a stateful app with a full cutover to fit another. -
S6. No node-pressure eviction. A pressured node takes no new work and keeps what it runs (3.20). Rejected: kubelet-style eviction by priority, which is preemption by another name.
-
S7.
deploy.endpoint_mode: vipis refused at render until the per-node L4 VIP exists;dnsrris honoured (3.15, 3.23). No catalog app sets either. Rejected: readingvipasdnsrr, which changes its meaning silently against decision 21; building the L4 VIP now, which nothing in the catalog needs. -
S8. Short-form and
mode: ingresspublished ports are refused at render, and the five catalog apps that publish ports are swept to long-formmode: host(3.16; the sweep is SHC-1081). Under Swarm those ports ride the routing mesh; under compose today they already publish on one node, so the sweep changes nothing that runs. Rejected: reading short form as host publication, which drops mesh reachability without saying so and breaks decision 21; rebuilding the routing mesh, a second forwarding plane beside Traefik. Superseded 2026-10-01:ports:in compose only declares, an app never self-publishes, and--porton the deployment is the one publication (3.16); the sweep shape is withdrawn. -
S9. The fence store, its reconciler, the
shc fenceCLI and explain live in ce; the multi-node engine loop, rescheduling and failover live in shc, as drain and volume failover do today (5.1). Rejected: all in shc, which leaves single-node CE estates with no built-in fences and no explain; all in ce, which moves the multi-node HA engine across the edition split. -
S10. Rebalancing is explicit only in v1 (
shc rebalance --plan|--apply, 3.19). Rejected: a rebalance rule attached to a scope now, which no estate has asked for; automatic rebalancing, which the rebalancer v1 ruling excludes. Amended for group members (Clay, 2026-10-01, mem shc-scale-and-cli-scope-model): members of a group that share a node move apart on their own when a node frees or joins (3.19, the separation pass). Only that separation runs on its own; load moves stay an operator’s request. -
S11. Cross-node names resolve through
dns_searchto the deployment’s zone, and a service split across nodes carries a per-replica name inside each projection (3.15). Rejected: baking remote names as aliases orextra_hosts, the render-time binding decision 22 removed; a per-node L4 VIP now (S7). Widened 2026-10-01: every entry is<service>-r<ordinal>, always, not only when a service spans nodes (3.6, 3.15). -
S12. The LeaderHub polls the leader every 1 s once the engine rides it (3.12). Rejected: keeping 5 s, which adds up to 5 s to every failover decision after an election.
-
S13. Default spread is soft
spread=nodefor stateless replicas andspread=node,max=1forstateful: true(3.6; the key wasidentity: stableuntil 2026-10-01). Rejected: one per node for every replicated service, which refuses--scale web=3on a two-node estate; a soft spread for members, which lets two ordinals of a clustered database share a node. -
S14. Liveness restarts a container unhealthy for 5 min (
x-shc-health.restart_after,0disables; 3.13). Rejected: Swarm’s immediate replacement, which crash-loops a database on a slow recovery; never restarting, which is how a container stayed unhealthy for 13.8 hours on corp. -
S15.
:rulesis its own HUD type (5.3). Rejected: folding rules into:fences: rules attach to scopes and deployments, not fences, and a tenant admin needs them tenant-scoped while fences are platform-wide. -
S16. A single writer is taken over on a STONITH confirm wherever the provider has a STONITH agent (Hetzner today). Elsewhere a lease-expiry takeover is allowed only on a node running the watchdog, after the elapsed stop window, and explain states the residual risk of a frozen VM resuming (3.10). Rejected: no automatic single-writer takeover without STONITH, which leaves Proxmox and bare-metal nodes with manual failover only; lease-expiry takeover without the watchdog, where a hung shcd leaves its writers running.
-
S17. Decision 23 stays strict. A rebooted node starts nothing, the per-node platform set included, until the control plane confirms what it owns and its leases are re-acquired (3.9). A node that reboots inside a partition therefore serves nothing until the heal. Rejected: starting the platform set from the last clusterfs at boot, which serves routes for a node that may no longer be a member or may have lost the platform assignment.
10. Stale and superseded
Section titled “10. Stale and superseded”Stale today:
- ce
modules/app/resource_reservations.go:13-19says no catalogdeploy.resourcesblock writes a reservation; four apps do now (onlyoffice, openarchiver, pulp, stirling-pdf). modules/quota/enforcement.go:37saysreservation_ratiodefaults to 0.25; cemodules/app/resource_reservations.go:125defaults it to 1.0.modules/quota/enforcement.gopresents three gates as the admission refusal; none has a production caller.subsystems/volume_failover.go:80-89describes a 45 s dwell; the dwell counts three qualifying ticks (:465-500), so it can authorize about 30 s after the first, and its compile-time assertion (:108) omits the stop time.
Superseded by this design when it lands:
modules/placement/placement.go:160“SHC has no HA”, and the scorer’s “SWARM DOES NOT GET A SCORE” (modules/placement/advisor.go:84, cemodules/stack/placement_seam.go).- ce
modules/app/restart_default.go(unless-stoppedfor every service that declares norestart:). - ce
modules/clone/move_presumed_dead.go:43-51, the manualdocker compose downon a returned node. - ce
modules/health/reconcile/errors.go:105-128, X200924RSTRMT’s “nothing will repair it”: the owning node’s agent does. docs/decisions/2026-08-11-rebalancer-v1-foundations.md§0 “no failover”: C1 and C2 add it, still with no autonomous rebalance.
11. Rulings of 2026-09-30 and 2026-10-01
Section titled “11. Rulings of 2026-09-30 and 2026-10-01”Clay’s rulings, folded in by SHC-1198. Each is a rule; the section that carries it is in brackets, and the full text of each lives in the memory files named on SHC-1198 and on the one-page summary.
- Deployment, not stack; app, not package (2026-09-30, 2026-10-01).
The catalog thing is an app. One app for one tenant in one environment is
a deployment, the running unit and its own network. “Stack” names only
the axis. The ladder is app, deployment, service, replica, compose entry
<service>-r<ordinal>, container. Existing code is not mass-renamed [vocabulary]. - Runtimes are
docker,podman,incusandapple, all four the architecture now; a runtime is the engine that takes a compose file; GPU, microVM and architecture are node capabilities; no Docker API shim, no Proxmox executor, and Proxmox HA never manages an SHC guest; pve02 and pve03 become bare Incus hosts only after the cutover (2026-09-30, 2026-10-01) [2.2, 3.13, 7]. - Three doors into shcd (2026-09-30): the unix socket for the local CLI, the mesh listener for nodes, the public listener for the Mac CLI, Terraform and the Console. No container reaches the node plane or the internal API [new-line simplification §3.12].
- The policy unit is the deployment (2026-09-30): same tenant open to
services that declare ports, on those ports; cross-tenant blocked and
refused at declaration; the system tier is master’s platform
environment; connections are wiring, not permission. The address plan
(2026-10-01): fabric
10.128.0.0/10, a bridge per tenant per node, a /20 default grant,10.64.0.0/10node-local,10.192.0.0/10control plane [new-line simplification §3.13]. ports:andexpose:only declare; an app never self-publishes;--hostand--porton the deployment are the only publication (2026-10-01) [3.16, 3.23, S8].- Identity is the pin; modules read files (2026-09-30): a platform container reads its own clusterfs section, with per-section revision directories (2026-10-01), and never dials the control plane; forward-auth’s gate call carries only the user’s JWT on the local public listener until it decides from files; no module credential [3.24, new-line simplification §3.12].
- No service mesh (2026-09-30, closed). No sidecars, no L7 mesh, no per-request policy; Traefik middlewares are the only retries and circuit breaking. Never re-propose.
- Two kernel WireGuard tunnels, never double wrapped (2026-09-30):
mesh0for the node plane,shc0for containers, coordinated by shcd from a peer table in rqlite; Nebula is today’s carrier and goes at stage 2, after the cutover (2026-10-01, SHC-1201) [rulings list, 7]. - The cutover lands bitspur on fresh VMs on both Gorilla boxes under Proxmox, docker runtime, app by app with both estates live, each app restored from a fresh restic backup, compose01 untouched as the rollback for two weeks and then retired; no replicated volumes in the first cut (2026-10-01) [3.6, 7].
- One
--scaleflag, three shapes, noshc scaleverb, nohome, nofailover(2026-10-01):web=3,pg=eu:2,us:1,proxy=per:locationon install and update; deployments addressed by flags, never a path; spread, regions and apart are engine-internal [1, 3.4, 3.6, 5.2]. - One compose entry per replica, always, named
<service>-r<ordinal>(2026-10-01); compose never scales; the bare name is a local alias only when every replica is local;x-shc-scale.stateful: truemarks distinct members and replacesidentity: stablewith no alias [3.6, 3.15, 6, S11, S13]. - Render once on the leader; nodes never render (2026-10-01): Jinja
with a
replicaobject at install or update, stored per replica with a digest; the materialiser copies; a daemon upgrade changes nothing until an explicit update [2.2, 3.24, 6]. - Swarm keys with a native form are translated with a warning that names it (2026-09-30, Q1) [3.23].
- Four pre-run checks on a node (2026-10-01): the revision is whole, the entry hash matches the stored digest, the replica is still this node’s at the current generation and epoch with its lease, cross-node dependencies are ready [2.2, 3.9].
- Resolver (2026-09-30): a small Go binary on miekg/dns per node, reads
resolver/zones, never dials shcd, answers ready replicas nearest first with TTL 5 s; containers getdnsplusdns_search [<deployment zone>, <environment zone>]; per-runtime wiring belongs to the executor [3.15]. - Health (2026-10-01): defined once in compose, run by the runtime (the node agent on apple), read by the node agent, recorded per replica; no healthcheck means running is ready; dampen, coalesce to one revision a second, reload only on a changed section digest [3.13].
- Leases (2026-10-01, SHC-1182): the holder stops well before anyone may take over, monotonic clock, lifetime longer than an election, a new leader honours existing leases for one lifetime [3.10].
- The per-node sealed secret cache is for external providers only (2026-10-01, SHC-1159): the node agent caches only values resolved from external vault providers (aws, 1password, hashicorp and the rest) for the replicas this node currently runs, in a root-only per-node file and never in rqlite; each value is sealed with the vault’s existing envelope scheme (a fresh AES-256-GCM DEK wrapped by the in-memory Argon2id KEK from unlock), so a node that cannot unlock its vault cannot read its cache, and an entry drops when its replica leaves the node. Local-provider secrets are not cached: every node already holds them, encrypted, in its rqlite copy [3.24, 3.26].
- The six gotchas and their verdicts (2026-10-01) [3.26].
- Open questions live in one table with a recommended answer beside each, and nothing ruled is in it [new-line simplification §9].
References
Section titled “References”- Multi-site estates and fences (SHC-1050), the base.
- Placement scoring and rebalancer v1 foundations.
- Builtin vs deploy placement.
- HUD types.
- Network fabric.
- SHC-1050 (fences), SHC-1079 (
shc fenceand the HUD fence views), SHC-1055 and SHC-1056 (fabric slices), SHC-1061 (bitspur across), SHC-1040 (per-service mount digest), SHC-973 (tenant on daemon calls).