Rotating nodes in and out
Hardware gets replaced. A cloud instance is resized by destroying it and creating a bigger one. A node dies overnight. This page is the operator runbook for all three: how a node joins, how one leaves, what decides whether it votes, and what happens to the apps running on it.
For first-time cluster construction see Multi-node cluster setup; for quorum-loss recovery see Disaster recovery. This page is the routine middle ground between them.
| What you want | Command |
|---|---|
| Add a node | shc node token then shc node join <token> |
| Move stacks off a node, keep it in the cluster | shc node drain <node> |
| Retire a reachable node | shc node leave (on it) or shc cluster rm <node> |
| Reap a node that is already gone | shc node forget <node> |
| Reap every confirmed-dead node | shc cluster repair |
Voters, non-voters, and why an even node count is fine
Section titled “Voters, non-voters, and why an even node count is fine”The control plane is raft over SQLite (rqlited). Raft cares about
voters, not nodes. A node’s role is fixed at enrolment, and the
rule is one line:
The voters are the largest odd number no greater than the control-plane fence’s members, capped at five. A node outside the fence is always a non-voter.
One member gives one voter, two give one voter and a non-voter, three or
four give three, five or more give five. The enrolment handler
(modules/cluster/enroll_handler.go)
grants a joiner a vote only inside the fence and only while the count is
under that target, and the suffrage pass
(modules/cluster/suffrage.go)
promotes and demotes as members come and go.
A non-voter is a full replica: it holds all the data and serves reads, but
it does not vote in elections and does not count toward quorum.
An even number of nodes is fine. This is the question the voter cap exists to answer. Four nodes is three voters plus one non-voter, an odd voter set under an even node count. You never hand-manage the split, and you never need to keep the node count odd:
| Nodes | Voters | Non-voters | Survives |
|---|---|---|---|
| 1 | 1 | 0 | nothing |
| 2 | 2 | 0 | nothing |
| 3 | 3 | 0 | 1 loss |
| 4 | 3 | 1 | 1 loss |
| 5 | 3 | 2 | 1 loss |
| 9 | 3 | 6 | 1 loss |
The one shape to avoid is two nodes, and it is even-numbered by coincidence, not by rule: the second node joins while the cluster holds only one voter, so it joins as a voter, and a two-voter set needs both alive to commit anything. Two nodes is strictly worse than one. Go from one to three.
Note that the “survives” column stops improving after three voters. More nodes buys capacity, replicas, and read throughput. It does not buy control-plane fault tolerance beyond one voter loss. That is the deliberate trade: a fixed, odd, small voter set is easier to reason about than an elastic one.
Reading the current split
Section titled “Reading the current split”shc node ls never shows suffrage in its table: plain renders name, state,
and leader, and -v adds host, runtime, and cores. Two ways to see it:
shc cluster nodes -v # the only human table with an is_voter column; -v selects itshc cluster status # raft_voters / expected_voters / voter_deficitshc cluster status is the one to watch. quorum_healthy alone reads
green even after the voter set has decayed, because the required-majority
figure shrinks along with it. voter_deficit is the field that notices:
it compares the live raft voter count against the policy target for the
current healthy-node count, and under text output it prints a warning line.
Adding a node
Section titled “Adding a node”The full procedure, prerequisites, and firewall rules are in Multi-node cluster setup. The short form:
# On any node already in the clustershc node token # single-use, default TTL 1h
# On the new host, with the daemon installed and running (do NOT run `shc init`)# --nic is optional (auto-detected, private-preferred); --public-address is# only needed when internet-facing DNS must resolve to this node.shc node join <TOKEN> --name <node-name> --nic <iface> --public-address <ip-or-host>Three things worth knowing before you run it:
- Suffrage is decided here. If the cluster already has three voters,
this node joins as a non-voter and stays one until
shc cluster rm --rebalancepromotes it. See replacing a voter below. This matters when you are replacing a voter rather than adding to the fleet. - The join can be refused by the licence. Enrolment meters node count
and CPU cores against the active licence and refuses with
X403031LICLIM, naming both the requested and the permitted figure. - The leader signs the joining node’s certificate, and the CA private key
comes from the vault. A daemon that has restarted comes back with its
vault locked, so the key load fails and the enrolment cannot be signed. If
a join fails right after a leader reboot, run
shc vault -p <password>on the leader and retry.
Verify from any node:
shc cluster nodes -v # the new node appears healthy, with its is_voter roleshc cluster status # leader named; no voter-deficit warningWhat happens to the apps on a node
Section titled “What happens to the apps on a node”Nothing moves by itself. SHC does not relocate deployments as a side effect of a node leaving, and it does not reschedule them when a node goes away. Relocation is an explicit verb with a wall-clock budget:
shc node drain <node>Leave --timeout off. The relocations are serial and a measured one
costs 40-50 seconds, so the daemon sizes the deadline from the number of
deployments it plans to move (timeouts.drain, 300s, per relocation, capped at
60 minutes). A figure passed on --timeout is the budget for the whole run,
so --timeout 300 is a five-minute cap that a multi-deployment drain may not
finish inside.
Drain marks the node unschedulable, then for each deployment bound to it asks
the placement engine for a target its --where and constraints allow. It stops
the deployment on the source and brings it up on the target. A stopped
deployment moves as it is, with its data, and stays stopped on the target; the
line before the drain names it. A deployment with no eligible target fails the
drain. --force stops those in place.
A re-run against an already-drained node is a no-op success.
A node that has just joined reads ready a few seconds before its first
clusterfs build lands. A drain typed in that window waits for that build,
up to a minute, before it stops the node’s global replicas; a node that
builds nothing in that minute fails the drain with X500382DRNCMP, naming
the wait.
If some stacks moved and others did not, the node is left
partially_drained (not draining) and the run reports X500406CLSDRP
with both counts; a re-run resumes, skipping the cutovers that already
landed. See the drain reference.
Source: modules/cluster/drain.go.
Drain does not remove the node from the cluster. It is still a member, still a voter if it was one; it just holds no work. That is exactly what you want for a reboot or an in-place upgrade.
The removal verbs refuse to strand work
Section titled “The removal verbs refuse to strand work”Retiring a node is a separate step, and every removal path checks occupancy first:
| Verb | Occupied node | Override |
|---|---|---|
shc cluster leave (self) | refuses, X409025LVDEPS | --force (strands them) |
shc cluster rm <node> | refuses, X400017NODDEP | --force (strands them) |
shc node forget <node> | refuses, X400017NODDEP | --force (strands them) |
shc cluster repair (sweep) | skips that node, keeps sweeping, counts it | none on the sweep; reap it with shc node forget <node> --force --yes |
A drained node is not occupied by the global replicas its drain stopped:
they have nowhere to move, so a plain shc cluster leave typed on that
node, shc cluster rm <node> and shc node forget <node> go through.
Any other row still refuses, and the refusal names the deployments in
the way.
The override strands. It is not an eviction and it is not a relocation:
the deployment rows are left naming a node that is no longer in the
cluster. Clear them on a surviving node with shc uninstall <stack>, or
reinstall the stacks elsewhere.
--force overrides the occupancy refusal. It does not bypass the raft
quorum floor.
A skipped row is counted. shc cluster repair prints
repaired cluster (found N; fixed N; removed N; skipped N) as a WARNING
when anything was skipped, so a sweep that reaped nothing does not read
like a clean one. It still exits 0: a skip is a refusal to act, not a
failure.
The removal gate checks deployment occupancy. Uninstall deployments left on a node before removing it.
Drain does not always discharge the gate. shc node drain moves
running and stopped deployments alike, and a deployment whose install
failed moves as it is and stays failed. Two things can still be on the
node afterwards. A failed install the drain could not move stays, and the
drain names it with X200047CLSDLF and the reason; the gate counts it, so
repair it with shc update <stack> or remove it with shc uninstall <stack>,
which is what the refusal’s remedy says. The global replicas the drain
stopped stay too, and the gate does not count them.
terraform destroy does not refuse; it warns. The provider’s node
Delete issues the plain forget first, and on the occupancy refusal emits a
warning carrying the daemon’s own listing, then re-issues with the
override. Refusing there would not save the deployments (the machine is
being deleted in the same apply either way); it would only add a stale raft
member on top of the orphaned rows, permanently inflating the surviving
cluster’s quorum denominator. So the destroy proceeds and tells you which
stacks it orphaned. Run shc node drain <node> before the destroy to avoid
it entirely.
The ruling behind all of it: a removal must refuse and name what is in
the way; it must never auto-relocate. Relocation stays shc node drain.
Removing a non-voter leaves a dangling raft member (v0.0.14)
Section titled “Removing a non-voter leaves a dangling raft member (v0.0.14)”Every removal verb evicts the raft member by looking the target up in the
live roster first, and at v0.0.14 that roster is voters only. The read
asks rqlited for /nodes without the nonvoters parameter
(core/rqlited/status.go), and rqlited
answers with voters alone unless asked for the rest. So the lookup never
matches a non-voter, the guarded eviction is skipped, and the removal falls
through to the registry write
(subsystems/rqlite_adapters.go):
shc cluster rm <non-voter>andshc node forget <non-voter>mark the rowremoved, report success, and leave the raft member configured.- Nothing reaps it afterwards. There is no membership reconciler, and the
shc cluster repairsweep re-enters the same lookup, so the sweep cannot clean up after itself either. - No SHC command shows you the leftover. Every roster read in the daemon
goes through that one voters-only call, so a dangling non-voter is
invisible to
shc cluster nodes,shc node ls, andshc cluster statusalike, andshc cluster nodeslists the registry, where the row is nowremovedand gone from the table.
Read the scope carefully: suffrage caps at three, so every node past the third is a non-voter for its whole life. On a cluster of four or more this is the ordinary case, not a corner. Removing a voter is unaffected; that lookup matches and the eviction fires.
The same voters-only roster also decides the scale-to-one engine flip, which is why the last-but-one check below is not simply “two nodes down to one”.
Pending. The roster read is being corrected to ask for non-voters, which repairs the eviction, the flip trigger, and a third caller besides. Not in v0.0.14. Until you are on a release that carries it, prefer to retire non-voters by rebuilding the cluster’s membership rather than trusting a
rm/forgetto have evicted them, and treat a node that returns and is told it is “already a member” as this defect rather than as a new fault.
Removing a node
Section titled “Removing a node”From the node itself (preferred)
Section titled “From the node itself (preferred)”shc node drain <node> # from anywhere; move the work off firstshc node leave # on the node being retiredshc node leave checks deployment occupancy and raft quorum before leaving.
It writes a mesh0 tombstone while it can still commit to raft. Peers retain
the tunnel until raft confirms removal. The departing node then tears down
mesh0 and its cache, retaining its WireGuard key for a rejoin. See
cluster join and leave.
From a surviving node
Section titled “From a surviving node”shc node leave <node> # ask the named node to leave (it runs the sequence above)shc cluster rm <node> # remove another node; refuses if occupied without --forceshc cluster leave is self-only by design. shc cluster rm is the
remote-removal verb.
Removing the node that ran shc init
Section titled “Removing the node that ran shc init”“Who ran init” is history the cluster does not depend on. There is no
founder, no seed node, no primary. Remove it exactly like any other node.
Specifically:
- Leadership is raft’s. It is re-elected continuously and has nothing
to do with genesis.
shc cluster statusnames whoever holds it now. - The cluster CA is replicated state, not a file on the genesis host.
It is minted once into the
name="cluster"certificates row with the private key escrowed in the vault under a ref, so whichever node holds leadership can sign a joining node’s certificate. - Join tokens are minted by whichever daemon you ask.
shc node tokenhas no “original leader” requirement.
Check the remaining roster before the genesis node goes:
-
Would the removal leave a one-member roster? When it does, the storage engine flips back to plain SQLite and schedules a daemon restart on the survivor. That is by design (SHC is single-node SQLite that becomes a cluster as you add nodes), and it is not instantaneous, so do not queue work against the survivor across that window.
The trap is which roster decides. At v0.0.14 the check tests the voters-only roster described above, not the node count. On a four-node cluster of three voters plus a non-voter, losing the voters one at a time and reaping each (which the quorum guard permits, because each removal on its own leaves a reachable majority) ends with a roster of just the survivor. The flip fires there, demoting the engine to plain SQLite while a live non-voter is still replicating from it. Until the roster fix ships, do not reap the second-to-last voter on a cluster that still carries live non-voters; retire the non-voters first, or rebuild.
Quorum
Section titled “Quorum”Raft commits a write when a majority of voters agrees. Majority is
floor(V/2) + 1 over the voter count, not the node count.
| Voters | Majority needed | Failures tolerated |
|---|---|---|
| 1 | 1 | 0 |
| 2 | 2 | 0 |
| 3 | 2 | 1 |
| 4 | 3 | 1 |
| 5 | 3 | 2 |
Four voters tolerate exactly what three do, which is the whole reason odd voter sets are preferred and the whole reason the target is three. In practice you will not reach four through joins: the cap prevents it.
Losing a majority is a trap, not just an outage
Section titled “Losing a majority is a trap, not just an outage”Take a three-voter cluster and lose two voters. The survivor cannot commit writes, which is expected. The part that surprises people: it also cannot remove the dead voters, because a membership change is itself a raft write and needs the same majority. The cluster cannot vote itself out of the hole it is in.
So the guarded removal path refuses rather than pretending. Every verb that
evicts a raft member (shc cluster rm, shc node forget, the
shc cluster repair sweep, and shc cluster leave) passes through one
non-bypassable quorum-floor guard first, which refuses with:
X409201CLSRMQ: too few voters would remain. Either this is the last voter, which no signal can ever make removable, or no liveness signal was available to prove the survivors are up and the conservative arithmetic applies.X409202CLSRMR: too few reachable voters would remain. This is the real wedge: removing a voter whose only surviving peer is dead.
--force overrides the occupancy and unhealthy-node gates. It has never
overridden this one and must not start.
repair --force is the escape, and it can lose data
Section titled “repair --force is the escape, and it can lose data”When the missing voters are genuinely, permanently gone:
shc cluster repair --dry-run # what the guarded sweep would reapshc cluster repair # reap confirmed-dead members, guard intactshc cluster repair --force # destructive: rewrite membership, sole leader--force rewrites raft membership so the local node is the sole member and
forces an election. Use it only when quorum is lost and the other voters
are truly dead, because it is unsafe in one specific way:
If the “dead” voters are actually alive on the other side of a network
partition, they have their own raft log, and this rewrite discards it.
Worse, if an operator runs repair --force on both sides of a partition
you get two clusters that each believe they are authoritative and that will
never reconcile. Confirm the other voters are destroyed, not merely
unreachable, before running it.
After it succeeds, treat the result as a new single-node cluster and rejoin the other nodes fresh. Do not try to un-evict a previously removed node; its raft state will conflict with the new leader’s log.
Replacing a voter
Section titled “Replacing a voter”rqlite fixes voter/non-voter at join time and exposes no way to change
it while the process runs. Suffrage is a field of the join request, read
once at start from -raft-non-voter. The engine underneath
(hashicorp/raft) does support in-place promotion; rqlite does not surface
it. There is no promote endpoint, no demote endpoint, and re-issuing a
join is silently ignored because the leader short-circuits on id and
address alone. SHC’s role-assignment seam therefore refuses with a typed
error rather than pretending
(subsystems/rqlite_adapters.go).
So SHC changes suffrage the only way rqlite allows: it evicts the member
from raft, restarts its rqlited with the other suffrage, and lets it
rejoin with its log intact. A daemon restart on its own promotes nothing.
The suffrage pass does this by itself. Each minute the leader counts the control-plane fence’s members and takes the largest odd number no greater than that, capped at five: one member gives one voter, two give one voter and a non-voter, three or four give three, five or more give five. It then makes at most one change toward that target. It promotes a reachable non-voter inside the fence, or demotes a voter outside the fence or past the target. It demotes nothing while a voter is unreachable or when the voters left would lose their quorum, and a voter that is down keeps its vote until it’s removed.
shc cluster rm of a voter is a plain removal. Its output and
shc cluster status say when it leaves the voters under target, and the
suffrage pass then promotes a non-voter inside the fence. Until it does,
the cluster runs short, so promote first when you replace a voter:
shc node join <TOKEN> # a 4th member: 4 still give 3 voters, so it joins as a non-votershc node drain old-votershc cluster rm --rebalance old-voter # promotes the new node first: 3 -> 4 -> 3 votersshc cluster status # confirm raft_voters == expected_voters--rebalance promotes a non-voter inside the control-plane fence, waits
for it to rejoin as a voter, and only then removes the named voter, so the
set never dips below target. It refuses when no non-voter inside the fence
can be promoted. If the promotion fails, the named voter is not removed;
re-run the command, or let the suffrage pass finish the promotion it finds
in flight.
Replacing a non-voter has no such constraint. Order does not matter, because the replacement was going to be a non-voter either way. At v0.0.14 it has a different problem instead: removing the old non-voter reports success without evicting its raft member, so the replacement is an addition to the membership rather than a swap within it. See the dangling-member defect.
The one-way door was measured against a live
rqlited, not assumed, and it is not something a future SHC release can simply switch on. It needs a change in what rqlite exposes.voter_deficitinshc cluster statustells you when a removal has left the set under target.
When the machine is already gone
Section titled “When the machine is already gone”For a reinstalled machine managed by a root using terraform/modules/core,
record its new SSH host key in the inventory, then run tofu plan and
tofu apply. The Rejoining a rebuilt machine warning names the node and
the evidence: the cluster lists it as unhealthy or absent, and the machine
answers SSH with no SHC node identity. Apply rejoins it under the same name
and keeps the deployment rows that place work on it. This works for any
node, including the first, module.core.shc_node.bootstrap[0], whose new
host key is what starts the check. No manual shc node forget, -replace or
state surgery is needed. A machine that does not answer SSH cannot supply
this evidence; the manual recovery below covers machines that are still gone.
Change vault_password in a separate apply, not in the one that rejoins the
first node.
A VM destroyed out of band, a disk that will not boot, a host that fell off
the network for good. You cannot drain it and you cannot run leave on it.
1. Confirm it is gone, not merely unreachable. This is the whole decision. A node that comes back after being forgotten has to rejoin as a fresh member, and if you guessed wrong about a partition you may be about to make a recoverable outage permanent.
shc cluster status # is quorum still healthy?shc node ls -v # host, runtime, and cores for the missing node2. Reap it from a healthy node.
shc node forget <node>This marks the registry row removed, and, if the gone node was a
voter, evicts its raft member. If it was a non-voter the eviction is
skipped and the member is left dangling; see
the defect above,
because the command reports success either way. If the gone node was a
voter and removing it would drop the reachable voter set below its
post-removal majority, the quorum-floor guard refuses with
X409202CLSRMR. That refusal is correct and it is telling you something
real: you are one voter away from an unrecoverable cluster. Bring a voter
back if you possibly can; otherwise escalate to shc cluster repair --force
and accept its terms.
3. Sweep, if several went at once.
shc cluster repair --dry-runshc cluster repairThe sweep reaps members that are unreachable or unseen for more than 24
hours, each through the same guard. Where the guard refuses, the action is
reported as quorum lost: run shc cluster repair --force to force-recover
rather than a bare error.
4. Deal with the stranded stacks. At v0.0.14 the removal does not touch deployment rows, so anything that was placed on the gone node still points at it. Those stacks are not running and will not be restarted anywhere. Reinstall them, or restore them from backup onto a surviving node. See the backup and restore flow.
5. Restore the voter set. If the node you lost was a voter, you are now below target. Join the replacement (with a voter gap open it enrols as a voter) and confirm:
shc node tokenshc node join <TOKEN> --name <replacement>shc cluster status # raft_voters back to expected_votersSee also
Section titled “See also”- Multi-node cluster setup: prerequisites, ports, first join
- Reaching estate nodes over SSH: where the cluster key lives, and why a node replacement invalidates the copy in your checkout
- Disaster recovery: quorum loss, forced recovery, restore from backup
- Upgrading SHC: rolling upgrades, which use drain but not removal
- Flow: cluster join: the protocol-level trace
- Architecture: cluster module: membership internals
- CLI reference:
nodeandcluster