Skip to content
SHC Docs

Rotating nodes in and out

Hardware gets replaced. A cloud instance is resized by destroying it and creating a bigger one. A node dies overnight. This page is the operator runbook for all three: how a node joins, how one leaves, what decides whether it votes, and what happens to the apps running on it.

For first-time cluster construction see Multi-node cluster setup; for quorum-loss recovery see Disaster recovery. This page is the routine middle ground between them.

What you wantCommand
Add a nodeshc node token then shc node join <token>
Move stacks off a node, keep it in the clustershc node drain <node>
Retire a reachable nodeshc node leave (on it) or shc cluster rm <node>
Reap a node that is already goneshc node forget <node>
Reap every confirmed-dead nodeshc cluster repair

Voters, non-voters, and why an even node count is fine

Section titled “Voters, non-voters, and why an even node count is fine”

The control plane is raft over SQLite (rqlited). Raft cares about voters, not nodes. A node’s role is fixed at enrolment, and the rule is one line:

The voters are the largest odd number no greater than the control-plane fence’s members, capped at five. A node outside the fence is always a non-voter.

One member gives one voter, two give one voter and a non-voter, three or four give three, five or more give five. The enrolment handler (modules/cluster/enroll_handler.go) grants a joiner a vote only inside the fence and only while the count is under that target, and the suffrage pass (modules/cluster/suffrage.go) promotes and demotes as members come and go. A non-voter is a full replica: it holds all the data and serves reads, but it does not vote in elections and does not count toward quorum.

An even number of nodes is fine. This is the question the voter cap exists to answer. Four nodes is three voters plus one non-voter, an odd voter set under an even node count. You never hand-manage the split, and you never need to keep the node count odd:

NodesVotersNon-votersSurvives
110nothing
220nothing
3301 loss
4311 loss
5321 loss
9361 loss

The one shape to avoid is two nodes, and it is even-numbered by coincidence, not by rule: the second node joins while the cluster holds only one voter, so it joins as a voter, and a two-voter set needs both alive to commit anything. Two nodes is strictly worse than one. Go from one to three.

Note that the “survives” column stops improving after three voters. More nodes buys capacity, replicas, and read throughput. It does not buy control-plane fault tolerance beyond one voter loss. That is the deliberate trade: a fixed, odd, small voter set is easier to reason about than an elastic one.

shc node ls never shows suffrage in its table: plain renders name, state, and leader, and -v adds host, runtime, and cores. Two ways to see it:

Terminal window
shc cluster nodes -v # the only human table with an is_voter column; -v selects it
shc cluster status # raft_voters / expected_voters / voter_deficit

shc cluster status is the one to watch. quorum_healthy alone reads green even after the voter set has decayed, because the required-majority figure shrinks along with it. voter_deficit is the field that notices: it compares the live raft voter count against the policy target for the current healthy-node count, and under text output it prints a warning line.


The full procedure, prerequisites, and firewall rules are in Multi-node cluster setup. The short form:

Terminal window
# On any node already in the cluster
shc node token # single-use, default TTL 1h
# On the new host, with the daemon installed and running (do NOT run `shc init`)
# --nic is optional (auto-detected, private-preferred); --public-address is
# only needed when internet-facing DNS must resolve to this node.
shc node join <TOKEN> --name <node-name> --nic <iface> --public-address <ip-or-host>

Three things worth knowing before you run it:

  • Suffrage is decided here. If the cluster already has three voters, this node joins as a non-voter and stays one until shc cluster rm --rebalance promotes it. See replacing a voter below. This matters when you are replacing a voter rather than adding to the fleet.
  • The join can be refused by the licence. Enrolment meters node count and CPU cores against the active licence and refuses with X403031LICLIM, naming both the requested and the permitted figure.
  • The leader signs the joining node’s certificate, and the CA private key comes from the vault. A daemon that has restarted comes back with its vault locked, so the key load fails and the enrolment cannot be signed. If a join fails right after a leader reboot, run shc vault -p <password> on the leader and retry.

Verify from any node:

Terminal window
shc cluster nodes -v # the new node appears healthy, with its is_voter role
shc cluster status # leader named; no voter-deficit warning

Nothing moves by itself. SHC does not relocate deployments as a side effect of a node leaving, and it does not reschedule them when a node goes away. Relocation is an explicit verb with a wall-clock budget:

Terminal window
shc node drain <node>

Leave --timeout off. The relocations are serial and a measured one costs 40-50 seconds, so the daemon sizes the deadline from the number of deployments it plans to move (timeouts.drain, 300s, per relocation, capped at 60 minutes). A figure passed on --timeout is the budget for the whole run, so --timeout 300 is a five-minute cap that a multi-deployment drain may not finish inside.

Drain marks the node unschedulable, then for each deployment bound to it asks the placement engine for a target its --where and constraints allow. It stops the deployment on the source and brings it up on the target. A stopped deployment moves as it is, with its data, and stays stopped on the target; the line before the drain names it. A deployment with no eligible target fails the drain. --force stops those in place. A re-run against an already-drained node is a no-op success.

A node that has just joined reads ready a few seconds before its first clusterfs build lands. A drain typed in that window waits for that build, up to a minute, before it stops the node’s global replicas; a node that builds nothing in that minute fails the drain with X500382DRNCMP, naming the wait.

If some stacks moved and others did not, the node is left partially_drained (not draining) and the run reports X500406CLSDRP with both counts; a re-run resumes, skipping the cutovers that already landed. See the drain reference. Source: modules/cluster/drain.go.

Drain does not remove the node from the cluster. It is still a member, still a voter if it was one; it just holds no work. That is exactly what you want for a reboot or an in-place upgrade.

Retiring a node is a separate step, and every removal path checks occupancy first:

VerbOccupied nodeOverride
shc cluster leave (self)refuses, X409025LVDEPS--force (strands them)
shc cluster rm <node>refuses, X400017NODDEP--force (strands them)
shc node forget <node>refuses, X400017NODDEP--force (strands them)
shc cluster repair (sweep)skips that node, keeps sweeping, counts itnone on the sweep; reap it with shc node forget <node> --force --yes

A drained node is not occupied by the global replicas its drain stopped: they have nowhere to move, so a plain shc cluster leave typed on that node, shc cluster rm <node> and shc node forget <node> go through. Any other row still refuses, and the refusal names the deployments in the way.

The override strands. It is not an eviction and it is not a relocation: the deployment rows are left naming a node that is no longer in the cluster. Clear them on a surviving node with shc uninstall <stack>, or reinstall the stacks elsewhere.

--force overrides the occupancy refusal. It does not bypass the raft quorum floor.

A skipped row is counted. shc cluster repair prints repaired cluster (found N; fixed N; removed N; skipped N) as a WARNING when anything was skipped, so a sweep that reaped nothing does not read like a clean one. It still exits 0: a skip is a refusal to act, not a failure.

The removal gate checks deployment occupancy. Uninstall deployments left on a node before removing it.

Drain does not always discharge the gate. shc node drain moves running and stopped deployments alike, and a deployment whose install failed moves as it is and stays failed. Two things can still be on the node afterwards. A failed install the drain could not move stays, and the drain names it with X200047CLSDLF and the reason; the gate counts it, so repair it with shc update <stack> or remove it with shc uninstall <stack>, which is what the refusal’s remedy says. The global replicas the drain stopped stay too, and the gate does not count them.

terraform destroy does not refuse; it warns. The provider’s node Delete issues the plain forget first, and on the occupancy refusal emits a warning carrying the daemon’s own listing, then re-issues with the override. Refusing there would not save the deployments (the machine is being deleted in the same apply either way); it would only add a stale raft member on top of the orphaned rows, permanently inflating the surviving cluster’s quorum denominator. So the destroy proceeds and tells you which stacks it orphaned. Run shc node drain <node> before the destroy to avoid it entirely.

The ruling behind all of it: a removal must refuse and name what is in the way; it must never auto-relocate. Relocation stays shc node drain.

Removing a non-voter leaves a dangling raft member (v0.0.14)

Section titled “Removing a non-voter leaves a dangling raft member (v0.0.14)”

Every removal verb evicts the raft member by looking the target up in the live roster first, and at v0.0.14 that roster is voters only. The read asks rqlited for /nodes without the nonvoters parameter (core/rqlited/status.go), and rqlited answers with voters alone unless asked for the rest. So the lookup never matches a non-voter, the guarded eviction is skipped, and the removal falls through to the registry write (subsystems/rqlite_adapters.go):

  • shc cluster rm <non-voter> and shc node forget <non-voter> mark the row removed, report success, and leave the raft member configured.
  • Nothing reaps it afterwards. There is no membership reconciler, and the shc cluster repair sweep re-enters the same lookup, so the sweep cannot clean up after itself either.
  • No SHC command shows you the leftover. Every roster read in the daemon goes through that one voters-only call, so a dangling non-voter is invisible to shc cluster nodes, shc node ls, and shc cluster status alike, and shc cluster nodes lists the registry, where the row is now removed and gone from the table.

Read the scope carefully: suffrage caps at three, so every node past the third is a non-voter for its whole life. On a cluster of four or more this is the ordinary case, not a corner. Removing a voter is unaffected; that lookup matches and the eviction fires.

The same voters-only roster also decides the scale-to-one engine flip, which is why the last-but-one check below is not simply “two nodes down to one”.

Pending. The roster read is being corrected to ask for non-voters, which repairs the eviction, the flip trigger, and a third caller besides. Not in v0.0.14. Until you are on a release that carries it, prefer to retire non-voters by rebuilding the cluster’s membership rather than trusting a rm/forget to have evicted them, and treat a node that returns and is told it is “already a member” as this defect rather than as a new fault.


Terminal window
shc node drain <node> # from anywhere; move the work off first
shc node leave # on the node being retired

shc node leave checks deployment occupancy and raft quorum before leaving. It writes a mesh0 tombstone while it can still commit to raft. Peers retain the tunnel until raft confirms removal. The departing node then tears down mesh0 and its cache, retaining its WireGuard key for a rejoin. See cluster join and leave.

Terminal window
shc node leave <node> # ask the named node to leave (it runs the sequence above)
shc cluster rm <node> # remove another node; refuses if occupied without --force

shc cluster leave is self-only by design. shc cluster rm is the remote-removal verb.

“Who ran init” is history the cluster does not depend on. There is no founder, no seed node, no primary. Remove it exactly like any other node. Specifically:

  • Leadership is raft’s. It is re-elected continuously and has nothing to do with genesis. shc cluster status names whoever holds it now.
  • The cluster CA is replicated state, not a file on the genesis host. It is minted once into the name="cluster" certificates row with the private key escrowed in the vault under a ref, so whichever node holds leadership can sign a joining node’s certificate.
  • Join tokens are minted by whichever daemon you ask. shc node token has no “original leader” requirement.

Check the remaining roster before the genesis node goes:

  • Would the removal leave a one-member roster? When it does, the storage engine flips back to plain SQLite and schedules a daemon restart on the survivor. That is by design (SHC is single-node SQLite that becomes a cluster as you add nodes), and it is not instantaneous, so do not queue work against the survivor across that window.

    The trap is which roster decides. At v0.0.14 the check tests the voters-only roster described above, not the node count. On a four-node cluster of three voters plus a non-voter, losing the voters one at a time and reaping each (which the quorum guard permits, because each removal on its own leaves a reachable majority) ends with a roster of just the survivor. The flip fires there, demoting the engine to plain SQLite while a live non-voter is still replicating from it. Until the roster fix ships, do not reap the second-to-last voter on a cluster that still carries live non-voters; retire the non-voters first, or rebuild.


Raft commits a write when a majority of voters agrees. Majority is floor(V/2) + 1 over the voter count, not the node count.

VotersMajority neededFailures tolerated
110
220
321
431
532

Four voters tolerate exactly what three do, which is the whole reason odd voter sets are preferred and the whole reason the target is three. In practice you will not reach four through joins: the cap prevents it.

Losing a majority is a trap, not just an outage

Section titled “Losing a majority is a trap, not just an outage”

Take a three-voter cluster and lose two voters. The survivor cannot commit writes, which is expected. The part that surprises people: it also cannot remove the dead voters, because a membership change is itself a raft write and needs the same majority. The cluster cannot vote itself out of the hole it is in.

So the guarded removal path refuses rather than pretending. Every verb that evicts a raft member (shc cluster rm, shc node forget, the shc cluster repair sweep, and shc cluster leave) passes through one non-bypassable quorum-floor guard first, which refuses with:

  • X409201CLSRMQ: too few voters would remain. Either this is the last voter, which no signal can ever make removable, or no liveness signal was available to prove the survivors are up and the conservative arithmetic applies.
  • X409202CLSRMR: too few reachable voters would remain. This is the real wedge: removing a voter whose only surviving peer is dead.

--force overrides the occupancy and unhealthy-node gates. It has never overridden this one and must not start.

repair --force is the escape, and it can lose data

Section titled “repair --force is the escape, and it can lose data”

When the missing voters are genuinely, permanently gone:

Terminal window
shc cluster repair --dry-run # what the guarded sweep would reap
shc cluster repair # reap confirmed-dead members, guard intact
shc cluster repair --force # destructive: rewrite membership, sole leader

--force rewrites raft membership so the local node is the sole member and forces an election. Use it only when quorum is lost and the other voters are truly dead, because it is unsafe in one specific way:

If the “dead” voters are actually alive on the other side of a network partition, they have their own raft log, and this rewrite discards it. Worse, if an operator runs repair --force on both sides of a partition you get two clusters that each believe they are authoritative and that will never reconcile. Confirm the other voters are destroyed, not merely unreachable, before running it.

After it succeeds, treat the result as a new single-node cluster and rejoin the other nodes fresh. Do not try to un-evict a previously removed node; its raft state will conflict with the new leader’s log.


rqlite fixes voter/non-voter at join time and exposes no way to change it while the process runs. Suffrage is a field of the join request, read once at start from -raft-non-voter. The engine underneath (hashicorp/raft) does support in-place promotion; rqlite does not surface it. There is no promote endpoint, no demote endpoint, and re-issuing a join is silently ignored because the leader short-circuits on id and address alone. SHC’s role-assignment seam therefore refuses with a typed error rather than pretending (subsystems/rqlite_adapters.go).

So SHC changes suffrage the only way rqlite allows: it evicts the member from raft, restarts its rqlited with the other suffrage, and lets it rejoin with its log intact. A daemon restart on its own promotes nothing.

The suffrage pass does this by itself. Each minute the leader counts the control-plane fence’s members and takes the largest odd number no greater than that, capped at five: one member gives one voter, two give one voter and a non-voter, three or four give three, five or more give five. It then makes at most one change toward that target. It promotes a reachable non-voter inside the fence, or demotes a voter outside the fence or past the target. It demotes nothing while a voter is unreachable or when the voters left would lose their quorum, and a voter that is down keeps its vote until it’s removed.

shc cluster rm of a voter is a plain removal. Its output and shc cluster status say when it leaves the voters under target, and the suffrage pass then promotes a non-voter inside the fence. Until it does, the cluster runs short, so promote first when you replace a voter:

Terminal window
shc node join <TOKEN> # a 4th member: 4 still give 3 voters, so it joins as a non-voter
shc node drain old-voter
shc cluster rm --rebalance old-voter # promotes the new node first: 3 -> 4 -> 3 voters
shc cluster status # confirm raft_voters == expected_voters

--rebalance promotes a non-voter inside the control-plane fence, waits for it to rejoin as a voter, and only then removes the named voter, so the set never dips below target. It refuses when no non-voter inside the fence can be promoted. If the promotion fails, the named voter is not removed; re-run the command, or let the suffrage pass finish the promotion it finds in flight.

Replacing a non-voter has no such constraint. Order does not matter, because the replacement was going to be a non-voter either way. At v0.0.14 it has a different problem instead: removing the old non-voter reports success without evicting its raft member, so the replacement is an addition to the membership rather than a swap within it. See the dangling-member defect.

The one-way door was measured against a live rqlited, not assumed, and it is not something a future SHC release can simply switch on. It needs a change in what rqlite exposes. voter_deficit in shc cluster status tells you when a removal has left the set under target.


For a reinstalled machine managed by a root using terraform/modules/core, record its new SSH host key in the inventory, then run tofu plan and tofu apply. The Rejoining a rebuilt machine warning names the node and the evidence: the cluster lists it as unhealthy or absent, and the machine answers SSH with no SHC node identity. Apply rejoins it under the same name and keeps the deployment rows that place work on it. This works for any node, including the first, module.core.shc_node.bootstrap[0], whose new host key is what starts the check. No manual shc node forget, -replace or state surgery is needed. A machine that does not answer SSH cannot supply this evidence; the manual recovery below covers machines that are still gone. Change vault_password in a separate apply, not in the one that rejoins the first node.

A VM destroyed out of band, a disk that will not boot, a host that fell off the network for good. You cannot drain it and you cannot run leave on it.

1. Confirm it is gone, not merely unreachable. This is the whole decision. A node that comes back after being forgotten has to rejoin as a fresh member, and if you guessed wrong about a partition you may be about to make a recoverable outage permanent.

Terminal window
shc cluster status # is quorum still healthy?
shc node ls -v # host, runtime, and cores for the missing node

2. Reap it from a healthy node.

Terminal window
shc node forget <node>

This marks the registry row removed, and, if the gone node was a voter, evicts its raft member. If it was a non-voter the eviction is skipped and the member is left dangling; see the defect above, because the command reports success either way. If the gone node was a voter and removing it would drop the reachable voter set below its post-removal majority, the quorum-floor guard refuses with X409202CLSRMR. That refusal is correct and it is telling you something real: you are one voter away from an unrecoverable cluster. Bring a voter back if you possibly can; otherwise escalate to shc cluster repair --force and accept its terms.

3. Sweep, if several went at once.

Terminal window
shc cluster repair --dry-run
shc cluster repair

The sweep reaps members that are unreachable or unseen for more than 24 hours, each through the same guard. Where the guard refuses, the action is reported as quorum lost: run shc cluster repair --force to force-recover rather than a bare error.

4. Deal with the stranded stacks. At v0.0.14 the removal does not touch deployment rows, so anything that was placed on the gone node still points at it. Those stacks are not running and will not be restarted anywhere. Reinstall them, or restore them from backup onto a surviving node. See the backup and restore flow.

5. Restore the voter set. If the node you lost was a voter, you are now below target. Join the replacement (with a voter gap open it enrols as a voter) and confirm:

Terminal window
shc node token
shc node join <TOKEN> --name <replacement>
shc cluster status # raft_voters back to expected_voters