Disaster recovery
This guide covers recovery from failure modes that normal node
operations (shc node drain, rolling upgrades) don’t handle:
specifically raft quorum loss, dead leader with no quorum
possible, and node evacuation when a host is permanently gone.
SHC’s clustered control plane is rqlite (Raft over SQLite): an external
rqlitedprocess each daemon supervises. Raft stays available as long as a majority of voting members can reach each other. Lose the majority and the cluster becomes read-only until a human intervenes. This document describes those interventions.
For normal multi-node operations see Multi-node cluster setup; for daemon upgrades see Upgrading SHC; for losing the whole site that holds the voters see Losing a site.
Before you start
Section titled “Before you start”-
Take a fresh backup of raft state on the surviving node(s) before attempting any recovery step. On each node:
Terminal window cp -a /var/lib/shc/rqlite /var/lib/shc/rqlite.before-recovery-$(date +%s)Recovery operations can be destructive. A cold copy of the rqlite data directory is the best escape hatch.
-
Identify the current state.
shc node lsandshc cluster statusshow who’s still alive:Terminal window shc node lsshc cluster statusThe output flags which nodes are voters and who (if anyone) is currently leader.
-
Identify which node should become leader. Prefer the node that was most recently a voting leader with the most up-to-date data. If you’ve been tracking raft log indexes in monitoring, pick the one with the highest index.
Scenario 1: Leader crashed, quorum still possible
Section titled “Scenario 1: Leader crashed, quorum still possible”Symptoms. shc node ls shows the leader as unhealthy but a
majority of voters are still healthy.
What happens. Raft automatically elects a new leader from the
remaining voters. You’ll see leadership move in shc node ls
within ~5–10 seconds (default Raft election timeout).
Your action. Usually none. Verify that another node took over. Once the crashed leader comes back, it rejoins as a follower and catches up from the log.
If the crashed node can’t be recovered, proceed to Scenario 3 below (evicting a dead node).
Scenario 2: Quorum lost, more than half the voters are down
Section titled “Scenario 2: Quorum lost, more than half the voters are down”Symptoms.
shc cluster statusanswers, with[X503021CLSQRM]naming the unreachable voters; the status verb is quorum-exempt- Writes fail:
[X500901OPFAIL]…503 Service Unavailable, message: leader not found - Reads fail too.
shc node lsreturns[X500899NODREPO]with the same 503. This is not a read-only degradation: the control plane is fully wedged - Logs mention
is_quorum_lostorX200918RCVOPW - Containers already running keep running. Both quorum losses are control-plane only; the data plane is untouched
Why. Raft requires a majority of the original voter set to commit writes. A 3-voter cluster can tolerate 1 failure; a 5-voter cluster can tolerate 2. Lose more than that, and Raft intentionally refuses writes rather than split-brain.
Option A: Wait for voters to come back
Section titled “Option A: Wait for voters to come back”If the missing voters just crashed and the hosts are recoverable (e.g. a power outage), bring them back. Raft picks up exactly where it left off. This is always the preferred option.
Option B: Force recovery on a surviving node
Section titled “Option B: Force recovery on a surviving node”Use when the missing voters are permanently gone (hardware failure, host destroyed) and you need the cluster to accept writes again.
★
shc cluster repair --forcenow works, and it is the preferred form. It performs exactly the rqlite procedure documented below (writing the one-memberpeers.jsonand restarting), using the node id and raft address the running engine already holds, so there is nothing to transcribe.Prefer it over doing this by hand, because it refuses in cases you cannot easily check yourself. It reads the raft roster and refuses (
X409403CLSPRT) unless every other configured voter can be shown to be dead: not merely unreachable, and not merely unmeasured (that is the partition case below, the one that forks the database), and refuses (X409407CLSRVU) rather than proceeding when it cannot evaluate that question at all. Neither refusal is bypassable by--force:--forceis what selects this verb in the first place.The manual procedure is kept below because it is what the verb does, because it is the path when the daemon itself will not start, and because an operator in an outage should be able to see the mechanism rather than trust a verb. It has no gate: you are the gate. See Consensus and recovery §5.2.
shc cluster repair --force --dry-run # asks every question the recovery asks, changes nothingshc cluster repair --force --yes★ This is potentially data-losing, and it can manufacture a split brain. Uncommitted writes on the missing voters are lost. Worse: if the other voters are alive but partitioned rather than dead, they still hold the OLD raft configuration in which they are a majority, so they will elect their own leader and keep committing while this node commits independently. Unreachable is not a synonym for dead. Confirm the other voters are genuinely gone before you run this.
rqlite: re-form a single-voter quorum
Section titled “rqlite: re-form a single-voter quorum”Proven on a 3-node lab against rqlite v10.2.7 with one surviving node.
Read the surviving node’s own -node-id and -raft-addr off the running
process first:
ps -o args= -C rqlited | tr ' ' '\n' | grep -A1 -E '^-(node-id|raft-addr)$'Then, on that node:
systemctl stop shc
printf '%s' '[{"id":"<node-id>","address":"<raft-addr>","non_voter":false}]' \ > /var/lib/shc/rqlite/raft/peers.json
systemctl start shcOn restart rqlite consumes the file (it is renamed to peers.info), forms a
single-voter configuration and elects itself. Verify:
shc cluster status # QUORUM HEALTHY true, VOTERS 1, VOTER DEFICIT trueshc node ls # answers again; dead nodes show unreachableWhat this preserves: the entire replicated database: node registry, deployments, config, vault, licence. Observed on the lab: a config row written before the outage read back correctly, all deployment rows survived, and every running container was untouched throughout.
What it costs: uncommitted writes on the dead voters, and voter membership: the survivors are demoted to a 1-voter set.
Afterwards: restore voter redundancy
Section titled “Afterwards: restore voter redundancy”A single voter has zero fault tolerance. Re-enrol eligible control-plane
nodes and check shc cluster status. SHC reconciles voter slots up to the
voter target: the largest odd count of eligible control-plane members,
capped at five. Confirm dead nodes are fenced before recovering quorum.
Option C: Restore from a backup
Section titled “Option C: Restore from a backup”If no surviving voter has current data (e.g. all voters’ data directories are corrupt), restore the raft state from the most recent backup:
- Stop SHC on all nodes.
- On the node you’ll make the new single-node cluster:
Terminal window systemctl stop shcrm -rf /var/lib/shc/rqlite# Restore your snapshot of /var/lib/shc/rqlite heresystemctl start shc - Verify
/statusreports healthy. - Rejoin other nodes per cluster-setup.
Be aware: backups of /var/lib/shc/rqlite taken while SHC was
running may be internally inconsistent. Prefer scheduled cold
backups (SHC stopped briefly) for true DR reference snapshots.
Scenario 2b: A voter held off mesh0 after losing its mesh0 cache
Section titled “Scenario 2b: A voter held off mesh0 after losing its mesh0 cache”A node whose raft served on its mesh0 address keeps {state}/mesh/cache.json
to bring mesh0 up before raft starts. If that file is gone, the node does not
move raft onto its underlay address: shcd starts degraded with
X500575MSHCAC and rqlited stays down.
While the node is held, its daemon answers from its boot store, the
single-node database as it stood when the node went onto raft, and not from
the cluster’s store. A read such as shc node ls or shc status lists those
old rows and prints the warning X200288STRBOT on stderr. Anything that would
write, shc tenant add or shc install for example, is refused with
X503287STRHLD, which repeats why the node is held and the way back. The same
holds for any node that started degraded on its boot store: a joiner with
nothing to join (X501035RQDNOJ) and a node whose rqlited did not come up in
time (X501030RQDDGR). shc cluster repair, shc node join --force and
shc init --force still run there.
Restore the file if you have it. Otherwise run, on that node:
shc cluster repair --dry-run # what it would rebuild; changes nothingshc cluster repair # asks, then rebuilds and restarts shcBefore it rebuilds anything it asks every other node of its roster (the raft
configuration and the signed membership table) over the peer status port, TCP
4004 at each node’s underlay address, with its own node certificate. It
rebuilds only when every one of them answers as itself, none is the raft
leader, none is on a newer raft term and none holds a newer membership table.
Any other answer refuses with X500575MSHCAC naming the node and what it
said:
- a node that leads or is on a newer term means the cluster went on without
this one, for example after
shc cluster repair --forceon another node. Rejoin it withshc node join --forcewhere the cluster still runs; - a node that does not answer, or has no address this node knows, refuses too, because it cannot be counted as not leading. Bring it back, or open TCP 4004 between the nodes, and run the repair again.
The rebuilt cache holds this node and every peer at the address it answered on; shc restarts a second later and raft serves on mesh0 again under the same raft id.
Scenario 3: Permanently evict a dead node
Section titled “Scenario 3: Permanently evict a dead node”Symptoms. A node is gone for good. Its row still shows in
shc node ls as unhealthy and the cluster is otherwise healthy.
Action. On a healthy voter:
shc node forget <node-id>This removes the node from Raft membership and SHC’s node inventory. Do not use this on a node that’s merely temporarily offline; it will have to rejoin as a fresh member if it ever comes back.
Source:
modules/node/commands/forget.go
(shc node forget).
Scenario 4: Draining a node for maintenance
Section titled “Scenario 4: Draining a node for maintenance”Symptoms. You want to take a node down cleanly (for a hardware swap, OS upgrade, or permanent retirement) without disrupting running deployments.
Action. On any healthy node:
shc node drain <node-id>Leave --timeout off unless you mean to cap the run: the relocations are
serial, so the daemon multiplies the per-relocation budget
(timeouts.drain, 300s) by the number of stacks it plans to move, up to 60
minutes. A figure you pass is the budget for the whole run, so the
--timeout 300 this page used to show is now a five-minute cap that a
multi-stack drain cannot finish inside.
What drain does (source:
modules/cluster/drain.go):
- Marks the node as draining (no new stacks scheduled here).
- Asks the placement engine for a target within the deployment’s fences
and selectors. A deployment with no eligible target fails the drain;
--forcepermits stopping it in place. - Relocates each deployment (stop on the source, bring up on the
target), or stops it in place under
--force. - Waits for the moves to complete, up to
--timeoutseconds when you named one, else up to the auto-sized budget above. - Returns a per-deployment report (migrated / stopped / failed).
If some moved and some did not, the node is left partially_drained
rather than draining and the run reports X500406CLSDRP with both
counts. Those cutovers have committed (the stacks are live on a peer)
and a re-run resumes, skipping them. Only a run where nothing moved
reports the blanket X500400CLSDRO.
Drain does not remove the node from the cluster. After drain, the node is idle but still a voter. To fully retire it:
shc node leave # run ON the draining node: removes it cleanly# OR, from another node:shc node forget <node-id>shc node leave is the preferred path: it removes the node from
the cluster gracefully from its own side, letting Raft pick up the
change consensus-style.
Scenario 5: A node’s certs have expired or are corrupted
Section titled “Scenario 5: A node’s certs have expired or are corrupted”Symptoms. The node can no longer authenticate to cluster peers; logs mention TLS verification failures or expired certificates.
Action. SHC auto-rotates node certs well before expiry (see
subsystems/impl/internal_cert_rotation.go, ce module).
If rotation failed:
# From any healthy node (orchestrates across the cluster):shc cluster cert rotate --type mtlsIf certs are irrecoverable, evict and rejoin:
# From a healthy node:shc node forget <broken-node-id>
# On the previously-broken node, after any broken state is cleared:shc node token # mint a fresh join token on a leadershc node join <token> --force # token is POSITIONAL; --force wipes the # stale cluster identity this node still holdsThe token carries the leader’s address and the cluster CA, so there is no
separate --leader to pass. --force is what lets a node that was already
clustered rejoin: without it the join refuses with X400021ALRCLS because
node.id and certs/node.pem are still on disk from the broken run. The wipe
happens only once the cluster has ADMITTED the node, so a refused attempt
leaves the node exactly as it was.
Verifying recovery
Section titled “Verifying recovery”After any recovery operation, run through this checklist:
shc node ls # all expected nodes healthy, one leadershc cluster status # no quorum warningsshc ls --all-tenants # all expected stacks appearThen run a read + write probe to confirm the DB is accepting writes:
shc tenant ls # readshc tenant add recovery-probe # writeshc tenant rm recovery-probePrevention
Section titled “Prevention”-
The voter count follows the voter target. That target is the largest odd count of eligible control-plane members, capped at five (
VoterTargetin cemodules/cluster/voters.go). The suffrage reconciler promotes and demotes one node at a time, by removing the member from raft and rejoining it. A member that is down keeps its vote. Two eligible members steady-state at one voter, which has no spare vote. -
Keep clocks synchronized (chrony / systemd-timesyncd). Clock skew > ~60s between voters breaks token validation and confuses leader election.
-
Monitor
shc node ls/shc.status.get: you want to catch a degraded cluster before losing quorum, not after. -
Control-plane snapshots run nightly at 02:00 UTC. The leader-only
db_snapshot_cronwrites a timestamped, consistent dump of the control-plane database to{state}/snapshots/control-plane/, a raft-consistent rqlite backup in cluster mode, a sqliteVACUUM INTOsingle-node. Both are online (no daemon stop). Tune the cadence withcrons.db_snapshot_cronand how many to keep withmaintenance.control_plane_snapshot_retention(default 7). Each snapshot lands on the node that led when it was taken, and the cluster records the newest one (node and file) in its store:health.control-plane-snapshotgrades that record from any node and names where the file is. When the cluster has none on record, or its newest is more than 26 hours old (one missed nightly run), the leader records its own newer snapshot or writes one within 10 minutes (db_snapshot_first_cron), so a fresh cluster doesn’t wait for 02:00.shc db snapshot createtakes one on the leader on demand, andshc db snapshot lslists a node’s snapshots with their kind, time and size. An on-demand snapshot is kept to the same retention apart from the nightlies and is not recorded, since the record grades the nightly schedule. To restore one, stop SHC, swap the dumped.sqlitein for the control-plane DB, clear the raft engine marker +rqlite/dir (as a demote does), and restart. Usebackupfor stack data.★ Use a CALENDAR spec, never an interval, if you change the cadence. This cron was spelled
every 24 hoursuntil 2026-08-18, which anchors to the moment the cron table was built, and the leader cron table is rebuilt from scratch on every become-leader. Any roll, restart or leadership flap inside the window restarted the countdown, so on an estate that deploys daily it never fired: the newest snapshot was 10 days old on one production estate and 11 on the other, with the cron registered and healthy throughout.0 2 * * *is immune. Verify withshc cron ls:NEXT RUN ATshould be a wall-clock hour, not an offset from now. -
Test recovery in staging. Force recovery has sharp edges; the first time you run it shouldn’t be during an outage.
shc cluster repair --force --dry-runruns every check the recovery runs and changes nothing: it prints the node it would seat and the members it would drop, or the refusal the recovery would give, so the shape of it can be rehearsed cheaply. -
shc doctor runchecks the age of the control-plane snapshot on the leader. An absent or empty snapshot directory is an error.
Related
Section titled “Related”- Multi-node cluster setup: normal cluster operations
- Losing a location: every node of one location gone; restoring its deployments elsewhere from their off-site backups
- Upgrading SHC: planned upgrades (rolling, drain-based)
- Flow: cluster join: how joins work at the protocol level
- Architecture: cluster module: recovery internals
- Error reference:
X200918RCVOPW: recovery-required event - Consensus and recovery for raft quorum and recovery procedures
- Error reference: the three force-recovery gates:
X409400CLSREC(a force-leader recovery is already in progress),X409405CLSLDR(another node answers as leader, so this node refuses to rewrite membership; recovery proceeds only when no leader is reachable, or when this node is the leader) andX500201CLSREC(the recovery itself failed)