Skip to content
SHC Docs

Disaster recovery

This guide covers recovery from failure modes that normal node operations (shc node drain, rolling upgrades) don’t handle: specifically raft quorum loss, dead leader with no quorum possible, and node evacuation when a host is permanently gone.

SHC’s clustered control plane is rqlite (Raft over SQLite): an external rqlited process each daemon supervises. Raft stays available as long as a majority of voting members can reach each other. Lose the majority and the cluster becomes read-only until a human intervenes. This document describes those interventions.

For normal multi-node operations see Multi-node cluster setup; for daemon upgrades see Upgrading SHC; for losing the whole site that holds the voters see Losing a site.


  1. Take a fresh backup of raft state on the surviving node(s) before attempting any recovery step. On each node:

    Terminal window
    cp -a /var/lib/shc/rqlite /var/lib/shc/rqlite.before-recovery-$(date +%s)

    Recovery operations can be destructive. A cold copy of the rqlite data directory is the best escape hatch.

  2. Identify the current state. shc node ls and shc cluster status show who’s still alive:

    Terminal window
    shc node ls
    shc cluster status

    The output flags which nodes are voters and who (if anyone) is currently leader.

  3. Identify which node should become leader. Prefer the node that was most recently a voting leader with the most up-to-date data. If you’ve been tracking raft log indexes in monitoring, pick the one with the highest index.


Scenario 1: Leader crashed, quorum still possible

Section titled “Scenario 1: Leader crashed, quorum still possible”

Symptoms. shc node ls shows the leader as unhealthy but a majority of voters are still healthy.

What happens. Raft automatically elects a new leader from the remaining voters. You’ll see leadership move in shc node ls within ~5–10 seconds (default Raft election timeout).

Your action. Usually none. Verify that another node took over. Once the crashed leader comes back, it rejoins as a follower and catches up from the log.

If the crashed node can’t be recovered, proceed to Scenario 3 below (evicting a dead node).


Scenario 2: Quorum lost, more than half the voters are down

Section titled “Scenario 2: Quorum lost, more than half the voters are down”

Symptoms.

  • shc cluster status answers, with [X503021CLSQRM] naming the unreachable voters; the status verb is quorum-exempt
  • Writes fail: [X500901OPFAIL] … 503 Service Unavailable, message: leader not found
  • Reads fail too. shc node ls returns [X500899NODREPO] with the same 503. This is not a read-only degradation: the control plane is fully wedged
  • Logs mention is_quorum_lost or X200918RCVOPW
  • Containers already running keep running. Both quorum losses are control-plane only; the data plane is untouched

Why. Raft requires a majority of the original voter set to commit writes. A 3-voter cluster can tolerate 1 failure; a 5-voter cluster can tolerate 2. Lose more than that, and Raft intentionally refuses writes rather than split-brain.

If the missing voters just crashed and the hosts are recoverable (e.g. a power outage), bring them back. Raft picks up exactly where it left off. This is always the preferred option.

Option B: Force recovery on a surviving node

Section titled “Option B: Force recovery on a surviving node”

Use when the missing voters are permanently gone (hardware failure, host destroyed) and you need the cluster to accept writes again.

★ shc cluster repair --force now works, and it is the preferred form. It performs exactly the rqlite procedure documented below (writing the one-member peers.json and restarting), using the node id and raft address the running engine already holds, so there is nothing to transcribe.

Prefer it over doing this by hand, because it refuses in cases you cannot easily check yourself. It reads the raft roster and refuses (X409403CLSPRT) unless every other configured voter can be shown to be dead: not merely unreachable, and not merely unmeasured (that is the partition case below, the one that forks the database), and refuses (X409407CLSRVU) rather than proceeding when it cannot evaluate that question at all. Neither refusal is bypassable by --force: --force is what selects this verb in the first place.

The manual procedure is kept below because it is what the verb does, because it is the path when the daemon itself will not start, and because an operator in an outage should be able to see the mechanism rather than trust a verb. It has no gate: you are the gate. See Consensus and recovery §5.2.

Terminal window
shc cluster repair --force --dry-run # asks every question the recovery asks, changes nothing
shc cluster repair --force --yes

★ This is potentially data-losing, and it can manufacture a split brain. Uncommitted writes on the missing voters are lost. Worse: if the other voters are alive but partitioned rather than dead, they still hold the OLD raft configuration in which they are a majority, so they will elect their own leader and keep committing while this node commits independently. Unreachable is not a synonym for dead. Confirm the other voters are genuinely gone before you run this.

Proven on a 3-node lab against rqlite v10.2.7 with one surviving node.

Read the surviving node’s own -node-id and -raft-addr off the running process first:

Terminal window
ps -o args= -C rqlited | tr ' ' '\n' | grep -A1 -E '^-(node-id|raft-addr)$'

Then, on that node:

Terminal window
systemctl stop shc
printf '%s' '[{"id":"<node-id>","address":"<raft-addr>","non_voter":false}]' \
> /var/lib/shc/rqlite/raft/peers.json
systemctl start shc

On restart rqlite consumes the file (it is renamed to peers.info), forms a single-voter configuration and elects itself. Verify:

Terminal window
shc cluster status # QUORUM HEALTHY true, VOTERS 1, VOTER DEFICIT true
shc node ls # answers again; dead nodes show unreachable

What this preserves: the entire replicated database: node registry, deployments, config, vault, licence. Observed on the lab: a config row written before the outage read back correctly, all deployment rows survived, and every running container was untouched throughout.

What it costs: uncommitted writes on the dead voters, and voter membership: the survivors are demoted to a 1-voter set.

A single voter has zero fault tolerance. Re-enrol eligible control-plane nodes and check shc cluster status. SHC reconciles voter slots up to the voter target: the largest odd count of eligible control-plane members, capped at five. Confirm dead nodes are fenced before recovering quorum.

If no surviving voter has current data (e.g. all voters’ data directories are corrupt), restore the raft state from the most recent backup:

  1. Stop SHC on all nodes.
  2. On the node you’ll make the new single-node cluster:
    Terminal window
    systemctl stop shc
    rm -rf /var/lib/shc/rqlite
    # Restore your snapshot of /var/lib/shc/rqlite here
    systemctl start shc
  3. Verify /status reports healthy.
  4. Rejoin other nodes per cluster-setup.

Be aware: backups of /var/lib/shc/rqlite taken while SHC was running may be internally inconsistent. Prefer scheduled cold backups (SHC stopped briefly) for true DR reference snapshots.


Scenario 2b: A voter held off mesh0 after losing its mesh0 cache

Section titled “Scenario 2b: A voter held off mesh0 after losing its mesh0 cache”

A node whose raft served on its mesh0 address keeps {state}/mesh/cache.json to bring mesh0 up before raft starts. If that file is gone, the node does not move raft onto its underlay address: shcd starts degraded with X500575MSHCAC and rqlited stays down.

While the node is held, its daemon answers from its boot store, the single-node database as it stood when the node went onto raft, and not from the cluster’s store. A read such as shc node ls or shc status lists those old rows and prints the warning X200288STRBOT on stderr. Anything that would write, shc tenant add or shc install for example, is refused with X503287STRHLD, which repeats why the node is held and the way back. The same holds for any node that started degraded on its boot store: a joiner with nothing to join (X501035RQDNOJ) and a node whose rqlited did not come up in time (X501030RQDDGR). shc cluster repair, shc node join --force and shc init --force still run there.

Restore the file if you have it. Otherwise run, on that node:

Terminal window
shc cluster repair --dry-run # what it would rebuild; changes nothing
shc cluster repair # asks, then rebuilds and restarts shc

Before it rebuilds anything it asks every other node of its roster (the raft configuration and the signed membership table) over the peer status port, TCP 4004 at each node’s underlay address, with its own node certificate. It rebuilds only when every one of them answers as itself, none is the raft leader, none is on a newer raft term and none holds a newer membership table. Any other answer refuses with X500575MSHCAC naming the node and what it said:

  • a node that leads or is on a newer term means the cluster went on without this one, for example after shc cluster repair --force on another node. Rejoin it with shc node join --force where the cluster still runs;
  • a node that does not answer, or has no address this node knows, refuses too, because it cannot be counted as not leading. Bring it back, or open TCP 4004 between the nodes, and run the repair again.

The rebuilt cache holds this node and every peer at the address it answered on; shc restarts a second later and raft serves on mesh0 again under the same raft id.

Symptoms. A node is gone for good. Its row still shows in shc node ls as unhealthy and the cluster is otherwise healthy.

Action. On a healthy voter:

Terminal window
shc node forget <node-id>

This removes the node from Raft membership and SHC’s node inventory. Do not use this on a node that’s merely temporarily offline; it will have to rejoin as a fresh member if it ever comes back.

Source: modules/node/commands/forget.go (shc node forget).


Scenario 4: Draining a node for maintenance

Section titled “Scenario 4: Draining a node for maintenance”

Symptoms. You want to take a node down cleanly (for a hardware swap, OS upgrade, or permanent retirement) without disrupting running deployments.

Action. On any healthy node:

Terminal window
shc node drain <node-id>

Leave --timeout off unless you mean to cap the run: the relocations are serial, so the daemon multiplies the per-relocation budget (timeouts.drain, 300s) by the number of stacks it plans to move, up to 60 minutes. A figure you pass is the budget for the whole run, so the --timeout 300 this page used to show is now a five-minute cap that a multi-stack drain cannot finish inside.

What drain does (source: modules/cluster/drain.go):

  1. Marks the node as draining (no new stacks scheduled here).
  2. Asks the placement engine for a target within the deployment’s fences and selectors. A deployment with no eligible target fails the drain; --force permits stopping it in place.
  3. Relocates each deployment (stop on the source, bring up on the target), or stops it in place under --force.
  4. Waits for the moves to complete, up to --timeout seconds when you named one, else up to the auto-sized budget above.
  5. Returns a per-deployment report (migrated / stopped / failed).

If some moved and some did not, the node is left partially_drained rather than draining and the run reports X500406CLSDRP with both counts. Those cutovers have committed (the stacks are live on a peer) and a re-run resumes, skipping them. Only a run where nothing moved reports the blanket X500400CLSDRO.

Drain does not remove the node from the cluster. After drain, the node is idle but still a voter. To fully retire it:

Terminal window
shc node leave # run ON the draining node: removes it cleanly
# OR, from another node:
shc node forget <node-id>

shc node leave is the preferred path: it removes the node from the cluster gracefully from its own side, letting Raft pick up the change consensus-style.


Scenario 5: A node’s certs have expired or are corrupted

Section titled “Scenario 5: A node’s certs have expired or are corrupted”

Symptoms. The node can no longer authenticate to cluster peers; logs mention TLS verification failures or expired certificates.

Action. SHC auto-rotates node certs well before expiry (see subsystems/impl/internal_cert_rotation.go, ce module). If rotation failed:

Terminal window
# From any healthy node (orchestrates across the cluster):
shc cluster cert rotate --type mtls

If certs are irrecoverable, evict and rejoin:

Terminal window
# From a healthy node:
shc node forget <broken-node-id>
# On the previously-broken node, after any broken state is cleared:
shc node token # mint a fresh join token on a leader
shc node join <token> --force # token is POSITIONAL; --force wipes the
# stale cluster identity this node still holds

The token carries the leader’s address and the cluster CA, so there is no separate --leader to pass. --force is what lets a node that was already clustered rejoin: without it the join refuses with X400021ALRCLS because node.id and certs/node.pem are still on disk from the broken run. The wipe happens only once the cluster has ADMITTED the node, so a refused attempt leaves the node exactly as it was.


After any recovery operation, run through this checklist:

Terminal window
shc node ls # all expected nodes healthy, one leader
shc cluster status # no quorum warnings
shc ls --all-tenants # all expected stacks appear

Then run a read + write probe to confirm the DB is accepting writes:

Terminal window
shc tenant ls # read
shc tenant add recovery-probe # write
shc tenant rm recovery-probe

  • The voter count follows the voter target. That target is the largest odd count of eligible control-plane members, capped at five (VoterTarget in ce modules/cluster/voters.go). The suffrage reconciler promotes and demotes one node at a time, by removing the member from raft and rejoining it. A member that is down keeps its vote. Two eligible members steady-state at one voter, which has no spare vote.

  • Keep clocks synchronized (chrony / systemd-timesyncd). Clock skew > ~60s between voters breaks token validation and confuses leader election.

  • Monitor shc node ls / shc.status.get: you want to catch a degraded cluster before losing quorum, not after.

  • Control-plane snapshots run nightly at 02:00 UTC. The leader-only db_snapshot_cron writes a timestamped, consistent dump of the control-plane database to {state}/snapshots/control-plane/, a raft-consistent rqlite backup in cluster mode, a sqlite VACUUM INTO single-node. Both are online (no daemon stop). Tune the cadence with crons.db_snapshot_cron and how many to keep with maintenance.control_plane_snapshot_retention (default 7). Each snapshot lands on the node that led when it was taken, and the cluster records the newest one (node and file) in its store: health.control-plane-snapshot grades that record from any node and names where the file is. When the cluster has none on record, or its newest is more than 26 hours old (one missed nightly run), the leader records its own newer snapshot or writes one within 10 minutes (db_snapshot_first_cron), so a fresh cluster doesn’t wait for 02:00. shc db snapshot create takes one on the leader on demand, and shc db snapshot ls lists a node’s snapshots with their kind, time and size. An on-demand snapshot is kept to the same retention apart from the nightlies and is not recorded, since the record grades the nightly schedule. To restore one, stop SHC, swap the dumped .sqlite in for the control-plane DB, clear the raft engine marker + rqlite/ dir (as a demote does), and restart. Use backup for stack data.

    ★ Use a CALENDAR spec, never an interval, if you change the cadence. This cron was spelled every 24 hours until 2026-08-18, which anchors to the moment the cron table was built, and the leader cron table is rebuilt from scratch on every become-leader. Any roll, restart or leadership flap inside the window restarted the countdown, so on an estate that deploys daily it never fired: the newest snapshot was 10 days old on one production estate and 11 on the other, with the cron registered and healthy throughout. 0 2 * * * is immune. Verify with shc cron ls: NEXT RUN AT should be a wall-clock hour, not an offset from now.

  • Test recovery in staging. Force recovery has sharp edges; the first time you run it shouldn’t be during an outage. shc cluster repair --force --dry-run runs every check the recovery runs and changes nothing: it prints the node it would seat and the members it would drop, or the refusal the recovery would give, so the shape of it can be rehearsed cheaply.

  • shc doctor run checks the age of the control-plane snapshot on the leader. An absent or empty snapshot directory is an error.