Disaster recovery

This guide covers recovery from failure modes that normal node operations (shc node drain, rolling upgrades) don’t handle — specifically raft quorum loss, dead leader with no quorum possible, and node evacuation when a host is permanently gone.

SHC’s clustered control plane is rqlite (Raft over SQLite) — an external rqlited process each daemon supervises. Raft stays available as long as a majority of voting members can reach each other. Lose the majority and the cluster becomes read-only until a human intervenes. This document describes those interventions.

For normal multi-node operations see Multi-node cluster setup; for daemon upgrades see Upgrading SHC.

1. Take a fresh backup of raft state on the surviving node(s) before attempting any recovery step. On each node:

   cp -a /var/lib/shc/rqlite /var/lib/shc/rqlite.before-recovery-$(date +%s)

   Recovery operations can be destructive. A cold copy of the rqlite data directory is the best escape hatch.

2. Identify the current state. shc node list and the /status endpoint show who’s still alive:

   shc node list
   curl http://127.0.0.1:4003/status

   The output flags which nodes are voters and who (if anyone) is currently leader.

3. Identify which node should become leader. Prefer the node that was most recently a voting leader with the most up-to-date data. If you’ve been tracking raft log indexes in monitoring, pick the one with the highest index.

Symptoms. shc node list shows the leader as unhealthy but a majority of voters are still healthy.

What happens. Raft automatically elects a new leader from the remaining voters. You’ll see leadership move in shc node list within ~5–10 seconds (default Raft election timeout).

Your action. Usually none. Verify that another node took over. Once the crashed leader comes back, it rejoins as a follower and catches up from the log.

If the crashed node can’t be recovered, proceed to Scenario 3 below (evicting a dead node).

Symptoms.

• shc node list hangs or errors (X500021CLSINT)
• The daemon returns 503 Service Unavailable on writes
• shc.status.get still answers (it is quorum-exempt) while writes fail closed
• Logs mention is_quorum_lost or X200918RCVOPW

Why. Raft requires a majority of the original voter set to commit writes. A 3-voter cluster can tolerate 1 failure; a 5-voter cluster can tolerate 2. Lose more than that, and Raft intentionally refuses writes rather than split-brain.

If the missing voters just crashed and the hosts are recoverable (e.g. a power outage), bring them back. Raft picks up exactly where it left off. This is always the preferred option.

Use when the missing voters are permanently gone (hardware failure, host destroyed) and you need the cluster to accept writes again. This is a potentially data-losing operation — any uncommitted writes on the missing voters are lost.

On a surviving node that you want to become the new leader:

shc cluster repair --force

The command:

1. Calls check_recovery_needed to confirm the cluster is actually in a broken state (warns if the cluster looks healthy — don’t run repair --force on a healthy cluster).
2. Prompts for confirmation with a data-loss warning.
3. Rewrites raft membership to just this node, forcing an immediate election.
4. Clears leader / quorum caches.
5. Returns a RecoveryResult indicating whether this node is now leader.

If the dead voters still have entries in the cluster’s node inventory, add --remove-other-nodes to also evict them:

shc cluster repair --force --remove-other-nodes

Source: modules/cluster/service.go (Service.ForceLeader, delegating to its ClusterRecoverer.RecoverCluster).

After repair --force succeeds, treat the result as a new single-node cluster and rejoin other nodes fresh. Do not attempt to “un-evict” previously removed dead nodes — their Raft state will conflict with the new leader’s log.

If no surviving voter has current data (e.g. all voters’ data directories are corrupt), restore the raft state from the most recent backup:

1. Stop SHC on all nodes.
2. On the node you’ll make the new single-node cluster:

   systemctl stop shc
   rm -rf /var/lib/shc/rqlite
   # Restore your snapshot of /var/lib/shc/rqlite here
   systemctl start shc

3. Verify /status reports healthy.
4. Rejoin other nodes per cluster-setup.

Be aware: backups of /var/lib/shc/rqlite taken while SHC was running may be internally inconsistent. Prefer scheduled cold backups (SHC stopped briefly) for true DR reference snapshots.

Symptoms. A node is gone for good. Its row still shows in shc node list as unhealthy and the cluster is otherwise healthy.

Action. On a healthy voter:

shc node forget <node-id>

This removes the node from Raft membership and SHC’s node inventory. Do not use this on a node that’s merely temporarily offline — it will have to rejoin as a fresh member if it ever comes back.

Source: modules/node/commands/forget.go (shc node forget).

Symptoms. You want to take a node down cleanly — for a hardware swap, OS upgrade, or permanent retirement — without disrupting running stacks.

Action. On any healthy node:

shc node drain <node-id> --timeout 300

What drain does (source: modules/cluster/drain.go):

1. Marks the node as draining (no new stacks scheduled here).
2. For each deployment, picks a peer node to relocate it to — respecting placement constraints: a swarm deployment pinned to a specific node (node.hostname== / node.id==) is only moved to a node that satisfies the pin. If no eligible node exists, that deployment is reported failed (no node satisfies the deployment's placement constraints) unless you pass --force, which stops it in place instead.
3. Relocates each deployment (stop on the source, bring up on the target), or stops it in place under --force.
4. Waits up to --timeout seconds for the moves to complete.
5. Returns a per-deployment report (migrated / stopped / failed).

Drain does not remove the node from the cluster. After drain, the node is idle but still a voter. To fully retire it:

shc node leave          # run ON the draining node — removes it cleanly
# OR, from another node:
shc node forget <node-id>

shc node leave is the preferred path — it removes the node from the cluster gracefully from its own side, letting Raft pick up the change consensus-style.

Symptoms. The node can no longer authenticate to cluster peers; logs mention TLS verification failures or expired certificates.

Action. SHC auto-rotates node certs well before expiry (see subsystems/impl/internal_cert_rotation.go, ce module; the cluster-CA rotation subsystem is subsystems/cluster_cert_rotation.go). If rotation failed:

# From any healthy node (orchestrates across the cluster):
shc cluster cert rotate --type mtls

If certs are irrecoverable, evict and rejoin:

# From a healthy node:
shc node forget <broken-node-id>

# On the previously-broken node, after any broken state is cleared:
shc node token                   # mint a fresh join token on a leader
shc node join --token <token> --leader <leader-addr>

After any recovery operation, run through this checklist:

shc node list                      # all expected nodes healthy, one leader
shc cluster status                 # no quorum warnings
shc stack list --all-tenants       # all expected stacks appear
curl http://127.0.0.1:4003/health  # { database: ok, queue: ok }

Then run a read + write probe to confirm the DB is accepting writes:

shc tenant list                    # read
shc tenant create recovery-probe   # write
shc tenant delete recovery-probe

• The voter set is capped at 3 automatically. SHC decides voter vs non-voter when a node enrols: nodes join as voters until the cluster has 3, then additional nodes join as read-only non-voters (full replicas that don’t vote). So a cluster of any size ≥ 3 steady-states at 3 voters (odd) — you don’t hand-manage the voter count. A non-voter is promoted only by removing and re-joining it (rqlite fixes the role at join). The one case still worth avoiding is a 2-node cluster (2 voters, no fault tolerance either way); prefer 1 or 3+.
• Keep clocks synchronized (chrony / systemd-timesyncd). Clock skew > ~60s between voters breaks token validation and confuses leader election.
• Monitor shc node list / shc.status.get — you want to catch a degraded cluster before losing quorum, not after.
• Control-plane snapshots run automatically. The leader-only db_snapshot_cron writes a timestamped, consistent dump of the control-plane database to {state}/snapshots/control-plane/ every 24 hours — a raft-consistent rqlite backup in cluster mode, a sqlite VACUUM INTO single-node. Both are online (no daemon stop). Tune the cadence with crons.db_snapshot_cron and how many to keep with maintenance.control_plane_snapshot_retention (default 7). To restore one, stop SHC, swap the dumped .sqlite in for the control-plane DB, clear the raft engine marker + rqlite/ dir (as a demote does), and restart. Use backup for stack data.
• Test recovery in staging. repair --force has sharp edges — the first time you run it shouldn’t be during an outage.

• Multi-node cluster setup — normal cluster operations
• Upgrading SHC — planned upgrades (rolling, drain-based)
• Flow: cluster join — how joins work at the protocol level
• Architecture: cluster module — recovery internals
• Error reference — X200918RCVOPW — recovery-required event
• Error reference — X400023LDRFRC — force-recovery (repair --force) validation