Multi-node cluster setup

An SHC cluster is a set of nodes running the shc daemon that share a Nebula mesh, a raft-replicated control plane DB (rqlited), and (by default) a Docker Swarm. Stacks are scheduled onto specific nodes, but state — tenants, installed stacks, backups, schedules, audit — is cluster-wide.

For the full procedural trace of what actually happens during join, see architecture/flows/cluster-join.md. This page is the operator-facing how-to.

Before adding nodes, confirm:

• Every node has followed Installation up through step 3 (the daemon is running). Skip step 4 (shc init) on every node except the first.
• Clock skew between nodes is < 500ms. Raft will struggle otherwise. Run timedatectl status (or equivalent) on each host and verify System clock synchronized: yes.
• If using the optional Nebula mesh (shc init --nebula / shc cluster mesh enable — off by default): UDP port 4242 is open between every pair of nodes. Firewalls, security groups, and cloud NSGs need to allow inbound/outbound UDP on 4242 between cluster members. Without the mesh, no 4242 rule is needed.
• TCP 4002 is open between every pair of nodes (raft), and free on each node for the raft listener. Raft does not ride the Nebula mesh: rqlited binds :4002 on the node’s cluster interface — the --nic pin from join, re-resolved to its current IPv4, else the auto-detected private-preferred interface (source: subsystems/rqlite_engine.go). On nodes with no private network between them that is the public interface, so firewalls must allow TCP 4002 between cluster members there too. Since #58, clusters initialized fresh run the raft plane (and the rqlited HTTP API on :4001) under mutual TLS — genesis writes RaftTLS: true and joiners inherit the leader’s posture; only clusters initialized before that flag stay plaintext on :4002 (for those, the firewall rule is the only protection).
• A network interface per node that the other nodes can reach. SHC auto-detects it (deterministic, private-preferred); override with --nic on join if wrong.

On the node you want to be the initial leader:

shc init --name <leader-name>

Verify the cluster has one HEALTHY node:

shc cluster status
shc cluster nodes

If shc init has already been run, see Architecture: cluster module for recovery / re-init.

shc node token

This prints a base64-encoded MeshJoinToken to stdout. Copy the whole blob. Properties of the token:

• Single-use — consumed atomically on first successful join.
• Time-limited — --ttl <seconds> on token generation overrides the default (consult the shc node token -h help text for the current default).
• Contains the Nebula CA cert, a short-lived enrollment cert/ key, and the cluster’s lighthouse addresses. No secrets beyond what’s needed for bootstrap.

Treat tokens as sensitive. Anyone with the blob can add a node to the cluster until it expires or is consumed.

On the node you’re adding (after completing steps 1-3 of Installation):

shc node join <TOKEN> \
    --name <node-name> \
    --nic <iface>                # optional; defaults to auto-detect (private-preferred)

What happens (summarized — full detail in flows/cluster-join.md):

1. Preflight — kernel prep, clock check, interface detection, host keypair generation.
2. Start a temporary Nebula with the enrollment cert from the token.
3. Wait for mesh connectivity to the lighthouses.
4. Call POST /enroll on the leader over the mesh. The leader allocates a mesh IP, signs a permanent mesh cert, registers the node in the cluster database, and returns join info for raft and Swarm.
5. Stop the temporary Nebula; install + start the permanent Nebula as a systemd service.
6. Join raft using the leader’s addresses.
7. Join Swarm if applicable (leader returns the join token in the same response).
8. Verify health and exit.

From here, the node is a first-class member. Stacks can be scheduled to it (shc install … --node <node-name>).

shc cluster nodes        # both nodes should show HEALTHY
shc cluster status
shc cluster ping         # round-trip over the mesh

Repeat steps 2 and 3 for each additional node. A new token must be minted each time — tokens are single-use.

From the node being removed:

shc node leave

This drains scheduled stacks, leaves Swarm, leaves Raft, removes the node row from the cluster database, and stops Nebula. Source: modules/node/commands/leave.go.

If the node is unreachable, run on a healthy node:

shc cluster rm <node-id>

This removes the node from the raft voter set and deletes its row. You’ll still need to shut down the actual host separately — SHC can’t reach it to stop the daemon.

UDP 4242 between joiner and lighthouses is probably blocked. Test:

# From joiner
nc -u -v <leader-mesh-physical-ip> 4242 < /dev/null
# From leader
sudo tcpdump -i any udp port 4242

The node registered in the cluster database but a subsequent step failed. Check logs on both nodes:

journalctl -u shc -u nebula -n 200
shc cluster logs          # if the daemon is reachable

Common causes: raft port (4002) blocked, clock skew, Swarm join failure with a Swarm-enabled cluster.

Tokens are single-use. Mint a new one with shc node token.

X200027CLKSKW — configure NTP/chrony and wait for sync. Raft will work with sub-500ms skew but is more reliable with tight sync.

• Flow: cluster join — the procedural trace
• Architecture: cluster module
• Architecture: nebula module
• Architecture: node module
• Security model — cluster-internal traffic
• CLI reference — cluster and node