Skip to content
SHC Docs

Multi-node cluster setup

An SHC cluster is a set of nodes running the shc daemon that share a raft-replicated control plane DB (rqlited) and mesh0 on UDP 4244. Mesh0 uses kernel WireGuard on Linux or userspace WireGuard on a utun on macOS. A Mac joins as a non-voter with no control-plane role. On the apple runtime it carries the shc fabric on a utun once its pf policy loads. Deployments are scheduled onto specific nodes, but state is cluster-wide. Which network carries what is recorded in the network layering decision.

For the full procedural trace of what actually happens during join, see architecture/flows/cluster-join.md. This page is the operator-facing how-to.


interactive: drag or click to explore

Before adding nodes, confirm:

  • Every node has followed Installation up through step 3 (the daemon is running). Skip step 4 (shc init) on every node except the first.
  • Clock skew between nodes is < 500ms. Raft will struggle otherwise. Run timedatectl status (or equivalent) on each host and verify System clock synchronized: yes.
  • UDP port 4244 is open between every pair of nodes (mesh0). Firewalls and security groups must allow inbound and outbound UDP 4244 between cluster members. UDP 4243 is also required when the shc0 fabric is on.
  • TCP 443 is reachable from the joiner to the cluster’s HTTPS entrypoint. Enrolment goes through Traefik into the public API with the join token.
  • TCP 4004 is open between every pair of nodes’ underlay addresses (the peer status port). It is the one TCP port off mesh0: a node held off mesh0 asks its peers there whether one of them leads before shc cluster repair rebuilds its mesh0 cache. Only a node certificate under the cluster CA completes its handshake.
  • TCP 4001/4002 (rqlite) and 4003 (internal API) use mesh0 addresses. Permit them between members on mesh0; they do not need public firewall openings. The --nic pin selects the underlay address the tunnels use and narrows the internal API listener to that interface and mesh0. See the two-tunnel design, sections 2 and 7.
  • A network interface per node that the other nodes can reach. SHC auto-detects it (deterministic, private-preferred); override with --nic on join if wrong.

On the node you want to be the initial leader:

Terminal window
shc init --name <leader-name>

Verify the cluster has one HEALTHY node:

Terminal window
shc cluster status
shc cluster nodes

If shc init has already been run, see Architecture: cluster module for recovery / re-init.


Terminal window
shc node token

This prints a shc-lan-v1. join token and its expiry. Copy the whole token. Properties of the token:

  • Single-use: consumed atomically on first successful join.
  • Time-limited: --ttl <seconds> on token generation overrides the default (consult the shc node token -h help text for the current default).
  • Carries the cluster CA cert, the known nodes’ addresses and a single-use bootstrap secret. The join uses HTTPS 443 through Traefik into the public API. The token’s trust bundle verifies the signed response.

Treat tokens as sensitive. Anyone with the blob can add a node to the cluster until it expires or is consumed.


On the node you’re adding (after completing steps 1-3 of Installation):

Terminal window
shc node join <TOKEN> \
--name <node-name> \
--nic <iface> # optional; defaults to auto-detect (private-preferred)

A node behind NAT (a VM on a host’s NAT bridge with ports forwarded, or a cloud instance whose public IP is 1:1 NAT) states the public IP the cluster reaches it at with --public-address <ip>, and gets its own --location. The token names each member at its own address and then at its public one, and the leader registers and signs the joiner’s public IP because that is where its request came from. Without --public-address the leader refuses with X403285ENRNAT, naming the address to state. Forward TCP 443 and UDP 4243 (shc0) and 4244 (mesh0) to the node.

What happens (summarized; full detail in flows/cluster-join.md):

  1. Preflight: token decode and expiry check, interface detection (--nic pins it).
  2. Call POST /api/join/node.enroll through Traefik on HTTPS 443 with the join token and the node’s mesh0 public key. The leader consumes the token, registers the node and returns a signed response with its mesh0 grant, internal-API certificate and mesh raft join addresses.
  3. Verify the signed response, bring up mesh0 from the grant, then start the daemon, which joins raft over mesh0.

From here, the node is a first-class member. The scheduler places deployments on it within their rules (shc install … --where 'node=<node-name>' holds one there).

The new node joins mesh0 as part of enrol. Each enrol response carries the joiner’s mesh0 address and peers. shcd coordinates the peer table. Use shc cluster mesh status to inspect the mesh.


Terminal window
shc cluster nodes # both nodes should show HEALTHY
shc cluster status
shc cluster ping # TCP round trip to each node's internal address (:4003)
shc cluster mesh diagnose # this node probes peer internal APIs over mesh0

Repeat steps 2 and 3 for each additional node. A new token must be minted each time; tokens are single-use.


From the node being removed:

Terminal window
shc node leave

The node must be free of deployments unless leave is forced. Leave marks its node row removed and removes it from raft membership. Source: modules/cluster/leave.go.

Leave tombstones the node’s mesh_peer row while raft is still reachable. After raft removal is confirmed, it takes local mesh0 down and clears its cache. Peers drop the departed member on their next sweep.

If the node is unreachable, run on a healthy node:

Terminal window
shc cluster rm <node-id>

This removes the node from the raft voter set and deletes its row. You’ll still need to shut down the actual host separately; SHC can’t reach it to stop the daemon. Its mesh_peer row is tombstoned; surviving peers remove its WireGuard key on their next sweep.


Run shc cluster mesh status and shc cluster mesh diagnose on the affected node. Check that mesh0 is up and the peer has a fresh handshake. The enrol response supplies the grant; the reconcile loop sweeps once per minute. Check UDP 4244 between the nodes’ underlay addresses:

Terminal window
# On the peer
sudo tcpdump -i any udp port 4244
# From the node, ping the peer's mesh address
ping <peer-mesh-ip>

Join succeeds but cluster nodes shows UNHEALTHY

Section titled “Join succeeds but cluster nodes shows UNHEALTHY”

The node registered in the cluster database but a subsequent step failed. Check logs on both nodes:

Terminal window
journalctl -u shc -n 200
shc cluster logs # if the daemon is reachable

Common causes: mesh0 connectivity failure, TCP 4002 blocked on mesh0, or clock skew.

Tokens are single-use. Mint a new one with shc node token.

X200027CLKSKW: configure NTP/chrony and wait for sync. Raft will work with sub-500ms skew but is more reliable with tight sync.