Multi-node cluster setup
An SHC cluster is a set of nodes running the shc daemon that share a
raft-replicated control plane DB (rqlited) and mesh0 on UDP 4244. Mesh0
uses kernel WireGuard on Linux or userspace WireGuard on a utun on macOS. A
Mac joins as a non-voter with no control-plane role. On the apple runtime
it carries the shc fabric on a utun once its pf policy loads. Deployments
are scheduled onto specific nodes, but state is cluster-wide. Which network
carries what is recorded in the network layering
decision.
For the full procedural trace of what actually happens during join,
see architecture/flows/cluster-join.md.
This page is the operator-facing how-to.
Prerequisites
Section titled “Prerequisites”Before adding nodes, confirm:
- Every node has followed
Installation up through
step 3 (the daemon is running). Skip step 4 (
shc init) on every node except the first. - Clock skew between nodes is < 500ms. Raft will struggle
otherwise. Run
timedatectl status(or equivalent) on each host and verifySystem clock synchronized: yes. - UDP port
4244is open between every pair of nodes (mesh0). Firewalls and security groups must allow inbound and outbound UDP 4244 between cluster members. UDP4243is also required when the shc0 fabric is on. - TCP 443 is reachable from the joiner to the cluster’s HTTPS entrypoint. Enrolment goes through Traefik into the public API with the join token.
- TCP 4004 is open between every pair of nodes’ underlay addresses (the
peer status port). It is the one TCP port off mesh0: a node held off mesh0
asks its peers there whether one of them leads before
shc cluster repairrebuilds its mesh0 cache. Only a node certificate under the cluster CA completes its handshake. - TCP 4001/4002 (rqlite) and 4003 (internal API) use mesh0 addresses.
Permit them between members on mesh0; they do not need public firewall
openings. The
--nicpin selects the underlay address the tunnels use and narrows the internal API listener to that interface and mesh0. See the two-tunnel design, sections 2 and 7. - A network interface per node that the other nodes can reach.
SHC auto-detects it (deterministic, private-preferred); override
with
--nicon join if wrong.
1. Initialize the first node
Section titled “1. Initialize the first node”On the node you want to be the initial leader:
shc init --name <leader-name>Verify the cluster has one HEALTHY node:
shc cluster statusshc cluster nodesIf shc init has already been run, see
Architecture: cluster module
for recovery / re-init.
2. Mint a join token on the leader
Section titled “2. Mint a join token on the leader”shc node tokenThis prints a shc-lan-v1. join token and its expiry. Copy the
whole token. Properties of the token:
- Single-use: consumed atomically on first successful join.
- Time-limited:
--ttl <seconds>on token generation overrides the default (consult theshc node token -hhelp text for the current default). - Carries the cluster CA cert, the known nodes’ addresses and a single-use bootstrap secret. The join uses HTTPS 443 through Traefik into the public API. The token’s trust bundle verifies the signed response.
Treat tokens as sensitive. Anyone with the blob can add a node to the cluster until it expires or is consumed.
3. Join from the new node
Section titled “3. Join from the new node”On the node you’re adding (after completing steps 1-3 of Installation):
shc node join <TOKEN> \ --name <node-name> \ --nic <iface> # optional; defaults to auto-detect (private-preferred)A node behind NAT (a VM on a host’s NAT bridge with ports forwarded,
or a cloud instance whose public IP is 1:1 NAT) states the public IP the
cluster reaches it at with --public-address <ip>, and gets its own
--location. The token names each member at its own address and then at
its public one, and the leader registers and signs the joiner’s public IP
because that is where its request came from. Without --public-address
the leader refuses with X403285ENRNAT, naming the address to state.
Forward TCP 443 and UDP 4243 (shc0) and 4244 (mesh0) to the node.
What happens (summarized; full detail in flows/cluster-join.md):
- Preflight: token decode and expiry check, interface detection
(
--nicpins it). - Call
POST /api/join/node.enrollthrough Traefik on HTTPS 443 with the join token and the node’s mesh0 public key. The leader consumes the token, registers the node and returns a signed response with its mesh0 grant, internal-API certificate and mesh raft join addresses. - Verify the signed response, bring up mesh0 from the grant, then start the daemon, which joins raft over mesh0.
From here, the node is a first-class member. The scheduler places
deployments on it within their rules (shc install … --where 'node=<node-name>'
holds one there).
The new node joins mesh0 as part of enrol. Each enrol response carries
the joiner’s mesh0 address and peers. shcd coordinates the peer table.
Use shc cluster mesh status to inspect the mesh.
4. Verify from the leader
Section titled “4. Verify from the leader”shc cluster nodes # both nodes should show HEALTHYshc cluster statusshc cluster ping # TCP round trip to each node's internal address (:4003)shc cluster mesh diagnose # this node probes peer internal APIs over mesh0Adding more nodes
Section titled “Adding more nodes”Repeat steps 2 and 3 for each additional node. A new token must be minted each time; tokens are single-use.
Removing a node
Section titled “Removing a node”Graceful (preferred)
Section titled “Graceful (preferred)”From the node being removed:
shc node leaveThe node must be free of deployments unless leave is forced. Leave marks
its node row removed and removes it from raft membership. Source:
modules/cluster/leave.go.
Leave tombstones the node’s mesh_peer row while raft is still reachable.
After raft removal is confirmed, it takes local mesh0 down and clears its
cache. Peers drop the departed member on their next sweep.
Forceful (dead node)
Section titled “Forceful (dead node)”If the node is unreachable, run on a healthy node:
shc cluster rm <node-id>This removes the node from the raft voter set and deletes its row.
You’ll still need to shut down the actual host separately; SHC
can’t reach it to stop the daemon. Its mesh_peer row is tombstoned;
surviving peers remove its WireGuard key on their next sweep.
Troubleshooting
Section titled “Troubleshooting”A node cannot reach a peer over the mesh
Section titled “A node cannot reach a peer over the mesh”Run shc cluster mesh status and shc cluster mesh diagnose on the
affected node. Check that mesh0 is up and the peer has a fresh handshake.
The enrol response supplies the grant; the reconcile loop sweeps once per
minute. Check UDP 4244 between the nodes’ underlay addresses:
# On the peersudo tcpdump -i any udp port 4244# From the node, ping the peer's mesh addressping <peer-mesh-ip>Join succeeds but cluster nodes shows UNHEALTHY
Section titled “Join succeeds but cluster nodes shows UNHEALTHY”The node registered in the cluster database but a subsequent step failed. Check logs on both nodes:
journalctl -u shc -n 200shc cluster logs # if the daemon is reachableCommon causes: mesh0 connectivity failure, TCP 4002 blocked on mesh0, or clock skew.
”bootstrap token already used”
Section titled “”bootstrap token already used””Tokens are single-use. Mint a new one with shc node token.
Clock skew warning
Section titled “Clock skew warning”X200027CLKSKW: configure NTP/chrony and wait for sync. Raft will
work with sub-500ms skew but is more reliable with tight sync.
See also
Section titled “See also”-
Decision: network layering: mesh0 for nodes, the fabric for containers, NetBird for people
-
Flow: cluster join: the procedural trace