Flow: cluster join
Layer 1. Mesh. Mesh
A joining node receives its mesh0 grant during enrolment, brings up mesh0,
then joins raft over its mesh address. Mesh0 uses UDP 4244, with kernel
WireGuard on Linux or userspace WireGuard on a utun on macOS. A Mac joins
as a non-voter, running the rqlited its archive carries at
lib/shc/bin/rqlited. The container fabric uses shc0 on UDP 4243, a
userspace device on a utun on a Mac on the apple runtime. Both tunnels
dial underlay addresses, so each packet is wrapped once.
Sources of truth:
- SHC
modules/node/commands/local_join.go,modules/cluster/enroll_rpc.go,modules/cluster/enroll_handler.goandmodules/cluster/join_client.go. - CE
modules/mesh,modules/cluster/mesh0_orchestrator.goandsubsystems/impl/mesh_reconcile.go. - Two WireGuard tunnels, sections 2, 3, 7 and 13.
1. Mint a join token
Section titled “1. Mint a join token”Run shc node token on a healthy cluster member. The daemon records the
bootstrap token in the single-use mesh_token ledger. The token carries
an expiry, cluster identity, join addresses and the cluster CA trust bundle
that pins the key used to verify the signed enrol response.
2. Prepare the joiner
Section titled “2. Prepare the joiner”shc node join <token> decodes and expiry-checks the token. The NIC
selection (--nic when supplied) determines the physical address reported
as the node’s underlay. It does not select the raft bind address.
The joiner generates its own mesh0 WireGuard key pair and sends only the
public key. The private key stays on the node. Cluster formation is guarded
by CE core/hostplat: join requires mesh (X501392CLJPLT without it).
Control-plane membership requires voter; an explicit fence or label
without that capability refuses with X501395ROLCAP.
3. Enrol over HTTPS
Section titled “3. Enrol over HTTPS”The joiner calls POST /api/join/node.enroll over HTTPS 443 through Traefik
into the public API. The bootstrap token authenticates the request. The
joiner verifies the response’s signature against the trust bundle in its
token before accepting certificates, grants or peer addresses.
A member that is restarting is not there for some seconds, and a join typed
right behind another one meets exactly that: the first join restarts the
leader for its storage engine, and every join recreates the edge on each
member. The joiner keeps trying every endpoint of its token for 90 seconds
while one of them refuses the connection, serves a certificate that does
not name its address, accepts the connection and gives no answer, answers
502, 503 or 504, or answers 429, in which case it waits as long as the join
route asks. A coded 503 from the handler is tried again the same way: a
member with no snapshot signer yet, or with a locked vault, answers with
that cause alone, after giving the token, the node row and the mesh0 row
back. Any other coded answer is the cluster’s refusal and ends the join at
once. When the window ends the join fails with X500023JNFAIL and says the
leader was still starting or restarting, with each endpoint’s last answer.
The leader’s enrol handler:
- Validates the request, including the mesh0 public key, and consumes the bootstrap token. Failed enrolment releases the consumed token for retry.
- Creates or adopts the node identity and records
underlay_addressfrom the reported physical IP. - Allocates the mesh0 grant under that public key. The
mesh_peerrow records the node’s key and address; the node’sinternal_addressbecomes its mesh address on port 4003. Failure to grant mesh0 fails enrolment. - Issues the separate internal-API mTLS certificate, including the mesh address in its SANs, and returns it with the mesh0 grant, fabric posture and raft join addresses on the mesh.
4. Activate the grant and join raft
Section titled “4. Activate the grant and join raft”The joiner validates the returned identity, certificate material, mesh0
grant and raft addresses before replacing local identity during --force.
It prepares the grant, persists the internal-API certificates and node id,
then activates the cached grant and writes the join marker. It brings up
mesh0 from that cache and records the bootstrap state before daemon startup.
mesh0 comes up before the daemon’s raft engine. rqlited binds and advertises the mesh address on TCP 4001/4002; the internal API uses the mesh address on TCP 4003. The joiner inherits the leader’s raft TLS posture. The leader decides its raft role from host capabilities and control-plane membership; a Mac is a non-voter.
5. Converge membership
Section titled “5. Converge membership”Identity on mesh0 is the WireGuard public key in mesh_peer. The leader
signs the peer table (CE modules/mesh/snapshot_sign.go, SHC-1286), and
shcd reconciles the device and local cache against membership. The mesh and
fabric reconcile loops each sweep once per minute. The two devices use
separate keys and underlay endpoints.
shc cluster mesh status reports the local device and its peers’ handshake
health. shc cluster mesh diagnose probes each peer’s internal API over
mesh0 from the node serving the command
(CE modules/cluster/mesh0_orchestrator.go).
6. Leave
Section titled “6. Leave”shc node leave refuses a node that still hosts deployments unless forced.
It writes its mesh_peer tombstone while it can still reach raft. Peers
retain its tunnel while raft still counts it.
After raft removal is confirmed, local mesh0 and its cache are removed;
the local WireGuard key remains for a rejoin. Survivors drop the departed
peer on their next sweep. The tombstoned row retains the address so it
cannot be allocated to another node.
A force removal of a node already dropped from raft tombstones its mesh
row and asks surviving peers to sweep immediately. Sources: SHC
modules/cluster/leave.go; CE modules/mesh/remove.go and repo.go.