Skip to content
SHC Docs

Flow: cluster join

Step-by-step trace of what happens when a new node joins an existing SHC cluster — from shc node token on the leader to the joining node becoming a healthy cluster member.

Source of truth:

  • modules/node/commands/local_token.go — the leader-side local mint fallback; modules/cluster/token_router.go — the daemon mint (POST /api/method/shc.cluster.token).
  • modules/node/commands/local_join.go — the shc node join command path.
  • modules/cluster/enroll_handler.go — the leader’s /internal/enroll endpoint.

Architecture overview: cluster module; security context: security model.

There is exactly ONE join credential: the LAN token (shc-lan-v1.), a CA-pinned bootstrap over any routable path. The Python-era mesh-token join (a throwaway Nebula instance reaching the leader over the overlay) was retired pre-1.0: joins ride the LAN token, and mesh membership is orchestrator-driven afterwards (shc cluster mesh enable is additive + idempotent).

Actors

Section titled “Actors”
ComponentRole
Leader nodeMints the join token; answers /internal/enroll on the :4003 internal API
Joining nodeRuns shc node join <token>
rqliteShared raft-replicated cluster DB the joining node will eventually become a member of
Docker Swarm (optional)Container orchestration mode — joiner gets the swarm token bundled with enrollment

Phase 1: Token minting (on the leader)

Section titled “Phase 1: Token minting (on the leader)”

Triggered by shc node token on a healthy leader. The CLI prefers the daemon’s POST /api/method/shc.cluster.token (so the single-use ledger records the token hash); with the daemon down it falls back to an unpersisted local mint. The encoded shc-lan-v1. token carries:

FieldPurpose
b (bootstrap token)Single-use secret; its hash is stored in the leader’s mesh_token ledger as “unused”
c (cluster id)Stable cluster identity
e (expires at)TTL bound (default 1h, --ttl)
j (join addrs)host:4002 per known node — the joiner rewrites these to host:4003 internal-API endpoints
caThe cluster CA cert — the joiner’s ONLY trust anchor for the enroll call

Tokens are time-limited and single-use: /internal/enroll consumes the ledger row atomically; a replay returns X401022TKNCNS, an expired token X401023TKNEXP.

Phase 2: Preflight (on the joiner)

Section titled “Phase 2: Preflight (on the joiner)”

LocalJoinLanCluster (local_join.go):

  1. Decode + expiry-check the token (X400022INVTKN on any defect — including a non-LAN prefix).
  2. Refuse to overwrite an existing cluster identity without --force (X400021ALRCLS); --force wipes node.id, certs/, cluster_join.json and the raft data dir, and drops the force-rejoin marker (#113/#121).
  3. NIC resolution — the shared core/nic detector picks the cluster-internal IPv4 (deterministic, private-preferred; --nic pins it and a pin that fails to resolve aborts the join with X400985NICNOF). This one address becomes the enroll physical_ip, the node-row internal_address, the raft bind+advertise, the internal-API bind, and the swarm advertise.

Phase 3: Enrollment call

Section titled “Phase 3: Enrollment call”

The joiner POSTs https://<leader>:4003/internal/enroll with FULL TLS verification against the token-embedded cluster CA (chain AND SAN — any enrolled node’s leaf chains to the same CA, so SAN pinning is what stops an enrolled node impersonating the leader). Body: bootstrap_token, node_name, physical_ip, runtime.

The leader (enroll_handler.go):

  1. Validates + consumes the bootstrap token (single-use ledger).
  2. Adopts the existing node id on a same-host rejoin (replaying its recorded node id — #88/#113), or mints node-<hex>; hostname collisions get a -N suffix.
  3. Signs the joiner’s :4003 leaf against the cluster CA with the physical IP + name/id SANs.
  4. Upserts the node row (internal_address = physicalIP:4003, state healthy, or transient joining on a rejoin).
  5. Probes the joiner’s raft port back from the leader (observation-only GAP-3 warn X503090CLSJNP).
  6. Returns the cert bundle + raft_join_addrs (the leader’s actual bound raft address) + optional swarm join token/manager addr.

Phase 4: Finalize on the joiner

Section titled “Phase 4: Finalize on the joiner”
  1. Persist certs (ca.pem, node.pem 0644 / node-key.pem 0600) — a missing leaf is FATAL and leaves no join marker.
  2. Persist node.id + cluster_join.json (the raft join addrs) — the daemon’s next boot reads these and joins raft.
  3. docker swarm join when the response carried swarm info (best-effort; --advertise-addr pinned to the same physical IP).
  4. Persist node_bootstrap.json (runtime, names, swarm identity, and the --nic pin + its resolved bind_ip when one was named) — consumed by node_self_register, the raft bind and the internal-API bind on the next daemon boot.
  5. Host activation: shc group membership + systemd enable/start.

On that daemon boot the rqlite_engine subsystem starts rqlited with -join against the cluster_join.json addresses (retried patiently while the leader restarts); the member comes in as a full voter.

Phase 5: Mesh (optional, orchestrator-driven)

Section titled “Phase 5: Mesh (optional, orchestrator-driven)”

Joining does NOT touch Nebula. On a mesh-enabled cluster the operator re-runs shc cluster mesh enable (or had run shc init --nebula on the first node): the leader signs a host cert per node and fans out /internal/mesh/install over the internal API, extending the overlay to the new member.

Phase 6: Leaving

Section titled “Phase 6: Leaving”

shc node leave reverses this (source: modules/node/commands/leave.go). It:

  1. Drains the node — evicts stacks scheduled here so they reschedule elsewhere.
  2. Leaves Swarm (if applicable).
  3. Removes itself from raft membership.
  4. Removes its row from the node table.
  5. Cleans up local cluster state.

Events

Section titled “Events”
EventWhen
A200900NDJOINNode joined cluster (leader emits after the row upsert)
A200908NDLEFTNode left cluster
X200027CLKSKWClock-skew warning during join
X400022INVTKNMalformed / expired / unknown token
X401022TKNCNSJoin token re-use blocked
X500023JNFAILEnrollment failed
X503090CLSJNPJoiner’s raft port unreachable back from the leader (observation-only reachability warn)

Trust model recap

Section titled “Trust model recap”
  • Join tokens are the only bearer credential in the flow; everything after enrollment is mTLS on :4003.
  • Tokens are minted by shc node token, which requires the platform-admin gate on the daemon route (see security).
  • The token-embedded cluster CA is the joiner’s trust anchor for the enroll call — full chain + SAN verification, never relaxed.
  • Post-join, every internal API call from this node is authenticated by its cluster-CA-signed leaf (plus the registry-membership gate).

See also

Section titled “See also”