Skip to content
SHC Docs

SHC Cluster Architecture

Layer 1. Mesh. Mesh

This document describes the cluster architecture for SHC, enabling secure communication, service discovery, and cross-node integrations.

Each node runs shcd and one container runtime. In a cluster, rqlited carries replicated state over mesh0, while nodes use shc0 for container traffic.

flowchart LR
A["shcd and rqlited on node A"] <-->|"mesh0: UDP 4244"| B["shcd and rqlited on node B"]
C["Containers on node A"] <-->|"shc0: UDP 4243"| D["Containers on node B"]
RuntimeExecutionFabric attachment
dockerDockershc0 on Linux, each container a routed /32 on a veth of its own
podmanPodman through the compose CLIshc0 on Linux, each container a routed /32 on a veth of its own. shc init warns that a podman cluster is single-node only
incusincus-composeshc0 on Linux, each instance a routed /32 on a routed NIC
appleApple’s container CLI on macOSshc0 on a utun, each replica on a /30 vmnet network of its own

All four runtimes are linked into shcd. Linux supplies the mesh, fabric and voter capabilities. macOS supplies mesh, adds fabric once its shc0 key is published after its pf policy loads (SHC-1623, SHC-1642), and joins as a non-voter. Sources: CE daemon/imports.go, core/hostplat and modules/fabric/types.go.

A Linux cluster node with the shc fabric and service VIPs, or a Mac node on the apple runtime, runs each replica as its own compose project. Other nodes run the deployment’s one project. Each replica runs on the node it is bound to: the leader finishes its project for that node and that node’s runner starts it (SHC-1426). See Runtime support and Traffic across runtimes.

In cluster mode the daemon supervises an external rqlited process (core/rqlited): SQLite replicated over Raft:

FeaturePortPurpose
HTTP API4001/tcpDatabase queries
Raft4002/tcpConsensus protocol

Cluster topology:

  • A joiner with the voter host capability in the control-plane fence takes a voter slot while the cluster is below its voter target: the largest odd number no greater than the eligible members, capped at five (ce/modules/cluster/voters.go). Every other joiner, including every Mac, enrolls as a non-voter (rqlited -raft-non-voter). Nodes are keyed by the string node id in {state}/node.id.
  • Leader elected via Raft consensus
  • Strong consistency for all writes

Data replicated through the raft engine:

  • Deployment, service and container state
  • Configuration (cascading scopes)
  • Vault secrets (encrypted)
  • Audit events

Leadership is derived from the raft leader itself (see below), not from lock rows in the database.

Secure inter-node communication:

PortPurpose
4003/tcpmTLS-authenticated internal API
4004/tcpPeer status port: mTLS, at the node’s underlay address

The peer status port answers one read-only function, cluster.peer_status: whether this node is the raft leader, its raft term, and the era and generation of the signed membership table it holds. It exists for the one caller that cannot reach the internal API, a node held off mesh0 (raft and the internal API both bind mesh0), which asks it of every other node before shc cluster repair rebuilds its mesh0 cache. The handshake requires a node certificate under the cluster CA, the function refuses a caller that is not a node of its signed table, and each source address is rate-limited.

Capabilities:

  • Container exec (WebSocket proxy)
  • Log streaming (SSE proxy)
  • Health checks
  • Deployment operations on remote nodes

HTTP routing and TLS termination:

  • Runs on nodes that need external access
  • Routes based on hostname rules
  • Automatic HTTPS via Let’s Encrypt
  • Supports HTTP-01 and DNS-01 challenges

On the shc fabric, shcd answers service names using the resolver’s current routing table. The renderer records provider names for integrations in the deployment’s stored spec; credentials and names come from plugs and sockets. A connection does not grant network permission across tenants.

Sources: CE modules/resolver, modules/app/fabric_render.go and modules/fabric/policy/doc.go.

mesh0 carries the node plane. shc0 carries container traffic between nodes. Every container is a routed /32 from its node’s /26 chunks, and the nftables reachability rules read its tenant as a label. Each tunnel dials the peer’s underlay address with its own keys. See Network fabric for the address plan and policy.

Terminal window
# On leader node, generate join token
shc node token
# On new node
shc node join <token>

The join process:

  1. The token carries the cluster endpoints, a single-use bootstrap secret and the cluster CA trust bundle.
  2. The new node calls POST /api/join/node.enroll through Traefik on HTTPS 443 with that token and its locally generated mesh0 public key.
  3. The leader registers the node and returns a signed response containing its mesh0 grant, internal-API certificate and mesh raft join addresses.
  4. The joiner verifies the response, brings up mesh0 and starts rqlited on its mesh address. The leader grants a voter slot only to an eligible control-plane member; a Mac joins as a non-voter.

Sources: SHC modules/cluster/enroll_rpc.go, enroll_handler.go and modules/node/commands/local_join.go.

RoleDescriptionRaft Role
LeaderRuns background tasks, coordinates operationsVoter
WorkerRuns containers, responds to leaderVoter if eligible and granted a slot; otherwise non-voter
Terminal window
shc cluster rm <node-id>

Requirements:

  • Node must have no running deployments
  • Relocate its deployments first by changing where they may run (shc update --where), or drain it

Events flow through the SSE broker and internal API:

sequenceDiagram
participant D as Docker Daemon
participant C as Event Collector
participant SSE as SSE Broker
participant INT as Internal API
participant N2 as Node 2 SSE
D->>C: Container event (start/stop/die)
C->>C: Enrich event with node_id
C->>SSE: Publish to events:{tenant} channel
Note over INT,N2: If clients on Node 2 need event
INT->>N2: Forward via internal API
N2->>N2: Publish to local SSE

SHC leadership follows the raft leader; there is no separate lock table. Each node’s leader_hub subsystem polls the raft engine’s reported leader id and compares it against the local node id ({state}/node.id); the matching node runs the leader-only work (ce subsystems/impl/leader_hub.go). Single-node installs are their own standing leader (no poller runs).

sequenceDiagram
participant HUB as leader_hub (each node)
participant DB as rqlited
loop Every leader_poll_interval
HUB->>DB: current raft leader id?
DB-->>HUB: leader id
HUB->>HUB: leader id == {state}/node.id → leader
end
Note over HUB: On a become/lose-leader edge the hub fans<br/>OnLeader out to every leader-aware subsystem
CertificateStoragePurpose
Cluster CAdatabase (encrypted, raft-replicated)Signs node certificates
Node certificatesFilesystemmTLS authentication

Automatic rotation:

  • Node certificates rotate 30 days before expiry
  • Zero-downtime via graceful reload
PortProtocolPurposeExposed To
4001TCPrqlite HTTP APICluster nodes over mesh0
4002TCPrqlite RaftCluster nodes over mesh0
4003TCPSHC Internal API (mTLS)Cluster nodes over mesh0
4004TCPSHC peer status (mTLS)Node underlay addresses
8000TCPSHC HTTP API, Traefik’s backendThis node, its system ingress, and operator mesh clients over mesh0
443TCPTraefik HTTPS, including enrolmentNode underlay addresses and public clients
4244UDPmesh0 (node plane)Node underlay addresses

A join needs 443 between nodes. A joining node enrols at the bootstrap’s POST /api/join/node.enroll on 443 before mesh0 exists, so a firewall between nodes that opens 4001 to 4004 and not 443 forms the first node and times every join out with X500023JNFAIL. The cloud modules’ inter-node firewall opens 443 with the plane ports.

PortProtocolPurposeExposed To
80TCPTraefik HTTPInternet
4243UDPshc0 (required when the fabric is on)Node underlay addresses

The node plane is mesh0 on UDP 4244, coordinated by shcd: kernel WireGuard on Linux, or userspace WireGuard on a utun on macOS. Each node holds one /32 from the cluster’s mesh range. Raft, the internal API and VIPs ride mesh0. Container traffic rides the separate shc0 fabric on UDP 4243. See Two WireGuard tunnels.

TierWhat it reaches
InternetOnly reaches Traefik (80/443); mesh0 and shc0 are between nodes
ClustermTLS-authenticated communication between nodes
LocalDocker socket never exposed to network

Key principles:

  1. mTLS for inter-node communication - All internal API calls are authenticated
  2. Raft for consensus - Distributed state with strong consistency (rqlite)
  3. No external dependencies - Cluster operates without internet access
  4. Private keys generated locally - Never transmitted

A deployment never names a node. Its --where says where it may run:

Terminal window
# Hold a deployment to one node
shc install postgres --where 'node=hel-02'
# Relocate it by replacing the expression; the update prints its plan first
shc update postgres --where 'node=hel-03'

Placement uses node labels and label expressions. --where limits eligible nodes and --prefer ranks them. shc node selector manages named node selectors; shc constraint sets constraints at cluster, tenant or deployment scope. Sources: CE modules/fence/commands (selector.go, constraint.go).

Every install, update and uninstall goes through shcd. An unreachable daemon refuses the operation with X503738DMNREQ.