Two WireGuard tunnels, never double wrapped (design)
Status: open design (SHC-1201). mesh0 and shc0 are on next, and the leader signs the peer table (SHC-1286); a follower that answers an enrol asks the leader to sign the joiner’s table (
mesh.enrol.sign, SHC-1668), which section 7 does not describe yet. Key rotation is built for both devices (SHC-1345, lane MESHROT):shc cluster mesh rotate [<node>][:<device>]runs section 8’s prepare, acknowledge, switch and activate, a peer keeps the old key until the rotating node has switched (SHC-1815), and the leader rotates each key pastcluster.key_rotationon a schedule. A Mac’s shc0 key does not rotate. Relay and punching (sections 6 and 11, SHC-1285) are built, shc0 takes each peer’s route MTU from its own probe (SHC-1284), and a Mac node runs shc0 with a fabric grant of its own once its pf policy loads (SHC-1623, SHC-1642). Not built: section 4’s ICMP confirmation under a temporary nftables rule and section 5’smesh.mtu.fragmentedcounter; confirmation on next is handshake-based. Clay accepted the target on 2026-09-30 (“ok 2 tunnels but not double wrapped”). On next (the new line) the node plane is mesh0 and Nebula is deleted (stage 2c, SHC-1201). main and release/v0.0 keep Nebula because the live estates run them, and the bitspur move is app by app onto new nodes. Stage 1 was ce !270 (tests in shc !513); slice 2 made it the only path and its switch is deleted (section 13).
Out of scope, by ruling: any service mesh or L7 proxy (closed, no-service-mesh). This is layer 3 only: kernel WireGuard, routes, bridges and nftables. There is no per-packet code of ours and no new tunnel protocol.
1. Where we are
Section titled “1. Where we are”A node runs two overlays today, one inside the other:
- Nebula (
nebula0, UDP 4242, tun MTU 1300) carries the node plane. Since SHC-1101 the node row’sinternal_addressis the node’s mesh address, so rqlite and the internal API (:4003) ride Nebula. Enrol and the mesh’s own repair calls dialunderlay_address(cecluster.UnderlayAddrFor). - shc0 (kernel WireGuard, UDP 4243) carries containers. ce
subsystems/impl/fabric_reconcile.godials every peer at itsinternal_address, which is now a Nebula address. So every cross-node container packet is WireGuard inside Nebula, andfabric.MTUis 1300 - 80 = 1220 for shc0 and every fabric bridge.
On the rig (section 10) the double wrap moves 0.72 to 0.96 Gbit/s container to container at MTU 1220, with Nebula busy about one core on each side. Kernel WireGuard dialling the underlay directly moves 4.0 to 4.5 Gbit/s at MTU 1420 to 1440 on the same two VMs.
2. Target
Section titled “2. Target”Two kernel WireGuard devices per node:
| mesh0 | shc0 | |
|---|---|---|
| carries | node plane: rqlite (raft and HTTP), internal API, VIPs | container traffic |
| address | one /32 per node from the control-plane range 10.192.0.0/10 | the node’s fabric grant .1, carried with the fabric prefix |
| AllowedIPs per peer | the peer’s /32, plus VIPs it holds | the peer’s fabric grant |
| UDP listen port | 4244 | 4243 (unchanged) |
| key | its own | its own (already in fabric_allocation.public_key) |
The ports are fixed per device and the same on every node. mesh0 never takes Nebula’s 4242, so deleting Nebula at stage 2c closes a port and moves nothing. Neither device uses 443: UDP 443 stays with Traefik, which may serve HTTP/3 later (not ruled). Enrol needs only TCP 443 outbound from the joiner (section 7) and opens no inbound port for itself. The tunnels then need UDP 4243 and 4244 between nodes, which a node behind NAT gets through its observed endpoints (section 11).
The fabric range is 10.128.0.0/10 with a per-node grant that is a genesis knob defaulting to /20, and node-local deployment networks come from 10.64.0.0/10 (ruled 2026-10-01). The rig keeps its 10.84.0.0/16 until it is rebuilt; nothing here renumbers a live cluster.
For a given peer both devices start from the same underlay address, so each packet is wrapped once. They are still two UDP flows with different ports: each roams on its own, can get its own NAT mapping and can hash onto its own ECMP path. Everything that is learned about a path (observed endpoint, confirmation, MTU) is therefore kept per device, and a result on mesh0 says nothing about shc0.
Keeping containers off the node plane. WireGuard’s cryptokey routing drops
a packet whose source is not in the sending peer’s AllowedIPs, so no shc0 peer
can claim a 10.192.0.0/10 source. That does not stop a legitimate fabric
source addressed to a node-plane destination. The enforcement is nftables, in
the per-node ruleset SHC-1202 compiles (ce modules/fabric/policy):
- forward: drop anything from a container-facing interface (shc0, the fabric bridges, deployment bridges, container veths) to mesh0 or to 10.192.0.0/10, and anything from mesh0 to a container-facing interface;
- input: a packet arriving on a container-facing interface reaches only the
host services declared for containers (the per-node resolver and the local
public listener); the internal API, rqlite and every other host port drop.
One narrow exception on shc0 itself: ICMP echo request and reply between a
peer grant’s
.1and this node’s.1(the confirmation and black-hole probes of sections 4 and 5), and the ICMP errors path MTU needs (fragmentation needed, packet too big, time exceeded); - input on mesh0: accept only from peer mesh addresses that are in the current table, the same ICMP set included.
The device split makes those rules interface matches rather than address lists, which is what keeps them short and auditable.
shcd is the coordinator: it owns the peer table, endpoint ranking, keys,
routes and probes, and drives both devices over netlink exactly as
modules/fabric already drives shc0 (wireguard.go, applier_linux.go).
3. The peer table, on the foundation ce already has
Section titled “3. The peer table, on the foundation ce already has”Nothing below is a new mechanism. The store, node liveness, request/response
to one node and fire-and-forget events are built in ce (memory
shc-foundation-already-built): core/db over rqlite, modules/nodelease,
modules/internal_rpc, modules/events. The mesh is rows plus a reconcile
loop, like the fabric.
Rows. Most of a peer is already stored:
node: id,internal_address,underlay_address,public_address,mesh_address,location,provider.fabric_allocation: the fabric grant, shc0’s public key, tombstones that keep a grant from ever being reused.
Stage 2a adds three tables, all in the single squashed initial migration (new line, no compat):
mesh_peer, keyed by node id, the identity:
| column | meaning |
|---|---|
node_id | PK, the node row |
kind | node or client (section 9) |
mesh_public_key | mesh0’s public key |
mesh_address | the /32 in 10.192.0.0/10, allocated at enrol, tombstoned forever like a fabric grant |
key_epoch | mesh0’s active key epoch |
updated_at |
mesh_rotation, keyed by (node id, device), a key rotation in progress on
one device (section 8): pending_public_key, epoch, prepared_at,
node_ready (the rotating node’s own acknowledgement). A device’s active key
lives where it always does: mesh_peer.mesh_public_key for mesh0,
fabric_allocation.public_key for shc0.
mesh_endpoint, one row per (observed node, device, observer), the paths:
| column | meaning |
|---|---|
node_id | the node this endpoint reaches |
device | mesh0 or shc0 |
observer | the node that saw it, or the node itself for an address it reports |
kind | lan, public or observed |
ip, port | the endpoint |
location | the observer’s location, for LAN ranking |
observed_at | refreshed while it stays true; rows older than 10 minutes are ignored |
Each node writes only rows where it is the observer, as conditional updates, and only on change or refresh. Liveness is not duplicated: a peer whose node lease has expired is still a peer (its tunnel may come back), and ranking reads the lease’s condition to try it last.
The signed table. Identity is what a node must not take from an
unauthenticated source, so the leader signs it. The signed object is a
snapshot of both devices’ membership: every mesh_peer row and every live
fabric_allocation row (grant and shc0 public key), every open
mesh_rotation row (node, device, epoch, pending public key), plus each
node’s identity key (below), with a generation (bumped on every join, rotation and
revocation), a content digest, issued_at and not_after = issued_at + 7 days. It is signed with the cluster signing key, the key every join token pins
(section 7). That is everything a node needs to bring both devices up before
rqlite is reachable. The leader re-signs at least hourly and on every
generation bump.
A node replaces its snapshot with one of a higher generation, or with one
of the same generation and digest and a later issued_at (a refresh that
only extends not_after). The same generation with a different digest is a
leader fault: the node keeps what it has and raises a doctor finding.
Identity keys are not WireGuard keys. WireGuard keys are Curve25519
Diffie-Hellman keys and are never used to sign. A node signs requests with its
existing cluster identity key, the key behind its internal-API client
certificate (ce modules/ca), which the snapshot binds to its node id. It is
rotated by the CA’s own procedure, never by a mesh key rotation, so a node
can always authenticate the recovery path in section 7 even halfway through
rotating its tunnel keys.
Endpoints are not signed: a wrong endpoint cannot impersonate a peer, because WireGuard authenticates every packet by key, and a node can only advertise endpoints as itself.
Change propagation. Reconcile is the source of truth, as for the fabric (a
node that was down converges when it returns, with no catch-up protocol). A
sweep runs every minute on every node and compares the store’s generation,
the snapshot and the kernel. To make a change usually take effect at once
rather than at the next tick, the writer also publishes mesh.peer.changed
through modules/events, which broadcasts to every peer with per-peer FIFO
and recovery on start; a receiving node runs its sweep early. A lost event
costs up to one sweep interval, never correctness.
Reads that must not need the leader. rqlite’s default read consistency
goes through the leader, so the mesh read path asks for local reads
explicitly (read consistency none) and accepts their staleness. Each node
also keeps the newest signed snapshot it has seen, from the store or from a
peer (section 7), and never replaces it with a lower generation.
4. Candidates, ranking and confirmation, per device
Section titled “4. Candidates, ranking and confirmation, per device”Gathering. For each device a node reports:
- LAN addresses: every global-unicast address on an interface that is not
SHC’s own (
mesh0,shc0, fabric and deployment bridges,nebula0while it exists, container veths), with the device’s port. - The configured
public_address, when the operator set one.
and records what it observes of others: Linux WireGuard sets a peer’s
endpoint to the outer source of each handshake or transport packet it accepts
from that peer, in processing order (after authentication and the replay
check), unless shcd sets it over netlink. So outside shcd’s own writes, the
endpoint node B’s shc0 holds for A is A’s shc0 flow as seen from B, address
and port.
B writes it as an observed row (node A, device shc0, observer B). No STUN
server is needed; the peers are the reflectors, per device.
Ranking, per peer and per device, highest first:
- Same site and a LAN candidate in a subnet this node also has: LAN
direct. Two nodes share a site when their location paths agree on the
cloud and the place, the first two segments (
gorillaservers/ogden/incus01and.../incus02do;hetzner/fsn1andhetzner/hel1do not). A same-site dial no handshake proves within five minutes is dropped for the next candidate, and tried again after an hour (SHC-2768). - Public direct: the peer’s
public_address, or a fresh observed endpoint for that device seen by a node in this node’s location. - Relay through a public node (section 6).
LAN-direct is decided per pair: home01 reaches home02 over the LAN and far01 over the public path at the same time.
Confirmation is an authenticated round trip, not a handshake. Setting a
new endpoint does not force a fresh handshake (an established session keeps
being used), so a handshake timestamp proves nothing about the new path, and
sampling the endpoint before and after a trial cannot prove it did not roam
away and back in between. A trial is therefore coordinated: the end that
wants it asks the other (mesh.trial, internal RPC over whatever path works
now) for an instant two seconds ahead and a trial window, and at that instant
both set exactly the endpoints they hold for each other, the candidate pair,
for its duration. Coordination alone does not stop a
packet already in flight, a NAT rebinding, or a tuple that used to belong to
another peer from arriving on another tuple, so for the window each end
admits only the candidate tuple to that device’s port: a temporary
nftables rule in prerouting drops every other UDP packet to the port, from any
peer. Any packet the device accepts during the window, from the peer under
trial or anyone else, came from the candidate tuple.
That briefly cuts the device’s other peers, so the window is kept short: the three probes go out together at its start, and it lasts twice the pair’s measured RTT plus 50 ms, capped at 400 ms. Their packets that drop are retransmitted by their own transports (TCP, raft), and 400 ms is well inside rqlite’s one-second heartbeat and election timeouts. For the same reason a trial runs only when something changed: this node set a new endpoint for the peer, WireGuard roamed it, or a kept endpoint failed (its handshakes stopped, or a trial of it failed). There is no periodic re-trial of an endpoint that keeps working: its handshakes are the evidence that it still does, and a re-trial of every endpoint every 10 minutes would drop a device’s other peers for about 20 seconds in every 10 minutes on a 50-node cluster. A node runs one trial per device at a time, leaves at least 2 seconds between trials so timeouts measured since last contact recover in between, and tries one peer at most every 30 seconds. An endpoint the kernel already holds when shcd starts is kept on its next handshake, since nothing changed, and so is any endpoint while no trial can run: a node without nftables (a Mac), a pair that is relayed, or a peer that does not answer the trial. Container UDP that drops in a window is lost, not retransmitted, which is the price of the attribution.
The probes are ICMP echoes carrying a fresh random identifier to the peer’s
address on that device (the peer’s mesh /32 on mesh0, its grant’s .1 on
shc0); the peer, under the same rule at its end, answers them. A reply
carrying the identifier that arrives inside the window entered this end over
the candidate tuple, and the probe it answers entered the peer’s end over the
candidate tuple at that end. The trial confirms the candidate when both ends
report a matched probe and reply inside the window. The rule is removed when
the window closes, success or not. A packet in flight from another tuple is
dropped; a NAT rebinding during the window makes the trial fail, which is the
right answer, and the next trial runs on fresh state.
When the peer cannot be reached by any path, a trial cannot be coordinated: it then confirms only that some path works, and shcd keeps whatever endpoint WireGuard roams to after the reply, recorded as observed, until a coordinated trial can rank it. On failure shcd moves down the list. A confirmed endpoint is sticky until confirmation fails three times in a row, so a flaky higher candidate never causes flapping. When WireGuard roams on its own, shcd reads the new endpoint back, records it as observed, and confirms it before keeping it.
5. MTU, per peer and per device, from the outer path
Section titled “5. MTU, per peer and per device, from the outer path”WireGuard never probes, and a tunnel MTU that is too large fails badly: on an IPv4 underlay Linux WireGuard sends its outer packets with DF clear, so an oversized inner packet is fragmented on the outer path and is either slow or, where a middlebox drops fragments, a black hole while small pings still work. For the same reason an in-tunnel ping with DF set proves only the inner limit: it can succeed through outer fragmentation. The probe must therefore measure the outer path, not the tunnel.
shcd borrows RFC 8899’s search rules (a base that is assumed to work, raise only on proof, black-hole detection, periodic re-probe) but probes outside the tunnel, from the host:
- What it sends: ICMP echo with DF set (IPv4) or an unfragmentable echo (IPv6), from this node’s underlay address to the peer’s confirmed endpoint address for that device. These are host packets on the underlay route, so the tunnel’s route and device MTU do not limit them; the search can go above whatever is installed.
- Search: start at the floor, an outer size of 1280 (the IPv6 minimum), and search upward to the egress link MTU. A size counts as proven after two replies; an unanswered size is not retried upward in the same round.
- Result: in-tunnel MTU = proven outer size less 60 (IPv4 underlay) or 80 (IPv6), written as a per-route MTU (below).
- When the floor fails (ICMP filtered, a NAT router answering for the peer, or a path below 1280): the result is “unproven” and the peer stays at the floor in-tunnel MTU, 1200. That floor is only safe on a path that carries 1280 bytes, which every IPv6 path must and every path we run does; an IPv4 path below 1280 is unsupported. A peer whose floor in-tunnel probe also fails is reported unreachable on that device and ranking moves to the next candidate or to relay.
- Re-probe every 10 minutes and whenever the device’s endpoint for that peer changes.
- Black-hole detector: shcd sends one in-tunnel probe at the installed size and one at the floor every minute per peer; when the full-size one fails twice while the floor answers, that peer’s route drops to the floor at once and the search restarts.
The MTU is therefore an estimate, and the design says so rather than claiming a proof. ICMP can hash onto another ECMP path than the UDP flow, a NAT router can answer for the peer, and on IPv4 Linux WireGuard cannot be made to set DF on its outer packets, so outer fragmentation cannot be prevented, only observed. Three detectors watch the estimate on the actual traffic:
- Fragments that arrive. Each node counts, in an nftables prerouting
chain at a priority before defragmentation, first fragments (MF set,
offset 0, which still carry the UDP header) from each peer’s endpoint to
each device’s port. A count means that peer’s packets toward this node are
too big. The receiver tells the sender directly,
mesh.mtu.fragmented(device, count)by internal RPC over mesh0 or the 443 path, never through the store, and the sender drops that route to the floor and restarts the search. - Fragments that are dropped upstream never reach the counter; the black-hole detector above sees the full-size probes fail while the floor answers.
- Fragments reassembled upstream (a middlebox) are invisible to both, but then traffic is delivered, so the cost is performance, not loss.
A path every detector is blind to is one that delivers. A path that fragments at the floor, or fails the floor, is reported unsupported on that device. IPv6 outer packets are never fragmented in flight, so on IPv6 the black-hole detector alone covers it.
The result is written as a per-route MTU: <peer grant> dev shc0 mtu N and
<peer /32> dev mesh0 mtu N. Linux honours a route’s explicit MTU for
forwarded packets (ip_dst_mtu_maybe_forward), which is what container
traffic is; no sysctl is needed. Each device’s own MTU is the largest of its
routes, and the fabric bridges are created at shc0’s MTU. The policy compiler
adds an MSS clamp on shc0 (tcp option maxseg size set rt mtu) so TCP never
depends on ICMP reaching a container: the forward chain of SHC’s own
ip shc_fabric table opens with it for every SYN in or out of shc0, and the
kernel takes rt mtu from the smaller of the route each way, which on shc0 is
the peer’s probed route. A Mac’s pf has no rt mtu and clamps nothing yet.
shc0 has its own probe on next (SHC-1284, ce subsystems/impl/fabric_reconcile.go):
each peer’s route starts at the floor and is raised only on a proven outer
path MTU. Refusals and the readiness row keep the egress link’s MTU.
6. Relay
Section titled “6. Relay”When a peer is reachable by neither LAN nor public direct, shcd routes the pair through a public node R by moving AllowedIPs, not by adding a protocol. Relay still needs UDP from both ends to R; it covers NAT and missing routes, not a blocked UDP.
A relayed pair (A, C) via R on one device is decided by the pair’s canonical owner (below) and applied by both ends, because cryptokey routing checks sources on receive:
- on A: C’s prefixes on that device (C’s /32 on mesh0, C’s grant on shc0) leave A’s peer entry for C and join A’s entry for R;
- on C: A’s prefixes leave C’s entry for A and join C’s entry for R. Without this C drops A’s packets arriving from R, because their source is in A’s AllowedIPs, not R’s;
- on R: nothing changes in WireGuard (R already has A and C as peers with their own prefixes); nftables on R accepts forwarding between those two peers’ prefixes on that device, for pairs R is the current relay for, and nothing else.
R must have a direct confirmed path to both ends on that device; a relay is never itself relayed, which is what keeps R’s own configuration unchanged and rules out routing loops.
One prefix has exactly one peer owner per device. shcd moves a prefix with one wgctrl configuration request that removes it from one entry and adds it to the other, but the kernel may apply a request across several messages, so the prefix can be briefly owned by neither entry (its packets drop for that instant). A request that fails part way leaves the device in some mix of old and new; the next sweep re-applies the full desired state, which is idempotent. Return routing needs nothing extra: each end’s kernel route to the other’s prefix is the connected device route, and WireGuard picks R by AllowedIPs.
Who decides, without quorum. A relay change must work exactly when a path
has just failed, possibly with quorum gone, so it does not go through the
store. Each pair has a canonical owner, the end with the lower node id; only
the owner decides. It picks R, the lowest node id among nodes that publish a
public address, have a confirmed direct path to it (a handshake within
ObservedHandshake) and are not relayed themselves; ranking by round-trip
time is not built. It sends mesh.relay.set(device, a, c, relay, generation)
by internal RPC to R over its direct tunnel, carrying the owner’s signed
proof; R checks that it has a direct confirmed path to the other end too,
forwards the call, and returns the other end’s acknowledgement.
Generations are per pair and per device. The owner keeps the last one it
issued on disk; each end keeps the last one it applied on disk. An end applies
a strictly higher generation, re-acknowledges an identical one without
reapplying it (so a lost acknowledgement is recovered by retrying the same
call), and ignores a lower one. Nothing goes to the store: every generation
lives in <state home>/mesh/relay.json (the owner’s issued ones, each end’s
applied ones, the pairs a relay forwards). A node that restarts reads its own
file, keeps the relays it had applied, and the owner’s next check confirms or
replaces them.
Failover. Detection runs on the mesh sweep, once a minute and on a kick.
A direct path counts as failed on a sweep only once its last handshake is
older than ObservedHandshake (3 minutes), or, for a pair that never
handshook, once this end has dialled it that long; three failed sweeps in a
row declare it dead. A path that dies is relayed about five to six minutes
after its last handshake. Changing the relay or going back to direct is then
the same call with the next generation; until both ends have applied it,
packets in one direction are dropped. The owner gives a candidate three
attempts with a 5-second timeout each. If it does not acknowledge, the pair
is reported down on that device and the next sweep’s decision starts from the
next candidate; a pair with no candidate is reported down as well.
While a pair is relayed the owner re-sends the applied generation to its relay
every sweep; three failed checks drop that relay from the candidates and the
pair is decided again, and three sweeps in a row with a fresh direct handshake
send it back to direct. Each end re-confirms the pair after applying.
A release (relay "") follows owner → current relay → other end, with the
owner’s signed proof unchanged. The other end applies and acknowledges through
the relay before the owner applies; only an owner with no applied relay sends
a release directly. If the relay’s own entry has no handshake within
ObservedHandshake for three consecutive sweeps and the pair’s direct entry
has a fresh handshake, each end drops the relay locally without RPC, persists
a local fallback at its held generation, and warns once. A higher owner
generation supersedes that fallback; the owner decides again from fresh health.
What R sees: node-plane traffic is TLS end to end (mTLS on :4003, rqlite’s
own TLS), so R relays ciphertext. Container traffic is plaintext at R, a node
of the same cluster that is already trusted with a tenant’s workloads when
they land there.
7. Bootstrap, freshness and the 443 path
Section titled “7. Bootstrap, freshness and the 443 path”Raft rides mesh0, so rqlite cannot bring the mesh up. Each node keeps a local
cache, /var/lib/shc/mesh/cache.json (root, 0600), rewritten after every
successful sweep: the newest signed snapshot, the endpoint rows, and per peer
and device the endpoint that last confirmed.
Boot order.
- shcd starts, reads the cache, verifies the snapshot’s signature against the
cluster signing key pinned at enrol, and checks
not_after. It brings up mesh0 and shc0 with the last confirmed endpoints before rqlited starts. - rqlited binds the mesh address and finds its peers.
- The first sweep against the store replaces the cache.
When the cache is not enough. If no cached endpoint confirms within 30
seconds, or the snapshot is past not_after, the node fetches a fresh
snapshot over HTTPS 443 through Traefik into the public API, the same
route enrol uses (built by SHC-1207 behind the one RPC dispatch). It resolves
the estate’s host name, whose A/AAAA records the dns module already publishes
for the public nodes, and asks any of them. The request is authenticated by
the node’s identity key (section 3), a signature over a fresh server nonce,
checked against the key the snapshot binds to that node id, so it keeps
working while the node’s tunnel keys rotate. The response is the signed
snapshot plus the serving node’s endpoint rows. Trust never rests on TLS: the node verifies the
snapshot’s signature with the pinned cluster key and refuses a lower
generation than it holds. A node answering this path serves the newest
snapshot it holds from its own cache, so the path works while quorum is
lost and before rqlited is up anywhere. There is no lighthouse role, and the
internal listener binds the mesh address only, never the underlay.
Enrol is the same route with a single-use, short-TTL join token instead of a node signature. The token pins the cluster signing key; the response (the new node’s grant, mesh address, the snapshot and the leader’s signature over the new row) is signed, so the joiner verifies it before trusting anything. The route is rate-limited per source and forward-auth lets exactly that path through.
Freshness and revocation. A snapshot is a statement of membership at a generation, not forever:
- a node applies a higher generation as soon as it sees one (event, minute sweep, any 443 fetch) and drops every peer the new snapshot no longer lists;
- a node holding a snapshot past
not_afterkeeps its existing tunnels up (the data plane is not torn down because the leader is gone) but accepts no peer it does not already have, raises a doctor finding and an alert, and polls the 443 path every minute; - local reads can be stale, and a healthy tunnel never triggers the 443 path, so every sweep also reconciles with peers: as part of the minute’s confirmation round a node asks each peer it has a direct mesh0 path to for its snapshot generation (one small RPC per peer) and fetches the snapshot from the first one that is higher;
- a revocation therefore reaches a node within one sweep (usually at once, through the event) after the newer snapshot reaches any node it has a direct mesh0 path to, or the store copy it reads. A node with no such path gets it when it next reaches the 443 path. The 7-day validity bounds how long a node can add peers from a stale table, not how long it keeps running;
- a cached peer answers but quorum is unreachable: tunnels come up from the cache, the node takes the highest generation any reachable peer serves, and joins, rotations and revocations wait for quorum. Nothing else does.
8. Keys
Section titled “8. Keys”Each device has its own static key. WireGuard already rotates session keys every two minutes, so static rotation is for containing a compromise, not for forward secrecy. WireGuard cannot hold two keys for one peer with the same AllowedIPs, so a rotation is break-before-make per pair; the design bounds the break and gives every miss a way back. On next the break on mesh0 is one handshake (SHC-1815): while a rotation is open every peer holds a second entry for the rotating node, under its pending key and with no AllowedIPs, and moves the node’s prefixes to it only once the node has switched. No peer drops the old key first.
Rotation (shc cluster mesh rotate [<node>][:<device>], and on a
schedule), one node and one device at a time. With no device named, mesh0
rotates and then shc0. The leader’s mesh_key_rotation cron starts at most
one rotation an hour, the oldest key past cluster.key_rotation (30d), each
key dated by a kv row written with its activation or the first tick that saw
it. A Mac’s shc0 key does not rotate: the Mac withdraws and republishes it
with its pf policy. Every step is keyed by (node,
device, epoch) and is idempotent, so any lost request or response is
recovered by sending it again:
- Prepare. The node generates the new key pair for that device and
writes the new private half to disk (root, 0600) beside the old one
before anything else, so a crash cannot leave the cluster expecting a
key the node no longer has. Over internal RPC it asks the leader to record
a
mesh_rotationrow (node, device, the new public key,epoch= active epoch + 1), retrying every 10 seconds. The leader writes the row, bumps the snapshot generation and publishesmesh.peer.changed. A retry with the same key and epoch is answered as success. - Acknowledge. The prepared rotation is in the signed snapshot, so every
peer that applies it adds the pending entry to its device and, once the
device holds it, answers
mesh.rotation.ack(node, device, epoch, digest of the pending public key), naming exactly what it acknowledges, stored as a row; an acknowledgement for another key or epoch counts for nothing. The entry under the current key is not touched, so the acknowledgement rides the link it leaves alone. The rotating node acknowledges too (node_ready), and only once it has read its own prepared row back. - Switch. When the rotating node and every node with a live node lease
have acknowledged, voters and non-voters alike, the leader records the
rotation as switching, signs that, and hands the table to the rotating
node. The node brings its device up under the new key and then answers,
naming the key its device runs. A non-voter is waited for too: one that
does not hold the pending key at the switch is cut off from the rotating
node, and with one voter that is its only way to the store. A node whose
lease is not ready holds nothing up, and neither does a non-voter that
mesh0 no longer counts. A node that has not acknowledged
after 2 minutes is named by
shc cluster mesh rotate(X504816MSHACK) and inshc cluster mesh status; nothing switches, and at 5 minutes the rotation is dropped like one withoutnode_ready, the old key still active everywhere. A switching rotation cannot be aborted. Each peer looks at its pending entry four times a second, and on a completed handshake moves the node’s AllowedIPs to it and drops the old entry in one write. The node asks every peer for its table right after it switches, so each of them sees that handshake at once. - Activate. On the rotating node’s answer, and only on an answer that
names the pending key, the leader writes the new key into the device’s
home (
mesh_peerorfabric_allocation), deletes the rotation row, bumps the generation and delivers the table to every node with a live lease.shc cluster mesh rotateanswersrotation activeonce that delivery is over. A prepare whose response was lost and that the node then abandoned (mesh.rotation.abort(node, device, epoch), or nonode_readywithin 5 minutes) expires, and the old key stays active. The rotation of a node that left mesh0 is dropped in any phase. - Recover. A peer that saw no handshake switches when it applies the activating generation: on the event, at its next sweep, at the peer reconciliation of section 7, or, when its confirmation of the other end fails, by fetching the snapshot over the 443 path (which does not ride mesh0; 10-second timeout, retried every 30 seconds). The rotating node signs that request with its identity key, not a tunnel key, so the recovery path accepts it in the middle of the rotation. Expected outage, not a bound: a pair that saw the handshake is down for that handshake and at most a quarter second more. When one end missed it, the pair is down until that end notices (its next sweep or a failed confirmation, within about 60 seconds), finds a node serving the current generation, and completes a handshake. Discovery asks the public nodes in turn, 10 seconds each, a stale answer counting as a failure, and repeats the round every 30 seconds; with k public nodes reachable and one of them current, that is at most k times 10 seconds per round, plus WireGuard’s own handshake retries (5 seconds apart). If no reachable node serves the current generation, the pair stays down until one does, and doctor names it. After activation the node drops the old private half at the next successful sweep.
Revocation is deleting the row (and tombstoning its addresses) at a new generation: every peer drops the entry on the event, the next sweep or the next 443 fetch (section 7). This replaces Nebula’s blocklist and certificate expiry.
9. macOS and the CLI
Section titled “9. macOS and the CLI”The darwin shcd embeds wireguard-go (golang.zx2c4.com/wireguard) in process
on utun devices and runs as a launchd system daemon. The control code is the
same; only the device seam differs (netlink on Linux, the wireguard-go device
and PF_ROUTE routes with -mtu on darwin).
- A Mac node (the apple runtime, SHC-1200) runs mesh0 from stage 2b, so it is a full member of the node plane. It runs shc0 and takes a fabric grant only once SHC-1200 settles how the apple runtime attaches containers to a bridge (open, section 15). Until then a Mac node’s containers are not on the fabric and the node holds no grant.
- The
shcCLI on an operator’s Mac joins mesh0 as a client (stage 2d):kind = clientinmesh_peer, a /32 from the client block 10.223.0.0/16 (the mesh pool’s last /16), no fabric grant, no shc0, no containers. The operator enrols it through the public API as a logged-in platform admin (shc cluster mesh client enrol): the CLI makes its own key, posts only the public half, and gets back its address and its gateways. There is no join token. An operator’s Mac already holds admin credentials, so a token would add a public entry point and protect nothing more (SHC-1288).revoketombstones the row the way a node’s removal does, andlslists clients. The client peers only with gateway nodes, the nodes with a public address. A gateway’s entry for the client allows exactly the client’s /32; the client’s entry for its current gateway allows the mesh pool.sudo shc cluster mesh client upholds the client’s utun in the foreground and refreshes its gateways every minute. Through a gateway a client reaches the public API listener (TCP 8000) on node mesh addresses and nothing else, masqueraded to the gateway’s mesh address, so no other node needs a route back to client addresses and every prefix keeps a single owner. The verdict is stateless, in mangle PREROUTING ahead of DNAT: the internal API, rqlite, every other port and every other client are dropped, and a node that is not a gateway drops all client traffic. A client never calls internal RPC and holds no certificate. It replaces NetBird for the Mac to cluster path only.
10. Measurement (rig, 2026-10-01)
Section titled “10. Measurement (rig, 2026-10-01)”iperf3 between two containers, one on home01 and one on home02. Both are VMs
on the same Proxmox host (pve02, 4 vCPU each, Debian 13, kernel 6.12), so the
“wire” is a host bridge, not a network. Throwaway devices, addresses and ports
beside the production Nebula, all removed afterwards; evidence and the change
ledger are in ~/.drover/state/evidence/SHC-1201/measure/. Throughput is the
sender’s rate from iperf3. Cores are each VM’s whole busy CPU over the run
(iperf3 included), sender / receiver.
| path | container MTU | 1 stream | 4 streams | reverse | RTT avg | cores, 1 stream (send / recv) | cores per Gbit, 1 / 4 streams |
|---|---|---|---|---|---|---|---|
| A. WireGuard inside Nebula (today) | 1220 | 721 Mbit/s | 950 | 960 | 0.63 ms | 1.74 / 2.56 | 6.0 / 4.9 |
| B. Nebula routes only, grant as unsafe_routes | 1300 | 1044 | 989 | 878 | 0.45 ms | 1.57 / 2.08 | 3.5 / 4.0 |
| B’. same at Nebula’s largest tun MTU | 1440 | 1160 | 995 | 1116 | 0.44 ms | 1.65 / 2.13 | 3.3 / 3.9 |
| C. WireGuard direct on the underlay | 1420 | 4035 | 4180 | 4098 | 0.39 ms | 1.26 / 2.22 | 0.86 / 0.92 |
| C’. same at the IPv4 MTU | 1440 | 4464 | 4372 | 0.43 ms | 1.37 / 2.27 | 0.82 / 0.86 |
Host to host for reference: 50 Gbit/s on the shared bridge, 1.26 Gbit/s over the production Nebula. Path MTU on the underlay was 1500 (DF 1472 passed, 1473 failed); container MTUs above were each verified with DF pings at the limit.
What it shows, for these two VMs: Nebula’s userspace forwarder topped out near 1.1 to 1.3 Gbit/s at about 1.4 cores per side whatever the MTU. Removing the inner WireGuard (B against A) gained 45 percent on one stream, 4 percent on four, and lost 9 percent in reverse, all under that ceiling. Kernel WireGuard on the underlay (C) was about four times faster and spent under one core per Gbit across both ends. What it does not show: a ceiling for other CPUs, real NICs or a WAN path, or bitspur’s workload capacity; it was one box with no wire. It is still enough to rank the options: Nebula-direct needs grants signed into certificates for a gain the two-tunnel design beats by a wide margin, so it is not worth building as an interim. It also measured UDP 4245 between the underlay addresses on the rig; that stage 1’s UDP 4243 is open on a given estate is a precondition to check there, not something this proves.
For the sequencing: the cutover rides path A, roughly 0.7 to 1 Gbit/s between two nodes’ containers on VMs like these. The workloads that would press on that (replicated volumes, bulk copies between nodes) are themselves after the cutover. Stage 1 is one config flip with no Nebula change, so it is the cheap lever if cross-node container throughput becomes a problem after the cutover.
11. NAT tiers
Section titled “11. NAT tiers”| tier | covers | when |
|---|---|---|
| 0. static | every node has a public address or a same-site LAN path; the house reaches in through a pfSense port forward | now (every SHC node today) |
| 1. observed | endpoint-independent NAT with a stable port (pfSense static-port outbound) | stage 2a (section 4 learns observed endpoints per device) |
| 2. punching | address-dependent NAT: while a pair is relayed and its direct handshake is not confirmed, the owner sends mesh.punch through the relay, which stamps both ends’ observed endpoints; both ends set them on the pair’s direct entry at one instant 2 seconds ahead, keepalive 25 s holds the mapping, and the usual three fresh handshakes release the relay | stage 2e, built (SHC-1285) |
| 3. relay | pairs with no direct path, symmetric NAT on both ends; still needs UDP to the relay | stage 2e, built (SHC-1285, section 6) |
Tiers 2 and 3 are on next (SHC-1285); nothing in stage 1 or 2a to 2c needs them for the estates we run.
12. Failure modes
Section titled “12. Failure modes”- Leader down or quorum lost. Tunnels run from the local tables and cache; only joins, rotations and revocations wait for quorum.
- Every cached endpoint dead (a node moved, a site renumbered). The 443 path (section 7).
- MTU black hole on a path. The detector drops that peer’s route to the floor; other peers keep theirs.
- Clock stepped backwards. WireGuard rejects handshake initiations whose TAI64N timestamp is older than the last one it accepted from that key, so a node whose clock jumped back cannot handshake until its clock passes the old value. A rotation (a new key) clears it; doctor flags the skew.
- UDP blocked on the underlay. Before it enrols, the joiner tries a
WireGuard handshake from a throwaway userspace device (wireguard-go on
netstack) to each member the token names, on the ports of the devices it
will run: both for a Linux node, mesh0 only for a Mac node without a
fabric grant. The member holds the throwaway key as a peer of its live
device, with no allowed IPs, for at most 15 seconds, armed by
underlay.probe.armon the 443 join route with the bootstrap token, checked and not spent. A handshake with any member passes the port. Otherwise the member’s counters for the key give the direction: nothing received refuses withX409598UDPOUT(from this node to the cluster), received and answered withX409599UDPBCK(from the cluster back to this node), each naming the port. The refusal comes before the token is spent. When no member can hold the key (an older member, a Mac member whose devices are userspace), the join enrols and the node publishesV300565UDPSKPwith the reason once its daemon runs. Doctor’sunderlay.udpruns the same check, armed on the same route under the node’s identity, for every peer with no handshake in the last three minutes. Relay does not help here. - A relay node dies. Confirmation fails on both ends of each relayed pair and the next generation picks another relay or a direct path (section 6).
- A rotation half applied. Section 8, step 4.
- A node returns under a new key while another voter is down. Until
rotation and the signed table land (SHC-1345 on SHC-1286), a peer that was
away cannot learn that key without the store, which rides mesh0, so the
pair stays down until raft answers on that peer again, and doctor
(
mesh.mesh0) names the peer and what it waits for. - Key compromise. Rotate the node, or revoke its row.
13. shc0 on the underlay
Section titled “13. shc0 on the underlay”Stage 1 (ce !270, tests shc !513) put this behind a cluster.fabric_endpoints
switch while Nebula carried the node plane. Slice 2 made a node’s
internal_address its mesh0 address, so dialling it would wrap every
container packet twice, and the switch is deleted (ce !458, tests shc
!718). There is one path:
- shc0 dials every peer at its underlay address, the host mesh0 dials too. A peer with no underlay address is left out of shc0;
- each node publishes an underlay readiness row in the
kvtable (namespacefabric, keyunderlay, scoped to its node id, value its own underlay link MTU less 80 and the peers its link cannot carry 1220 toward, with a 5-minute expiry the sweep refreshes at half life); - a pair runs above 1220 only while both rows are fresh and neither refuses the other. The per-route MTU is the smallest of the two published values and what this node’s shc0 prober has proven the outer path toward the peer carries (SHC-1284): a peer starts at the floor and is raised only on proof. Every other peer runs at 1220, which every path carries;
- a failed readiness read keeps the last good read until each row expires.
Both ends dial the same underlay hosts, so WireGuard roaming has no second path to move a pair onto, and the settle, leave and roaming machinery stage 1 needed is gone with the switch.
Every intermediate state is safe. The applier creates a fresh shc0 at 1220. On each sweep it lowers first: the device MTU and every owned route whose MTU goes down or goes away change before the WireGuard endpoints change; then the endpoints; then every MTU that goes up. An error at any step returns with the lower value in place, which every path can carry. Owned per-peer routes carry rt protocol 167, are added exclusively and deleted only with that protocol, so a route an operator or another daemon owns is never replaced or removed; a destination someone else holds pins shc0 at 1220 and the sweep reports it. It needs UDP 4243 reachable between underlay addresses.
The bridges. A tenant bridge is created at shc0’s MTU when that is above 1220. Docker and podman cannot change a network’s MTU in place, so a bridge at the wrong MTU is dropped and recreated only on a drained node no replica is pinned to and no container, running or stopped, names; until then its containers keep the MTU they were created at.
14. Stage 2: mesh0, Nebula removed
Section titled “14. Stage 2: mesh0, Nebula removed”Five steps, each its own MR, each behind the previous one being proven.
- 2a. mesh0 beside Nebula (Linux nodes). The
mesh_peer,mesh_rotationandmesh_endpointtables, the signed snapshot, per-device candidates, ranking, confirmation, the outer-path probe, the cache and the 443 fetch. mesh0 comes up on 4244 with its 10.192.0.0/10 address and carries nothing but probes and diagnostics (shc cluster mesh diagnosegrows a mesh0 column). shc0 takes its per-route MTU from its own probe instead of stage 1’s link MTU less 80. Rollback: turn the mesh0 switch off; shcd deletes mesh0 and its routes, and shc0 returns to section 13’s rule (link MTU less 80 between peers whose readiness rows are both fresh, 1220 otherwise) on the next sweep, through the same lower-first ordering. - 2b. the node plane moves to mesh0, and Mac nodes join it. New clusters
do it at genesis: the genesis node’s
internal_addressis its mesh0 address and rqlited binds it. An existing new-line cluster moves node by node: drain, leave, re-enrol onto mesh0, the same verbs the Nebula join uses today. A Mac node runs mesh0 from here. Rollback: re-enrol that node onto Nebula, which is still installed. - 2c. delete Nebula.
modules/nebula, thenebula_meshtable, the lighthouse and Nebula certificate verbs, and the bundled binary go; UDP 4242 closes. mesh0 stays on 4244, so no port moves.shc cluster meshkeeps its name and gainsrotate. Rollback: none in place. Roll back the release on fresh nodes and restore, the same procedure as the cutover. - 2d. the operator client. Client enrol and revocation, gateways and their masquerade (section 9). Rollback: revoke the client rows; nodes are unaffected.
- 2e. NAT tiers 2 and 3. Punching and relay (sections 6 and 11), built on next (SHC-1285). Rollback: turn relay selection off; pairs fall back to direct candidates, which is all tier 0 needs.
15. Open
Section titled “15. Open”- Relay choice by round-trip time, and declaring a dead direct path in seconds rather than over three one-minute sweeps (section 6): not built, not ruled.
- Whether Traefik serves HTTP/3 on UDP 443 (Clay asked, not ruled). Nothing here depends on it.