SHC Network Fabric

Status: slice 1 (reachability) implemented and lane-driveable; slices 2 and 3 are still design. Written 2026-07-16, slice 1 built 2026-07-31, made reachable from terraform the same day.

What exists today, behind shc init --fabric shc and inert on every other cluster:

• the cluster-scoped fabric choice, validated against the topology matrix at init and at enroll, and stored in a durable raft-replicated cluster_fabric row (modules/fabric/posture.go);
• per-node /24 IPAM with permanent tombstoning (modules/fabric/repo.go);
• the shc0 kernel WireGuard device and its peer set, converged on every node by the fabric_reconcile subsystem — plus the local fabric bridge and the host forwarding / no-NAT glue;
• the renderer fabric target: pinned ipv4_address per service and extra_hosts carrying today’s alias strings for cross-node references (modules/app/fabric_render.go);
• shc cluster fabric status — peers, allowed-IPs, endpoints, handshakes, interface state, MTU and the /24 + pin map, with private keys structurally excluded;
• the terraform surface that makes all of the above reachable on real infrastructure: bootstrap.fabric / bootstrap.fabric_cidr on modules/core and modules/shc/hetzner, fabric / fabric_cidr on shc_cluster / shc_node / data.shc_cloudinit, and the hetzner lane’s SHC_FABRIC switch (see Driving it live).

Not built: the nftables policy compiler and default-drop (slice 2), and the Podman runtime adapter (slice 3). Intra-fabric policy is therefore OPEN, which is what slice 1 is defined to be. See Implementation notes for where slice 1 made a concrete choice this design left open.

The rest of this document is the design as written, and remains the authority for the unbuilt slices.

It describes a cluster-scoped network fabric, owned by the SHC daemon, that provides cross-node container connectivity without Docker Swarm. It is the enabling design for two currently-impossible topologies:

• Multi-node Podman clusters (Podman has no overlay networking at all)
• Multi-node Docker clusters without swarm membership
• Mixed clusters — Docker-compose nodes and Podman-compose nodes in one cluster, communicating transparently

For today’s networking model (scope networks, aliases, Traefik), see core/networking.md. For cluster membership and the internal mTLS mesh, see overview.md.

SHC’s only cross-node data plane today is the Swarm overlay driver. Compose-runtime stacks are placed on a single node but interconnect across nodes by attaching to attachable overlay networks; the Swarm runtime (docker stack deploy) requires overlay networks outright. Either way, every node must be joined to a swarm, because overlay networks need Swarm’s control plane for VXLAN coordination and cross-node IPAM.

Podman cannot participate in any of this:

• Netavark (Podman’s network backend) is explicitly single-host; the maintainers state multi-host is out of scope (netavark discussion #674).
• The CNI backend that once allowed flannel-style overlays was removed in Podman 5.
• The only serious community attempt at a Podman overlay driver — a WireGuard netavark driver (netavark PR #472, extracted to netavark-wireguard-plugin) — stalled as a static-config plugin (v0.1.0, 2023, unmaintained) because a network driver alone cannot do the control-plane half of the job: allocating IPs across hosts, distributing peer keys, and updating peers as workloads churn.

That missing control plane is the thing SHC already is: a raft-backed cluster store for cluster-wide allocation, an mTLS internal API for distribution, per-node daemons with reconcile loops, and a renderer that owns every compose file. The fabric design exploits exactly this.

Swarm overlay bundles four properties. The fabric must account for each:

The L2 point deserves emphasis: Swarm overlay is L2 in name only. Its control plane distributes per-container MAC/IP/VTEP mappings and programs static ARP/FDB entries precisely so it never floods; real broadcast and multicast do not work on overlay networks. Applications that run on Swarm overlay today therefore need only unicast IP reachability — which is what a routed L3 fabric provides. Nothing that works today requires the L2 illusion.

The fabric is a routed kernel-WireGuard network built at the host layer, not a container-runtime network driver and not a VXLAN L2 clone. Three arguments, in decreasing order of weight:

1. Control-plane state scales with nodes, not containers. A Swarm-style L2 clone requires programming ARP/FDB entries for every container on every host, updated on every container start, stop, and move — cluster-wide cached state that goes stale (the failure class behind the traefik_public scope-mismatch bug and the stale in-memory trust “bad certificate” incident, multiplied by container churn). The routed design’s distributed state is one route and one WireGuard peer per node; it changes only on enroll/remove. Per-container state lives solely in the local bridge IPAM of one host.

2. Runtime neutrality. A netavark plugin is Podman-only; a libnetwork plugin is Docker-only. A host-layer fabric sits beneath both: from either runtime’s perspective, the rendered network is a plain local bridge with a pinned subnet. One implementation covers Docker-without-swarm and Podman identically; the per-runtime residue is firewall glue only.

3. One encapsulation, precedented architecture. WireGuard is L3, so the routed model needs no second encapsulation (a VXLAN clone would still need WireGuard underneath for encryption — Swarm’s own answer, IPsec ESP, is exactly what a provider firewall silently dropped in live testing). Routed-over-WireGuard is the same shape as flannel’s host-gw/wireguard-native backends and k3s’s wireguard-native; pure-routed is Calico’s and Cilium’s default posture. The industry converged here for the same stale-L2-state reasons.

Fabric is a cluster-scoped choice at shc init (--fabric swarm|shc|none), stored in the cluster store. It determines which runtimes are legal; enroll and install validate against it.

Two constraints are forced, not chosen:

• shc fabric excludes the Swarm runtime. Swarm services can only attach to swarm-scoped networks; docker stack deploy cannot target the fabric.
• swarm fabric excludes Podman nodes. Podman cannot join a swarm or attach to Docker overlay networks.

none = single node is a product simplification, not physics: the SHC control plane already spans nodes without any app fabric, so a no-fabric multi-node cluster over a trusted LAN is technically possible — but then SHC cannot promise encrypted cross-node app traffic, and the fabric is cheap. The invariant multi-node ⇒ fabric keeps the matrix honest.

Fabric migration (swarm → shc) is deliberately out of scope; see Open questions.

Per node, the daemon owns three host-level things, none visible to the container runtime:

1. One kernel WireGuard interface (shc0), address 10.42.<node>.1/16 (CIDR configurable at init to avoid collisions), listening on a fabric UDP port. Peers = every other node, each with AllowedIPs = 10.42.<peer>.0/24, endpoint = the peer’s internal_address. Managed via netlink (wgctrl-go), no config files, no shelling out.

2. Plain bridge networks with pinned subnets carved from the node’s /24 — these are what rendered compose files reference. Docker and Podman consume them identically (--subnet / compose ipam config). Bridge MTU is set to underlay MTU − 60 (typically 1420).

3. Routing and firewall glue. The kernel routes 10.42.<peer>.0/24 → shc0 automatically via AllowedIPs. The daemon inserts forwarding accepts (the DOCKER-USER chain on Docker; netavark’s chains on Podman) and a masquerade exemption for the fabric CIDR so cross-node traffic keeps real source IPs.

Packet path: container on node 1 (10.42.1.34) → local bridge → host routing table → shc0 → encrypted UDP to node 2 → its shc0 → its bridge → container (10.42.2.7). Kernel end to end, one encapsulation.

The hierarchical scope networks and alias formats from core/networking.md are preserved:

• _d / _e / _t networks each remain a local bridge per node, with per-node subnets allocated from the node’s /24.
• Same-node members of a scope network resolve each other exactly as today, via the runtime’s local DNS (embedded DNS / aardvark-dns) — unchanged.
• Cross-node members are wired by the renderer: the referenced service gets a pinned ipv4_address, and consumers get extra_hosts entries carrying the same alias strings used today ({service}, {service}.{stack}, {environment}.{service}.{stack}). App configs are identical under either fabric; only the resolution mechanism differs.
• The tenant _t network keeps its ingress role: Traefik joins the local _t bridge on its own node and reaches remote backends via their fabric IPs. This lands in the existing per-host posture machine (modules/app/inject_traefik.go), which already performs node-aware dialing for the Keycloak backchannel.

Scope semantics (“who may reach whom”) therefore stop being enforced by network membership and become policy — see Isolation below.

“Compose file” means two different things in SHC, and the fabric affects only one of them.

App-author surface — unchanged. apps/<name>/compose.yaml templates, app.yaml, vars.yaml, and connection declarations are fabric-agnostic. Authors declare services and named networks exactly as today; the renderer has always owned network injection, so nothing an author writes encodes the fabric. An app packaged once deploys on either fabric.

Rendered output — differs per fabric. What SHC writes to the deploy directory is where the fabric choice materializes:

The difference is observable only by inspecting the rendered file or /etc/hosts inside a container — application configuration sees the same hostnames either way.

Runtime behavior parity. The shc fabric only ever runs compose runtimes, and compose already ignores swarm-only deploy: keys (placement, swarm replica semantics) today — so a stack that deploys as a compose stack on a swarm-fabric cluster behaves identically on an shc-fabric cluster. The swarm runtime (docker stack deploy) is the one surface with no shc-fabric equivalent, which the fabric matrix forbids up front.

The genuinely new UX seam, stated once more so it isn’t a surprise: alias→IP bindings are fixed at render time, so moving a stack triggers a quick automated re-render of its connected dependents — “move” touches more stacks than it does under swarm DNS.

Additions to the cluster store (leader-allocated, CAS-guarded, matching existing allocator patterns; nothing here is dqlite-specific — the design carries unchanged across a store migration):

Private keys are generated on the node and never leave it; only public keys are stored.

Join. shc node join handles the fabric end to end — no per-node manual steps, no key ceremony. Fabric enrollment extends the existing join pipeline (flows/cluster-join.md: token bootstrap → mTLS enrollment → membership) rather than adding a new flow:

1. The joining node generates its WireGuard keypair locally (the private key never leaves the node) and includes the public key in its enrollment request.
2. The leader allocates the node’s /24 from the fabric CIDR and writes pubkey + subnet + fabric IP to the node row.
3. Every existing node’s daemon — via the same reconcile loops that already react to membership changes — observes the new row and adds one shc0 peer. The joining node’s daemon syncs the full peer list, creates its bridges, and installs its firewall rules.
4. Full mesh established.

Teardown is symmetric: node remove deletes one row and every daemon’s next reconcile drops the peer. Because convergence is reconcile-based rather than event-based, a node offline during another node’s join converges when it returns — the same self-healing shape as the rest of the daemon. (Overall this mirrors Nebula enrollment in modules/nebula/, minus CA, certificates, and lighthouse — which is most of that module’s ~4,700 lines.)

Install / render. The renderer emits: bridge networks with pinned subnet + MTU; ipv4_address pins for referenced services; extra_hosts for consumers; Traefik backends pointed at fabric IPs.

Move. The stack lands on the new node; its pinned IPs are reallocated from the new node’s /24; stacks connected to it get a light re-render (their extra_hosts changed). SHC already re-renders the moved stack; the ripple to connected stacks is new cost, and it is this design’s main UX regression versus Swarm DNS (accepted for v1 — see Open questions for the stable-VIP escape hatch).

Node loss / removal. Peer removed everywhere on the next reconcile. The node’s /24 is tombstoned, never reused until explicitly reclaimed — a stale AllowedIPs entry pointing at a reused subnet is this design’s one stale-state trap, and tombstoning eliminates it.

Daemon boot. Idempotent reconcile: ensure interface → sync peers → ensure bridges → ensure firewall rules → start stacks. Same pattern as existing infrastructure reconcile (subsystems/impl/infrastructure_reconcile.go).

This is the largest new component and the one place the fabric is harder than Swarm, not easier.

Swarm overlay provided isolation for free: a container could only reach networks its service joined. A routed fabric is all-to-all reachable by default, so isolation must be rebuilt explicitly. SHC has complete knowledge to do so — it renders every scope membership and every connection — and compiles that graph into per-node nftables rules:

• allow within-deployment (_d co-members), within-environment (_e), within-tenant (_t) flows per current scope semantics;
• allow each declared connection: (source stack’s IP set) → (target’s pinned IP : port);
• allow Traefik → backend flows;
• default-drop everything else crossing the fabric.

With multiple tenants on one fabric, default-drop is not polish — it is the tenant boundary, replacing what _t network membership enforces today. The compiler is a real subsystem with its own unit tests (graph → ruleset) and live tests (cross-tenant probes must fail).

Per-runtime residue lives here too: Docker’s DOCKER-USER chain and default MASQUERADE (needs a more-specific no-NAT rule for the fabric CIDR) versus netavark’s firewall handling on Podman. This is an adapter of rules, not a second implementation.

Only inbound traffic rides the fabric bridge for free: a pinned service receives on its fabric address, but its outbound fabric flows used to leave via the container’s default-route network (a scope bridge), where Docker’s MASQUERADE rewrote the source to the node’s shc0 device address (the .1) before the packet crossed the fabric — so a policy rule matching the consumer’s declared source never fired. Real container sources cannot simply be un-NAT’d either: WireGuard’s cryptokey routing accepts only sources inside the sending peer’s AllowedIPs (= its node /24), and scope-bridge subnets are runtime-assigned and collide across nodes.

Three options were weighed, with the tenant boundary as the deciding criterion:

1. Match the masqueraded host address — rejected. The .1 is shared by every tenant on the node; a scheme that cannot distinguish two tenants’ consumers on the same node fails the floor.
2. Consumer-side enforcement at egress on the source node (where the pre-masquerade container source is still visible) combined with target-side default-drop — rejected. It keeps the wire tenant-blind (the target can only trust node-granular sources), so enforcement collapses onto one side: a single flushed egress chain silently opens every tenant on that node. It also needs the runtime-assigned scope-bridge subnets as compiler input — per-node-colliding state the store does not carry, exactly the stale-cached-state class this design avoids.
3. RULED: consumers reach the fabric as themselves. The renderer makes the fabric bridge every pinned service’s default-route network (gw_priority on the fabric attachment, modules/app/fabric_render.go in ce), so a fabric-bound flow leaves via the fabric bridge sourcing from the container’s own pin, and the no-NAT exemption carries that source intact across the wire. The compiled rule shape is unchanged, both chains verify the same truthful source (two-sided enforcement stands), and a runtime that ignores gw_priority fails closed: its flows arrive wearing the device address, which matches no accept — reachability breaks loudly, isolation never leaks silently. Structurally, the compiler refuses any declared source space covering a node device address (X400365FABTEN), so the masqueraded shape can never be legalized back into an accept set.

Requires Docker Engine 28+ / Compose v2.33+ (gw_priority); the shc fabric is opt-in at init and those floors predate it on every supported install. Podman/netavark honoring of gw_priority is a slice-3 verification item.

Scope note: the compiled table polices only shc0-crossing forward flows, so a container’s export to the host-published collector endpoint (172.17.0.1:4318, the -i otel=OTEL path) is untouched — that flow is host-input plus DNAT-forward, and slice-1’s DOCKER-USER fabric-source accept carries the forwarded leg. No explicit allow is needed on fabric estates.

The isolation this compiler enforces is still a CROSS-NODE property. What changed on 2026-08-01 is that it now holds against a LYING same-node neighbour as well as an honest one. It is still NOT a boundary against an ordinary connect() between two containers that share a node, so do not describe it as a same-node tenant boundary on any user-facing surface.

The gap is tracked in modules/fabric/policy/doc.go (the authority) and gates the multi-tenant isolation claim, not this wiring. It was broader than raw-packet forgery, in three stacked ways; one is now closed:

1. Plain reachability, no raw packets — STILL OPEN. Every pinned container on a node sits on that node’s one shared fabric bridge (a single L2 segment, container-to-container traffic on by default), and bridge-local traffic never traverses shc0 where the policing hooks — so an ordinary connect() from another tenant’s container on the same node to a provider pin succeeds today. Closing this needs per-tenant bridges or bridge-family policing of the whole scope graph; source identity alone cannot do it.
2. Source forgery of fabric-crossing flows — CLOSED (ce modules/fabric/antispoof.go, 2026-08-01). See below.
3. Bridge crossing — STILL OPEN. The owned SHC-FABRIC fabric-source accept precedes Docker’s inter-bridge isolation, so a truthfully-sourced fabric flow can still be forwarded into another tenant’s scope bridge on the same node.

The 2-node live lane on 2026-08-01 turned (2) from a predicted weakness into a measured one, and it was worse than this document used to say (lane log /var/tmp/shc-lane-state/w4-fabric.md, artifacts under /var/tmp/shc-lane-state/w4fab/):

• No privilege was required. CAP_NET_RAW is in Docker’s default capability set. A stack SHC itself rendered, with CapAdd=[] and no privileged, reported CapEff: 00000000a80425fb — bit 13 set (w4fab/probe-gh-setup.txt).
• The forge completed. A forged SYN from tenant evil’s container on node 2, wearing the pin of default/default/cli4 — a different tenant’s container co-located on node 2 — reached default/default/srv on node 1, and the victim answered SYN/ACK (w4fab/probe-f-diagnose.txt).
• Two-sided enforcement was no defence — both chains matched the same forged source.
• The per-bridge source guard did not cover it: it is ! -s <node-fabric-/25> -i br-<fabric> -j DROP, cross-bridge only.

Nothing at the ip layer can separate those packets: a truthful packet from pin P and a forged one claiming P are byte-identical on the wire. The only thing that differs is the bridge port they entered the host on, so that is where the binding lives — an owned bridge-family table, shc_fabric_guard:

table bridge shc_fabric_guard {
    set fabric_pin      { type ipv4_addr }
    set fabric_portbind { type ipv4_addr . ifname }
    chain guard {
        type filter hook prerouting priority -300; policy accept;
        ip  saddr @fabric_pin ip  saddr . iifname != @fabric_portbind counter drop
        arp saddr ip @fabric_pin arp saddr ip . iifname != @fabric_portbind counter drop
    }
    chain applied { counter comment "applied sha256:…" }
}

Design points that are load-bearing, each with the reason recorded in the ce file doc:

• Bridge family, because same-bridge traffic never reaches the ip forward hook. It also means the guard sees EVERY local bridge, which closes a vector the per-bridge guard never could: a container on a scope bridge forging a fabric pin.
• The binding is read from inside the container’s own netns — the address and its peer ifindex together — because that is the one source the adversary cannot write. The fdb and the neighbour table were rejected: they are learned from the wire, so a hostile neighbour could poison them into binding the victim’s address to the ATTACKER’s port, turning a spoofing gap into a remote DoS of the victim. The engine is asked exactly one thing: which container PIDs exist.
• Read the rule as “a BOUND address must arrive on its port”. An unknown port cannot forge a bound pin, so an attacker gains nothing by restarting; an unbound address is unpoliced, so a just-created container is never blackholed.
• The ARP half is not decoration. Without it a neighbour answers ARP for the victim’s pin, the host’s neighbour table moves, and cross-node traffic for the victim lands on the attacker — the same boundary crossing by interception instead of emission.
• Staleness runs the other way: a RECREATED container gets a new veth and its own truthful traffic is dropped until the binding follows. That is why the guard has its own 2-second loop (AntiSpoofTickInterval) behind a host-link fingerprint, instead of riding the fabric sweep’s minute. The tail case is the one way this can take a workload down: the table is kernel state and outlives the daemon (deliberately — shc0 is left up on stop for the same reason), so a container recreated while shcd is DOWN stays blackholed on the fabric for the whole outage, healing on the first tick after the daemon returns. Tearing the guard down on shutdown was rejected: it would open the boundary for exactly as long as an upgrade takes.
• Failure is loud, never silent: X200368FABASP warns every tick when the guard is not in force, and every address the normalizer refuses to bind is named in that warning. That specifically includes the case where the node has no fabric bridge at all — the observer reports it rather than rendering an empty table, because an empty guard loads, verifies its own applied marker and polices nothing, which reads green from the kernel while every pin on the node stays forgeable. “Nothing to guard” and “nothing guardable” must not look alike.
• The gateway that finds the bridge comes from BridgeGateway, not from a formula recomputed at the guard. The bridge is located by its gateway address and by nothing else, so a second opinion about where that address sits is a second chance to go quietly inert.

An address observed behind two ports is left unbound, because binding it to the wrong one blackholes the honest container. That is the right call and it is also a primitive: a container that can add an address inside its own netns — needing CAP_NET_ADMIN, not in docker’s default set (unlike the CAP_NET_RAW this guard is about), but something an SHC app may legitimately request — can put its victim’s pin on its own interface. The observer honestly sees the conflict, declines to bind, and the victim’s pin goes from guarded to unpoliced; a forged source for it then passes. Measured, not theorised: 0/4 dropped with the contention in place, against 4/4 before it.

The mitigation is that it is loud and distinct. Contention raises X200369FABCTN, naming the address and both claiming ports, separately from an ordinary unusable observation — two ports asserting one address is never routine, so treat a repeat as an incident. Binding to the first observation was rejected (a race the attacker wins by starting earlier); binding to both was rejected (it re-opens forgery from a port that now legitimately holds the address). Closing it properly means not letting a container hold an address the allocator never gave it, which is a runtime-admission concern rather than a bridge-family one, and is not part of this change.

Proof: modules/fabric/antispoof_kernel_linux_test.go rebuilds the attack in a throwaway user+net namespace and runs the real observer, renderer and nft applier through it — legitimate traffic and truthful raw packets pass with zero drops, three forged SYN and two forged ARP are dropped and counted. It FAILS rather than skips when it cannot run.

⚠ Two things the hermetic proof does NOT establish, both owed to a live re-run on a real docker node:

1. The hook ordering. -300 is chosen to stay ahead of br_netfilter, which hooks bridge prerouting at NF_BR_PRI_BRNF (0) and re-injects IPv4 frames with NF_STOLEN on a path that does not re-run the bridge chain list. That interaction was measured while writing the guard and did NOT reproduce on kernel 6.12 with br_netfilter loaded and bridge-nf-call-iptables=1: two identical counting chains at -300 and +200 both saw all four forged frames. So the ordering is defence against documented kernel behaviour this kernel did not exhibit — confirm on a node.
2. The guard under a real engine. The observer’s one engine question (ps / inspect {{.State.Pid}}) is verb-compat tested but has not run against a live docker or podman daemon.

The owed live re-run is probe (f) inverted — same command, opposite expectation. On a 2-node fabric estate, from a second tenant’s container co-located with the victim:

# on the node hosting BOTH the attacker and the victim pin
nft list table bridge shc_fabric_guard          # sets populated, marker present
docker exec <attacker-ctr> python3 /tmp/spoof.py <victim-pin> <remote-pin> 80
nft list table bridge shc_fabric_guard | grep 'forged pin source'
#   -> counter packets MUST be > 0, and the tcpdump on shc0 MUST be empty
tcpdump -ni shc0 "host <victim-pin>" -c 5 -w /dev/null   # expect: nothing

spoof.py is /var/tmp/shc-lane-state/w4fab/spoof.py verbatim. A probe that merely TIMES OUT proves nothing about which rule stopped it — the counter is the evidence.

⚠ Nothing in CI will tell you this section has gone stale. The guard is bridge-family and lands in ce package fabric, which package policy does not import. Whoever changes the same-node posture again must update this section, ce modules/fabric/policy/doc.go (the authority) and the CHANGELOG’s fabric posture note by hand, in the same change.

Isolation proofs must therefore still place prober and provider on different nodes; the same-node cross-tenant probe is expected to connect and is recorded as gap (1) — live-confirmed in w4fab/probe-e-samenode.txt. The same-node spoofed probe is now expected to be dropped.

• No Swarm runtime on the shc fabric (forced; see matrix).
• Move ripples re-render connected stacks (Swarm DNS previously absorbed this).
• No cross-node dynamic DNS — resolution is baked at render time. A container started outside SHC’s renderer does not get fabric discovery. (Out of scope by design: everything on the fabric is SHC-rendered.)
• No broadcast/multicast — unchanged in practice from Swarm overlay, which also does not support them.
• MTU 1420 on fabric networks (WireGuard overhead).
• Rootful runtimes only. Rootless Podman/Docker cannot attach to host bridges this way; out of scope.
• No fabric migration in v1 (swarm → shc conversion).

Each slice is independently provable live:

1. Slice 1 — reachability (BUILT). Fabric up + IPAM + renderer target + rendered connections, open intra-fabric policy, 2-node Docker-compose-only proxmox cluster. Proof: cross-node connection works; move re-renders correctly; node loss/rejoin converges.
2. Slice 2 — isolation. Policy compiler + default-drop. Proof: declared flows pass, undeclared and cross-tenant probes fail. Live-run 2026-08-01 on a 2-node hetzner estate — results below.
3. Slice 3 — Podman. Podman runtime adapter + mixed Docker/Podman cluster e2e.

Two cpx31 nodes, shc init --fabric shc + shc node join, two tenants placed on both nodes. Full log /var/tmp/shc-lane-state/w4-fabric.md; artifacts under /var/tmp/shc-lane-state/w4fab/. Recorded as run, not as hoped:

The one functional claim the lane could not finish. What is proven: a cross-node flow between two same-tenant stacks connects, full 3-way handshake on shc0, carried by the compiled scope accept — in both directions between the two nodes. What is not proven: that a declared connection whose two ends live on different nodes connects, because no such connection could be created.

Two product rules collide. X400956NDLOCL — a compose install executes on the local node — forces the consumer install to run on the consumer’s node; and the plug/dependency resolver is node-local, so that install fails X404906STKNOF (“stack srv not found in tenant default environment default”) while shc ls on that same node lists the provider as running. It is not a leader/follower split: the same bind run on the leader for a provider on the follower fails identically. Co-located binds succeed. The shc move workaround also fails — the re-render on the target node drops the fabric attachment and the integration, and the pin table keeps homing the deployment on the source node, which then leaks asymmetrically into the compiled policy.

A follow-up must make the resolver cluster-aware (or make the install path node-agnostic) and only then re-run this probe. Until it does, “declared connections work across nodes” is unproven, and the positive control’s success is not a substitute for it.

Overall scale: a multi-week program comparable to the storage-volumes work. Design risk concentrates in the policy compiler and the move-ripple; the WireGuard/IPAM/renderer parts are contained.

Where slice 1 had to pick something this design left open, it picked these. They are recorded here because a later slice will have to live with them.

Defaults. Fabric CIDR 10.84.0.0/16, UDP port 4243 — both chosen against open question 4. 10.42.0.0/16 (the illustrative value above) collides with k3s pod CIDR on shared hardware, and 51820 is commonly already occupied on an operator’s host; 4243 sits next to Nebula’s 4242 in SHC’s own port family.

One bridge per node, not one per scope network. The design says “plain bridge networks with pinned subnets carved from the node’s /24”. Slice 1 carves ONE — shc_fabric — and spends the node’s address budget on pinned service addresses instead, which is what open question 3’s “~254 pinned addresses per node” already assumed. Same-node discovery is untouched: it still runs over the existing per-provider networks and the runtime’s own DNS, and the fabric address is strictly additive.

The /24 is split, because two things want the .1. shc0 carries the node’s .1 with the SUPERNET prefix (that connected /16 route is what sends sibling /24s into the tunnel), so the bridge cannot also claim it. The bridge takes the upper half — 10.84.N.128/25, gateway .129, pins from .130 — which is disjoint from the device address and more specific than the /16, so the node’s own container traffic never round-trips through WireGuard. No route juggling: both routes fall out of the interface addresses.

Every service is pinned, not just reference targets. Narrowing to “cross-node reference targets” would require knowing the whole cluster’s future connection graph at the provider’s render time. Pinning everything costs one address out of 125 per service and means a later -i against an already-installed stack needs no re-render of it.

Pins are a cluster row (fabric_pin), not a derived hash. The two ends are rendered by different nodes at different times — the provider allocates, and a consumer installed later somewhere else reads it back — so a hash would have to be collision-free forever with no way to observe a collision.

Consumers are not on the fabric bridge; only targets are. A consumer reaches a pinned address through its own bridge’s gateway and the host routing table, so its traffic is masqueraded to its host’s fabric address. Connectivity is unaffected (conntrack un-NATs the reply), but the consumer’s real source IP is not visible at the target — which slice 2’s policy compiler will need. Putting consumers on the fabric too is the obvious fix and is slice 2’s problem to make.

The firewall glue is load-bearing in slice 1, despite the open policy. Docker sets the filter FORWARD policy to DROP and only accepts traffic into a bridge through its own DOCKER chain, which holds one rule per published port — so without an explicit accept, a packet arriving on shc0 for a container is dropped while the device is up and the peer answers ping. Slice 1 inserts an ACCEPT in DOCKER-USER (traversed first, and terminating) plus a POSTROUTING RETURN for fabric→fabric traffic, re-asserted on every reconcile tick because docker network create re-inserts docker’s own rules ahead of ours.

The posture is replicated, not leader-local. The first cut read it from the leader’s own node_bootstrap.json, which a node that JOINED does not have — so a leader failover made fabric enrollment silently skip. The cluster_fabric row is now the source of truth and the file is the pre-raft fallback only.

Slice 1 shipped built but UNREACHABLE: nothing in terraform/ mentioned the fabric, so shc init --fabric shc could not be produced by any lane and the design’s central claim — cross-node reachability without swarm — had no regression test on real hardware. The terraform surface closes that.

The input. bootstrap.fabric ("swarm" | "shc" | "none") and bootstrap.fabric_cidr on modules/core and every substrate front door that forwards bootstrap (modules/shc/hetzner today). Both default to "", which the module folds to null, which the provider renders as NO flag — so a config that never names a fabric plans byte-identically to the pre-fabric module. That property is pinned by terraform/modules/core/tests/fabric_input.tftest.hcl.

The lowering. bootstrap.fabric → shc_cluster.fabric → shc_node.bootstrap.fabric → shc init --fabric <mode>. Only the bootstrap node carries it: the fabric is cluster-scoped and every joiner reads it back out of the cluster_fabric row, so shc node join has no --fabric flag and the provider refuses one on a join node. --fabric-cidr rides --fabric shc alone (the CLI refuses it otherwise), and data.shc_cloudinit carries the same pair for the self-init path.

The firewall. fabric = "shc" adds UDP 4243 — the shc0 listener — to modules/shc/hetzner’s node-to-node plane port set. Peers dial each other at their INTERNAL address, the one raft and the internal API already use, so on a private-network cluster the rule is inert; on the public-plane shape it is the difference between a working fabric and handshakes that vanish with no diagnostic.

The lane. One env var on the standing hetzner tier:

HCLOUD_TOKEN=... SHC_FABRIC=shc SHC_TEST_NODE_COUNT=2 \
  SHC_TEST_CLUSTER_NAME=shc-fabric make test/hetzner
HCLOUD_TOKEN=... SHC_TEST_CLUSTER_NAME=shc-fabric make test/hetzner/destroy

That provisions a real two-node compose-runtime estate on the shc fabric and then asserts, per node, that shc0 is present and up, that the peer set is complete, and that no peer is still on a zero handshake — then pings one node’s fabric address from the other over shc0. The ping is the assertion that matters: device state and peer config can all look right while every packet is being dropped. SHC_FABRIC_CIDR overrides the supernet. See tests/hetzner/README.md.

The runtime must be compose or podman — the shc fabric excludes the swarm runtime, refused at plan time by module core’s fabric_selection check and by shc_node’s ValidateConfig, and earlier still by the lane script.

• Control plane over the fabric on bare metal. Largely already addressed by #58 raft mTLS: since the RaftTLS gate landed, a fresh cluster runs raft :4002 (and the rqlited HTTP API :4001) under mutual TLS by default (modules/cluster/service.go genesis posture). Only legacy pre-#58 clusters — whose node_bootstrap.json predates the field — still ride :4002 in plaintext, protected only by hand-firewalling. For those, and as defense-in-depth for everyone else, once every node has shc0 the cluster store’s raft transport and the internal API can bind fabric IPs and inherit encryption.
• Retire the Nebula module. The fabric subsumes the Nebula mesh’s role (encrypted node-to-node transport) with a strictly smaller machine — no CA, no lighthouse, no cert rotation, kernel data path. One mesh in the product, not two.
• WAN membership over the fabric. The bind/advertise-split proposal documented why edge-NAT clouds cannot join today (one fused bind≡advertise address rides every cluster surface) and recommended a Nebula-first WAN join; this design supersedes that recommendation — with Nebula retired, the WAN member story is the fabric over WAN. The proposal’s surviving primitive, a per-node advertise address distinct from the bind, is the input this design consumes for NAT’d WAN peers: shc0 peer endpoints must be dialable, so “where peers dial me” needs decoupling from “where I listen” (natively expressible on rqlited via -raft-adv-addr, which also serves the control-plane-over-fabric follow-on above).

1. Stable service VIPs. A per-service stable fabric IP tier (node-independent, rerouted on move) would eliminate the move ripple at the cost of per-service routing state. Deferred until the ripple demonstrably hurts.
2. Fabric migration. swarm → shc conversion via rolling re-render of every stack is conceivable; nothing in v1 should bend to enable it.
3. Subnet sizing. /16 fabric with /24 per node caps at ~254 nodes and ~254 pinned addresses per node — comfortably above current targets, but the CIDR is configurable at init and should be sized there, since renumbering later is a cluster-wide re-render.
4. Fabric port and CIDR defaults. Pick defaults that avoid the common collisions (WireGuard 51820 may already be in use on operator hosts; 10.42.0.0/16 collides with k3s defaults on shared hardware).

• netavark discussion #674 — maintainers: “Podman/Netavark is single node only”
• netavark PR #472 — the WireGuard driver attempt, redirected to the plugin API
• b-m-f/netavark-wireguard-plugin — the surviving (stale) artifact of that attempt
• Podman basic networking tutorial — no multi-host networks statement
• flannel host-gw / wireguard-native, k3s wireguard-native, Calico/Cilium native routing — prior art for routed container fabrics