Daemon bind/advertise split (2026-07 mixed-cluster finding)

RECONCILED against the network-fabric design (2026-07-17). This doc carries two proposals, and they now have different fates:

• Nebula-first join — SUPERSEDED by network-fabric.md (2026-07-16). The fabric’s strategic follow-ons retire the nebula module outright (one mesh in the product: routed kernel WireGuard — no CA, no lighthouse, no cert rotation), so the recommendation below — options (b)/(c): enroll-carries-mesh-bundle, the mesh_join token split, nebula-before-shcd ordering — is a dead end and must not be built. The WAN member story is now “fabric over WAN”: a NAT’d worker gets cluster reachability from the fabric, and the fabric design absorbs the WAN endpoint question.
• The bind/advertise config split — SURVIVES independently, as the primitive the fabric can consume: fabric peer endpoints derive from internal_address, so a WAN node behind edge NAT still needs “where peers dial me” decoupled from “where I listen”. The problem analysis below (one fused address rides every cluster surface; seven edge-NAT clouds excluded) remains the reference for that work.

Engine note: written against dqlite; storage has since moved to rqlite. Option (a)‘s “the engine cannot express an advertise address” costing is obsolete — rqlited natively takes -raft-adv-addr/-http-adv-addr, and internal/core/rqlited/manager.go already carries the (unpopulated) RaftAdvAddr/HTTPAdvAddr knobs. The bind≡advertise fusion now lives only in SHC’s single chooser (resolveClusterBindAddr, internal/subsystems/impl/rqlite_engine.go).

Posture update (2026-07-17): the plaintext-:4002 framing below is SUPERSEDED for new clusters. #58 landed gated raft mutual TLS: a from-empty genesis always writes RaftTLS: true (internal/modules/cluster/service.go), joiners inherit the leader’s posture over enroll, and rqlited then runs both the raft plane (:4002) and its HTTP API (:4001) under mTLS (internal/core/rqlited/manager.go). Only legacy clusters whose node_bootstrap.json predates the field stay plaintext — this doc’s “new security finding” analysis remains accurate for those, and is the reference the RaftTLS field doc points at. The historical analysis below is kept verbatim.

The live mixed-provider proof (exoscale lighthouse + hetzner worker, joined over the public internet) succeeded ONLY because both clouds put the public IPv4 directly on the guest NIC. The originally-requested gcp/azure/openstack workers cannot participate at all: SHC binds AND advertises exactly one interface-local IPv4, and on edge-NAT clouds the public IP never exists on the NIC. This proposal decides how the platform separates “where I listen” from “where peers dial me” so WAN/mixed clusters stop being gated on the four direct-public-IP clouds.

ONE address, chosen once, rides every cluster-internal surface (core/nic is deliberately the single chooser — internal/core/nic/nic.go package comment):

• dqlite raft bind+advertise: resolveDqliteBindAddr (internal/subsystems/impl/dqlite_manager.go:85) picks the —nic pin’s live resolve, else the recorded bind_ip, else the nic heuristic post-filtered by a LOCAL bind probe (bindable(), :107). The result feeds DqliteOptions.BindAddr → raftAddr → app.WithAddress(n.raftAddr) (internal/modules/cluster/init.go:241).
• enroll physical_ip: the worker advertises advertiseAddress(nicName) (internal/modules/node/commands/local_join.go:124; resolution chain in local_token.go:195 — —nic hard-wins, then $SHC_ADVERTISE_ADDR, then node_bootstrap bind_ip, then the shared heuristic).
• node-row internal_address: the leader stamps physicalIP:4003 verbatim (internal/modules/cluster/enroll_handler.go:240) and SANs the same IP into the joiner’s internal-API leaf (:201-207).
• every leader→worker dial-back derives from that row: raft membership projection host:4002 (internal/subsystems/impl/dqlite_membership.go:100), the GAP-3 promotability probe physicalIP:4002 (enroll_handler.go:280), mesh-install POSTs to internal_address :4003 (internal/modules/cluster/mesh_orchestrator_impl.go:164 + Enable’s Peer.Post fan-out), health poller DialAddress (internal/modules/health/node_poller.go:42), voter-reachability probe (dqlite_manager.go:46), backup export peers (internal/modules/backup/export_peers.go:32), and the daemon token mint’s join_addrs (internal/modules/cluster/token_router.go:342).
• swarm advertise + node_bootstrap bind_ip are pinned to the SAME physicalIP (local_join.go:249, :269-271), so the next daemon boot binds raft to it too.

On gcp/azure/aws/digitalocean/openstack/linode/cloudstack the public IP is 1:1-NAT’d at the edge — bindable() can never pass on it, so such nodes can only advertise their VPC-private address, and every dial-back above fails cross-cloud. The networking.private must be true validations in exactly those seven modules (terraform/modules/shc/*/variables.tf) are honest fail-louds over this, not policy. Only exoscale/hetzner/scaleway/vultr (public IPv4 on the NIC) can host public-plane WAN nodes today — the shape the mixed proof was forced into.

Two corrections to the raw finding, confirmed by reading the vendored go-dqlite v3.0.4:

• “app.WithAddress gossips the BIND address” is precisely inverted: WithAddress is the ADVERTISED identity (info.yaml, raft membership), and the plain-TCP path then BINDS that same string (app/app.go:188 nodeBindAddress = info.Address). Same fusion, opposite arrow. There is NO app-level WithBindAddress; dqlite.WithBindAddress exists only on the low-level Node API (node.go:55) the app layer owns. The only app-level decoupling hooks are WithTLS (binds a TCP proxy — on info.Address again, app/app.go:277) and WithExternalConn (app/options.go:53): the caller owns listen+dial, and the accept side speaks an HTTP-101-upgrade preamble that is NOT wire-compatible with plain-TCP peers.
• $SHC_ADVERTISE_ADDR is not enroll-only: a second, independent advertiseAddress in the app module reads it for DNS targets (internal/modules/app/dns_lifecycle.go:264). Neither reaches the raft bind — that path has its own SHC_DQLITE_BIND_ADDR, which moves bind AND advertise together.

One NEW finding the investigation surfaced: SHC passes no app.WithTLS, so raft :4002 is the PLAINTEXT, unauthenticated dqlite wire protocol. The mixed proof opened it 0.0.0.0/0 under a comment claiming “all mTLS/TLS-authenticated” — wrong for 4002. Any public-plane WAN cluster today exposes its replicated state DB to the internet. This weighs heavily in the option choice below.

A per-node advertise address (flag/env/enroll field) distinct from the bind, carried into dqlite, enroll, node row, swarm, mesh-install.

What it actually costs: the vendored app API cannot express it on the plain-TCP path (bind ≡ WithAddress, above). The choices are (1) adopt app.WithExternalConn and own the listener+dialer — a raft TRANSPORT replacement whose upgrade preamble breaks mixed old/new clusters, i.e. a flag-day epoch bump; (2) fork/patch go-dqlite to add an app-level bind split — a carried patch on the storage engine; both large. And the prize is small: every worker STILL needs a 1:1-NAT public IP (cost: per-address fees on gcp/aws/azure; several clouds now bill idle+attached both), still needs :4002/:4003 inbound from the internet, and the raft plane stays plaintext. The provider would also need per-node advertise plumbing (cloud-init flags, node rows, in-place repair) mirroring what —public-address already does for ingress.

Invert the ordering: a WAN worker meshes FIRST, then joins the cluster THROUGH the mesh. Its nebula IP (10.42.0.0/16, allocator internal/modules/nebula/service.go:212) becomes the one address on every surface: enroll physical_ip = mesh IP → internal_address = meshIP:4003 → raft bind+advertise = mesh IP (node_bootstrap bind_ip; nebula1 is a real local interface, so bindable() passes and dqlite’s fused bind≡advertise stops being a constraint — we change WHICH address, not the arity).

This requires inverting mesh bring-up for the joiner: today mesh install is a leader→worker POST to :4003 (mesh_orchestrator_impl.go Enable), which itself needs the dial-back that NAT breaks. Under (b) the ENROLL response carries the mesh bundle — the leader (sole holder of the nebula CA key) allocates the mesh IP, signs the host cert, and returns cert+key+CA+ lighthouse entries; EnrollResponse.MeshIP already exists on the wire (internal/modules/cluster/types.go:145, never populated today) and the joiner already declares it (local_join.go enrollResponseBody). Join flow becomes: enroll over the leader’s PUBLIC :4003 (worker→leader always works) → write nebula config, start nebula, verify lighthouse handshake → persist cluster_join.json with mesh join addrs + bind_ip=meshIP → daemon starts, raft binds the mesh. NAT traversal is nebula’s job: the worker punches OUT to the lighthouse; the worker needs ZERO inbound rules and — decisively — NO public IP at all (egress-only/NAT instances).

Wins: kills per-worker public IPs (cost + attack surface); encrypts the raft plane (closes the plaintext-:4002 exposure — nebula is Noise-based); the mixed proof observed mesh IPs stable across worker replacement (allocation is persisted per node row and the enroll adopt-row path reuses the id); the seven module validations stop being load-bearing. Costs: nebula becomes a hard dependency of WAN membership (systemd ordering: nebula before shcd on mesh-joined nodes); the lighthouse must be public and healthy for NEW handshakes (existing tunnels survive; the lighthouse store already supports pinning, mesh_orchestrator_impl.go:196); swarm’s data plane rides vxlan-over-nebula (MTU headroom must be configured); throughput on the DB/API plane takes the overlay tax.

(b)‘s mechanism, selected per cluster: topology.kind (a terraform-layer concept — the core module dispatches lan/wan/lb and only “wan” creates shc_mesh, terraform/modules/core/main.tf:35) decides whether the token mints a mesh-first join. LAN/homelab clusters keep today’s physical path byte-for-byte — no nebula dependency for the laptop install. The genesis leader keeps its physical public address (it is the lighthouse and must be public anyway); raft membership happily mixes per-entry addresses.

(b) is the right mechanism and (c) is its right scope; pure (b) would tax LAN installs with a mesh they don’t need, and (a) buys the smallest win for the largest surgery (transport flag-day or engine fork, per-worker public IPs retained, raft still plaintext). The pitch: a WAN worker joins through the mesh it was going to get anyway — the platform stops needing the public IP to be on the NIC because it stops needing a public IP at all, the plaintext raft plane becomes encrypted as a side effect, no go-dqlite API is bent, and the single-address invariant (one chooser, one address on every surface) survives intact. LAN clusters see zero change.

• node row: NO new columns. internal_address’s SEMANTIC is sharpened to “the cluster-plane dialable address” (meshIP:4003 on mesh-joined nodes); mesh_ip keeps recording the overlay address; public_address stays ingress-only (dns_target reads it; internet DNS already fail-closes on internal_address, internal/modules/app/dns_target.go:317).
• enroll request: a mesh_join bool (omitempty — old leaders reject unknown keys, so a new CLI against an old leader falls back to the physical path loudly). physical_ip stays REQUIRED and carries the underlay IP for lighthouse static-host-map purposes.
• enroll response: populate the existing mesh_ip field; add mesh cert/key/ CA + lighthouse entries (mirrors today’s /internal/mesh/install payload, mesh_orchestrator_impl.go:368). DqliteJoinAddrs become mesh addresses.
• token payload: join_addrs today double as ENROLL endpoints (the joiner rewrites host:4002 → host:4003, local_join.go:133) AND raft fallback; under (c) the mint (token_router.go:342 derives from node rows) must emit the leader’s PUBLIC/physical endpoints for the enroll hop — mesh addresses are undialable pre-mesh. Split the fields: enroll_addrs (physical, public) vs join_addrs (raft; response wins anyway, local_join.go:216).
• internal-API leaf SANs: sign the mesh IP (leader-side, enroll_handler.go :201 — it’s allocated in the same call) alongside the physical IP.
• provider: —public-address is untouched (ingress). The core module’s topology.kind=wan threads a mesh-join bit into the token/cloud-init; the seven networking.private must be true validations RELAX from “wan unsupported” to “wan rides the mesh; private plane required only for the underlay” — the lighthouse-hosting module still requires a NIC-local public IP (today’s four clouds, or any cloud’s NAT-free LB in front of UDP 4242 later).
• health/rollup, voter probes, exports, mesh RPCs: all dial internal_address — they follow the row with no code change. GAP-3’s probe becomes meaningful again cross-cloud (it probes the mesh address, which the leader can actually reach).

go-dqlite has NO online member-address update: the client protocol offers Add/Assign/Remove/Transfer/Describe/Weight only; app.New HARD-FAILS when WithAddress disagrees with info.yaml (app/app.go:120 “address … does not match”); the only address rewrite is dqlite.ReconfigureMembershipExt — the OFFLINE whole-config surgery force-leader recovery already uses (internal/modules/cluster/init.go RecoverCluster). The #113 force-rejoin preserves the node’s raft id but re-enters the SURVIVING entry — i.e. the OLD address; it cannot flip a live member onto the mesh. So:

• existing nodes flip only via full remove + re-enroll (shc node rm → shc node join --force with a mesh token), one worker at a time, leader last-or-never (the leader keeps its physical address as lighthouse);
• or a maintenance-window ReconfigureMembershipExt across ALL nodes at once — supported but all-stop, reserved for disaster shapes;
• per the no-backcompat doctrine this is acceptable: mesh-first applies to new joins, the only live WAN cluster is the throwaway proof, and the doc states plainly that there is no in-place flip.

Hermetic (CI, root-gated like other privileged suites): a netns fake-NAT rig — three namespaces (leader | router doing 1:1 DNAT/MASQUERADE | worker whose “public” IP lives on the router, not its NIC), real shcd binaries. Assert the TODAY failure first (join enrolls, GAP-3 X503090CLSJNP fires on the leader, worker wedges at Spare), then the mesh-first join converging to Voter through the same NAT. Unit tier: enroll-handler mesh branch (allocation + bundle + SAN), token enroll_addrs/join_addrs split round-trip, LocalJoinLanCluster ordering (nebula verified up BEFORE cluster_join.json lands), resolveDqliteBindAddr resolving the nebula interface via the —nic pin.

Live acceptance: the mixed-cluster scratch-root pattern (~/selfhosted-cloud/mixed-cluster-test) rebuilt as gcp worker + exoscale lighthouse — the EXACT pair that cannot work today — with the gcp instance deliberately public-IP-free (networking.private=true, egress via Cloud NAT, zero inbound rules). Green = 2-node cluster healthy, worker promoted Voter, an app deployed across both, worker destroyed+replaced re-enrolls with a stable mesh IP. Keep the exoscale+hetzner public-plane proof as the regression twin so the physical path stays honest.

• No go-dqlite transport rewrite or fork (WithExternalConn/WithTLS paths stay unused); the plaintext-:4002 exposure is closed by the mesh, not by adding raft TLS.
• No in-place address migration tooling for existing members (see limits).
• No multi-lighthouse HA / lighthouse failover redesign — the pinning store exists; follow-up.
• No swarm-over-mesh performance work beyond setting MTU headroom.
• No IPv6 cluster plane.
• No change to ingress/DNS semantics: public_address and the dns_target ladder are untouched.
• No mesh requirement for LAN topologies or single-host installs — the physical path remains the default and the mesh path is opt-in per cluster, not per node.