One virtual IP per service (design)
Status: open design (SHC-1238). Allocation, the VIP table and the per-node applier are built, and so is the sidecar case (SHC-1298): every rule of section 1’s SHC-1298 paragraph has code and tests in ce (
fabric_pin.members,SettleNamespaces, member backends inmodules/vip/section.go,MemberReady, the yield rule on both the nft and the pf compilers,X400240PODNET,X400241PODPUB). Its live proof on the rig is still owed. Clay, 2026-10-02: an SHC service (one compose service and its replicas) always has exactly one virtual IP, like a Kubernetes Service. “Routing wise they all share a single virtual IP … I feel like that’s just how that should work.” The authoredstatefulflag that would have taken the VIP away is rejected. Groups, members and aliases on top of replicas are not ruled and are out of scope here, as are external load balancing (--port,--host) and the mesh.
1. Shape
Section titled “1. Shape”A service web of deployment acme/prod/shop has one address, say
10.224.0.17. A container anywhere in the cluster that connects to
10.224.0.17:8080 lands on one ready replica of web, on port 8080 of that
replica’s fabric pin. The address never changes while the service exists: not
on update, not on restart, not when a replica moves.
The VIP exists only as a match in nftables. No interface carries it, no route
points at it, and it never appears on the wire: the node the client runs on
rewrites the destination to a replica’s pin in nat prerouting, and the packet
crosses the fabric addressed to that pin. So the VIP needs no WireGuard
AllowedIPs, no ARP and no route, and it works the same on one node or fifty.
Services of one network namespace (SHC-1298). A service that joins
another’s network namespace (network_mode: service:web, a sidecar in the same
pod) has no address of its own: it listens on its owner’s. It still has its own
VIP. The owner’s pin row records it as a member with the container ports it
declares (fabric_pin.members; a second row cannot carry the same address,
because (node_id, ordinal) is unique), the VIP table gives the member’s VIP
the owner’s pin as its backend, judged by the member’s own container, and the
tenant policy opens the member’s declared ports on that address. The two names
split the address by port: each VIP yields, and so refuses, the ports another
service of its namespace declares and it does not, counted as port ranges (a
port one names inside the other’s range is carried by both) and over every
service of the namespace, with a VIP yet or not. A flow rewritten before a
port changed hands ends on the next flow pass. A ready owner with a dead member
takes no traffic on the member’s VIP, and a route to the member waits for the
member’s own newest container, never the owner’s. A member that declares no
port gets what any portless service gets. A member is never published or
routed under its owner’s name: a --host route to the owner on a port only a
member declares finds no candidate (X400936HSTNOP), a --port publish naming
the owner for such a port, or naming a member at all, is refused
(X400241PODPUB), and a member whose owner has no fabric address is refused,
ports or none (X400240PODNET), since its VIP could have no backend. A port the
owner declares itself stays the owner’s to publish and route. Each render
settles the deployment’s pins against its namespaces, under a per-deployment
lock on its node: a service that now joins another’s loses the pin it held as
an owner, and a pin whose service left the document stops naming the
document’s services as members (other names it holds, such as a replica
entry’s, stay). Why a VIP of its own and not the owner’s: Clay ruled that a
service always has exactly one VIP, and that a replica is a pod whose members
move as one. Answering the member’s name with the owner’s VIP would leave a
service with no VIP and make its name an alias of another service’s, and
aliases are not ruled. A VIP over the owner’s pins keeps both rulings and asks
nothing new of either, so the alias form goes to Clay before anyone builds it.
2. Addresses and allocation
Section titled “2. Addresses and allocation”Range. 10.224.0.0/11, the upper half of the control-plane /10 ruled on
2026-10-01. shc init refuses a fabric CIDR that overlaps it
(X400351FABCID). A cluster recorded with one before that keeps its fabric,
and its posture is still adopted, but it runs no VIPs: nothing is allocated,
the section’s table names no fabric, LiveVIPs answers nothing, the policy’s
error accept stays pin-only (section 6), and the leader warns once
(X409239VIPOVL) until the fabric moves to a disjoint supernet. Such a
cluster, and one off the shc fabric, runs no replica projects either
(SHC-1379): a replica’s peers and Traefik dial its service’s VIP, and nothing
falls back to its pins. The daemon declines them (X200769PODOFF), naming the
VIP module’s own refusal as the reason shc doctor shows: X409249VIPNSF off
the shc fabric, X409250VIPOVR on an overlap, X409243VIPUNR for a posture
that does not read, and X409251VIPNAP on a node whose platform runs no VIP
applier (every darwin node until SHC-1381). A daemon that did decide to run
them asks again before every pass that can start a replica (Runner.VIPs,
over the posture as the switch reads it), and a refusal starts nothing; a
stop, a removal and a prune do not ask.
One predicate,
compile.OverlapsDefaultRange, decides all of it, over one reading of the
CIDR: fabric.ParseCIDR. Go’s two parsers disagree on some spellings
(10.224.0.0/012 parses with net only), so Posture.Normalize rewrites the
CIDR to its canonical masked form for every consumer, a CIDR nothing can read
counts as overlapping, and a node refuses a table whose fabric overlaps the
range before it loads a rule. The lower half,
10.192.0.0/11, stays with mesh addresses. That is
2,097,152 addresses; an address whose last octet is 0 or 255 is never handed
out, because some client stacks refuse them.
Allocated on the leader, stored in rqlite. New table service_vip, in the
single squashed migration:
| column | meaning |
|---|---|
id | uuid |
deployment_key | tenant/environment/stack, the pin ledger’s key |
service_name | the compose service, never a replica entry (web, not web-r0) |
service_id | the service row it was handed to; AUTOINCREMENT, never reused |
deployment_id | that row’s deployment row; AUTOINCREMENT, never reused |
slot | offset into the range; the address is range + slot |
seq | allocation order, one above the highest any row holds |
address | the rendered address, for readers that do not do the math |
created_at, updated_at | |
released_at | when the service went away, a UTC datetime like every store time; NULL while it lives |
UNIQUE (slot) and UNIQUE (seq) over every row, live or released. UNIQUE
(deployment_key, service_name) over live rows (released_at IS NULL). Same
doctrine as fabric_pin: no interactive transaction, every write
self-contained, a constraint or 0-rows outcome rereads and retries. Each write
puts clusterfs.BumpStmt in the same batch, so the VIP and the revision that
publishes it replicate together.
Who allocates. A leader-gated sweep, every 5 s:
- every
servicerow joined to its livedeploymentrow that has no live VIP gets one; - every live VIP whose
(deployment, service, service_id, deployment_id)no longer joins is released:released_atis stamped and the row stays. A service removed and added again under the same name has a new service id, a reinstall has a new deployment id, and a service row thatRescueOrphanedServiceRowrelinks to a reinstalled deployment has a new pair, so each gets a new VIP even when no sweep ran in between.
The sweep reads the live VIPs before the services: a service missing from the
later read is gone for good, because its ids never come back. A release names
the allocation by its seq, which no other allocation ever carries, and
proves in the same WHERE that the lifetime it was handed to is gone (no
service row with that id, name, deployment row and key). An allocation
re-asserts the same identity for itself. So no writer, however stale its
read, takes a VIP from a service that still exists. No order between
lifetimes is assumed: a relinked row keeps an older service id under a newer
deployment. One service that
cannot get a VIP holds none of the others back.
The install and update path may call the same idempotent Ensure so a fresh
service has its VIP before its first replica starts; the sweep is what makes it
true regardless. A failed read releases nothing. A service row is upserted by
(deployment_id, name), so an update never makes a service disappear for a
moment and never costs it its VIP.
Order and reuse. No address is reused before the range is exhausted. The
allocator is a cursor that only moves forward through slots no row has ever
held: the newest row (highest seq) names the last slot handed out, and the
next allocation takes the first never-held usable slot after it. Released rows
stay as tombstones and are skipped, so for the first two million allocations
every VIP is a fresh address.
Only when no never-held slot is left does the allocator reuse, and then it
takes the tombstone released longest ago, provided that release is older than
VIPQuarantine (24 hours); it deletes the tombstone and writes the new row in
one batch. With no such tombstone the allocation is refused (range full). An
address therefore comes back only after every other address in the range has
been handed out once, oldest release first.
That makes the contract explicit. A VIP belongs to its service for as long as the service exists. A client that keeps an address after its service is uninstalled reaches nothing until the range is exhausted, and after that may reach whichever service holds it next; resolving the name again is how a client stays right, which the 5 s TTL is for. Every node that is converging current revisions ends tracked flows to a VIP no longer live in its table (section 6). A node partitioned from the store keeps routing from its last revision until its clusterfs catches up, which it shares with pins and routes; the oldest release first rule puts the most time possible between that node’s last view and any reuse.
Keyed on (deployment, service) and the service row, and on nothing about
placement, the VIP survives an update, a restart and a relocation unchanged
(AC1): an update upserts the service row by (deployment_id, name) and keeps
its id. An uninstall releases it; a reinstall under the same name gets a new
one.
VIPs are allocated only on a cluster whose fabric posture is shc. Without the
fabric there are no pins to route to, the section’s table names no fabric, and
every node’s applier leaves its host alone (section 7).
3. The clusterfs file
Section titled “3. The clusterfs file”A new clusterfs section, vip, rendered on every node like resolver and
traefik. It writes one file, vip/services.json:
{ "range": "10.224.0.0/11", "fabric": "10.128.0.0/10", "nodes": { "n1": { "location": "fsn" }, "n2": { "location": "fsn" }, "n3": { "location": "hel" } }, "services": [ { "deployment": "acme/prod/shop", "service": "web", "vip": "10.224.0.17", "backends": [ { "replica": "web-r0", "pin": "10.128.16.130", "node": "n1", "running": true, "ready": true }, { "replica": "web-r1", "pin": "10.128.32.130", "node": "n2", "running": true, "ready": false } ] } ]}-
backendsare the service’s live pins: the pin filed underwebor under a replica entryweb-r<N>, with the node it sits on. -
readyneeds positive evidence about that backend’s own replica, from a source that records it under the same key the pin is filed under:- a pin filed under the service name (
web) is the one container of that service; pins are unique per service, so no other container can carry the address. Its evidence is the newest container row the node recorded for that deployment, service and node. - a pin filed under a replica entry (
web-r0) is that replica’s, and container rows cannot name it: the recorder files a replica’s container under its base service and keeps no ordinal. The assignment store cannot yet either: its state is the pod replica’s, and a pod’s readiness is taken from the members that declare ports, which need not includeweb. So a replica-entry backend needs member-specific evidence, reported per member of each pod replica at the current generation, and until nodes report that (SHC-1144, over SHC-1216’s replica records) it is never ready. A replica-entry VIP carries no traffic before its evidence exists. Whoever wires that report bumps the clusterfs revision in the same batch as an accepted change of a member’s running or ready state, as the container recorder does, so the VIP republishes on the change and not at the next refresh.
Ready means running, and healthy or without a healthcheck (the node agent’s damper has already held back a flapping health word). No evidence means not ready, so a new or replacement pin takes no traffic before its container is up, and there is no fallback from a replica entry to its service, so a healthy sibling never speaks for another replica. The node must also be ready and the deployment serving. This is stricter than the resolver’s predicate today, which reads an unobserved replica as ready and ORs a service’s containers on a node; the two should converge on this one.
- a pin filed under the service name (
-
runningis whether that same evidence says the replica runs at all, whatever its health. It decides which flows survive (section 6). The container recorder bumps the clusterfs revision when a container’s running state changes as well as when its readiness does, so a running but unhealthy container that exits is republished at once, not at the next refresh. -
Sorted throughout, so an unchanged table has an unchanged section digest and the node does nothing.
The file also carries the cluster’s published ports (SHC-1353), under
published:
"published": [ { "deployment": "acme/prod/shop", "service": "web", "port": 8443, "container": 443, "protocol": "tcp", "nodes": ["n1", "n2"] }]- One entry per published port, read from the ledger: the
published_portscolumn of each deployment’s row.portis the port a listening node takes connections on andcontainerthe port the replicas listen on.spreadis set for a port published with+spread. nodesis the listen set, resolved on every build from the port’s selector (or the cluster defaultpublish.listen, or every node), the node labels and the live pins. A label, a replica or a node that moves takes the listeners with it at the next build.- No two entries share a node, port and protocol. Install, update and
shc node labelrefuse an overlap. Where one arises anyway (a replica moves, a location changes) the older deployment keeps the node and the younger listens on the rest of its set. - A port of a service with no live VIP is left out.
The section is cluster-wide content. Each node compiles its own ruleset from it (section 4), which is the only node-specific step.
4. The node’s nftables document
Section titled “4. The node’s nftables document”The node applier reads current/vip/services.json, compiles it for itself and
converges the kernel with the fabric’s existing discipline: one nft -f
transaction (declare, delete, recreate the table), an applied marker chain
carrying the document’s version, read back from the kernel. A marker that
already matches is a strict no-op.
table ip shc_vip { map dispatch { type ipv4_addr : verdict elements = { 10.224.0.17 : goto vip_10_224_0_17 } } set hairpin { type ipv4_addr . ipv4_addr elements = { 10.128.16.130 . 10.128.16.130 } } chain prerouting { type nat hook prerouting priority dstnat; policy accept; ip daddr vmap @dispatch } chain output { type nat hook output priority dstnat; policy accept; ip daddr vmap @dispatch } chain postrouting { type nat hook postrouting priority srcnat; policy accept; ct status dnat ip saddr . ip daddr @hairpin counter masquerade ct status dnat ct original ip daddr 10.224.0.0/11 fib saddr type local counter masquerade } chain refuse_forward { type filter hook forward priority -30; policy accept; ip daddr 10.224.0.0/11 meta l4proto tcp counter reject with tcp reset ip daddr 10.224.0.0/11 counter reject with icmp type port-unreachable } chain refuse_output { ... same two rules on the output hook ... } chain vip_10_224_0_17 { counter dnat ip to 10.128.16.130 comment "acme/prod/shop/web local" } chain applied { counter comment "applied sha256:..." }}Published ports. A node in a published port’s nodes gets three more
rules in the same table:
chain prerouting { type nat hook prerouting priority dstnat; policy accept; iifname != { "shc0", "mesh0" } fib daddr type local jump published ip daddr vmap @dispatch } chain postrouting { ... oifname "shc0" ct status dnat ct mark & 0x10000000 == 0x10000000 counter masquerade } chain published { tcp dport 8443 counter ct mark set ct mark or 0x10000000 dnat to 10.128.16.130:443 comment "acme/prod/shop/web 8443/tcp local" }A published port is a rule, never a socket and never a runtime’s own publish:
the rendered compose has no ports: entry for it. A connection that arrives
for any of the node’s own addresses on the port, from anywhere but the fabric
device and the control-plane mesh, is rewritten to a ready replica’s pin on the container port. The
target is where the service’s VIP goes from that node (nearest first, below),
so a replica on the listening node takes it and sees the client’s own address.
With +spread it is every ready replica. A flow that leaves by the fabric
device wears the node’s fabric address, the only source a peer’s AllowedIPs
admit from it.
The rule that rewrites also sets the conntrack mark the fabric policy reads
(policy.PublicationCtMark). It is one rule, so the marked flows are exactly
the flows the listener published, and nothing else in SHC sets the bit. The
policy admits a client from outside the fabric to a pin on another node only
with the rewrite and the mark (section 6). A client that dials the pin itself,
or that another table rewrote to it, carries no mark and is dropped; the
kernel lab measures both.
With no ready replica the node has no rule for the port, and its own stack refuses the connection. A node outside the set has none of the three rules. The table is IPv4, so a published port does not answer on a node’s IPv6 address, and it does not answer a connection the node makes to itself (nat prerouting never sees one).
An app’s own ports: entries are not publishes. On a cluster that runs VIPs
the render takes them out and declares their container ports under
expose:, and the install warns (X200141PUBTPL) with the --expose that
would publish each. The system Traefik’s 80 and 443, and its SSH entrypoint
when one is set, are the one exception: they become plain publishes with the
listen selector node=@(service=traefik), so a node listens on them exactly
while it holds a Traefik replica and always sends the connection to that
replica, which sees the client’s address. A process on a node reaches
Traefik through its service VIP (vip.LocalAddr): the keycloak and OTLP
backchannels and the certificate probes dial it, since a node’s loopback no
longer answers 443.
Address-only DNAT. dnat ip to <pin> rewrites the address and keeps the
port, so the VIP is the service’s address on every port. Which ports a client
may use is the tenant policy’s call (section 6), exactly as if it had dialled
the pin. Inside a deployment everything talks; across deployments only declared
ports pass.
Nearest first, per node. Of the service’s ready backends the node keeps the
first non-empty tier: those on this node, else those under the deepest location
level this node shares with any ready backend, else all of them. Levels compare
whole path segments; an empty location shares no level. Several in the chosen
tier are spread per connection with
numgen random mod N map { 0 : pin0, 1 : pin1 }; conntrack pins each flow to
its first choice. Every dispatch rule carries a counter and a comment naming
the tier (local, location, any), which is what the kernel proof reads
(AC5).
No ready backend means no dispatch element. The packet stays addressed to
the VIP and refuse_forward answers at once with a TCP reset or ICMP port
unreachable, the code a connected UDP socket fails on. The same rule catches a
VIP nobody owns, and refuse_output answers a process in the node’s own
namespace the same way, which also keeps its VIP traffic off the uplink.
Hairpin. A replica that dials its own service’s VIP and is picked gets a
packet from itself to itself. Two things make that work, and Kubernetes needs
the same two. First, br_netfilter treats a DNAT whose new destination sits on
the bridge the packet came from as a bridged DNAT and hands the frame back to
the bridge with the ingress port as its device; the bridge then delivers to
the destination port only if that is a different port or the port is in
hairpin_mode. So the applier turns hairpin_mode on for every port of every
fabric bridge, and checks it again on every tick, because a recreated container
comes back on a new veth. Second, without a source rewrite the replica’s reply
would stay inside its own namespace and never be un-NATted, so postrouting
masquerades exactly the pairs (pin . pin) of this node’s chosen local
backends, after the forward verdict, and the reply comes back through the
host.
Two different containers on the same bridge need neither: the frame goes to a
different port, and the backend’s bridged reply passes through conntrack
because br_netfilter (net.bridge.bridge-nf-call-iptables=1) hands bridged
IPv4 to the ip hooks, which restores the VIP as its source. The applier checks
that sysctl and reports a node without it as not converged; every runtime’s
node setup must turn it on (docker does).
A node’s own processes (SHC-1378). shcd, an operator’s curl on the node
and a network_mode: host container send from the node’s namespace, which
never crosses prerouting. chain output hooks nat output with the same
dispatch map, so such a flow is dispatched exactly as a container’s on that
node: same tiers, same counters. One thing differs. The kernel routes a local
packet before nat output rewrites it, so the flow already wears the source of
the route to the VIP, which is the uplink’s address. A replica on another node
would see that address arrive on shc0, where the peer’s AllowedIPs refuse it,
and a replica on this node would see the uplink instead of its bridge. So
postrouting masquerades a DNATted flow whose original destination is in the
VIP range and whose source is one of the node’s own addresses
(fib saddr type local). The flow then wears the outgoing device’s address:
the bridge’s for a local pin, shc0’s for a pin elsewhere. Those are the sources
a node process gets when it dials the pin itself.
5. Readiness and drain
Section titled “5. Readiness and drain”A backend is in a node’s DNAT set only while ready is true, so a replica that
is starting, unhealthy, not yet observed, or on a node the cluster no longer
sees healthy receives no new connection through the VIP (AC4). Readiness
changes already bump the clusterfs revision, coalesced to at most one a second;
the vip section’s digest changes only when its content does, and only then
does the node re-apply.
This assumes one container per pin at a time, which holds while a replica has one incarnation. The member-specific report that replica-entry pins wait for (section 3) is per incarnation, with draining as a state, so a start-first update that runs two incarnations side by side reads the right one.
Drain before stop (SHC-1155) takes an incarnation out of every routing table
(Traefik pools, resolver answers, VIP maps) and waits for the routing nodes to
confirm before it stops the copy. The VIP side of that: a draining incarnation
is not ready, and after every converge, including one that found the kernel
already right, the applier records the clusterfs revision whose vip section
its kernel now holds. A node whose table did not change still advances, so it
never holds a drain back. How that per-node revision reaches the rollout is
SHC-1155’s to choose; the applier exposes it.
As built (SHC-1295), every caller that stops or replaces a replica goes
through one primitive, vip.Drainer. A drain ends early only by its own
holder’s Release (the verb knows what it drained runs again) or when every
live pin it named is released, checked once; otherwise it holds one lease
after the verb returns, beaten by its holder, then the holder removes it and
bumps; a dead holder’s row is ended by the sweep.
What that guarantees: what a verb stops is out of every routing node’s table before anything is stopped, and stays out for one lease (60 s) after the verb returns. After that, routing follows the nodes’ records. The limit is a node whose recorder is more than a lease behind: its record still says running when the hold ends and the stopped pin is routed to until the record catches up. The fix for that is a desired state the table reads (SHC-1349), not a longer hold.
Markwrites onerouting_drainrow (deployment key, narrowed by service, replica entry or node when set) under an id it mints, and advances the clusterfs revision. The table build carries every backend a row names as not ready, whatever its replica reports, for as long as the row exists. TheDrainthatMarkreturns is the row’s one holder for its whole life: no caller names a row, no drain replaces or ends another’s, so two overlapping stops never undo each other and no deployment status is borrowed as the drain mark. A drain that has to survive a daemon restart is the rollout engine’s own later change (a holder generation on the row).Waitasks every routing node (RPCvip.drained), in parallel, re-reading the routing nodes each round, until each says its kernel holds, converged, a table at the drain’s revision or later with none of those backends ready. A table older than the drain never confirms it, and a node with no applier is not drained: its kernel keeps the last table. After 30 s (placement.DefaultDrainTimeout) the stop goes ahead, X200240VIPDRT names the nodes that had not confirmed, andshc doctorcounts it (vip.drain).BeginisMarkthenWait; the rollout engine, which steps, callsMarkand thenPendingeach step.- A drain whose routing nodes cannot be read or whose row cannot be written
did not start: no node took anything out of its table. Its
Errsays so (X503247VIPDRS) and the verb refuses its stop and stops nothing. A row that is written but whose revision could not be advanced did start: no node can confirm a row no revision carries, soWaittries the bump again each round and waits for it like any other, up to the same 30 s. - The row is a lease. Its holder beats it every 20 s. The leader’s VIP sweep
removes a row whose beat it has watched stand still for a minute, by its own
clock, and advances the revision so every node drops it together; it says so
(X200244VIPDRX) and
shc doctorcounts it. A holder that died is back in routing a lease and one sweep after its last beat under a stable leader, plus however long the leader’s reads and its delete take. The sweeper stamps a beat it first sees with the end of its read and measures up to the start of the next one, so a slow store stretches a lease and never shortens one; a new leader has watched nothing and gives every row a whole lease, so a lease runs out only under a leader that leads for longer than a minute. Releaseremoves the row at once, for what the caller knows is running: a restart the engine answered 0 to and whose target set, read back, is not empty.Holdis for every other outcome: a stop that went through, any verb that failed (a failure does not show that nothing stopped), a restart that matched no container or whose target set could not be read back, a teardown, a verb refused because its drain lapsed. A verb calls it as its last act. When every live pin the drain named is released, read once from the pin ledger, the row goes now: an uninstall, a move’s leg. Otherwise the row holds for one lease fromHold’s return (after the ledger read, which may be slow), still beaten, and then its holder removes it. Nothing a node observed ends a drain: a container record, a replica report and the deployment’s status all follow observations (a recordedstoppedis lifted by one live container, SHC-321), and an observation made before a stop can be written after it. A pin is released only by the operation that owns it (a teardown, a move, a job that ended), never by a recorder. The cost: a stop and then a start inside a minute stays out of routing until the hold ends, for a human’sshc stopthenshc start(SHC-1340) and for a stop-quiesced backup shorter than a minute (SHC-1350).- Nothing but its holder ends a drain whose operation still runs: there is no
cap on a drain’s life, which would put what the operation is stopping back
in routing under it.
shc doctor(vip.drain) names a drain held longer than 15 minutes on the daemon that holds it. The verbs end their drain on every return path, a panic included. - A drain whose row lapses under it (the sweep removed it, or its daemon could
not reach the store for a lease) says so:
Lapsed.Waitstops waiting. A verb that has not stopped anything yet refuses with X503246VIPDRN and stops nothing; a stop already under way cannot be taken back, so the drain says X200245VIPDRO and writes its row again as soon as the store answers. A row no revision carries yet (the bump failed, atMarkor after the row was written again) is bumped again on every beat until one goes through.
Callers today: shc stop and shc restart (the stack or one service),
shc uninstall (the teardown, and the stop its safety backup may make
first), a backup’s pause-to-stop escalation, and a move’s leg teardown, which
drains its own node only because the key is shared with the copy on the other
node. A leg that names no node runs on this one and drains this node; a node
that cannot say which one it is refuses the teardown (X503248VIPDRU).
6. Tenant policy and conntrack
Section titled “6. Tenant policy and conntrack”Policy holds through the DNAT (AC7). nat prerouting runs at priority
-100, before the forward hook where the tenant policy chains sit (SHC-1202,
priority -10). By the time the policy sees a VIP packet its destination is the
replica’s pin and the real port. A VIP is therefore exactly as reachable as the
pins behind it: same tenant on declared ports, own deployment on every port,
another tenant never, except the system tier’s accept classes. The bridge
anti-spoof guard judges sources, which a DNAT does not touch. The node-plane
table’s drop of container traffic to the control-plane range only meets VIP
packets that were not DNATted, which refuse_forward has already answered.
The same holds for a node’s own process. Its flow to a VIP is rewritten at nat
output and leaves with the source a flow to the pin would have, so the policy
judges the two alike. A pin on the process’s own node is reached without
crossing the forward hook, and no policy rule sees it. A pin on another node
sits behind that node’s fabric_ingress, which admits no node address today:
the flow is dropped there by the default-drop, through the VIP exactly as
through the pin. The kernel lab names that rule for both.
One policy rule does need to know: the ICMP error accept. The policy lets a pin
raise an error only about a flow addressed to it, matched as
ct original ip daddr <pin>. On the client’s node the flow’s original
destination is the VIP, so an ICMP error the replica raises about a VIP flow
(a UDP port unreachable, a fragmentation needed) is dropped there. The accept
becomes ct original ip daddr { <pin>, 10.224.0.0/11 } ct reply ip saddr <pin> ip saddr <pin>: the pin answers the flow, and the flow was sent to the pin or
to a VIP. A flow without a DNAT matches exactly as before. ct reply ip saddr
alone would also admit the pin’s errors about any other DNAT to it, a
published port or a hand rule, which the old match refused; the VIP range is
the only destination added, and the kernel lab proves that other DNAT stays
dropped.
On a fabric that overlaps the VIP range the range is not added
(Rule.VIPFlows is false, and the policy’s verify pass re-derives it). VIPs
are off there, and the range holds fabric addresses, a node’s own device
address among them, so naming it would admit a pin’s error about a published
port on such an address. The accept is the pin-only form from before VIPs. A
second kernel lab runs that fabric (10.224.0.0/12) and proves the other DNAT
stays dropped; it is red with the range always named.
Flows end with their replica (AC4) and with their VIP (AC2). A NAT binding
lives in its conntrack entry, so replacing the shc_vip table never breaks an
established flow, and also never moves one. After every apply, and once a
minute, the VIP applier runs its own pass over the kernel’s table through
SHC-1232’s modules/fabric/conntrack. Each DNATted flow whose original
destination is in the VIP range names a frontend, its VIP on every port of
its protocol, whose targets are the VIP’s running backends; conntrack.End
deletes only dumped entries, by reply tuple and kernel ID. Such a flow ends
when
- that VIP is not live in the table (released or unknown), or
- its backend is not a
runningbackend of that VIP.
A backend that is unready or draining but still running keeps its established flows, like Kubernetes. A stopped one loses them on each node once that node has materialised the revision that says so: with the store reachable, that is the recorder’s observation of the exit, at most a second of revision coalescing, the one-second clusterfs poll, and the apply and pass that follow the changed section. A node that missed a publication recovers on its next successful clusterfs refresh, which materialises, applies and runs the pass; the minute pass only repeats the judgement over what the node already holds, for entries created after the last one. A node cut off from the store has no bound and keeps its last revision. SHC-1232’s own pass already ends a flow whose backend pin was released and one the policy no longer accepts; it judges by the backend, so VIP flows are covered there too.
7. shcd restarts and failure modes
Section titled “7. shcd restarts and failure modes”- No Go in the data path (AC8). The kernel’s nft table and conntrack are the only forwarding state. shcd computes and installs; it is never on a packet’s path.
- shcd restart. The table stays in the kernel while shcd is down. On start
the applier reads
current/vip/services.jsonfrom disk, needs nothing from the control plane, compiles and finds the marker already matching: no-op. If the table changed while shcd was down, one atomic replace applies the new one, and established flows keep their bindings through it. - Partition. A node cut off from the store keeps serving its last revision, the same as the resolver.
nft flush rulesetby hand. The marker goes with the rules; the next tick re-applies.- Leader change. Allocation state is rows, not memory. A new leader’s sweep reads the same rows; two racing allocators collapse on the UNIQUE indexes.
- No fabric in the revision. A node with no clusterfs yet, a revision
without the
vipsection, or a cluster whose posture is notshchas a table that names no fabric. The applier then loads no rules, sets no hairpin or sysctl and runs no flow pass; it only drops ashc_viptable an earlier revision installed. The posture is read from the materialised revision, so this holds offline and across a restart. A container’s packet to the range meanwhile meets the node plane’s drop of the control-plane range.
8. DNS
Section titled “8. DNS”<service>.<stack>.<environment>.<tenant>.shc.internal, the bare name through
dns_search, and a plug’s provider name all answer with the service’s VIP
(AC6). The answer does not depend on readiness: the VIP itself refuses fast
when no replica is ready, and a cached answer stays right through every
reschedule. Replica-level names stay out of scope until groups and members are
ruled. shcd answers DNS (SHC-1213); that lane owns the change, reading the VIP
from the store or the vip section.
9. Status
Section titled “9. Status”shc cluster fabric vips [-s stack] [-e environment] [-o json] lists each
service’s VIP, its backends with node and readiness, and the tier this node
dispatches to, plus the version this node’s kernel holds. It reads the same
section the applier compiles, so what it prints is what the kernel does (AC1).
10. Code
Section titled “10. Code”ce:
modules/vip/compile: pure. The file types, range math, tier selection and the nft renderer. No store, no exec.modules/vip: theservice_vipstore and allocator, the leader sweep, the clusterfs section, and the node applier (nft converge, bridge port hairpin, the flow pass, the recorded revision). Readiness is injected, so the package does not importmodules/app.ent/schema/service_vip.goand the squashed migration.subsystems/impl/service_vip.go: two subsystems,service_vip_alloc(the allocation sweep, leader only) andservice_vip(the node applier, every node). The section is added to the clusterfs subsystem’s list.modules/vip.LiveVIPs: the live VIPs through a clusterfs build’s reader, for the resolver section (SHC-1213). It answers nothing while VIPs are off, so a name never resolves to an address no node routes.modules/cluster: thevipsroute and command.
shc: tests under tests/_ce/modules/vip, the subsystem catalog count, the pin
bump, and the kernel proof.
11. Proof
Section titled “11. Proof”Unit tests, table-driven, red first: the allocator (cursor order, uniqueness,
release, tombstones, wrap with the top slot still held, the 24 h floor, range
full, concurrent allocators), the sweep (allocate missing, release gone, a
failed read releases nothing), the section (backend mapping for web and
web-r<N> pins, positive readiness, running, stable ordering), the flow pass
predicate, and the compiler (each tier, no ready backend, hairpin elements,
address-only DNAT, name and comment validation, version stability).
A kernel lab in Go, gated like the fabric’s existing ones (root, Linux): three network namespaces stand in for three nodes in two locations, each with its own bridge, nftables and conntrack, routed over veth pairs. It loads the compiled VIP documents together with the policy the SHC-1202 compiler renders for the same graph, and proves AC3 (cross-node), AC4 (unready gets nothing new, a stopped backend’s flow ends), AC5 (counters on the local and location tiers), AC7 (another tenant is dropped after the DNAT), the hairpin, a same-bridge pair and AC8 (a long-lived connection survives a table replace). It also proves the ICMP cases: a replica’s error about a VIP flow reaches the client, a replica’s error about its own flow through its VIP comes back to it, an error about any other DNAT to the pin is dropped, and another tenant cannot forge one. A counter put on the responder accept in the lab says that rule took each error, and with the accept removed the cross-node one is dropped. (The hairpin’s error is a packet from the replica to itself once the masquerade is undone, which its deployment’s own accept would pass too.)
A multi-node proof script, POSIX sh, for any Debian 13 hosts reachable over ssh: it builds the documents with the same compiler, sets up WireGuard, bridges and namespace containers on each, and runs the same checks with counters and transcripts, the hairpin and the three ICMP cases included. The live proof on a real cluster is owed by the rig lane (SHC-1123).
12. What other lanes change
Section titled “12. What other lanes change”- SHC-1213 (shcd DNS): answer service names and provider names with the VIP (section 8).
- SHC-1202 (tenant policy, merged): the ICMP error accept binds the
responder as
ct reply ip saddrand admits a VIP as the flow’s original destination beside the pin itself (section 6). Nothing else. - SHC-1232 (pins, conntrack): nothing required; the VIP’s flow pass uses its conntrack package.
- SHC-1155 (drain before stop): read the applier’s recorded revision as the VIP’s routing confirmation (section 5).
- SHC-1144 / SHC-1216 (replica projects): keep filing each replica entry’s
pin as
<service>-r<N>under the deployment key; the section maps those to the service. Replica-entry backends stay unready until nodes report each pod member’s own state per incarnation, with a clusterfs bump on every accepted change (section 3).
13. Open
Section titled “13. Open”Nothing here needs a ruling to build. Two follow-ups are worth naming: whether
a UDP flow to a backend that went unready but still runs should end (Kubernetes
ends it; here it keeps going until the backend stops), and whether
shc cluster fabric explain should accept a VIP as an end and judge it against
the backend the asking node would pick.