Swarm-native rebalance: the measurements, and the chooser they point at
Archived, superseded by the new line (SHC-1092), which removed Swarm.
Swarm-native rebalance: the measurements, and the chooser they point at
Section titled “Swarm-native rebalance: the measurements, and the chooser they point at”Status: PROPOSED. Nothing here is built. Seven owner rulings landed on 2026-08-11 while this lane ran; all are honoured below (§2).
Read first: docs/decisions/2026-08-11-rebalancer-v1-foundations.md is
RULED and BUILT and outranks this page on every fact it states.
Baselines. shc main 824b20930+, ce main 0796595. Swarm behaviour was
measured on a disposable 3-node docker-in-docker swarm (inner engine 29.7.2,
host 29.4.1). Estate facts were read read-only over SSH against the live
nodes; no estate was mutated by this lane. Claims are marked MEASURED,
RECORDED, PROPOSED, or RULING-OWED.
1. Three corrections that change the brief, and one live hazard
Section titled “1. Three corrections that change the brief, and one live hazard”1.1 ★★ Both estates DO run swarm. The prior “zero swarm stacks” is wrong.
Section titled “1.1 ★★ Both estates DO run swarm. The prior “zero swarm stacks” is wrong.”MEASURED, live, both estates:
bitspur (hel1…bitspur.com, shc 0.0.49) | promanager (hel1…promanager.vip, shc 0.0.50) | |
|---|---|---|
| nodes | 3 (compose01, swarm01, swarm02) | 4 (compose01, compose02, swarm01, swarm02) |
| swarm initialised | yes, quorate | yes, quorate |
| deployments | 65 | 31 |
| swarm-runtime deployments | 4 | 3 |
The swarm stacks are real production: bitspur dtrack, gitlab, ollama,
penpot; promanager dtrack, gitlab, penpot. This is not a bootstrap
situation; it is a working swarm carrying live workloads.
1.2 ★★ THE LIVE HAZARD: all 7 are armed for a silent data fork, today
Section titled “1.2 ★★ THE LIVE HAZARD: all 7 are armed for a silent data fork, today”Every one of those 7 swarm deployments carries node-local (local-driver)
named volumes (MEASURED, e.g. corp_prod_dtrack_dtrack-data,
corp_prod_penpot_penpot-data, gitlab-{backups,config,data,logs}), and
no deployment on either estate has pinned_node set to a swarm node
(MEASURED).
So: 7 production deployments, unpinned, backed by node-local volumes, on a runtime that reschedules automatically on node failure. §5 demonstrates what that produces: the service comes up blank and serving, with the real data orphaned on the dead node. Nothing in SHC prevents it and nothing surfaces it.
This is armed independently of this proposal. It is the single most important finding of the lane, and it deserves its own remediation regardless of whether a rebalancer is ever built. The remedy is cheap: pin them (§4.6 measures what a pin buys).
★★ And the same deployments have already caused a real incident of the
adjacent kind. RECORDED at ce/modules/app/deployment_anchor_reclaim.go:6-13:
on bitspur, 2026-08-10, dtrack, ollama and penpot recorded swarm01 as
their anchor while their deployment directories existed only on compose01,
and nothing on the estate could back them up, refused from both ends. Three
of the four hazard-bearing stacks, the same names again. A deployment that is
both un-backuppable and fork-armed has no recovery path at all, which
moves item 1 of §11.1 from “cheap hygiene” to “the thing to do first.”
★ A second, related instance already ships: ce/daemon/services_swarm_verbs.go:695-710
(swarmVerbRestart) runs service update --force over every service in the
stack, with no eligibility check and no placement pin. shc restart on any
of those 7 stacks is the fork mechanism, reachable from the CLI today, firing
at whatever probability §4.1 measures.
1.3 ★ The estates are dramatically lopsided: the motivating evidence
Section titled “1.3 ★ The estates are dramatically lopsided: the motivating evidence”MEASURED per-node deployment counts:
| bitspur | promanager | ||
|---|---|---|---|
| compose01 | 50 | compose01 | 28 |
| swarm01 | 3 | compose02 | 0 |
| swarm02 | 0 | swarm01 | 2 |
| swarm02 | 1 |
compose01 carries 50 of 53 placed deployments on bitspur and 28 of 31
on promanager. promanager’s compose02 is completely empty: zero
deployments, added 2026-08-09 explicitly as a second compose landing zone.
The coordinator’s prediction was that placement is an artifact of who answered
the phone. It is: compose01 is the API front door on both estates.
1.4 The advisor correction: wired on main, absent from both estates
Section titled “1.4 The advisor correction: wired on main, absent from both estates”The claim reaching this lane was that SetPlacementAdvisor has zero
callers. That is false on shc main, and I am not softening it:
shc main:daemon/wiring/cluster_placement_ledger.go:43callsstack.SetPlacementAdvisor(placement.DBAdvisor{DB: pool.Reader()}), reached fromdaemon/wiring/quota_wire.go:66. It is wired in the daemon (nil in CE and in every CLI process, which is documented behaviour).
But the prediction still holds for the estates, for a different reason:
modules/placement/advisor.golanded on main 2026-08-11 01:58 UTC.- It is ABSENT from
v0.0.49(bitspur, tagged 2026-08-10 09:07) and fromv0.0.50(promanager, tagged 2026-08-10 23:45).
So nothing has ever chosen a node on either estate; §1.3 is exactly what
that looks like, while on main the chooser now exists and is wired. The gap
is a release, not a call site.
2. The rulings, and the boundary they draw
Section titled “2. The rulings, and the boundary they draw”| # | ruling |
|---|---|
| A | The mover is swarm’s, not SHC’s: “leverage what swarm does for rebalancing.” |
| B | Nothing moves running work on its own. Advisory + explicit operator verb. |
| C | Node-pool segregation: a node is compose or swarm, never both. |
| D | SHC picks candidates; swarm picks destinations. SHC never implements a scheduler. |
| E | One chooser. Install is its degenerate case (§3). |
| F | Runtime changes are explicit only: evacuate never changes runtime (§8). |
| G | Placement is a layer above data movement, and asks it only for a cost (§3.2). |
★ The boundary, stated so it cannot be misread:
| automatic? | |
|---|---|
| Placement at INSTALL: where NEW work lands | YES, wanted. |
| Relocation of RUNNING work | NEVER automatic. Ruling B. |
| Reaction to node FAILURE | Not a policy choice, substrate behaviour, differs per runtime (§7). |
3. ★★ The chooser is the deliverable; everything else is a caller
Section titled “3. ★★ The chooser is the deliverable; everything else is a caller”Ruling E collapses the design. There is one node chooser with four consumers:
shc installwith no--nodeshc moveshc node evacuate <node>shc rebalance
Build it once, all four get it. Test it once, all four are covered. A design that produces two selection paths is wrong. The chooser, not the rebalancer, is the primary deliverable, and its test suite carries the weight of all four verbs, so it should be mutation-proven properly.
3.1 Install is the degenerate case
Section titled “3.1 Install is the degenerate case”The chooser takes the runtime domain + a data-cost term:
- at install there is no data yet → cost is zero for every candidate → it reduces to pure balance.
- at move / evacuate / rebalance the data exists and has a location → cost is non-zero outside the storage domain.
Same logic, one extra input that happens to be empty at install.
3.2 The interface to move is a COST QUERY, not a mechanism
Section titled “3.2 The interface to move is a COST QUERY, not a mechanism”Placement asks the move abstraction what a landing would cost; it never asks how to do it. The answer is one of:
- free: the data is already reachable there
- rebind: the volume re-attaches, no copy
- N bytes: a real capture-and-restore
Placement scores on that number. Mechanism stays below the line. ★ If placement reaches into move internals, the design is wrong.
3.3 ★★ The unifying abstraction: storage domain
Section titled “3.3 ★★ The unifying abstraction: storage domain”Every declared volume has a storage domain: the set of nodes that can reach its bytes without copying. Local → one node. Cloud volume → the nodes in its AZ. Shared / object-backed → the whole estate.
Placement works against two domains at once:
- runtime domain: HARD. Which nodes may run this at all: swarm pool vs compose node, the fabric matrix. Never crossed automatically (ruling F).
- storage domain: SOFT, BUT PRICED. Inside it a move is free; outside it costs a copy.
Choose from the runtime domain, prefer the storage domain, pay to leave it.
Two things fall out, which is the test of the model:
- ★★ Pinning stops being a separate concept. A pin is a storage domain of size one with an expensive exit. And “unpinned still means pinned to all the swarm nodes” is the same statement about the runtime domain. One model replaces three rules; the placement matrix is derivable, not stated alongside it.
- ★ Rebalance becomes conservative for free. With a real cost term, a slightly-better-balanced node needing 500 GB moved loses to a slightly worse node where the volume rebinds. No anti-churn policy needs bolting on; thrashing is expensive by construction, which composes with ruling B.
★ The wrinkle: the domain is STATE, not a static property. After a copy the data lives elsewhere and the domain moves with it. It must be read at plan time, and the plan must show the cost it computed, so an apply can detect that reality changed underneath it and refuse rather than proceed on stale arithmetic, the same discipline as reporting observed-vs-intended placement (§6.3).
3.4 ★★ Whether this is a small addition or a real build: MEASURED: a REAL BUILD
Section titled “3.4 ★★ Whether this is a small addition or a real build: MEASURED: a REAL BUILD”The ingredients are better than expected. The query does not exist, and the model as stated covers a class of volume that zero shipped apps use.
There is no general volume→node reachability query. AttachableNodes (the
name the schema comment itself uses) exists only as that comment
(ce/ent/schema/volume.go:65); it is not a function anywhere in either repo.
What exists are single-node predicates, all inside the deployment-move
classifier:
ClassifyMountDecision(ce/daemon/clone_move_adapters.go:270): pure, exported, unit-tested. But it answers “is this ONE mount rebindable onto this ONE node?” →{Rebindable, CrossProvider, NotRebindable}. It never returns a node set.nodeInLocation(:635) andnodeSupportsProvider(~:500-582): both single-node.- ★ The failover path’s same-AZ logic is private and duplicative:
pickSameAZNode(ce/subsystems/impl/volume_failover_node.go:126) is unexported, returns the first match and stops (:151-153), a “find one”, not a “list all”, and re-derives location comparison independently. Two implementations of “same location” already exist and share no code.
The per-volume facts do exist: location (ent/schema/volume.go:68,
best-effort, NULL by design for local and RWX shared), provider (:55),
access_mode (:57), size_bytes (:58). Shared-vs-local is the
best-built part: providerTypeShared (modules/storage/storageapi/types.go:51-55)
with IsShared() at :68/:277, guarded by a source-parsing test that reddens
if a new provider constant lacks an entry.
★★ But the decisive finding is that the data model does not describe the
volumes that actually exist. A volume row is created only via
daemon/volume_binder.go:132, reached only for a mount carrying an
x-shc-volumes binding, and zero apps in either catalogue declare
x-shc-volumes (measured across apps/ and ce builtin/). Every real mount
today is a host bind under <deployDir>/mounts/<name> or a bare docker named
volume, and neither gets a row, a location, an access mode, or a size. The
classifier says so itself (clone_move_adapters.go:114-122): the --storage
stamp “is a CLAIM, not a fact … Only a volume row is evidence.”
Likewise per-volume size is real only for bound volumes; volumes_deployed.go:29-36
states “SizeBytes is best-effort and left 0 here”, so the byte figure the
plan wants to print (§8.3) is not available for the mounts that matter.
The verdict, in three tiers:
| tier | work | cost |
|---|---|---|
| small | hoist nodeInLocation + nodeSupportsProvider into NodesReaching(volumeID) []string; make pickSameAZNode call it | ~a day |
| medium | make the domain a total function including the NULL-location local/shared cases | real but bounded |
| ★ large | the domain is currently over an empty set. Either write a volume row for every persistent mount including local/host-bind, or build the domain over a different substrate (compose mount graph + deployment_node) | from-scratch design |
So §3.3 is a real build, and the honest headline is that its elegance is ahead of its substrate. The abstraction is still right; it is what makes pinning derivable and anti-churn structural, but it should be costed as new machinery, not as a refinement of what exists.
4. What docker service update --force actually does: MEASURED
Section titled “4. What docker service update --force actually does: MEASURED”4.1 Relocation is stochastic, undirected, and mostly does not happen
Section titled “4.1 Relocation is stochastic, undirected, and mostly does not happen”20 sequential force-updates, 1-replica stateless service, idle 3-node swarm, each update waited to completion by task ID:
| force-updates issued | 20 |
| updates that changed node | 3 (15%) |
| landing distribution over 21 tasks | swl1 ×18, swl3 ×2, swl2 ×1 |
| longest run without moving | 15 consecutive force-updates |
Methodology note, recorded because it changed the answer. A first pass measured 5-in-12 and was wrong. Polling
service ps --filter desired-state=runningright after a--detached update can read the outgoing task, andservice pstruncates task IDs while--no-truncdoes not, so a naive join silently returns nothing. Both were fixed and--task-history-limitraised from its default of 5 to 80. The uncorrected 42% figure is recorded only so nobody re-derives it.
4.2 The gap is 4.1 seconds, and it is a real outage
Section titled “4.2 The gap is 4.1 seconds, and it is a real outage”Data-plane probe (not control-plane inference): a continuous request loop from
outside the swarm against the routing-mesh published port, one force-update per
run, default --update-order stop-first:
| run | probes | failed | outage (last OK → first OK) |
|---|---|---|---|
| 1 | 5321 | 9 | 4128 ms |
| 2 | 5402 | 9 | 4123 ms |
| 3 | 4589 | 9 | 4150 ms |
Failures were contiguous in all three runs: one clean window, no flapping. 4.1 s ± 30 ms, image pre-pulled on every node, so this is a floor.
Since 0 of 91 apps declare replicas:, every app is single-replica, so
this gap is unavoidable on this path. The plan must print the number, not the
adjective. --update-order start-first runs two tasks concurrently, the exact
failure mode this design exists to prevent, so it is sound only for the
stateless-eligible set; whether it is the default there is RULING-OWED.
4.3 ★★ Swarm ranks by TASK COUNT, not commitment, so the nudge misses
Section titled “4.3 ★★ Swarm ranks by TASK COUNT, not commitment, so the nudge misses”The most consequential measurement in the lane.
A rebalance advisory is naturally expressed in commitment ratio, the one
capacity metric SHC can compute durably (§9). Terrain built deliberately against
that: swl1 holding one task reserving 40 of 48 CPUs (~83% committed, fewest
tasks); swl2 and swl3 each holding six tiny tasks (~1% committed, most
tasks). Subject reserving 4 CPUs, started on swl1, unpinned. Twelve bare
force-updates:
| relocations | 0 of 12 |
| landing distribution over 13 tasks | swl1 ×13 |
The task never left the 83%-committed node with two nearly-empty nodes available throughout.
★ Mechanism: swarm’s spread strategy ranks candidates by task count;
resource reservations are a fit filter, not a ranking term. swl1 had the
fewest tasks and won every round. And because stop-first releases the
outgoing task’s reservation before the new one is scheduled, the source node
is always a valid candidate again, which also explains §4.1’s stickiness.
Consequences, all design-level:
- SHC’s advisory metric (commitment) and swarm’s decision metric (task count) are different quantities. A plan can be right about which node is overloaded and still watch swarm put the task straight back.
- Under ruling D the plan’s to-node is a prediction, and a weak one.
- A no-op is the expected case, not the exception.
4.4 A DIRECTED move works, and it conflicts with ruling D
Section titled “4.4 A DIRECTED move works, and it conflicts with ruling D”service update --constraint-add node.hostname==<target>: the task landed on
exactly the requested node, first attempt, control plane 2487 ms.
SHC already does this: ce/daemon/backup_swarm_replay.go (pinSwarmStackToNode),
whose header states the problem verbatim. That file is the implementation
reference.
★ The conflict: a directed move is SHC choosing the destination, which
ruling D forbids, and it adds the node.hostname== constraint that
ce/modules/stack/placement_seam.go:29-37 forbids in terms. But §4.3 measured
that the undirected form does essentially nothing. RULING-OWED, three ways:
- (a) bare force, honestly reported: faithful to ruling D; most invocations cost 4.1 s and move nothing; a no-op reads as “swarm evaluated and declined.”
- (b) directed, pin retained: an operator naming a node is an explicit placement intent, and under §3.3 naming a node IS the pin. Smallest mechanism, truest description.
- (c) directed, add-then-remove: no residual pin, but a pinned window and a second 4.1 s gap.
This is the central open question of the design.
4.5 When it does not move at all
Section titled “4.5 When it does not move at all”- A hard-pinned service never relocates (§4.6).
- A single eligible node: nowhere to go; a pure restart costing 4.1 s. The
landed advisor already refuses this at install
(
modules/placement/advisor.go:104-106). - Insufficient reservable resources elsewhere: swarm declines rather than overcommits. INFERRED from documented behaviour; not separately tested.
- A drained or paused node is not a candidate, §6, the basis of pool
segregation. Reuse the advisor’s predicate verbatim:
cluster_state IN ('healthy','degraded') AND (availability IS NULL OR availability = 'active')(advisor.go:62-67).
4.6 A pin turns a silent fork into a visible outage
Section titled “4.6 A pin turns a silent fork into a visible outage”Service hard-pinned with --constraint node.hostname==swl2:
- Six sequential force-updates: all six re-placed onto swl2, a restart with no relocation risk, same 4.1 s.
- With swl2 then drained: task went
Pending, “no suitable node (scheduling constraints not satisfied on 2 nodes; 1 node not available for new tasks)”, service 0/1. Restoring swl2 returned it to 1/1 on the same node, volume intact.
A pin converts a silent data fork into a visible, self-healing outage. This is the remedy for §1.2 and it is available today.
5. The data fork: demonstrated, not reasoned about
Section titled “5. The data fork: demonstrated, not reasoned about”A 1-replica service with an ordinary docker-managed named volume, exactly what
SHC emits for an unbound swarm data mount
(ce/modules/app/install_runner.go:4092-4118, the swarmRuntime && !hasBinding
branch, whose comment says it keeps the named-volume reference so “swarm
provisions it per-node on whichever node the task lands”). With 0 of 91 apps
declaring x-shc-volumes, this is the default path for 74 of 91 apps and
all 215 named volumes, and for all 7 live swarm deployments (§1.2).
MEASURED:
task started on swl1; wrote CUSTOMER-DATA-DO-NOT-LOSE to /data/ledger.txtvolume inventory before: swl1: 1 swl2: 0 swl3: 0
directed move swl1 -> swl2 (landed on swl2, 2487 ms)
what the app sees on swl2: entries in /data : 0 cat /data/ledger.txt : <<NO SUCH FILE: DATA FORK>>the bytes, still on swl1: ledger.txt / CUSTOMER-DATA-DO-NOT-LOSEvolume inventory after: swl1: 1 swl2: 1 swl3: 0The application came up blank and serving. Nothing failed, nothing errored, no health check fired; nginx is perfectly healthy with an empty data directory, and so is almost every real app whose probe answers before it reads user data.
★ The worst possible shape of failure: a silent, healthy-looking service serving an empty database while the real data sits one node away with nothing pointing at it.
And the same thing happens with no operator involved. MEASURED: an unpinned volume-backed service whose node was drained was rescheduled automatically by swarm onto another node and came up against a fresh empty volume. That is §1.2’s hazard, reproduced.
5.1 The RWO pin is real, and currently a no-op
Section titled “5.1 The RWO pin is real, and currently a no-op”ce/modules/app/swarm_placement.go:104-160 (applyVolumePlacement) pins an RWO
x-shc-volume service to its attachment node and clamps it to one replica,
because “a second replica or a reschedule to another node is two writers =
corruption.” Sound mechanism, right model, and guarded at :105 by
if len(placements) == 0 { return }. With 0 of 91 apps declaring
x-shc-volumes, it is a no-op estate-wide.
6. Node-pool segregation: the primitive, VERIFIED
Section titled “6. Node-pool segregation: the primitive, VERIFIED”Ruling C, implemented with the native primitive
docker node update --availability drain, so SHC enforces nothing at placement
time and swarm enforces it because it cannot do otherwise.
★ Why native, and why not a shadow ledger: swarm has no API to account for containers it did not schedule. You cannot tell swarm “12 GB on this node belongs to a compose stack.” On a mixed node swarm’s capacity view is simply wrong and it will overcommit. Segregation makes swarm’s view complete by construction.
MEASURED, all properties confirmed, with a node promoted to manager first so the raft claim could be tested on a drained manager:
| requirement | result |
|---|---|
| drained node stays a full member and raft voter | Availability=Drain, Status=Ready, ManagerStatus=Reachable, State=ready, still counted among managers |
| scheduler structurally cannot place on it | 20 replicas → 10 + 10 on the two active nodes, zero on the drained one |
| an explicit constraint to the drained node | refused: Pending, “…1 node not available for new tasks” |
| non-swarm containers on the node are untouched | a plain docker run container stayed running, StartedAt unchanged across the drain |
| reversibility | restoring active scheduled the pending task within ~14 s |
The fourth row is load-bearing: draining a node does not disturb its compose workload at all. Declaring an existing compose node into the compose pool is a metadata operation, not an outage.
6.1 The migration is already done: MEASURED
Section titled “6.1 The migration is already done: MEASURED”Compose-runtime deployments sitting on
swarm*-named nodes: 0 on bitspur, 0 on promanager.
Naming and occupancy already agree. Nothing has to move before the compose
nodes can be drained. No node on either estate is currently drain or
pause; all are Active.
But two things must be handled first:
- ★★ After segregation the swarm pool has ZERO managers on both estates.
bitspur’s only swarm manager is
compose01; promanager’s arecompose01+compose02. Drain does not disable manager duties so it works, but bitspur runsswarm_managers: 1, no manager redundancy at all. Promoting a swarm-pool node to manager is owed alongside the drain. - ★ bitspur’s two control planes are split: the raft leader and
background_work_nodeisswarm01, while the swarm manager iscompose01. Do not assume one node is “the” control node.
6.2 Two data-quality traps found while measuring
Section titled “6.2 Two data-quality traps found while measuring”- ★★ The
nodecolumn is the ANCHOR, not the task host, and reading it as occupancy is a real trap. bitspur’sshc lsreportsgitlaboncompose01, butdocker stack psshows both its tasks onswarm02; the two estates even disagree about the same stack (promanager recordsgitlabonswarm02). This is not a defect; the code states the taxonomy explicitly (ce/subsystems/impl/backup_scheduler_anchor.go:8-29): ANCHOR is the node whose filesystem holds the deployment directory (deployment.recorded_node_id); PLACEMENT is the node(s) containers actually run on, “legitimately NOT single-valued under swarm.” A swarm reschedule moves placement and leaves the directory where it is, so the anchor staying put is the truth. But anyone computing occupancy fromshc lswill conclude a swarm stack sits on a compose node, and §9.5 shows which table to read instead. - Orphan empty named volumes for compose apps already exist on swarm nodes:
corp_prod_librechat_db_*,corp_prod_sonarqube_*, measuring 4K–32K, while the live deployments run oncompose01. Harmless leftovers today, but exactly the shape that silently adopts stale state if those apps were ever scheduled there.
6.3 The costs, stated so they do not surprise a future reader
Section titled “6.3 The costs, stated so they do not surprise a future reader”- Capacity partitioning. Pools cannot bin-pack across the estate; compose work cannot borrow swarm headroom. Accepted knowingly, but a real loss.
- ★ The pools are NOT symmetric. traefik (→
traefik-init) and postgres (→postgres-guard) are both swarm-untranslatable, so a swarm pool cannot host its own ingress or its own system store. Those stay on compose nodes and route into the pool. A naive “drain everything that isn’t a swarm node” would strand the estate’s own ingress. - Denominator reconciliation owed. This lane measured 91 repo-committed apps, 37 strong-condition; a sibling lane reports 102 packages, 38 refused. Different denominators (apps vs packages, staged builtins in or out). Reconcile before either is quoted as the catalogue size.
7. Eligibility, and the census that makes it honest
Section titled “7. Eligibility, and the census that makes it honest”7.1 The predicate
Section titled “7.1 The predicate”Eligible for an operator-invoked swarm relocation iff: runtime == swarm;
no persistent local state (no mounts, or every data mount is a declared
rebindable volume); no depends_on condition; not never-move; and
more than one candidate in the swarm pool.
7.2 The census: MEASURED across all 91 catalogue apps
Section titled “7.2 The census: MEASURED across all 91 catalogue apps”91 repo-committed apps (96 on disk, the extra 5 gitignored staged builtins), 321 services. Jinja resolved as a union of all branches, a deliberate superset, so nothing is under-counted.
| data-persistence bucket | count |
|---|---|
| (a) no mounts of any kind | 17 |
(b) declares x-shc-volumes (rebindable) | 0 |
| (c) named volumes, undeclared, the fork hazard | 74 |
| (d) bind-mount-only | 0 |
215 distinct named volumes across 369 mounts. Every host bind in the catalogue
is a :ro config or platform bind, zero binds of mutable host state.
| N_total | 91 |
| N_swarm_runnable (minus 37 strong-condition apps) | 54 |
| N_eligible (full predicate, today) | 17 |
★ The honest number is smaller than 17. Ten of the 17 are external-*
adapter packages plus s3, front packages describing an off-estate resource,
declaring zero services; there is no container to force-update. Of the
remaining 7, two are test fixtures.
The shipped, customer-facing eligible set is 5 apps: affine-rs,
gitlab-huly, huly-nextcloud, identity-broker, nextcloud-picker: one
server and four single-service bridges, every one stateless by construction
because it delegates all state to a plugged provider.
5 of 91 apps. 7 of 321 services (~2.2%).
★ And none of them is currently deployed on either estate’s swarm pool. The 7 live swarm deployments are all in bucket (c); every one would be REFUSED. So the rebalancer’s answer on both estates today is “nothing to move”, not because swarm is absent (§1.1) but because everything on it would fork.
7.3 The never-move annotation exists and is dead
Section titled “7.3 The never-move annotation exists and is dead”ce/modules/cluster/placement.go:127 defines
HasHardNodePin([]PlacementConstraint) bool: “the case that makes a
deployment unmovable by drain”, typed, documented, unit-tested, zero
non-test callers. PROPOSED: give it a call site rather than inventing a
parallel annotation. Under §3.3 it is derivable anyway: a hard pin is a
storage domain of size one.
8. Three verbs, three orthogonal axes
Section titled “8. Three verbs, three orthogonal axes”★★ shc node evacuate <node> is the chooser with that node’s capacity forced
to ZERO. Same chooser, same plan-then-apply, same buckets, same refusals.
| verb | does | never does |
|---|---|---|
shc doctor | reports that something is wrong | acts |
shc rebalance | spreads load across the swarm pool | touches compose |
shc node evacuate <node> | gets everything off ONE node | runs unattended |
Complementary, not overlapping: swarm reschedules its own pool’s workloads when a node dies, so evacuate matters precisely for what rebalance cannot touch: compose deployments and pinned swarm ones.
8.1 The three axes, and what evacuate may touch
Section titled “8.1 The three axes, and what evacuate may touch”- node change: placement
- runtime change: packaging; EXPLICIT ONLY (ruling F)
- data movement: follows from (1)
Evacuate touches ONLY (1). A runtime change is not a placement decision, it is a REPACKAGING: the deployment re-renders through enrichment, may hit the swarm-untranslatable set, and changes how it is supervised and how traefik reaches it. It must never occur as a side effect of draining.
★ Axis (2) is independent of (1): shc move --runtime swarm with NO node
change is a pure repackaging with ZERO data movement, probably the cheapest,
safest way to try swarm on any given app, and the natural migration path for
the catalogue.
★★ The sharp edge, and the test that pins it. If a compose-only app’s node is drained and every remaining node is in the swarm pool, evacuate must REFUSE that deployment and say why. It must NOT switch the runtime to make the problem disappear. A deployment whose runtime domain is empty after drain is REFUSED, not converted; write it, and mutation-prove it.
★★ The gate this ruling depends on does not exist: MEASURED, and worse than
reported. “Assuming the app supports that runtime” means
requires.runtimes, and the key does not appear in any Go type in either
repo: no yaml:"requires" struct tag exists anywhere, and the app.yaml decode
targets (modules/app/bundled.go:13-19, detail.go:61-67,
modules/adapter/metadata.go:17-44) omit it entirely. It is a comment that
happens to be YAML. Nothing validates it either: requires.runtimes: [banana]
passes, and so does misspelling the key or deleting the block.
★ And mailcow’s pin would be invisible even to a correct reader. Only three
packages declare a runtime restriction, in two different spellings:
ivatar (apps/identity/ivatar/app.yaml:18) and dependency-track
(apps/security/dependency-track/app.yaml:40) use modern requires.runtimes,
while mailcow uses the legacy top-level runtimes:
(apps/email/mailcow/app.yaml:13). The only enforcement anywhere is a single
hand-written bats assertion inside dependency-track’s own tests
(tests/tests.bats:344-345), one app policing its own file, for a rule mailcow
violates.
The canonical reading exists on ce branch docs/swarm-ordering-wrapper
(modules/app/required_runtimes.go): RequiredRuntimes :81,
RequiredRuntimesFromYAML :95, PreferredRuntime :157,
RuntimeAcceptable :179: zero production callers, and it reads both
spellings (:100-105), so it is the only code in existence that would see
mailcow’s pin. Build the gate on those; do not re-derive the semantics.
★★ And where a check does fire, it fires in the wrong phase: CONFIRMED, with
two corrections that make it worse. The durable row is written synchronously
in Phase 1 (service.go:585, status queued), the goroutine is launched at
router.go:1454 and genuinely detached (it swaps to
context.Background() at :1459, surviving the response) and the
swarm-untranslatable preflight runs inside it at install_runner.go:1558
(not :1552, which is the guard). The caller gets HTTP 200 plus an operation
id (router.go:1538); the refusal arrives later over SSE.
The worse part: the daemon has a marker for “refused before anything was
created”: RefuseDeploy / IsDeployRefusal
(modules/app/install_refusal.go:74/:83), which makes the QUEUED intent row
be deleted rather than stamped. X400073SWMXLT is not wrapped in it and
is not on the refusal family list (:37-49), so the row is stamped FAILED and
kept, with an audit event, for an install that created no directory, no
compose file and no container. The CE swarm EE gate at install_runner.go:190
is wrapped, so requesting swarm on CE leaves no debris while requesting swarm
on EE with an untranslatable app does.
No Phase-1 gate can catch this cheaply: CheckRuntimeMatchesNode
(service.go:422) is cross-engine only and does not distinguish compose from
swarm, and CheckNodeTargetPlacement (:429) explicitly exempts swarm. The
two facts that could refuse early (the app’s declared runtimes, and its
depends_on graph) are respectively unparsed and only knowable after a full
render. A move --runtime that cannot work must be rejected before anything
is written down, which means wiring the Q2 reader is the prerequisite.
8.2 ★ The naming collision is already live
Section titled “8.2 ★ The naming collision is already live”shc node drain <node> SHIPS TODAY, declared at
ce/core/eestub/manifest.go:139 as “drain workloads from a node”; it already
means evacuate. Ruling C spends availability: drain on pool
membership. Two live, opposite meanings for one word, and the codebase is
already tangled: shc modules/cluster/drain.go:300 comments on “the swarm-side
drain”, and drain_swarm_test.go:4 records that “shc node drain wrote
SHC’s own availability column.”
PROPOSED: rename the evacuation verb to shc node evacuate; surface
swarm availability as pool membership (shc node pool <node> compose|swarm),
never as “drain”. Per the standing no-backcompat doctrine, rename outright
rather than keeping an alias. Leaving both meanings live is the one option that
should be rejected outright.
8.3 The UX: plan then apply, three buckets
Section titled “8.3 The UX: plan then apply, three buckets”Bare shc rebalance computes and prints, changing nothing. Plan is the
default because the destructive reading of a bare verb is the one typed by
accident. --apply acts, honouring a concurrency cap.
PLAN (2 would move · 2 pinned · 2 refused)
WOULD MOVE identity-broker/prod swarm01 -> swarm02 (predicted) because commit_ratio(swarm01)=0.91 > 0.80 ; commit_ratio(swarm02)=0.22 data free, no persistent state cost ~4.1s outage, single replica, stop-first recreate note destination is swarm's choice, not ours; may decline
PINNED gitlab/prod node=compose01 reason runtime: compose to move by hand: shc move gitlab/prod --to compose02 dtrack/prod node=swarm01 reason node-local volume: dtrack-data data 2.1 GB capture+restore if moved
REFUSED mailcow/prod refused depends_on: service_healthy (31 edges), cannot run on swarm nextcloud/prod refused 13 undeclared named volumes; relocation would fork data remedy declare x-shc-volumes, or move on compose with `shc move`★ A refusal is shown, never silently omitted; a plan that noticed nothing
must not look like a plan that found nothing.
★ A pinned compose deployment is listed with the exact shc move command to
fix it by hand. The move machinery already exists; do not reimplement it and
do not run it from here.
★ Cost is a measured number, and data cost is stated in bytes (§3.2). A
500 GB gitlab volume and a 200 MB config volume are completely different
operator decisions and must not print identically.
8.4 ★★ Report OBSERVED placement, never intent
Section titled “8.4 ★★ Report OBSERVED placement, never intent”Under ruling D swarm picks the destination, and §4.3 measured that the
prediction is weak. So --apply must read actual task placement after the
force-update settles and diff it against the before-state.
A no-op must print as a no-op. This project has already been burned by this
exact shape: shc update printed the past tense of the verb off a
{"success": true} payload that only meant “no error was raised”, making a
no-op and a real upgrade byte-identical on stdout and in the exit code
(RECORDED). Rebalance is the same trap with worse consequences.
APPLIED identity-broker/prod swarm01 -> swarm02 MOVED (4.2s outage) nextcloud-picker/prod swarm01 -> swarm01 NO CHANGE: swarm evaluated and kept it here8.5 Two stated properties
Section titled “8.5 Two stated properties”- Nowhere-to-go refuses PER DEPLOYMENT with a reason, and evacuate still moves everything else. A wholesale abort because one of twelve is stuck is the wrong shape.
- ★ Evacuation is ONE-WAY. When a repaired node returns, nothing comes back on its own; rebalance will eventually spread swarm work onto it, but pinned deployments stay where evacuate put them. This is correct (automatic return = churn) but must be a stated property, not a surprise.
- A pinned swarm service that evacuates comes up pinned to a different single node, same shape, different address. It does not become free-floating because it was forced to move.
8.6 doctor reports and refuses to act
Section titled “8.6 doctor reports and refuses to act”doctor may say “placement is imbalanced on 3 deployments; run
shc rebalance”. It must not relocate anything, and --fix must not reach
this code.
The argument is this project’s own history. shc doctor --fix on a swarm
stack would have spawned a second, placement-unconstrained copy of every
service onto the same named volumes, on a cron, unattended (RECORDED; P0 fixed
2026-08-06, ce 8fe9cba / shc 4b61191dd). Bitspur was armed and never
fired, surviving only because nobody typed the flag shc doctor run
advertises as the remedy. Its recorded lesson is adopted verbatim:
“a doctor prescription that mutates deployment rows feeds the reconciler’s remediation queue. Any future diagnosis that writes
statusis one hop from an automatedcompose up.”
★ Note the four swarm stacks that incident found armed on bitspur (dtrack, gitlab, penpot, ollama) are exactly the four in §1.2. The same deployments keep turning up on the wrong side of the same hazard.
9. Node failure: the scenario table
Section titled “9. Node failure: the scenario table”Ruling B governs deliberate relocation of healthy work. It says nothing about a node dying, and the answer differs per runtime. An operator who reads “SHC never moves my workloads automatically” and infers “so nothing moves when a node dies” will be wrong on swarm and right on compose.
9.1 The axis is not “what happened to the node”
Section titled “9.1 The axis is not “what happened to the node””It is can you reach the bytes, and do you actually know it is dead:
| situation | bytes reachable | known dead | operation |
|---|---|---|---|
| planned decommission / resize | yes | alive | evacuate, fully planned |
| overloaded but healthy | yes | alive | rebalance |
| degraded but alive (disk full, OOM) | yes | alive | evacuate, the best failure case |
| dead, disk re-attachable | via the volume | yes | re-attach elsewhere, then move |
| dead, disk gone | no | yes | restore from backup, not a move at all |
| unreachable, might be alive | unknown | NO | ★★ FENCE FIRST, then it becomes one of the two above |
9.2 ★★ The row that must REFUSE
Section titled “9.2 ★★ The row that must REFUSE”An unreachable node is not a dead node. Evacuating a partitioned-but-live node holding node-local data produces two copies of one service writing to two copies of one dataset.
Do not reimplement this judgement. ce/subsystems/impl/volume_failover.go
already has it: leader-only, lease expiry plus a 45 s dwell or real
STONITH proof via the hcloud STONITH agent, same-AZ, fail-closed five ways, off by
default. evacuate must REQUIRE that proof and refuse without it. Note it
is scoped to RWO declared volumes and therefore fires for nothing today:
deliberately, since that is the only class where the single-writer question has
a provable answer.
9.3 The honest bottom of the table
Section titled “9.3 The honest bottom of the table”For a node dying but still alive, clone-then-cut-over is strictly safer than move; the source is retained until the target is proven. Not new machinery: the existing move flow’s data gate is the commit point; this is the same operation with the source kept longer.
★ A deployment with node-local storage and no backup, on a node whose disk is
gone, has NO operation available. Not a refusal to be worked around; there is
nothing to move. evacuate must say that in those words. Silence, a generic
error, or a failed-move report is the worst possible output.
9.4 Capture ordering: stop→capture, not capture→stop
Section titled “9.4 Capture ordering: stop→capture, not capture→stop”★ A live volume capture is inconsistent; this project already carries a quiesce-before-capture fix precisely because of live-volume tar trips (RECORDED). Ordering must be: quiesce → capture → transfer → verify ON THE TARGET → release source.
★★ The commit point is after verification on the target, not after transfer,
and the source data is retained until the target is proven; that is what
makes a failed move a no-op instead of a loss. Confirm the implemented
dataPhase/verifyPhase order actually does this rather than assuming it.
★ Two-pass capture is the UX win worth costing. Downtime need not be proportional to volume size: capture live first to move the bulk, then stop and capture only the delta, making downtime proportional to what changed during the copy. On a large volume that is the difference between a minute and an hour. Determine whether the current machinery is single-pass; if so, cost the two-pass variant. Either way the plan must state expected downtime per move (§8.3).
9.5 ★★ Does SHC notice a swarm-initiated reschedule? MEASURED: yes, and by a better route than expected
Section titled “9.5 ★★ Does SHC notice a swarm-initiated reschedule? MEASURED: yes, and by a better route than expected”A split answer, and the good half is genuinely good.
deployment_node DOES converge, without ever calling the swarm API.
ReconcileOccupancy (ce/modules/app/deployment_node_repo.go:308) runs
leader-only every 60 s from ce/subsystems/impl/container_reconciler.go:281. It
asserts anchors, upserts observations from the container table, and, the
safety property, prunes only on positive evidence, for unanchored rows on a
currently reachable node reporting no container (:377-385). The container
table is filled first-hand per node by container_recorder_node, event-driven
off the docker event bus with a ~30 s floor.
The design states that this covers swarm reschedules, and why
(deployment_node_repo.go:297-301): the recorder gates on the shc.* label
set, not the compose-project label, and swarm task containers carry those
labels, so “every node records the swarm tasks running on it and the set
converges without docker service ps.” Worst-case convergence ≈ 90 s.
Its stated invariant is the right one: a stale or absent observation “can only
ever ADD refusals; it can never remove one.”
deployment.recorded_node_id does NOT track reschedules, by design, per
the anchor-vs-placement taxonomy in §6.2. Its only writers are Migrate
(deployment_repo.go:236, explicit move/drain) and ReanchorDeployment
(deployment_anchor_reclaim.go:129, disk-evidence-driven), and
ReconcileOccupancy deliberately re-asserts it rather than correcting it.
★ So the rule for the chooser is precise: read deployment_node /
ObservedNodes, never recorded_node_id. The reduction to use is
core/placement.Of(pinned, observed, runtime, recorded)
(ce/core/placement/placement.go:121), which already gates the recorded
fallback to host-bound runtimes and never applies it to swarm (:112-120), and
whose enrichment refuses to substitute the recorded node for a missing
observation: “EMPTY IS AN ANSWER, AND IT IS THE DEFAULT”
(observed_nodes_enrich.go:12-16).
★ The “scoring against fiction” risk is therefore not live today, because
in ce nothing scores at all (core/placement/ is a 135-line display reduction;
scheduling_source.go is a Prometheus projection). The scorer lives in shc
(modules/placement), wired through the ce seam (§1.4). The risk becomes live
the moment the chooser (§3) is built, and it is avoided by reading the right
table.
★★ But the confusion has already cost the estate once, and it is the §1.2
set again: bitspur, 2026-08-10: dtrack, ollama and penpot anchored to
swarm01 with their directories only on compose01, un-backuppable from both
ends (deployment_anchor_reclaim.go:6-13).
10. What survives ruling B
Section titled “10. What survives ruling B”Kill switch, hysteresis and thrash protection are gone (nothing fires by itself); §3.3 makes anti-churn structural anyway. These remain:
10.1 Default OFF, and OFF means do not evaluate. A scoped key in the 5-level
cascade (Deployment → Stack → Environment → Tenant → Global,
ce/core/scope/scope.go:81-97). Copy deploy.placement_scoring
(shc modules/placement/advisor.go:26-32) verbatim, including “off MEANS
DO NOT SCORE, NOT ‘score and ignore’”, checked before a single query is
issued. ★ One inversion: placement_scoring leaves the feature ON when
unreadable; a verb that relocates production workloads must fail the other way:
unreadable means refuse.
10.2 A concurrency cap on --apply. max_in_flight = 1 fleet-wide and per
node. shc node drain already moves one deployment at a time
(shc modules/cluster/drain.go:208-259). For cluster-wide exclusion the house
primitive is the transaction-free conditional UPDATE, claimCAS
(ce/modules/task/service.go:447-486).
10.3 The audit trail, which ruling B makes simpler and better. It records
what was recommended and what the operator chose. An A-prefixed
registered event class lands a hash-chained audit row on every emit
(ce/core/log/typed.go:126) with no schema change, no migration and no license
gate. Existing move codes A200601STKMOV / A400601MOVFAL are the join point.
★ The explain data already exists and is thrown away. placement.Score
returns a Decision carrying All []Scored, each with per-term
{Name, Normalised, Weight, Abstained, Reason}: candidates considered, those
rejected, and why. drain.go:619 discards it. Persisting it is the audit
trail; it needs keeping, not designing. It must carry both the prediction and
the observed outcome (§8.4), plus the computed data cost (§3.3), or it
cannot distinguish “moved as planned” from “swarm declined” from “priced against
a stale domain”.
What the trigger could have read, had ruling B not removed it: StatsCache
is in-memory, per process, never persisted, empty until the first tick, with
no TTL (the “15 s” is a consumer threshold at
modules/hud/builder/builder.go:33). ★ Its cross-node freshness clock is
broken: PeerNodeStatsCollector merges a peer sample via UpdateNode,
which restamps UpdatedAt = time.Now() (peer_nodes.go:110 +
stats_cache.go:131), so a stale peer sample always looks fresh. There is no
disk column on node, no remote disk fact, no scrape endpoint and no
in-process TSDB. The one durable, replicated metric is the commitment
ratio, and §4.3 is the caveat that swarm does not schedule on it.
11. The residue
Section titled “11. The residue”| class | count | why it cannot move | what would change it |
|---|---|---|---|
| Undeclared named volumes | 74 of 91 apps, 215 volumes, including all 7 live swarm deployments | Relocation forks the data (§5) | Declare x-shc-volumes and the volume must be genuinely rebindable. A declaration over node-local disk changes nothing. |
depends_on conditions | 37 of 91 | Cannot run on swarm at all | Rewrite the dependency as in-container waiting, per app. |
| traefik, postgres | system-critical | swarm-untranslatable | Nothing near-term; the pool cannot host its own ingress or system store. |
| Compose deployments | 59 of 65 (bitspur), 26 of 31 (promanager) | Out of scope by construction | The compose shc move path. |
| Multi-replica scale-out | all 91 | x-shc-scale.max: 1 on all 65 that declare it | Per-app concurrency-safety work. Not a placement problem. |
The load-bearing residue is the first row, and its remedy is not the one it
looks like. Declaring x-shc-volumes on an app whose data lives on node-local
disk does not make it movable; it makes SHC aware that it is not. Real
movability needs a detachable block volume, or the compose path’s
stop-capture-ship-verify.
★ The swarm-native path can never move stateful work; it can only move work
that has no state to move. That is not a gap to close later; it is the shape
of the primitive the owner chose, and choosing it was correct. The alternative
was SHC building a second scheduler over swarm’s, which R1 and
placement_seam.go both refuse.
11.1 Recommended order of work
Section titled “11.1 Recommended order of work”- ★★ Pin the 7 live swarm deployments (§1.2). Independent of everything here, cheap, and closes an armed data-loss hazard, on deployments three of which were also recently un-backuppable. §4.6 measures the trade.
- ★ Fix
swarmVerbRestart(§1.2): a live, ungated instance of the same hazard. - ★ Wire the
requires.runtimesreader (§8.1) from the branch that already has it, and move the untranslatable refusal into Phase 1 wrapped inRefuseDeploy. Small, and it is the prerequisite for ruling F’s sharp edge. - Ship the chooser (§3): one build, four consumers, and the thing that
makes §1.3’s lopsidedness stop recurring. It must read
deployment_node, neverrecorded_node_id(§9.5). - Pool segregation (§6): the migration is already clear; promote a swarm-pool manager first.
shc rebalance/shc node evacuateon top of the chooser.
11.2 Owed rulings
Section titled “11.2 Owed rulings”- §4.4: the central one. Bare force (faithful to ruling D, mostly a no-op), directed with the pin retained, or directed add-then-remove?
- §8.2: the
draincollision. Rename toevacuate; do not leave two live meanings. - ★★ §3.4: the storage domain is a REAL BUILD, now measured. The ruling
owed is no longer “which is it” but “is it funded”: its bottom tier
requires either writing a
volumerow for every persistent mount, or building the domain over a different substrate. Until then the byte figure the plan wants to print (§8.3) is unavailable for the mounts that matter. - §4.2: is
start-firstthe default for the stateless-eligible set? - §9.2: is
volume_failover.gothe home for a compose-side node-loss answer? - §6.3: reconcile the 91/37 and 102/38 denominators.
- §8.1: should the two
runtimes:spellings be reconciled catalogue-wide (mailcow is on the legacy one), and enforced by something better than one app’s own bats file?
Answered during this lane, no longer owed: §9.5: SHC does observe
swarm-initiated reschedules, via deployment_node, within ~90 s.