Skip to content
SHC Docs

Swarm-native rebalance: the measurements, and the chooser they point at

Archived, superseded by the new line (SHC-1092), which removed Swarm.

Swarm-native rebalance: the measurements, and the chooser they point at

Section titled “Swarm-native rebalance: the measurements, and the chooser they point at”

Status: PROPOSED. Nothing here is built. Seven owner rulings landed on 2026-08-11 while this lane ran; all are honoured below (§2).

Read first: docs/decisions/2026-08-11-rebalancer-v1-foundations.md is RULED and BUILT and outranks this page on every fact it states.

Baselines. shc main 824b20930+, ce main 0796595. Swarm behaviour was measured on a disposable 3-node docker-in-docker swarm (inner engine 29.7.2, host 29.4.1). Estate facts were read read-only over SSH against the live nodes; no estate was mutated by this lane. Claims are marked MEASURED, RECORDED, PROPOSED, or RULING-OWED.


1. Three corrections that change the brief, and one live hazard

Section titled “1. Three corrections that change the brief, and one live hazard”

1.1 ★★ Both estates DO run swarm. The prior “zero swarm stacks” is wrong.

Section titled “1.1 ★★ Both estates DO run swarm. The prior “zero swarm stacks” is wrong.”

MEASURED, live, both estates:

bitspur (hel1…bitspur.com, shc 0.0.49)promanager (hel1…promanager.vip, shc 0.0.50)
nodes3 (compose01, swarm01, swarm02)4 (compose01, compose02, swarm01, swarm02)
swarm initialisedyes, quorateyes, quorate
deployments6531
swarm-runtime deployments43

The swarm stacks are real production: bitspur dtrack, gitlab, ollama, penpot; promanager dtrack, gitlab, penpot. This is not a bootstrap situation; it is a working swarm carrying live workloads.

1.2 ★★ THE LIVE HAZARD: all 7 are armed for a silent data fork, today

Section titled “1.2 ★★ THE LIVE HAZARD: all 7 are armed for a silent data fork, today”

Every one of those 7 swarm deployments carries node-local (local-driver) named volumes (MEASURED, e.g. corp_prod_dtrack_dtrack-data, corp_prod_penpot_penpot-data, gitlab-{backups,config,data,logs}), and no deployment on either estate has pinned_node set to a swarm node (MEASURED).

So: 7 production deployments, unpinned, backed by node-local volumes, on a runtime that reschedules automatically on node failure. §5 demonstrates what that produces: the service comes up blank and serving, with the real data orphaned on the dead node. Nothing in SHC prevents it and nothing surfaces it.

This is armed independently of this proposal. It is the single most important finding of the lane, and it deserves its own remediation regardless of whether a rebalancer is ever built. The remedy is cheap: pin them (§4.6 measures what a pin buys).

★★ And the same deployments have already caused a real incident of the adjacent kind. RECORDED at ce/modules/app/deployment_anchor_reclaim.go:6-13: on bitspur, 2026-08-10, dtrack, ollama and penpot recorded swarm01 as their anchor while their deployment directories existed only on compose01, and nothing on the estate could back them up, refused from both ends. Three of the four hazard-bearing stacks, the same names again. A deployment that is both un-backuppable and fork-armed has no recovery path at all, which moves item 1 of §11.1 from “cheap hygiene” to “the thing to do first.”

★ A second, related instance already ships: ce/daemon/services_swarm_verbs.go:695-710 (swarmVerbRestart) runs service update --force over every service in the stack, with no eligibility check and no placement pin. shc restart on any of those 7 stacks is the fork mechanism, reachable from the CLI today, firing at whatever probability §4.1 measures.

1.3 ★ The estates are dramatically lopsided: the motivating evidence

Section titled “1.3 ★ The estates are dramatically lopsided: the motivating evidence”

MEASURED per-node deployment counts:

bitspurpromanager
compose0150compose0128
swarm013compose020
swarm020swarm012
swarm021

compose01 carries 50 of 53 placed deployments on bitspur and 28 of 31 on promanager. promanager’s compose02 is completely empty: zero deployments, added 2026-08-09 explicitly as a second compose landing zone.

The coordinator’s prediction was that placement is an artifact of who answered the phone. It is: compose01 is the API front door on both estates.

1.4 The advisor correction: wired on main, absent from both estates

Section titled “1.4 The advisor correction: wired on main, absent from both estates”

The claim reaching this lane was that SetPlacementAdvisor has zero callers. That is false on shc main, and I am not softening it:

  • shc main:daemon/wiring/cluster_placement_ledger.go:43 calls stack.SetPlacementAdvisor(placement.DBAdvisor{DB: pool.Reader()}), reached from daemon/wiring/quota_wire.go:66. It is wired in the daemon (nil in CE and in every CLI process, which is documented behaviour).

But the prediction still holds for the estates, for a different reason:

  • modules/placement/advisor.go landed on main 2026-08-11 01:58 UTC.
  • It is ABSENT from v0.0.49 (bitspur, tagged 2026-08-10 09:07) and from v0.0.50 (promanager, tagged 2026-08-10 23:45).

So nothing has ever chosen a node on either estate; §1.3 is exactly what that looks like, while on main the chooser now exists and is wired. The gap is a release, not a call site.


2. The rulings, and the boundary they draw

Section titled “2. The rulings, and the boundary they draw”
#ruling
AThe mover is swarm’s, not SHC’s: “leverage what swarm does for rebalancing.”
BNothing moves running work on its own. Advisory + explicit operator verb.
CNode-pool segregation: a node is compose or swarm, never both.
DSHC picks candidates; swarm picks destinations. SHC never implements a scheduler.
EOne chooser. Install is its degenerate case (§3).
FRuntime changes are explicit only: evacuate never changes runtime (§8).
GPlacement is a layer above data movement, and asks it only for a cost (§3.2).

★ The boundary, stated so it cannot be misread:

automatic?
Placement at INSTALL: where NEW work landsYES, wanted.
Relocation of RUNNING workNEVER automatic. Ruling B.
Reaction to node FAILURENot a policy choice, substrate behaviour, differs per runtime (§7).

3. ★★ The chooser is the deliverable; everything else is a caller

Section titled “3. ★★ The chooser is the deliverable; everything else is a caller”

Ruling E collapses the design. There is one node chooser with four consumers:

  1. shc install with no --node
  2. shc move
  3. shc node evacuate <node>
  4. shc rebalance

Build it once, all four get it. Test it once, all four are covered. A design that produces two selection paths is wrong. The chooser, not the rebalancer, is the primary deliverable, and its test suite carries the weight of all four verbs, so it should be mutation-proven properly.

The chooser takes the runtime domain + a data-cost term:

  • at install there is no data yet → cost is zero for every candidate → it reduces to pure balance.
  • at move / evacuate / rebalance the data exists and has a location → cost is non-zero outside the storage domain.

Same logic, one extra input that happens to be empty at install.

3.2 The interface to move is a COST QUERY, not a mechanism

Section titled “3.2 The interface to move is a COST QUERY, not a mechanism”

Placement asks the move abstraction what a landing would cost; it never asks how to do it. The answer is one of:

  • free: the data is already reachable there
  • rebind: the volume re-attaches, no copy
  • N bytes: a real capture-and-restore

Placement scores on that number. Mechanism stays below the line. ★ If placement reaches into move internals, the design is wrong.

3.3 ★★ The unifying abstraction: storage domain

Section titled “3.3 ★★ The unifying abstraction: storage domain”

Every declared volume has a storage domain: the set of nodes that can reach its bytes without copying. Local → one node. Cloud volume → the nodes in its AZ. Shared / object-backed → the whole estate.

Placement works against two domains at once:

  • runtime domain: HARD. Which nodes may run this at all: swarm pool vs compose node, the fabric matrix. Never crossed automatically (ruling F).
  • storage domain: SOFT, BUT PRICED. Inside it a move is free; outside it costs a copy.

Choose from the runtime domain, prefer the storage domain, pay to leave it.

Two things fall out, which is the test of the model:

  • ★★ Pinning stops being a separate concept. A pin is a storage domain of size one with an expensive exit. And “unpinned still means pinned to all the swarm nodes” is the same statement about the runtime domain. One model replaces three rules; the placement matrix is derivable, not stated alongside it.
  • ★ Rebalance becomes conservative for free. With a real cost term, a slightly-better-balanced node needing 500 GB moved loses to a slightly worse node where the volume rebinds. No anti-churn policy needs bolting on; thrashing is expensive by construction, which composes with ruling B.

★ The wrinkle: the domain is STATE, not a static property. After a copy the data lives elsewhere and the domain moves with it. It must be read at plan time, and the plan must show the cost it computed, so an apply can detect that reality changed underneath it and refuse rather than proceed on stale arithmetic, the same discipline as reporting observed-vs-intended placement (§6.3).

3.4 ★★ Whether this is a small addition or a real build: MEASURED: a REAL BUILD

Section titled “3.4 ★★ Whether this is a small addition or a real build: MEASURED: a REAL BUILD”

The ingredients are better than expected. The query does not exist, and the model as stated covers a class of volume that zero shipped apps use.

There is no general volume→node reachability query. AttachableNodes (the name the schema comment itself uses) exists only as that comment (ce/ent/schema/volume.go:65); it is not a function anywhere in either repo. What exists are single-node predicates, all inside the deployment-move classifier:

  • ClassifyMountDecision (ce/daemon/clone_move_adapters.go:270): pure, exported, unit-tested. But it answers “is this ONE mount rebindable onto this ONE node?” → {Rebindable, CrossProvider, NotRebindable}. It never returns a node set.
  • nodeInLocation (:635) and nodeSupportsProvider (~:500-582): both single-node.
  • ★ The failover path’s same-AZ logic is private and duplicative: pickSameAZNode (ce/subsystems/impl/volume_failover_node.go:126) is unexported, returns the first match and stops (:151-153), a “find one”, not a “list all”, and re-derives location comparison independently. Two implementations of “same location” already exist and share no code.

The per-volume facts do exist: location (ent/schema/volume.go:68, best-effort, NULL by design for local and RWX shared), provider (:55), access_mode (:57), size_bytes (:58). Shared-vs-local is the best-built part: providerTypeShared (modules/storage/storageapi/types.go:51-55) with IsShared() at :68/:277, guarded by a source-parsing test that reddens if a new provider constant lacks an entry.

★★ But the decisive finding is that the data model does not describe the volumes that actually exist. A volume row is created only via daemon/volume_binder.go:132, reached only for a mount carrying an x-shc-volumes binding, and zero apps in either catalogue declare x-shc-volumes (measured across apps/ and ce builtin/). Every real mount today is a host bind under <deployDir>/mounts/<name> or a bare docker named volume, and neither gets a row, a location, an access mode, or a size. The classifier says so itself (clone_move_adapters.go:114-122): the --storage stamp “is a CLAIM, not a fact … Only a volume row is evidence.”

Likewise per-volume size is real only for bound volumes; volumes_deployed.go:29-36 states “SizeBytes is best-effort and left 0 here”, so the byte figure the plan wants to print (§8.3) is not available for the mounts that matter.

The verdict, in three tiers:

tierworkcost
smallhoist nodeInLocation + nodeSupportsProvider into NodesReaching(volumeID) []string; make pickSameAZNode call it~a day
mediummake the domain a total function including the NULL-location local/shared casesreal but bounded
★ largethe domain is currently over an empty set. Either write a volume row for every persistent mount including local/host-bind, or build the domain over a different substrate (compose mount graph + deployment_node)from-scratch design

So §3.3 is a real build, and the honest headline is that its elegance is ahead of its substrate. The abstraction is still right; it is what makes pinning derivable and anti-churn structural, but it should be costed as new machinery, not as a refinement of what exists.


4. What docker service update --force actually does: MEASURED

Section titled “4. What docker service update --force actually does: MEASURED”

4.1 Relocation is stochastic, undirected, and mostly does not happen

Section titled “4.1 Relocation is stochastic, undirected, and mostly does not happen”

20 sequential force-updates, 1-replica stateless service, idle 3-node swarm, each update waited to completion by task ID:

force-updates issued20
updates that changed node3 (15%)
landing distribution over 21 tasksswl1 ×18, swl3 ×2, swl2 ×1
longest run without moving15 consecutive force-updates

Methodology note, recorded because it changed the answer. A first pass measured 5-in-12 and was wrong. Polling service ps --filter desired-state=running right after a --detached update can read the outgoing task, and service ps truncates task IDs while --no-trunc does not, so a naive join silently returns nothing. Both were fixed and --task-history-limit raised from its default of 5 to 80. The uncorrected 42% figure is recorded only so nobody re-derives it.

4.2 The gap is 4.1 seconds, and it is a real outage

Section titled “4.2 The gap is 4.1 seconds, and it is a real outage”

Data-plane probe (not control-plane inference): a continuous request loop from outside the swarm against the routing-mesh published port, one force-update per run, default --update-order stop-first:

runprobesfailedoutage (last OK → first OK)
1532194128 ms
2540294123 ms
3458994150 ms

Failures were contiguous in all three runs: one clean window, no flapping. 4.1 s ± 30 ms, image pre-pulled on every node, so this is a floor.

Since 0 of 91 apps declare replicas:, every app is single-replica, so this gap is unavoidable on this path. The plan must print the number, not the adjective. --update-order start-first runs two tasks concurrently, the exact failure mode this design exists to prevent, so it is sound only for the stateless-eligible set; whether it is the default there is RULING-OWED.

4.3 ★★ Swarm ranks by TASK COUNT, not commitment, so the nudge misses

Section titled “4.3 ★★ Swarm ranks by TASK COUNT, not commitment, so the nudge misses”

The most consequential measurement in the lane.

A rebalance advisory is naturally expressed in commitment ratio, the one capacity metric SHC can compute durably (§9). Terrain built deliberately against that: swl1 holding one task reserving 40 of 48 CPUs (~83% committed, fewest tasks); swl2 and swl3 each holding six tiny tasks (~1% committed, most tasks). Subject reserving 4 CPUs, started on swl1, unpinned. Twelve bare force-updates:

relocations0 of 12
landing distribution over 13 tasksswl1 ×13

The task never left the 83%-committed node with two nearly-empty nodes available throughout.

★ Mechanism: swarm’s spread strategy ranks candidates by task count; resource reservations are a fit filter, not a ranking term. swl1 had the fewest tasks and won every round. And because stop-first releases the outgoing task’s reservation before the new one is scheduled, the source node is always a valid candidate again, which also explains §4.1’s stickiness.

Consequences, all design-level:

  • SHC’s advisory metric (commitment) and swarm’s decision metric (task count) are different quantities. A plan can be right about which node is overloaded and still watch swarm put the task straight back.
  • Under ruling D the plan’s to-node is a prediction, and a weak one.
  • A no-op is the expected case, not the exception.

4.4 A DIRECTED move works, and it conflicts with ruling D

Section titled “4.4 A DIRECTED move works, and it conflicts with ruling D”

service update --constraint-add node.hostname==<target>: the task landed on exactly the requested node, first attempt, control plane 2487 ms.

SHC already does this: ce/daemon/backup_swarm_replay.go (pinSwarmStackToNode), whose header states the problem verbatim. That file is the implementation reference.

★ The conflict: a directed move is SHC choosing the destination, which ruling D forbids, and it adds the node.hostname== constraint that ce/modules/stack/placement_seam.go:29-37 forbids in terms. But §4.3 measured that the undirected form does essentially nothing. RULING-OWED, three ways:

  • (a) bare force, honestly reported: faithful to ruling D; most invocations cost 4.1 s and move nothing; a no-op reads as “swarm evaluated and declined.”
  • (b) directed, pin retained: an operator naming a node is an explicit placement intent, and under §3.3 naming a node IS the pin. Smallest mechanism, truest description.
  • (c) directed, add-then-remove: no residual pin, but a pinned window and a second 4.1 s gap.

This is the central open question of the design.

  • A hard-pinned service never relocates (§4.6).
  • A single eligible node: nowhere to go; a pure restart costing 4.1 s. The landed advisor already refuses this at install (modules/placement/advisor.go:104-106).
  • Insufficient reservable resources elsewhere: swarm declines rather than overcommits. INFERRED from documented behaviour; not separately tested.
  • A drained or paused node is not a candidate, §6, the basis of pool segregation. Reuse the advisor’s predicate verbatim: cluster_state IN ('healthy','degraded') AND (availability IS NULL OR availability = 'active') (advisor.go:62-67).

4.6 A pin turns a silent fork into a visible outage

Section titled “4.6 A pin turns a silent fork into a visible outage”

Service hard-pinned with --constraint node.hostname==swl2:

  • Six sequential force-updates: all six re-placed onto swl2, a restart with no relocation risk, same 4.1 s.
  • With swl2 then drained: task went Pending, “no suitable node (scheduling constraints not satisfied on 2 nodes; 1 node not available for new tasks)”, service 0/1. Restoring swl2 returned it to 1/1 on the same node, volume intact.

A pin converts a silent data fork into a visible, self-healing outage. This is the remedy for §1.2 and it is available today.


5. The data fork: demonstrated, not reasoned about

Section titled “5. The data fork: demonstrated, not reasoned about”

A 1-replica service with an ordinary docker-managed named volume, exactly what SHC emits for an unbound swarm data mount (ce/modules/app/install_runner.go:4092-4118, the swarmRuntime && !hasBinding branch, whose comment says it keeps the named-volume reference so “swarm provisions it per-node on whichever node the task lands”). With 0 of 91 apps declaring x-shc-volumes, this is the default path for 74 of 91 apps and all 215 named volumes, and for all 7 live swarm deployments (§1.2).

MEASURED:

task started on swl1; wrote CUSTOMER-DATA-DO-NOT-LOSE to /data/ledger.txt
volume inventory before: swl1: 1 swl2: 0 swl3: 0
directed move swl1 -> swl2 (landed on swl2, 2487 ms)
what the app sees on swl2: entries in /data : 0
cat /data/ledger.txt : <<NO SUCH FILE: DATA FORK>>
the bytes, still on swl1: ledger.txt / CUSTOMER-DATA-DO-NOT-LOSE
volume inventory after: swl1: 1 swl2: 1 swl3: 0

The application came up blank and serving. Nothing failed, nothing errored, no health check fired; nginx is perfectly healthy with an empty data directory, and so is almost every real app whose probe answers before it reads user data.

★ The worst possible shape of failure: a silent, healthy-looking service serving an empty database while the real data sits one node away with nothing pointing at it.

And the same thing happens with no operator involved. MEASURED: an unpinned volume-backed service whose node was drained was rescheduled automatically by swarm onto another node and came up against a fresh empty volume. That is §1.2’s hazard, reproduced.

5.1 The RWO pin is real, and currently a no-op

Section titled “5.1 The RWO pin is real, and currently a no-op”

ce/modules/app/swarm_placement.go:104-160 (applyVolumePlacement) pins an RWO x-shc-volume service to its attachment node and clamps it to one replica, because “a second replica or a reschedule to another node is two writers = corruption.” Sound mechanism, right model, and guarded at :105 by if len(placements) == 0 { return }. With 0 of 91 apps declaring x-shc-volumes, it is a no-op estate-wide.


6. Node-pool segregation: the primitive, VERIFIED

Section titled “6. Node-pool segregation: the primitive, VERIFIED”

Ruling C, implemented with the native primitive docker node update --availability drain, so SHC enforces nothing at placement time and swarm enforces it because it cannot do otherwise.

★ Why native, and why not a shadow ledger: swarm has no API to account for containers it did not schedule. You cannot tell swarm “12 GB on this node belongs to a compose stack.” On a mixed node swarm’s capacity view is simply wrong and it will overcommit. Segregation makes swarm’s view complete by construction.

MEASURED, all properties confirmed, with a node promoted to manager first so the raft claim could be tested on a drained manager:

requirementresult
drained node stays a full member and raft voterAvailability=Drain, Status=Ready, ManagerStatus=Reachable, State=ready, still counted among managers
scheduler structurally cannot place on it20 replicas → 10 + 10 on the two active nodes, zero on the drained one
an explicit constraint to the drained noderefused: Pending, “…1 node not available for new tasks”
non-swarm containers on the node are untoucheda plain docker run container stayed running, StartedAt unchanged across the drain
reversibilityrestoring active scheduled the pending task within ~14 s

The fourth row is load-bearing: draining a node does not disturb its compose workload at all. Declaring an existing compose node into the compose pool is a metadata operation, not an outage.

6.1 The migration is already done: MEASURED

Section titled “6.1 The migration is already done: MEASURED”

Compose-runtime deployments sitting on swarm*-named nodes: 0 on bitspur, 0 on promanager.

Naming and occupancy already agree. Nothing has to move before the compose nodes can be drained. No node on either estate is currently drain or pause; all are Active.

But two things must be handled first:

  • ★★ After segregation the swarm pool has ZERO managers on both estates. bitspur’s only swarm manager is compose01; promanager’s are compose01 + compose02. Drain does not disable manager duties so it works, but bitspur runs swarm_managers: 1, no manager redundancy at all. Promoting a swarm-pool node to manager is owed alongside the drain.
  • ★ bitspur’s two control planes are split: the raft leader and background_work_node is swarm01, while the swarm manager is compose01. Do not assume one node is “the” control node.

6.2 Two data-quality traps found while measuring

Section titled “6.2 Two data-quality traps found while measuring”
  • ★★ The node column is the ANCHOR, not the task host, and reading it as occupancy is a real trap. bitspur’s shc ls reports gitlab on compose01, but docker stack ps shows both its tasks on swarm02; the two estates even disagree about the same stack (promanager records gitlab on swarm02). This is not a defect; the code states the taxonomy explicitly (ce/subsystems/impl/backup_scheduler_anchor.go:8-29): ANCHOR is the node whose filesystem holds the deployment directory (deployment.recorded_node_id); PLACEMENT is the node(s) containers actually run on, “legitimately NOT single-valued under swarm.” A swarm reschedule moves placement and leaves the directory where it is, so the anchor staying put is the truth. But anyone computing occupancy from shc ls will conclude a swarm stack sits on a compose node, and §9.5 shows which table to read instead.
  • Orphan empty named volumes for compose apps already exist on swarm nodes: corp_prod_librechat_db_*, corp_prod_sonarqube_*, measuring 4K–32K, while the live deployments run on compose01. Harmless leftovers today, but exactly the shape that silently adopts stale state if those apps were ever scheduled there.

6.3 The costs, stated so they do not surprise a future reader

Section titled “6.3 The costs, stated so they do not surprise a future reader”
  • Capacity partitioning. Pools cannot bin-pack across the estate; compose work cannot borrow swarm headroom. Accepted knowingly, but a real loss.
  • ★ The pools are NOT symmetric. traefik (→traefik-init) and postgres (→postgres-guard) are both swarm-untranslatable, so a swarm pool cannot host its own ingress or its own system store. Those stay on compose nodes and route into the pool. A naive “drain everything that isn’t a swarm node” would strand the estate’s own ingress.
  • Denominator reconciliation owed. This lane measured 91 repo-committed apps, 37 strong-condition; a sibling lane reports 102 packages, 38 refused. Different denominators (apps vs packages, staged builtins in or out). Reconcile before either is quoted as the catalogue size.

7. Eligibility, and the census that makes it honest

Section titled “7. Eligibility, and the census that makes it honest”

Eligible for an operator-invoked swarm relocation iff: runtime == swarm; no persistent local state (no mounts, or every data mount is a declared rebindable volume); no depends_on condition; not never-move; and more than one candidate in the swarm pool.

7.2 The census: MEASURED across all 91 catalogue apps

Section titled “7.2 The census: MEASURED across all 91 catalogue apps”

91 repo-committed apps (96 on disk, the extra 5 gitignored staged builtins), 321 services. Jinja resolved as a union of all branches, a deliberate superset, so nothing is under-counted.

data-persistence bucketcount
(a) no mounts of any kind17
(b) declares x-shc-volumes (rebindable)0
(c) named volumes, undeclared, the fork hazard74
(d) bind-mount-only0

215 distinct named volumes across 369 mounts. Every host bind in the catalogue is a :ro config or platform bind, zero binds of mutable host state.

N_total91
N_swarm_runnable (minus 37 strong-condition apps)54
N_eligible (full predicate, today)17

★ The honest number is smaller than 17. Ten of the 17 are external-* adapter packages plus s3, front packages describing an off-estate resource, declaring zero services; there is no container to force-update. Of the remaining 7, two are test fixtures.

The shipped, customer-facing eligible set is 5 apps: affine-rs, gitlab-huly, huly-nextcloud, identity-broker, nextcloud-picker: one server and four single-service bridges, every one stateless by construction because it delegates all state to a plugged provider.

5 of 91 apps. 7 of 321 services (~2.2%).

★ And none of them is currently deployed on either estate’s swarm pool. The 7 live swarm deployments are all in bucket (c); every one would be REFUSED. So the rebalancer’s answer on both estates today is “nothing to move”, not because swarm is absent (§1.1) but because everything on it would fork.

7.3 The never-move annotation exists and is dead

Section titled “7.3 The never-move annotation exists and is dead”

ce/modules/cluster/placement.go:127 defines HasHardNodePin([]PlacementConstraint) bool: “the case that makes a deployment unmovable by drain”, typed, documented, unit-tested, zero non-test callers. PROPOSED: give it a call site rather than inventing a parallel annotation. Under §3.3 it is derivable anyway: a hard pin is a storage domain of size one.


★★ shc node evacuate <node> is the chooser with that node’s capacity forced to ZERO. Same chooser, same plan-then-apply, same buckets, same refusals.

verbdoesnever does
shc doctorreports that something is wrongacts
shc rebalancespreads load across the swarm pooltouches compose
shc node evacuate <node>gets everything off ONE noderuns unattended

Complementary, not overlapping: swarm reschedules its own pool’s workloads when a node dies, so evacuate matters precisely for what rebalance cannot touch: compose deployments and pinned swarm ones.

8.1 The three axes, and what evacuate may touch

Section titled “8.1 The three axes, and what evacuate may touch”
  1. node change: placement
  2. runtime change: packaging; EXPLICIT ONLY (ruling F)
  3. data movement: follows from (1)

Evacuate touches ONLY (1). A runtime change is not a placement decision, it is a REPACKAGING: the deployment re-renders through enrichment, may hit the swarm-untranslatable set, and changes how it is supervised and how traefik reaches it. It must never occur as a side effect of draining.

★ Axis (2) is independent of (1): shc move --runtime swarm with NO node change is a pure repackaging with ZERO data movement, probably the cheapest, safest way to try swarm on any given app, and the natural migration path for the catalogue.

★★ The sharp edge, and the test that pins it. If a compose-only app’s node is drained and every remaining node is in the swarm pool, evacuate must REFUSE that deployment and say why. It must NOT switch the runtime to make the problem disappear. A deployment whose runtime domain is empty after drain is REFUSED, not converted; write it, and mutation-prove it.

★★ The gate this ruling depends on does not exist: MEASURED, and worse than reported. “Assuming the app supports that runtime” means requires.runtimes, and the key does not appear in any Go type in either repo: no yaml:"requires" struct tag exists anywhere, and the app.yaml decode targets (modules/app/bundled.go:13-19, detail.go:61-67, modules/adapter/metadata.go:17-44) omit it entirely. It is a comment that happens to be YAML. Nothing validates it either: requires.runtimes: [banana] passes, and so does misspelling the key or deleting the block.

★ And mailcow’s pin would be invisible even to a correct reader. Only three packages declare a runtime restriction, in two different spellings: ivatar (apps/identity/ivatar/app.yaml:18) and dependency-track (apps/security/dependency-track/app.yaml:40) use modern requires.runtimes, while mailcow uses the legacy top-level runtimes: (apps/email/mailcow/app.yaml:13). The only enforcement anywhere is a single hand-written bats assertion inside dependency-track’s own tests (tests/tests.bats:344-345), one app policing its own file, for a rule mailcow violates.

The canonical reading exists on ce branch docs/swarm-ordering-wrapper (modules/app/required_runtimes.go): RequiredRuntimes :81, RequiredRuntimesFromYAML :95, PreferredRuntime :157, RuntimeAcceptable :179: zero production callers, and it reads both spellings (:100-105), so it is the only code in existence that would see mailcow’s pin. Build the gate on those; do not re-derive the semantics.

★★ And where a check does fire, it fires in the wrong phase: CONFIRMED, with two corrections that make it worse. The durable row is written synchronously in Phase 1 (service.go:585, status queued), the goroutine is launched at router.go:1454 and genuinely detached (it swaps to context.Background() at :1459, surviving the response) and the swarm-untranslatable preflight runs inside it at install_runner.go:1558 (not :1552, which is the guard). The caller gets HTTP 200 plus an operation id (router.go:1538); the refusal arrives later over SSE.

The worse part: the daemon has a marker for “refused before anything was created”: RefuseDeploy / IsDeployRefusal (modules/app/install_refusal.go:74/:83), which makes the QUEUED intent row be deleted rather than stamped. X400073SWMXLT is not wrapped in it and is not on the refusal family list (:37-49), so the row is stamped FAILED and kept, with an audit event, for an install that created no directory, no compose file and no container. The CE swarm EE gate at install_runner.go:190 is wrapped, so requesting swarm on CE leaves no debris while requesting swarm on EE with an untranslatable app does.

No Phase-1 gate can catch this cheaply: CheckRuntimeMatchesNode (service.go:422) is cross-engine only and does not distinguish compose from swarm, and CheckNodeTargetPlacement (:429) explicitly exempts swarm. The two facts that could refuse early (the app’s declared runtimes, and its depends_on graph) are respectively unparsed and only knowable after a full render. A move --runtime that cannot work must be rejected before anything is written down, which means wiring the Q2 reader is the prerequisite.

8.2 ★ The naming collision is already live

Section titled “8.2 ★ The naming collision is already live”

shc node drain <node> SHIPS TODAY, declared at ce/core/eestub/manifest.go:139 as “drain workloads from a node”; it already means evacuate. Ruling C spends availability: drain on pool membership. Two live, opposite meanings for one word, and the codebase is already tangled: shc modules/cluster/drain.go:300 comments on “the swarm-side drain”, and drain_swarm_test.go:4 records that “shc node drain wrote SHC’s own availability column.”

PROPOSED: rename the evacuation verb to shc node evacuate; surface swarm availability as pool membership (shc node pool <node> compose|swarm), never as “drain”. Per the standing no-backcompat doctrine, rename outright rather than keeping an alias. Leaving both meanings live is the one option that should be rejected outright.

8.3 The UX: plan then apply, three buckets

Section titled “8.3 The UX: plan then apply, three buckets”

Bare shc rebalance computes and prints, changing nothing. Plan is the default because the destructive reading of a bare verb is the one typed by accident. --apply acts, honouring a concurrency cap.

PLAN (2 would move · 2 pinned · 2 refused)
WOULD MOVE
identity-broker/prod swarm01 -> swarm02 (predicted)
because commit_ratio(swarm01)=0.91 > 0.80 ; commit_ratio(swarm02)=0.22
data free, no persistent state
cost ~4.1s outage, single replica, stop-first recreate
note destination is swarm's choice, not ours; may decline
PINNED
gitlab/prod node=compose01 reason runtime: compose
to move by hand: shc move gitlab/prod --to compose02
dtrack/prod node=swarm01 reason node-local volume: dtrack-data
data 2.1 GB capture+restore if moved
REFUSED
mailcow/prod refused depends_on: service_healthy (31 edges), cannot run on swarm
nextcloud/prod refused 13 undeclared named volumes; relocation would fork data
remedy declare x-shc-volumes, or move on compose with `shc move`

★ A refusal is shown, never silently omitted; a plan that noticed nothing must not look like a plan that found nothing. ★ A pinned compose deployment is listed with the exact shc move command to fix it by hand. The move machinery already exists; do not reimplement it and do not run it from here. ★ Cost is a measured number, and data cost is stated in bytes (§3.2). A 500 GB gitlab volume and a 200 MB config volume are completely different operator decisions and must not print identically.

8.4 ★★ Report OBSERVED placement, never intent

Section titled “8.4 ★★ Report OBSERVED placement, never intent”

Under ruling D swarm picks the destination, and §4.3 measured that the prediction is weak. So --apply must read actual task placement after the force-update settles and diff it against the before-state.

A no-op must print as a no-op. This project has already been burned by this exact shape: shc update printed the past tense of the verb off a {"success": true} payload that only meant “no error was raised”, making a no-op and a real upgrade byte-identical on stdout and in the exit code (RECORDED). Rebalance is the same trap with worse consequences.

APPLIED
identity-broker/prod swarm01 -> swarm02 MOVED (4.2s outage)
nextcloud-picker/prod swarm01 -> swarm01 NO CHANGE: swarm evaluated and kept it here
  • Nowhere-to-go refuses PER DEPLOYMENT with a reason, and evacuate still moves everything else. A wholesale abort because one of twelve is stuck is the wrong shape.
  • ★ Evacuation is ONE-WAY. When a repaired node returns, nothing comes back on its own; rebalance will eventually spread swarm work onto it, but pinned deployments stay where evacuate put them. This is correct (automatic return = churn) but must be a stated property, not a surprise.
  • A pinned swarm service that evacuates comes up pinned to a different single node, same shape, different address. It does not become free-floating because it was forced to move.

doctor may say “placement is imbalanced on 3 deployments; run shc rebalance”. It must not relocate anything, and --fix must not reach this code.

The argument is this project’s own history. shc doctor --fix on a swarm stack would have spawned a second, placement-unconstrained copy of every service onto the same named volumes, on a cron, unattended (RECORDED; P0 fixed 2026-08-06, ce 8fe9cba / shc 4b61191dd). Bitspur was armed and never fired, surviving only because nobody typed the flag shc doctor run advertises as the remedy. Its recorded lesson is adopted verbatim:

“a doctor prescription that mutates deployment rows feeds the reconciler’s remediation queue. Any future diagnosis that writes status is one hop from an automated compose up.”

★ Note the four swarm stacks that incident found armed on bitspur (dtrack, gitlab, penpot, ollama) are exactly the four in §1.2. The same deployments keep turning up on the wrong side of the same hazard.


Ruling B governs deliberate relocation of healthy work. It says nothing about a node dying, and the answer differs per runtime. An operator who reads “SHC never moves my workloads automatically” and infers “so nothing moves when a node dies” will be wrong on swarm and right on compose.

9.1 The axis is not “what happened to the node”

Section titled “9.1 The axis is not “what happened to the node””

It is can you reach the bytes, and do you actually know it is dead:

situationbytes reachableknown deadoperation
planned decommission / resizeyesaliveevacuate, fully planned
overloaded but healthyyesaliverebalance
degraded but alive (disk full, OOM)yesaliveevacuate, the best failure case
dead, disk re-attachablevia the volumeyesre-attach elsewhere, then move
dead, disk gonenoyesrestore from backup, not a move at all
unreachable, might be aliveunknownNO★★ FENCE FIRST, then it becomes one of the two above

An unreachable node is not a dead node. Evacuating a partitioned-but-live node holding node-local data produces two copies of one service writing to two copies of one dataset.

Do not reimplement this judgement. ce/subsystems/impl/volume_failover.go already has it: leader-only, lease expiry plus a 45 s dwell or real STONITH proof via the hcloud STONITH agent, same-AZ, fail-closed five ways, off by default. evacuate must REQUIRE that proof and refuse without it. Note it is scoped to RWO declared volumes and therefore fires for nothing today: deliberately, since that is the only class where the single-writer question has a provable answer.

For a node dying but still alive, clone-then-cut-over is strictly safer than move; the source is retained until the target is proven. Not new machinery: the existing move flow’s data gate is the commit point; this is the same operation with the source kept longer.

★ A deployment with node-local storage and no backup, on a node whose disk is gone, has NO operation available. Not a refusal to be worked around; there is nothing to move. evacuate must say that in those words. Silence, a generic error, or a failed-move report is the worst possible output.

9.4 Capture ordering: stop→capture, not capture→stop

Section titled “9.4 Capture ordering: stop→capture, not capture→stop”

★ A live volume capture is inconsistent; this project already carries a quiesce-before-capture fix precisely because of live-volume tar trips (RECORDED). Ordering must be: quiesce → capture → transfer → verify ON THE TARGET → release source.

★★ The commit point is after verification on the target, not after transfer, and the source data is retained until the target is proven; that is what makes a failed move a no-op instead of a loss. Confirm the implemented dataPhase/verifyPhase order actually does this rather than assuming it.

★ Two-pass capture is the UX win worth costing. Downtime need not be proportional to volume size: capture live first to move the bulk, then stop and capture only the delta, making downtime proportional to what changed during the copy. On a large volume that is the difference between a minute and an hour. Determine whether the current machinery is single-pass; if so, cost the two-pass variant. Either way the plan must state expected downtime per move (§8.3).

9.5 ★★ Does SHC notice a swarm-initiated reschedule? MEASURED: yes, and by a better route than expected

Section titled “9.5 ★★ Does SHC notice a swarm-initiated reschedule? MEASURED: yes, and by a better route than expected”

A split answer, and the good half is genuinely good.

deployment_node DOES converge, without ever calling the swarm API. ReconcileOccupancy (ce/modules/app/deployment_node_repo.go:308) runs leader-only every 60 s from ce/subsystems/impl/container_reconciler.go:281. It asserts anchors, upserts observations from the container table, and, the safety property, prunes only on positive evidence, for unanchored rows on a currently reachable node reporting no container (:377-385). The container table is filled first-hand per node by container_recorder_node, event-driven off the docker event bus with a ~30 s floor.

The design states that this covers swarm reschedules, and why (deployment_node_repo.go:297-301): the recorder gates on the shc.* label set, not the compose-project label, and swarm task containers carry those labels, so “every node records the swarm tasks running on it and the set converges without docker service ps.” Worst-case convergence ≈ 90 s. Its stated invariant is the right one: a stale or absent observation “can only ever ADD refusals; it can never remove one.”

deployment.recorded_node_id does NOT track reschedules, by design, per the anchor-vs-placement taxonomy in §6.2. Its only writers are Migrate (deployment_repo.go:236, explicit move/drain) and ReanchorDeployment (deployment_anchor_reclaim.go:129, disk-evidence-driven), and ReconcileOccupancy deliberately re-asserts it rather than correcting it.

★ So the rule for the chooser is precise: read deployment_node / ObservedNodes, never recorded_node_id. The reduction to use is core/placement.Of(pinned, observed, runtime, recorded) (ce/core/placement/placement.go:121), which already gates the recorded fallback to host-bound runtimes and never applies it to swarm (:112-120), and whose enrichment refuses to substitute the recorded node for a missing observation: “EMPTY IS AN ANSWER, AND IT IS THE DEFAULT” (observed_nodes_enrich.go:12-16).

★ The “scoring against fiction” risk is therefore not live today, because in ce nothing scores at all (core/placement/ is a 135-line display reduction; scheduling_source.go is a Prometheus projection). The scorer lives in shc (modules/placement), wired through the ce seam (§1.4). The risk becomes live the moment the chooser (§3) is built, and it is avoided by reading the right table.

★★ But the confusion has already cost the estate once, and it is the §1.2 set again: bitspur, 2026-08-10: dtrack, ollama and penpot anchored to swarm01 with their directories only on compose01, un-backuppable from both ends (deployment_anchor_reclaim.go:6-13).


Kill switch, hysteresis and thrash protection are gone (nothing fires by itself); §3.3 makes anti-churn structural anyway. These remain:

10.1 Default OFF, and OFF means do not evaluate. A scoped key in the 5-level cascade (Deployment → Stack → Environment → Tenant → Global, ce/core/scope/scope.go:81-97). Copy deploy.placement_scoring (shc modules/placement/advisor.go:26-32) verbatim, including “off MEANS DO NOT SCORE, NOT ‘score and ignore’”, checked before a single query is issued. ★ One inversion: placement_scoring leaves the feature ON when unreadable; a verb that relocates production workloads must fail the other way: unreadable means refuse.

10.2 A concurrency cap on --apply. max_in_flight = 1 fleet-wide and per node. shc node drain already moves one deployment at a time (shc modules/cluster/drain.go:208-259). For cluster-wide exclusion the house primitive is the transaction-free conditional UPDATE, claimCAS (ce/modules/task/service.go:447-486).

10.3 The audit trail, which ruling B makes simpler and better. It records what was recommended and what the operator chose. An A-prefixed registered event class lands a hash-chained audit row on every emit (ce/core/log/typed.go:126) with no schema change, no migration and no license gate. Existing move codes A200601STKMOV / A400601MOVFAL are the join point.

★ The explain data already exists and is thrown away. placement.Score returns a Decision carrying All []Scored, each with per-term {Name, Normalised, Weight, Abstained, Reason}: candidates considered, those rejected, and why. drain.go:619 discards it. Persisting it is the audit trail; it needs keeping, not designing. It must carry both the prediction and the observed outcome (§8.4), plus the computed data cost (§3.3), or it cannot distinguish “moved as planned” from “swarm declined” from “priced against a stale domain”.

What the trigger could have read, had ruling B not removed it: StatsCache is in-memory, per process, never persisted, empty until the first tick, with no TTL (the “15 s” is a consumer threshold at modules/hud/builder/builder.go:33). ★ Its cross-node freshness clock is broken: PeerNodeStatsCollector merges a peer sample via UpdateNode, which restamps UpdatedAt = time.Now() (peer_nodes.go:110 + stats_cache.go:131), so a stale peer sample always looks fresh. There is no disk column on node, no remote disk fact, no scrape endpoint and no in-process TSDB. The one durable, replicated metric is the commitment ratio, and §4.3 is the caveat that swarm does not schedule on it.


classcountwhy it cannot movewhat would change it
Undeclared named volumes74 of 91 apps, 215 volumes, including all 7 live swarm deploymentsRelocation forks the data (§5)Declare x-shc-volumes and the volume must be genuinely rebindable. A declaration over node-local disk changes nothing.
depends_on conditions37 of 91Cannot run on swarm at allRewrite the dependency as in-container waiting, per app.
traefik, postgressystem-criticalswarm-untranslatableNothing near-term; the pool cannot host its own ingress or system store.
Compose deployments59 of 65 (bitspur), 26 of 31 (promanager)Out of scope by constructionThe compose shc move path.
Multi-replica scale-outall 91x-shc-scale.max: 1 on all 65 that declare itPer-app concurrency-safety work. Not a placement problem.

The load-bearing residue is the first row, and its remedy is not the one it looks like. Declaring x-shc-volumes on an app whose data lives on node-local disk does not make it movable; it makes SHC aware that it is not. Real movability needs a detachable block volume, or the compose path’s stop-capture-ship-verify.

★ The swarm-native path can never move stateful work; it can only move work that has no state to move. That is not a gap to close later; it is the shape of the primitive the owner chose, and choosing it was correct. The alternative was SHC building a second scheduler over swarm’s, which R1 and placement_seam.go both refuse.

  1. ★★ Pin the 7 live swarm deployments (§1.2). Independent of everything here, cheap, and closes an armed data-loss hazard, on deployments three of which were also recently un-backuppable. §4.6 measures the trade.
  2. ★ Fix swarmVerbRestart (§1.2): a live, ungated instance of the same hazard.
  3. ★ Wire the requires.runtimes reader (§8.1) from the branch that already has it, and move the untranslatable refusal into Phase 1 wrapped in RefuseDeploy. Small, and it is the prerequisite for ruling F’s sharp edge.
  4. Ship the chooser (§3): one build, four consumers, and the thing that makes §1.3’s lopsidedness stop recurring. It must read deployment_node, never recorded_node_id (§9.5).
  5. Pool segregation (§6): the migration is already clear; promote a swarm-pool manager first.
  6. shc rebalance / shc node evacuate on top of the chooser.
  1. §4.4: the central one. Bare force (faithful to ruling D, mostly a no-op), directed with the pin retained, or directed add-then-remove?
  2. §8.2: the drain collision. Rename to evacuate; do not leave two live meanings.
  3. ★★ §3.4: the storage domain is a REAL BUILD, now measured. The ruling owed is no longer “which is it” but “is it funded”: its bottom tier requires either writing a volume row for every persistent mount, or building the domain over a different substrate. Until then the byte figure the plan wants to print (§8.3) is unavailable for the mounts that matter.
  4. §4.2: is start-first the default for the stateless-eligible set?
  5. §9.2: is volume_failover.go the home for a compose-side node-loss answer?
  6. §6.3: reconcile the 91/37 and 102/38 denominators.
  7. §8.1: should the two runtimes: spellings be reconciled catalogue-wide (mailcow is on the legacy one), and enforced by something better than one app’s own bats file?

Answered during this lane, no longer owed: §9.5: SHC does observe swarm-initiated reschedules, via deployment_node, within ~90 s.