Skip to content
SHC Docs

Rebalancer v1 foundations: four rulings, and what was measured wrong on the way

Superseded by New-line runtime. The text below is the record as of its date.

Status: RULED and BUILT. Everything on this page is landed code with a guard, or a ruling recorded where the question was asked. Nothing here is a proposal. The design pages it builds from are docs/proposals/2026-08-11-{placement-ownership-fork,capacity-substrate,placement-policy,rebalance-trigger-and-safety}.md; where this page and one of those disagree, this page is what was measured and the proposal is the earlier draft.

Baselines. shc main 2a5800aaf, ce main ba0252a. Every figure below was read out of those trees or produced by a test run in this session.


0. What v1 is, and the line it does not cross

Section titled “0. What v1 is, and the line it does not cross”

v1 is placement plus the substrate underneath it. It chooses where a first install lands, and it builds the facts a future rebalancer would need to read. It contains:

  • no autonomous trigger,
  • no rebalance timer,
  • nothing that moves a live deployment,
  • no failover.

The reason for the line is not caution in general, it is one specific asymmetry. A bad first placement costs a deployment sitting on a node the operator would not have picked, and it is undone by the next install. A bad rebalance costs a full stateful cutover (stop, archive, ship, restore, health gate, data gate), and the revert window closes at the commit point, after which failures roll forward. The two are not comparable risks and they do not belong in one release.


1. R1: RULED: runtime: swarm is LEGACY / SECOND-CLASS

Section titled “1. R1: RULED: runtime: swarm is LEGACY / SECOND-CLASS”

Recorded where it was asked, in docs/architecture/cluster/swarm-contention-map.md §8. The evidence, re-measured in this session rather than inherited:

claimmeasured
catalog compose files99
carrying depends_on conditions Swarm cannot express (service_healthy / service_completed_successfully)37
ce builtins carrying the same2, and one of them is builtin/traefik, the system ingress
Swarm-runtime stacks on either estate0
Swarm-runtime installs verified end to end, ever0

grep -rl 'service_healthy\|service_completed_successfully' apps --include=compose.yaml | wc -l → 37 of 99.

A runtime that cannot host 37% of the catalogue or the ingress, has no production instance, and has never been proven end to end is not a supported production configuration whatever the code says. Swarm is not deleted and not broken; it is where new capability does not go.

★ The ruling is acted on in code, in both directions. The install-time placement scorer declines to have an opinion when the runtime is swarm; Swarm has a scheduler, SHC now feeds it the reservations it needs to use one, and two control planes choosing over one fleet is not a better scheduler. Meanwhile the capacity substrate is deliberately runtime-neutral: every figure is computed from SHC’s own replicated tables and its own mesh, so none of it requires Swarm to exist.

R2 of the same map (“should SHC write deploy.resources.reservations? no code should be written before this is ruled”) is marked OVERTAKEN there: SHC has been writing them since resource_reservations.go landed.


2. The node capacity column did not need re-adding. It needed READING.

Section titled “2. The node capacity column did not need re-adding. It needed READING.”

The brief for this lane, and scheduling_source.go’s own comment, both said the column was dropped 2026-07-22 for want of a writer and that “a future resource-aware scheduler re-adds the column together with a writer”.

MEASURED: the columns came back over a week ago, with three writers each, and the scheduling source never learned about them.

columnmigrationwriters
node.cpu_cores20260730032524_node_cpu_cores.sqljoiner at enroll (modules/cluster/service.go:977), boot upsert (subsystems/impl/node_self_register.go:187), per-node fact-refresh tick (:332)
node.memory_bytes20260804022217_auto.sqlsame three (service.go:982, node_self_register.go:192, :363)

So there is no migration in this lane and there must not be one. The migration doctrine (make migrate, never flatten) was the recorded reason to be careful here, and the v0.0.26 brick is why: a table-rebuild migration inside a transaction hits FK 1811 and the daemon refuses to boot on every node. That risk was never on this path.

2.1 What the un-read column was publishing

Section titled “2.1 What the un-read column was publishing”

ListNodes scanned three columns and left capacity at its zero value. SchedulingMetricsCollector is wired unconditionally (ce subsystems/impl/collectors.go:341), so for every node whose row was not being read:

  • shc_node_cpu_capacity_millicores = 0
  • shc_node_cpu_scheduled_millicores = a real non-zero sum
  • shc_node_cpu_schedulable_millicores = 0 − scheduled = a published NEGATIVE quantity of schedulable CPU
  • both *_schedule_utilization_ratio gauges = never emitted, because their capacity > 0 guard was the only thing standing between the fabricated zero and a division by it

★ The gauge that would have exposed the contradiction was the one the contradiction suppressed.

Fixed. The columns are read; NULL stays NULL through the scan; NodeSchedule grows CPUCapacityKnown / MemoryCapacityKnown / DiskCapacityKnown; and everything derived from a node’s SIZE is withheld unless that size was measured. What SHC knows from its own rows, what a node has been promised, how many stacks sit on it, still publishes unconditionally. A measured oversubscription still publishes its negative schedulable figure, because that one is true and is the thing a rebalancer exists to fix.

The local-node runtime probe is now explicitly local-only. A remote node must never inherit this box’s core count.


3. The commitment operand is the RESERVATION x REPLICAS

Section titled “3. The commitment operand is the RESERVATION x REPLICAS”

collectScheduled summed CPULimitMillicores / MemoryLimitBytes, once per service. Wrong twice:

  1. The limit is the deliberately-oversold CEILING. The commitment is the reservation, which is also the only figure Swarm’s own scheduler consults, and the one enforcement.go’s header already argued for on the request side. Summing ceilings and calling the difference “schedulable” reports the intended overcommit as a breach.
  2. It ignored desired_replicas, while the quota charge over the very same table multiplies by it. A 3-replica service was committing a third of what it holds.

Both fixed, with a guard that names each wrong answer separately so a regression cannot land quietly on either.


4. The multipliers: RULED, and the brief’s numbers were not the shipped ones

Section titled “4. The multipliers: RULED, and the brief’s numbers were not the shipped ones”

The brief stated the model as reservation = quota x 1.0, limit = quota x 1.5 and described that as what ships. MEASURED, ce main shipped reservation = quota x 0.25, limit = quota x 1.0, the owner’s shape with a competing design’s values, the residue of a derivation that read the app author’s limit as its input.

The owner’s numbers are restored. What that changes, numerically:

shipped (0.25 / 1.0)ruled (1.0 / 1.5)
committed memory, per unit of granted quota25%100%
the live estates’ ~20.68 GiB of authored ceilings, as commitment, on a 30 GiB node~5.17 GiB (17%)~20.68 GiB (69%)
shc_node_memory_schedulable_bytes on that node~24.83 GiB~9.32 GiB
rendered container ceiling, vs authored1.0x1.5x
that node’s applied ceilings20.68 GiB31.02 GiB
tenant accounting chargethe quotathe quota (unchanged; the charge is the quota, always)

★ It is a 4x swing on the commitment axis, not the 50% the brief assumed, and it is the difference between a capacity gate that can never fire and one that means something.

★ The two constants are ONE decision. Raising the reservation to 1.0 while leaving the limit at 1.0 renders reservation == limit on every service; docker accepts it, and it tells the kernel never to reclaim a container below its own hard cap. A floor jammed against its own ceiling is not a floor. That state is neither model; it is the gap between them.

Blast radius and rollback, stated rather than glossed. Every service’s effective ceiling is 50% larger on its next render, install or update; a daemon roll re-renders nothing, so this arrives one deploy later per stack. Ceilings are not allocations and both axes were already oversubscribed by authorship alone (18.20 cores on 16). A ceiling going UP refuses nothing and OOM-kills nothing that fits today; what it costs is that a runaway container can take 50% more RAM before the kernel stops it. Rollback is one key: shc config set deploy.limit_ratio 1.0 at global scope.

  • A refused limit ratio was clamping to the DEFAULT, not the minimum. Invisible while both were 1.0. At 1.5 it hands an operator who typed 0.5, plainly asking for tighter caps, a ceiling 50% looser than the minimum, as the consequence of refusing their input, while X200948QTALRT says the value “was clamped to 1.0”. Fixed with an explicit minLimitRatio, and the transform now distinguishes the zero value (unset → default) from a stated-but-illegal ratio (→ minimum).
  • ★ The whole-catalogue blast-radius guard was SELF-CONSISTENT on both axes. It computed its expected ceiling by calling the very scaler it was testing, so a mutation making limits ignore the ratio entirely passed over all 329 services. That hole predates this change; it was invisible while the multiplier was 1 and both numbers agreed. The multiply is now done in the test.

TestCatalogueEffectiveLimitsDoNotMove is re-pointed, not deleted, and the new form is strictly stronger: every service’s ceiling must equal its authored figure times exactly defaultLimitRatio. “Does not move” caught movement in either direction only while the multiplier was 1; “moves by exactly the multiplier” catches drift at any multiplier, and catches a service scaled twice.


5. Total-vs-free: three gates, and they do NOT get the same answer

Section titled “5. Total-vs-free: three gates, and they do NOT get the same answer”

The question was open in docs/decisions/2026-08-04-physical-capacity-gates.md, and enforcement.go’s header said why it could not be closed there: the fix “needs an operand no signature here accepts, a call-site change, not a change in this file.” The operand now exists.

gateanswerwhy
EnforceNodeCapacityFREE (total − committed)“Is there room right now.” Against a total, the first deployment and the fortieth got the same verdict, and a node carrying 15 of its 16 cores admitted a 15-core request.
EnforceLargestReplicaFitsNodeTOTAL, unchanged”Is any node physically big enough for ONE replica.” A 16 GiB service does not fit an 8 GiB node whether that node is empty or full. Subtracting usage could only make it refuse what would fit once something stops. It is also the only one of the three that catches a failure with no other detector.
EnforceSwarmPoolCapacityTOTAL, unchanged, understood as a FLOOR”The fleet cannot hold this at all.” A necessary condition, not a sufficient one.

Derived, not materialised, and that is the design rather than a shortcut: every input is already replicated and three of the four CASCADE from deployment, so an uninstall releases the commitment by deleting the rows it was computed from. Nothing to credit back, nothing to drift, no release path to get wrong. The alternative was measured before it was rejected: a materialised ledger is usage on a second axis, and usage needed its check and its charge folded into one statement batch after eight concurrent charges all read used = 0 and all admitted, plus a reconcile cron and a repair verb.

Three properties, each with its own mutation-proven guard:

  • reservation x replicas, never the limit;
  • the ANCHOR set UNION’d with recorded_node_id (so a live deployment can never occupy nothing), and UNION, not UNION ALL, because the ordinary compose deployment has both halves and counting it twice reports every compose node at exactly double;
  • no status filter. A stopped deployment still holds its bytes and wants its memory back the instant it starts. This is the deployment_node prune rule with the sign of the danger flipped: there, a derived-empty set produced a PERMIT; here it produces an ADMIT. Both are the unsafe direction, so the rule is the same: absence of observation must never reduce a commitment.

★ total known + committed unknown is not committed = 0. That is the unknown-as-zero collapse one layer out and it fails worse: it admits everything instead of refusing everything. An axis is judgeable only when the total is known and the commitment is. A zero commitment across zero deployments stays a measurement (the node is empty); a zero across deployments whose reservation columns were never written does not.

A fourth gate: a system headroom floor. total − committed >= request admits a node to exactly 100% reserved, and every real scheduler keeps a floor for the daemon, rqlite, the docker engine and the OS. It is not written because its numbers are not answerable from a desk: it cannot be a percentage (10% of a cpx32 and 10% of a cx53 are different amounts of daemon, and the daemon’s footprint does not scale with the box), so it must be an absolute floor, and an absolute floor invented rather than measured is a refusal threshold nobody can defend. Measuring the daemon’s own footprint on both estates is the prerequisite.


6. Placement: the scorer left the drain orchestrator

Section titled “6. Placement: the scorer left the drain orchestrator”

pickTarget (shc modules/cluster/drain.go) was the entirety of SHC’s placement intelligence: sort by the count of running deployment ROWS, tie-break by node id, prefer a runtime match. Unexported, reachable from one caller, never consulted at install.

It counted the wrong thing. Over this catalogue the largest app declares 32.62 GiB of authored memory limits and the smallest 0.125, a 261x spread, so “one gitlab” and “one link shortener” weigh the same to a counter.

shc modules/placement is that function extracted and made capacity-aware. Every property of pickTarget worth keeping is kept and the tests say so by name: deterministic id tie-break, runtime-match-beats-everything, a pure function over a slice, and (the one whose loss would be the whole regression) the batch tally folded back in between picks, so placing N deployments spreads them instead of stacking all N on whichever node was quietest when the batch started.

Three rules that are easy to get backwards:

  • An unknown axis ABSTAINS and the remaining weights renormalise. For a scorer the unknown-as-zero failure is inverted from a gate’s and just as bad: a node whose memory_bytes is NULL scored as zero headroom never receives a deployment and silently leaves the fleet; scored as infinite it receives everything. A node that can answer nothing falls back to the counts, which is exactly today’s pickTarget, so the degraded behaviour is the current behaviour rather than a new one.
  • ★ An abstaining candidate must not set the SCALE. Found while mutation-testing: the placeholder zero an abstaining node carries was reaching the min/max computation, so one unmeasured node rescaled a nearly-full node and an empty one from 1000 apart to 32 apart. The scorer kept working and quietly stopped discriminating. Same class as a pool sum that sums only the members who answered.
  • It cannot refuse. A scorer that can refuse is a gate, and the two verbs have opposite costs of refusal: a drain that refuses strands deployments on the node the operator is emptying. Admission refusal is the capacity gates’ job, on a separate operand with a separate posture.

Default posture is SPREAD, weights memory 40 / cpu 20 / count 20 / tenant 10 / app 10. Memory outweighs CPU two to one because over-limit memory KILLS and over-limit CPU throttles. Bin-pack is the same function with the three resource weights negated: one scorer, one explain output, one test surface. The weights are a starting point and are not tuned; tuning them without production placement data would be fitting noise.

6.1 Install-time placement, and the door one line away from the right one

Section titled “6.1 Install-time placement, and the door one line away from the right one”

resolveInstallTarget synthesises a descriptor with no NodeID for a stack that does not exist yet, and the back-fill fills it from the request: absent --node, the daemon that received the API call. That was the whole policy: the machine you typed the command on.

The window is two conditions and both matter: the operator named no node (an explicit --node is an instruction, not a hint) and the stack does not exist yet (a stack that exists already has bytes somewhere, and re-placing it here would be an unannounced move with no data behind it).

★ The result goes in the ROUTING slot, never the PIN slot. Writing the scored node into req.NodeID instead of target.NodeID lands it in StackRepoRow.PinnedNode and, through app.PlacementIntent, in deployment.pinned_node_id. On compose that looks harmless because it is what already happens; on swarm it manufactures a hard constraint the operator never asked for, the exact distinction StatedNodeID exists to keep expressible, and one an earlier two-way fold in that file destroyed once already. The existing fold is left alone and the derived node is applied after it, to target.NodeID only; recordStackRow’s existing firstNonEmpty(req.NodeID, target.NodeID) picks it up for the Node column while PinnedNode stays empty.

The advisor also declines on a single-candidate fleet: one candidate is not a decision, and a scorer that “chooses” on a single-node estate starts logging placement decisions where there is no placement to decide.

Kill switch: deploy.placement_scoring ∈ on | off, default on, through the ordinary 5-level cascade. off means do not score, not “score and ignore”; a kill switch that still runs the code is a kill switch that can still be crashed by the code. An unreadable knob leaves scoring ON, because a disabled feature and a broken one look identical from outside.


Every guard in this lane asserts a non-empty corpus before it asserts anything else, and every one was mutation-proven: the mutation was applied, the test was watched to fail naming the defect, and the tree was restored.

guardmutation that must kill itcaught
TestListNodesReadsCapacityColumnsstop scanning cpu_coresyes
TestNodeScheduleNeverPublishesNegativeSchedulablesameyes
TestCollectScheduledSumsReservationTimesReplicasrevert to the limit operandyes
TestCommitmentDoesNotDoubleCountTheAnchorAndTheScalarUNION ALLyes
TestCommitmentCountsStoppedDeploymentsadd a status = 'running' filteryes
TestEnforceNodeCapacityComparesAgainstFreecompare against totalyes
TestBatchSpreadsAcrossNodesmake Commit a no-opyes
TestUnmeasuredNodeIsNeitherStarvedNorFloodedzero-fill an abstained axisyes
TestAbstainingCandidateDoesNotSetTheScalelet the abstained placeholder into min/maxyes
TestFirstInstallTakesTheScoredNodeAsROUTINGNotAsAPINwrite the scored node into req.NodeIDyes
TestAnExistingStackIsNeverRePlaceddrop the “does not exist yet” conditionyes
TestAnExplicitNodeIsNeverOverriddendrop the operator-pin conditionyes
TestAdvisorSkipsDrainingAndQuarantinedNodesdrop availability; drop the cluster_state allowlistyes (both)
TestAdvisorRefusesToScoreSwarmdrop the swarm guardyes
TestAdvisorUnreadableKnobLeavesScoringOninvert the kill-switch defaultyes
TestCatalogueEffectiveLimitsMoveByExactlyTheRatiolimits ignore the ratio; ratio applied twice; reservation scaler halvedyes (all three, over 329 services)

★ Two mutations that did NOT kill their guard first time, both fixed rather than explained away: the catalogue guard asked the code under test what it should have done (self-consistent on both axes), and the scorer let an abstaining candidate set the normalisation scale.


  • No autonomous trigger, no timer, no move. §0.
  • No fourth gate. §5.3; its numbers need a measurement nobody has taken.
  • No durable utilization digest. The scorer reads intent (replicated, answerable while a node is unreachable), never a live sample. A rebalancer that plans on instantaneous samples thrashes; that axis is phase 2 and no gate needs it.
  • No per-node disk fact. node has no disk column and the local probe answers only for the local node, so shc_node_disk_capacity_bytes is now SKIPPED for remote nodes rather than published as a zero. The move’s staging preflight still measures only the executor’s disk and cannot ask a remote node; that capability is owed to the move lane.
  • No shc placement explain command. The data is carried out of the scorer (Decision.All holds every candidate’s score and every abstained term with its reason); nothing renders it yet.
  • docs/architecture/cluster/swarm-contention-map.md §8, where R1 is asked and now answered
  • docs/decisions/2026-08-04-physical-capacity-gates.md: the three gates and the total-vs-free question this page closes
  • shc modules/quota/commitment.go, enforcement.go: the ledger and the gates
  • shc modules/placement/: the scorer and the live advisor
  • ce modules/stack/placement_seam.go: the install-time seam
  • ce modules/app/resource_quota.go, resource_reservations.go: the multipliers