SHC scheduling and autoscaling (Design)
Layer 6. Services. Services
Status: ruled, and none of it built. Every section here is the design, not a description of running code. Clay answered the seven open choices on 2026-11-11 and took the recommendation every time, so this is settled rather than proposed; section 12 is the record of what was put to him. The card is FZ-10588. The build is the seven batches of section 13, one card each, in that order: batch 1 FZ-10622, batch 2 FZ-10623, batch 3 FZ-10624, batch 4 FZ-10625, batch 5 FZ-10626, batch 6 FZ-10627, batch 7 FZ-10628.
It builds on the scheduler’s mechanics design and does not replace it.
../proposals/2026-09-29-shc-scheduler.md
(SHC-1058) is the scheduler itself: the assignment store, the engine’s passes,
failover, rollout, explain, the surfaces and the build order. This is the
resource model that sits on top of it. That document’s section 3 lists what was
missing: a caller, with the fit as stage 2 of the engine; a system reserve per
node; the commitment derived from assignments, surge and global replicas
included; and the fit check inside the bind. Sections 2 and 3 answer each of
those. Section 13’s Kept list, the engine, the scope ladder, the atomic
bind, the freeze gate, movability, both marks, selfclean, the heartbeat and the
portal’s guards, is a reference to that document’s machinery. Where the two
differ, this document is the later statement. Three differences are substantive
rather than incidental, and they are named in
section 11 so no reader has to
find them by collision.
Lane SCHED. The code was read at ce dev 93216b9e8883 and shc dev 5633c3b7f5b4, and every cite is a line at one of those two commits rather than at either branch’s head. A line with a cite describes the code as it stood there, and a cite that has moved since is still the evidence this was ruled on. A path carries its repo in front, ce or shc. A line marked NEW, or with no cite inside a target section, is the design.
Inputs: both designs and both critiques were complete. The maps were present and not re-read. Cites come from the truth critique’s checked column plus spot checks made here.
1. The model in one page
Section titled “1. The model in one page”A person who sets nothing learns two words: quota and pending. An operator who tunes learns eight, in one grammar.
- Quota. The size of the box one replica of a service runs in. One number per resource: CPU, memory, GPU, scratch, and one per volume. It always exists. It comes from a flag, else the app’s compose file, else a built-in default. It is never set at a scope.
- Request. A percentage of the quota. Placement counts it and nothing else does.
- Limit. A percentage of the quota. The kernel or the storage enforces it and nothing else does.
- Scope. Where a percentage is set: deployment, stack, environment, tenant, node, location, global. The ladder exists (ce core/scope/scope.go:61).
- Pending. A replica with no node, with the reason. It stays until it fits.
- Disruption. One setting per deployment:
rollingornever. - Budget. A tenant’s total cap. It exists today under the name
quota.cpu,quota.memory,quota.disk(shc modules/quota/config_effects.go:61). Two things are called quota today. Choice 6 renames this one. - Autoscale settings. In the portal: on, most servers, allowed sizes, cooldown.
Rules (--where, --prefer, --scale, --storage) exist and are not new.
With no settings: every service has a quota, memory is never oversold, CPU is shared 4 to 1, no replica is stopped to save a node, nothing moves unless a node is lost or the operator asks, and a replica that doesn’t fit is pending and visible.
| Kubernetes | SHC |
|---|---|
| requests, limits (two numbers per resource per container) | one quota, with request and limit as percentages |
| LimitRange | the same percentage keys at a wider scope |
| ResourceQuota | the tenant budget (exists, shc modules/quota/config_effects.go:61) |
| PriorityClass, preemption | none to learn. System apps go first and die last. Nothing preempts (ce core/placement/engine.go:164-172) |
| PodDisruptionBudget | deploy.disruption |
| taints and tolerations, affinity | --where, --prefer (exist) |
| HPA | none. A person types --scale |
| VPA | none. A person types the quota |
| Cluster Autoscaler or Karpenter | the portal, reading one document |
The task’s list has 11 names. The brief’s list has 14: it adds QoS classes, StorageClass, PV and PVC, splits taints from tolerations, and has no HPA or VPA. Against either list SHC has 8 nouns and 0 object kinds. The honest claim is 2 concepts for a user who sets nothing and 8 for one who tunes, not “5 against 11”.
2. Resources
Section titled “2. Resources”The table
Section titled “The table”| resource | quota is per | request default | limit default | at the limit |
|---|---|---|---|---|
| CPU | service, cores | 25% | 100% | slowed, never killed |
| memory | service, bytes | 80% | 100% | that container is killed and restarted in place |
| disk, a volume | volume, bytes | 100% | 100% | the write fails where the storage enforces it |
| disk, scratch | service, bytes | 100% | 100% | the write fails where the runtime enforces it |
| GPU | service, whole devices of one kind | fixed 100% | fixed 100% | not applicable |
Memory is 80 and 100, Clay’s ruling of 2026-11-11: “I feel like it should be a little bit compressed”. A full node then carries limits summing to 1.25 x allocatable, so it fits about a quarter more than it holds. That margin is the point. Its cost is named rather than hidden: section 4’s node-saving stop is armed on every node at defaults, where at 100 and 100 it would never arm at all. CPU is 25 and 100 because a CPU shortage slows and never kills. Disk is 100 and 100 because a 25% disk request puts 510 GB of limits on a 128 GB pool by default, and every volume can stay inside its limit while the pool fills.
The percentages are bounded, and the bound is part of the model. Clay’s ruling, 2026-11-11: “the limit is the quota … you can make the limit greater than 100% but you can’t make it less than 100% … and then the request have to be a percentage lower than 100%, lower or up to 100%”.
- A limit percentage below 100% is refused. The quota is the floor of the ceiling, so a quota is never a number the container cannot reach.
- A request percentage above 100% is refused. A replica never reserves more than its own box.
- So
request <= quota <= limit, always, however the three are written. An explicit value is held to the same order: a deploy flag that sets a request above the quota, or a limit below it, is refused at parse with both numbers printed. - A compose file that authors only a reservation has that reservation taken as the quota, which at defaults makes it the limit too. Authoring only a request therefore buys a hard cap at that number, not a soft one.
The bound is what makes the defaults legible. At a 100% request and a 100% limit a node cannot be oversold at all, and that is the fixed point everything else is measured from. The defaults do not sit there: memory’s request is 80%, so a full node is oversold by a quarter, on purpose. Every step away from the fixed point is a named percentage in one place, raising a limit above 100% or dropping a request below it, rather than an emergent property of what happened to land where.
Today: limit is 1.5 x quota and reservation 1.0 x (ce modules/app/resource_quota.go:203, ce modules/app/resource_reservations.go:111). A service that authors nothing runs uncapped (ce modules/app/resource_quota.go:504-506). Placement zeroes a request derived from the app’s own limit (ce modules/app/replica_projects.go:524-531). Disk and GPU have no quota.
How the quota is set
Section titled “How the quota is set”Per service, per resource, first match wins:
- A flag on install or update:
--cpu-quota web=2,--memory-quota web=4Gi(exist, ce modules/app/commands/install.go:351-352),--gpu-quota web=nvidia:1,--scratch-quota web=5Gi,--storage data=200Gi(NEW for GPU and scratch). - The app’s compose file: the authored limit. With no limit, the authored reservation. For GPU the authored device count. For a volume the authored size.
- The built-in default: memory 1Gi, CPU 1, 10Gi per volume, 2Gi scratch, GPU 0.
That is his order with one swap (section 11, item 1). There is no scoped quota. deploy.cpu_quota and deploy.memory_quota exist as scoped config today (ce core/config/default.yaml:1552-1553) and are deleted, because a tenant row deploy.memory_quota 512M would override every app’s own size. Clay: “you don’t have default quotas at different scopes”.
The quota and its source (flag, app, default) are stored on the service row. The prior-marker rung (ce modules/app/resource_quota.go:24-29) is deleted. An app update re-seeds only a quota whose source is app or default.
Compose numbers do nothing else. The render writes limit = quota x limit% and request = quota x request% into each replica’s spec. An authored oom_score_adj is dropped the same way.
Percentages and explicit values
Section titled “Percentages and explicit values”Scoped config carries percentages only:
shc config set deploy.cpu_request 50% # globalshc config set -t acme deploy.memory_request 100% # tenantshc config set -e dev deploy.memory_request 50% # environment: densershc config set --node burst-1 deploy.memory_limit 150% # anything on this nodeshc config set --location hel1/rack2 deploy.cpu_request 100% # anything in this locationshc config get -t acme -s gitlab -e prod deploy.memory_request # value and winning scopeKeys: deploy.{cpu,memory,disk,scratch}_request and deploy.{cpu,memory,disk,scratch}_limit. They replace deploy.limit_ratio, deploy.reservation_ratio, deploy.cpu_reservation_ratio and deploy.memory_reservation_ratio (ce core/config/default.yaml:1441-1488), which are deleted.
An explicit value is passed at deploy, per service, and is rare:
shc update -t acme -s gitlab -e prod --memory-limit web=12Gishc install nextcloud -t acme -e prod --cpu-request web=75%--{cpu,memory}-request and --{cpu,memory}-limit take service=value, where the value is a percentage or an amount. An amount ignores the quota for that figure. The tenant budget charges the larger of the quota and the request, so an explicit 8Gi request on a 1Gi quota can’t hold 8 GiB while charging 1. A scoped key takes no amount: the narrowest tier is the deployment (ce core/scope/scope.go:61), so deploy.memory_limit 12Gi on a deployment would cap every one of its services at 12Gi. An app can author none of these. The package lint refuses them.
Scopes and precedence
Section titled “Scopes and precedence”The existing ladder carries it: deployment, stack, environment, tenant, node, location, global, narrowest first (ce core/scope/scope.go:61). Workload tiers outrank place tiers. So a node row is a fallback: it applies to whatever no tenant, environment, stack or deployment row covers. Per-node ratios are resolved today in ce modules/app/quota_node.go:60-75. The install scope itself carries no node (ce modules/app/install_runner.go:2229-2233).
Two axes resolve like this. The quota never depends on the node. The percentages are looked up once per candidate node, so the fit computes a request per candidate, and the limit written into a replica’s spec is the one for the node it bound to.
A node operator who wants a floor no tenant can touch has the operator keep in the fit (ce modules/scheduler/scheduler.go:1120-1129).
Example 1: a node with 4 cores and 8 GiB filling up
Section titled “Example 1: a node with 4 cores and 8 GiB filling up”Fit capacity is the node’s totals less outside-SHC use, brokered containers and the operator keep (ce modules/scheduler/scheduler.go:1120-1129). Say that is 200m and 0.6 GiB. Allocatable: 3800m and 7.4 GiB.
The node reserve has one name, and it is the operator keep. That is the one
that is built, in roomExpr, with the markers SHC-1812, SHC-763 and SHC-2634 on
the comment that states it (ce modules/scheduler/scheduler.go:1120-1129).
scheduler.system_reserved, the per-node reserve the 2026-09-29 scheduler
design proposed at 1 GiB plus 5% of memory and 500m CPU, is not adopted and is
not in the code: git grep -iE "system_reserved|SystemReserved" over ce at the
pinned commit finds nothing in Go or in yaml. One mechanism, and this is its
name.
Four services install in order. Each authors only limits.
| service | authored limit | quota | CPU request (25%) | memory request (80%) |
|---|---|---|---|---|
| web | 2 CPU, 2 GiB | 2, 2 GiB | 500m | 1.6 GiB |
| db | 1 CPU, 1 GiB | 1, 1 GiB | 250m | 0.8 GiB |
| cache | 0.5 CPU, 0.5 GiB | 0.5, 0.5 GiB | 125m | 0.4 GiB |
| gitlab | 4 CPU, 4 GiB | 4, 4 GiB | 1000m | 3.2 GiB |
After web, db and cache: CPU 500 + 250 + 125 = 875m of 3800m. Memory 1.6 + 0.8 + 0.4 = 2.8 GiB of 7.4, free 4.6 GiB.
gitlab: CPU 875 + 1000 = 1875m, fits. Memory needs 3.2 GiB and 4.6 is free, so it binds. All four run. Memory limits on the node now sum to 2 + 1 + 0.5 + 4 = 7.5 GiB against 7.4 GiB allocatable, so the node is oversold by 0.1 GiB and section 4’s node-saving stop is armed. CPU limits sum to 7.5 cores on 4.
This one node is the whole 80% trade. At a 100% memory request the same four apps give requests of 2.0 + 1.0 + 0.5 = 3.5 GiB, gitlab needs 4.0 with 3.9 free, and it is pending, class room, short 0.1 GiB, with limits summing to 3.5 GiB and no burst able to exhaust the node. 80% is exactly the difference between a fourth app running and a fourth app waiting, bought for 0.1 GiB of oversell and an armed stop. Clay chose that on 2026-11-11.
Today all four bind: each placement request is zero (ce modules/app/replica_projects.go:524-531) and the rendered limits are 1.5 x 7.5 = 11.25 GiB on an 8 GiB node (ce modules/app/resource_quota.go:203).
The dial goes further in either direction with one key. shc config set -e dev deploy.memory_request 100% puts this environment back to no oversell, and gitlab waits. shc config set -e dev deploy.memory_request 50%: requests become 1.0 + 0.5 + 0.25 + 2.0 = 3.75 GiB of 7.4, room for a fifth and sixth app, and limits well past allocatable. The operator chooses, in one named place, and shc doctor prints the resulting ratio.
The FZ-10385 run by the same arithmetic: about 29 GB of limits on one 8 GB node (brief), which at an 80% request is 23.2 GB of requests. Each node takes 7.4 GB of requests. With a cap of 2 servers: 2 x 7.4 = 14.8 GB of requests bound, 8.4 GB pending, and the portal records at_cap. Both nodes stay reachable. Fitting all of it needs 23.2 / 7.4 = 3.14, so 4 servers.
Example 2: a catalog app that authors only limits
Section titled “Example 2: a catalog app that authors only limits”A service authors limits: {cpus: "1", memory: 1G} and no reservation.
- Today: rendered limit 1.5G, rendered reservation 1G, placement request 0 (ce modules/app/resource_quota.go:203, ce modules/app/replica_projects.go:524-531).
- Target: quota 1 CPU and 1G. Memory request 1G x 80% = 819M, memory limit 1G. CPU request 1000m x 25% = 250m, CPU limit 1 core. The container is capped at the number its packager wrote. The app needs no edit.
The node from example 1 holds 7.4 / 0.8 = 9 by memory and 3800 / 250 = 15 by CPU, so 9. Limits then sum to 9 GiB on 7.4 GiB allocatable: 1.6 GiB oversold, two more replicas than a 100% request would have placed. With deploy.memory_request 50% on the environment: request 512M, limit still 1G, 7.4 / 0.5 = 14 by memory and 15 by CPU, so 14. Limits sum to 14 GiB on 7.4. That 6.6 GiB is the risk the operator bought on top of the 1.6 the default already carries.
Three more cases. An app that authors reservation 512M and limit 2G gets quota 2G. An app that authors only a 512M reservation gets quota 512M. An app that authors nothing gets 1Gi and 1 CPU, enforced.
Example 3: an app with a GPU
Section titled “Example 3: an app with a GPU”Node: 16 cores, 64 GiB, 2 NVIDIA GPUs. Allocatable after 200m and 0.6 GiB: 15800m, 63.4 GiB, 2 devices.
Service worker authors reservations.devices: [{driver: nvidia, count: 1, capabilities: [gpu]}] and limits: {cpus: "4", memory: 16G}. Installed with --scale worker=3.
Per replica: GPU 1, CPU request 4000m x 25% = 1000m, memory request 16 GiB x 80% = 12.8 GiB.
- Replica 1: GPU 1 of 2, CPU 1000m, memory 12.8 GiB. Bound.
- Replica 2: GPU 2 of 2, CPU 2000m, memory 25.6 GiB. Bound.
- Replica 3: CPU 3000m of 15800m fits, memory 38.4 of 63.4 GiB fits, GPU needs 1 and 0 are free. Pending, class
room, asknvidia: 1, 12.8 GiB, 1000m. GPU is the only thing stopping it, which is the point of counting devices.
The portal looks for an allowed size with at least 1 NVIDIA device and 12.8 + 0.6 = 13.4 GiB. With none in the list it records no_size and buys nothing.
Today all three bind, because GPU fit is kind presence with no count (ce core/placement/needs.go:143-150).
3. Placement and pending
Section titled “3. Placement and pending”One engine. placement.Engine serves install, update, scale, drain, failover and volume provisioning (ce core/placement/engine.go:60-65). The target keeps it and removes its second opinion.
What the fit counts. Per candidate node, the sum of requests of every replica bound there, against allocatable:
- CPU and memory requests. Allocatable is today’s expression (ce modules/scheduler/scheduler.go:1120-1129).
- GPU: free devices of the asked kind.
- Disk: images to pull, scratch requests and node-disk volume requests against the node’s pool, extending today’s image and sized-volume charge (ce core/placement/place.go:345-368).
- Fabric address room, as today.
What changes:
serviceReservestops zeroing (ce modules/app/replica_projects.go:524-531). The request is always quota x request%.- The commitment ledger’s steer and the bind-anyway as
OverCapacitygo (ce core/placement/place.go:425-454), withledgerShortandawaitLedger(ce modules/fence/admit_ledger_room.go:21-62). There is one number. - A node whose capacity is unknown takes no new bind, reason
capacity_unknown. Today unknown fits everything (ce core/placement/place.go:798-807). - A bound replica keeps its node across an update. Today it stays only if its new spec passes the fit there (ce core/placement/place.go:387-388), and rollout is stop-first (ce core/placement/rollout.go:46). Without this rule, the first update of a limits-only app on a packed node stops the old replica and then finds no room for the new one. NEW: an update whose rules still allow the node never unbinds for room. The node is marked over-requested, takes no new bind, and
shc doctornames it.
The memory mark (95% demand, clear under 90% for 2 minutes, ce core/placement/memory.go:14-18) and the disk mark (80% used and under 100 GiB free, ce core/placement/disk_room.go:29-32) stay as filters. They read live node values.
Ranking. Start-failure nodes last, fewest sharings, least soft-spread excess, most --prefer hits, then NEW: a base node before an autoscale node, then the scorer, then the lowest node id. The scorer is set once at engine construction, so failover lands where an install would. Today failover passes none and takes the lowest id (shc subsystems/failover_pass.go:221, ce core/placement/place.go:758-765). The base-first term is what lets bought servers empty: a spread-only ranking sends every new replica to the emptiest node, which is the server just bought, and it never reaches zero. BinPackWeights has no caller (shc modules/placement/placement.go:182) and is deleted.
Replica spread. Unchanged. Replicas of a service prefer separate nodes. Group members prefer separate nodes and can require it (ce core/placement/engine.go:242-247).
Pending. A replica state in the assignment store beside the seven that exist (ce modules/scheduler/scheduler.go:36-44). The record holds the rendered spec, the ask per resource, one reason per node from the engine’s trace, a class, the closest node and its shortfall, since, and the leader epoch. Because it holds the spec and not the install request, an install that carries a credential is held like any other. Today those are never held (ce modules/app/install_capacity.go:16-24).
Two classes:
room: a new empty node inside the replica’s rules would take it. The engine decides this by running its filters, without the fit, against a synthetic node built from the rules (location, architecture, runtime, GPU kind). It needs no size list. Whether any size is big enough is the portal’s question.rule: no such node would. A volume’s reach, a runtime, or a rule that names nodes that are all excluded. The portal never buys for it.
Refuse or pend: a verb whose rules match no node that exists in the cluster, in any state, is refused at once with X409171FNCEMP (ce modules/fence/admit.go:46-48). That is a typing error. Rules that match at least one node, none of which can take the replica now, give pending.
An install, update or --scale change with a pending replica is accepted and answers with the pending code and the reason. The deployment reads pending when no replica is bound and degraded when some are. Accepting writes the deployment and its specs only. Volumes, addresses and DNS records are created at the first bind, so uninstalling a pending deployment removes rows and nothing else.
This replaces the whole-install hold and its replay (ce modules/fence/admit_await.go:25, ce modules/capacity/capacity.go:280), the failover hold word pending, and the refusal of an update or scale-up for room (ce modules/fence/admit_service.go:383-386).
Who retries, when. The leader’s placement pass. It runs when a node becomes ready, a replica stops or is removed, capacity changes, a quota or percentage changes, storage grows, and once a minute (today’s tick, ce modules/capacity/leader.go:20-29). Order: system apps, then oldest first. Each bind is the store’s atomic batch that re-checks fit (ce modules/scheduler/scheduler.go:5-9), so two asks racing for the last room give one bound and one pending. At most 6 binds a minute through the freeze limiter (ce modules/freeze/freeze.go:158-160), and none while the control plane is frozen.
To stop a large pending replica starving behind small ones: the oldest room replica holds its closest node. No younger replica binds there until it binds or is removed. One hold per cluster.
Pending never expires. The 90 minute drop (ce modules/capacity/capacity.go:33-35) goes. A cluster condition turns red after 10 minutes. The operator cancels by uninstalling or lowering --scale.
Priorities. Four exist: low, normal, high, system (ce core/placement/engine.go:167-172), and low and high are parsed by name (ce core/placement/priority.go:31-33). In the target only system changes anything: system apps are placed first and die last. Nothing preempts. Low and high rank nothing, so they should be deleted. What reads them today was not traced.
4. Killing
Section titled “4. Killing”| who decides | when | what happens |
|---|---|---|
| the kernel | a container passes its memory limit | that container is killed. The node agent restarts it in place, 100 ms doubling to 1 minute (ce modules/nodeagent/supervisor.go:12-16). It never moves: another node would kill it at the same limit |
| the runtime, the storage | CPU limit, disk limit | CPU is slowed. The write fails. Nothing is stopped |
| the node agent | the node passes its memory mark while limits on it sum to more than allocatable | stops the one replica furthest over its request, then restarts it in place with the same backoff |
| the kernel, as the backstop | the node runs out before the agent acts | kills by a score the daemon wrote at start |
| the daemon | a drain, an autoscale release of an empty node, a lost lease on a single writer (ce modules/watchdog/watchdog.go:194-248) | stops gracefully, inside the disruption setting |
Limits contain, requests rank. At its own limit a container is the kernel’s business. To save a node, limits can’t choose: two replicas with a 4 GiB limit and a 2 GiB request, one using 3.9 GiB and one 2.1, are both under their limit. The order is by use over request: 1.9 GiB over goes before 0.1 GiB over.
Why the node agent and not only the kernel. The kernel scores processes, not containers. On an 8 GiB node, replica A is one process, request 4 GiB, using 3.9. Replica B is 20 processes of 0.3 GiB, request 2 GiB, using 6. With oom_score_adj = 1000 - 1000 x request / node memory, A’s process scores 500 + 487 = 987 and each of B’s scores 750 + 37 = 787. The kernel kills A, which is inside its request, and spares B, which is 4 GiB over. So the kernel alone gets multi-process replicas wrong. The node agent reads each replica’s cgroup memory.current, a property read, and picks by the whole replica. It acts on any node where a limit sits above a request, which at the 80% memory default is every node. At 100 and 100 it would never arm; that is what the 80% ruling traded away, deliberately.
The backstop. A spike can beat any loop. So every replica also gets, at render: memory.oom.group = 1, so a kernel kill takes the whole container and the restart is clean (rg "oom.group|oom_group" ce/core ce/modules: not found today), and oom_score_adj = 1000 - 1000 x request / node memory, clamped to 2..999, system apps -900. oom_score_adj is honoured on every runtime but apple (ce core/composenorm/keys.go:442). On incus the request is already written as memory.low (ce core/executor/incus/translate.go:2350-2352). The incus exec helper writes a planned oom_score_adj today (ce core/executor/incus/execenv/main.go:815). Where that plan’s value comes from was not traced.
At defaults. Limits sum to 1.25 x allocatable, so a node can run out of memory it counted: every replica inside its own limit, all peaking at once, is 25% more than the node has. That is the case the node agent exists for, and at 80% it is a real case rather than a theoretical one. A node can still also run out from memory SHC didn’t count: outside-SHC use that grew after the bind, tmpfs, shcd. The operator keep is the setting for that. shc doctor shows outside use above the keep, and the node’s committed-to-allocatable ratio beside it.
After a stop. Restart in place. A node-memory stop is never counted as a start failure and never moves the replica: moving for memory pressure is churn. The kernel’s two kinds of kill are told apart from the cgroup’s memory.events: oom counts hits on the container’s own limit, oom_kill counts kills of any kind. Today the classifier has one cause, oom (ce modules/app/container_exit_cause.go:58), so this is new code. Status says “killed at its 1Gi limit, 3 times” or “stopped: node out of memory”.
Never stopped for pressure: system apps, shcd, and by the node agent any replica inside its request.
The disruption setting. deploy.disruption, scoped config, set at any workload tier.
rolling(default): a voluntary mover takes one replica of a service at a time and waits for it to be ready elsewhere.never: no voluntary mover touches it. A drain reports the node blocked and names the deployment.shc cluster drain --forceoverrides, and then stops these deployments last, after everything else has left the node.
Voluntary movers: drain, the separation pass, rebalance. Not governed: a lost node, a kernel kill, and the deployment’s own update or rule change, which the operator asked for and which roll one at a time already (ce core/placement/rollout.go:46). Drain moves already ask the freeze gate (ce daemon/services_extra.go:2814-2819). NEW: they also pass its 6 a minute limiter, so a node with 30 deployments empties over 5 minutes and not at once. Rebalance never runs on its own (shc modules/rebalance/rebalance.go:1-5).
The unit is the deployment because that is the narrowest scope tier (ce core/scope/scope.go:61). It replaces the compose key x-shc-disruption.max_unavailable, which only drain reads and where only 0 matters (shc modules/cluster/drain_disruption.go:74-79). No catalog compose sets it (rg -l x-shc-disruption shc/apps: 0). An app doesn’t set policy.
Voluntary disruption, and what a pod disruption budget is here
Section titled “Voluntary disruption, and what a pod disruption budget is here”Clay asked whether disruption budgets are baked or configurable. Both, split: the mechanism is baked, the value is the operator’s, and an app cannot set it.
A Kubernetes PDB exists because a cluster moves pods on its own. SHC does not. There is no move verb, automatic moves freeze under pressure, and nothing is consolidated, ever. So the only voluntary disruptions in this design are an update roll and a drain, and a budget over those two is the whole need.
That is why the setting is two values and not a number:
rolling, the default: one replica at a time, the next not touched until the last is back and healthy. This ismaxUnavailable: 1by another name, and for a deployment of N replicas it is the same guarantee.never: none of the cluster’s own movers touches it, and a drain of a node carrying one of these stops and names the deployment. The operator’s own update is not governed: it rolls as any update does, and it is how the setting is changed back.
Set per deployment, by the operator only. A package cannot seed it, because a disruption budget is a statement about how much the owner of the data can lose and the packager does not know that. There is no minAvailable: 3, because with no move verb the only thing that could violate it is a roll, and rolling already bounds a roll at one.
5. Disk
Section titled “5. Disk”| storage | request counts against | limit enforced by | if it can’t be enforced |
|---|---|---|---|
| cloud block volume | the provider. Request = limit = size | the device size | always enforced |
| zfs dataset, btrfs subvolume, LINSTOR | the pool’s free bytes | the store’s own quota | always enforced |
| node disk on xfs with prjquota | the disk’s free bytes | xfs project quota | always enforced |
| node disk on ext4 or a plain directory | the disk’s free bytes | nothing | status prints “limit not enforced: ext4” |
| nfs, cephfs | nothing on the node | the share’s quota if it has one | printed as not enforced |
| scratch (writable layer, temporary volumes) | the node’s pool | docker and podman storage_opt where the driver takes it | not enforced on incus and apple today (ce core/composenorm/keys.go:465), printed |
| images | the node’s pool, as today (ce core/placement/place.go:345-368) | not applicable | not applicable |
The enforced set is what runs today, and each result already says hard or soft and why (shc modules/quota/disk_cap.go:100-193).
No scan, ever. The limit is enforced by the storage or it isn’t enforced. The request reserves by declared size. Usage is read only where it is a property read. The hourly du walk on ext4 (shc modules/quota/crons.go:18-19) stays a tenant accounting job and feeds no placement or stop decision.
Request below 100%. Allowed as an operator’s dial on a store that allocates on write. A block volume ignores it: the device is allocated whole.
Over the limit. The write fails. Nothing is stopped and nothing moves, for volumes and for scratch. A full volume moved is a full volume on another node. Stopping a replica for scratch would need its usage, and overlay scratch without storage_opt has no usage read short of a scan.
Disk pressure is about the pool, not the replica.
- Past the mark, the node takes no new bind (ce core/placement/disk_room.go:29-32). Selfclean removes images and leftovers, never a volume (ce modules/selfclean/selfclean.go:1-37).
- NEW: the node enters the capacity document’s
disk_shortlist, and at 90% a condition turns red. - Nothing running is stopped.
The fix is more storage on that node, and a person does it. The portal has no path to attach storage: its grow path queues only scale_up of servers (shc frappe/apps/portal/portal/autoscale.py:219-236), and an attach path was not found. So a disk shortfall on a node is reported, not bought for. A new replica that is pending for disk is class room and can get a new server.
The honest limit: on a pool with no native quotas one volume can still fill the disk, and if shcd’s state shares that disk the node suffers. The guarantee needs zfs, btrfs or xfs with project quotas. shc doctor says which each node has.
Growth. shc update --storage vol=size grows in place and never shrinks (ce modules/app/volume_grow.go:3-8). Nothing grows by itself.
When data moves between nodes.
| the replica mounts | cost | who may start it |
|---|---|---|
| nothing, or scratch only | free. Scratch is recreated empty (ce core/placement/durability.go:38-41) | any mover |
| a shared volume | stop, start | any mover |
| a block volume that reaches the target | stop, detach, attach, start (ce modules/clone/move_strategy.go:1-58) | failover and the start-failure move. Not separation or rebalance |
| node disk, or block that reaches no other node | copy: stop, tar, send, unpack (ce daemon/clone_replica.go:680-713) | an operator only |
| another provider | backup and restore | an operator only |
The rule: data is copied between nodes only when an operator asked for a change that requires it. Two asks exist: changing the deployment’s rules so its replica is outside them, and draining its node. Both print the cost first: bytes, the downtime estimate, and spool room, which is checked before anything stops (ce modules/clone/preflight_space.go:3-40). A drain copies one volume at a time. Failover, disk pressure, autoscale release, rebalance and separation never copy. A replica with local data on a lost node waits for the node (ce core/placement/movability.go:13-44).
Worked cost: 100 GiB at 100 MiB/s is 1,024 s a pass. Tar, send and unpack are three passes: about 51 minutes down.
6. GPU
Section titled “6. GPU”- Quota. A whole number of devices of one kind per replica. Seeded from the compose
reservations.devicescount, or--gpu-quota worker=nvidia:2. Default 0.count: allmeans every device of that kind on the node, so the fit needs a node with none in use. - Request and limit. Both equal the quota. There are no GPU percentage keys.
- Counting. NEW: a node reports its devices, kind and id. Today it reports kinds only and the fit is presence, so two GPU services can land on one device (ce core/placement/engine.go:86-90, ce core/placement/needs.go:143-150).
- Binding. The leader’s atomic bind names the device ids, so two replicas never share one.
- Per runtime. Docker reads the device reservation. Podman and incus get CDI devices. Apple has none and reports zero devices.
limits.devicesis refused today “until a device fence exists” (ce core/composenorm/keys.go:489-492): that is a gap in the code and this closes it. - No sharing. One device, one replica. No time slicing and no fractions.
- Autoscale. The pending ask carries
gpu: {kind, count}. The portal buys only a size that has that kind.
7. Failure table
Section titled “7. Failure table”| failure | state shown | retry | who sees it | what unblocks it |
|---|---|---|---|---|
| cannot place, room | replica pending, class room, ask and shortfall | leader pass on each capacity event and each minute, no expiry | the verb’s output, shc status, shc ls, red condition at 10 minutes, the capacity document, the portal | a node with room, a smaller quota or request, a replica leaving |
| cannot place, rule | pending, class rule, the rule or volume named | same | same. The portal never buys | the operator changes --where, --scale or the volume |
| cannot start | failed, with the start error | agent backoff. 5 failed starts in 5 minutes penalise the node for that spec and the replica rebinds elsewhere if one fits (ce core/placement/startfailure.go:16-19) | shc status, health | a fixed image or config. An update clears the penalty |
| out of memory at its own limit | degraded, “killed at its 1Gi limit, 3 times” | restart in place, 100 ms to 1 minute. Never moved | shc status, health | a larger --memory-quota |
| node full, memory | the stopped replica reads “stopped: node out of memory”. The node takes no new bind at the mark (ce core/placement/memory.go:14-18) | restart in place | shc status, shc doctor | a higher memory request percentage on that scope, or a larger keep |
| node full, disk | node in disk_short, new binds refused with disk_room | selfclean. None for volumes | shc doctor, the document, the portal banner | storage added to that node, or volumes removed |
| cannot stop | replica stopping. Stop timeout 10 s, kill 5 s later (ce modules/watchdog/watchdog.go:225-248) | the node agent retries each reconcile. The mover that asked doesn’t start its target. A drain stays partially_drained (shc modules/cluster/drain.go:811-814) and the node isn’t released | the drain or update output, shc status, the document’s blocked_by | the runtime answers, or the node is rebooted |
| cannot move | the replica is back on its source. The source copy is kept until commit (ce modules/scheduler/move_incarnations.go:3-9) | none by itself. A re-run goes forward, --rollback goes back (relocation design) | the verb’s output, shc status | room or spool on the target, then a re-run |
| node lost | ”node unreachable since”. Failover after a grace of 60 s for stateless replicas and 5 minutes for group members (ce core/placement/grace.go:41-42). A replica with local data waits. A replica with no target is pending | each scheduler tick, behind the freeze gate | shc ls, shc status | the node returns, or a node with room and reach exists |
| portal unreachable | cluster condition “portal unreachable since”. Pending stays pending | heartbeat retries. Nothing is bought, cordoned or deleted | shc doctor, shc cluster status | the portal answers. Nothing running depends on it |
| the provider refuses a server | pending stays. The portal’s last answer is shown on the pending replica: at_cap, provider_refused, no_size, licence, bought_no_fit | one cooldown after a failed add (shc frappe/apps/portal/portal/autoscale.py:164-177). The refused name is reused so no second server is billed | the portal banner, shc status | a raised cap, a fixed account, an allowed size |
8. Autoscaling
Section titled “8. Autoscaling”Base node and bought server. A base node is a server the operator put in the cluster. It joined because a person ran a join, and it leaves when a person removes it. A bought server is one the portal’s autoscaler created by itself, because a replica was pending with nowhere to go, and it is the autoscaler’s to delete again once it is empty. Same software, same membership, same everything a replica can see. The only difference is who decided it should exist, and therefore who is allowed to make it stop existing.
One thing turns on that difference: whether something irreplaceable may live on it. A volume on a base node is on a server nobody will take away without asking. A volume on a bought server is on a server the autoscaler is entitled to return, and if that is allowed then the server can never actually be returned, so the cluster keeps paying for it with no exit. That is what choice 5 is about.
The scheduler never calls a cloud. The cluster publishes one document in the heartbeat. It replaces today’s report, which carries one summed CPU and memory figure per held install and the single reason no_fit (ce modules/capacity/capacity.go:24, :40-49), none of which the portal reads as numbers (shc frappe/apps/portal/portal/autoscale.py:120-124).
capacity: version: 2 generated_at: RFC 3339 epoch: integer # leader epoch; an older one is ignored autoscale: bool reserve: {cpu_millis, memory_bytes, disk_bytes} # what an empty node loses to keep and system apps pending: # one per pending replica - id: tenant/environment/stack/service/index since: RFC 3339 class: room | rule together: string # replicas that must share a node ask: {cpu_millis, memory_bytes, disk_bytes, gpu: {kind, count}} needs: {location, arch, runtime} # what a new node must be short: {node, resource, amount} # the closest existing node portal: string # the portal's last answer, echoed back nodes: - node, group: base | autoscale, location state: ready | cordoned | drained allocatable: {cpu_millis, memory_bytes, disk_bytes, gpus: [{kind, count}]} requested: {cpu_millis, memory_bytes, disk_bytes, gpus: [{kind, count}]} holds: {replicas, volumes} empty_since: RFC 3339 | null blocked_by: [code] disk_short: - node, pool_bytes, used_bytes, requested_bytes, short_bytesUnits: millicores, bytes, whole devices, RFC 3339 times. All of it comes from the assignment store and node rows, none from the metrics store.
What adds a server. The portal, when: autoscale is on, the document is under 150 s old, a room entry is over 2 minutes old (today’s guards, shc frappe/apps/portal/portal/autoscale.py:25-29, :73-76), no add was made for that entry, the cluster is under “Most servers” (shc frappe/apps/portal/portal/caps.py:159-164), and the licence covers the cores of the chosen size. That last check extends the licence check that already runs before a buy (shc frappe/apps/portal/portal/autoscale.py:81-84). Cores are otherwise checked at join, after the server is billed, which is the FZ-10155 failure.
Which size. The cluster’s autoscale settings list allowed sizes. The list defaults to the provision’s own size, which is all the portal buys today (shc frappe/apps/portal/portal/clouds.py:302-304). The portal groups room entries by needs, takes the largest ask, and picks the cheapest allowed size whose capacity less reserve holds it. It fills that one server first-fit to know which entries it covers, and buys one server per tick. An ask no size holds is answered no_size and never bought for.
Worked: asks of 4.0, 2.0 and 1.0 GiB, reserve 0.6 GiB, sizes 4, 8 and 16 GiB. The largest needs 4.0 + 0.6 = 4.6 GiB, so 4 GiB is out. The 8 GiB size holds 7.4: 4.0 + 2.0 + 1.0 = 7.0 fits. One 8 GiB server.
If an entry is still pending after its server joined, the portal answers bought_no_fit and doesn’t buy for it again.
What removes one. Only an autoscale-group node that has been empty for the cooldown (30 minutes, ce modules/capacity/settings.go:20). NEW: empty means no replica of any tenant except global system replicas, and no volume, stopped replica or kept source copy. Today it means no replica outside the master tenant (ce modules/capacity/capacity.go:127-130), so a release can delete the last copy of a volume. The cluster cordons and drains the node, which refuses anything but global replicas (shc modules/cluster/autoscale.go:18-37), and reports drained. The portal deletes one drained node per tick (shc frappe/apps/portal/portal/autoscale.py:100-111), and only names in its autoscale set (shc frappe/apps/portal/portal/autoscale.py:107). Base servers are never removed.
Three new rules:
- No cordon while autoscale is off or the portal has been silent for 150 s. Today “off” makes every idle node drainable (ce modules/capacity/leader.go:81), and the leader never reads portal reachability.
- A cordoned or drained node the portal hasn’t deleted is uncordoned when a pending
roomreplica fits it. Today the cordon is one way (ce modules/capacity/leader.go:289-300). The cluster binds to it only after a heartbeat that reported itreadywas acknowledged, and the portal deletes only a node that its newest document lists asdrained. - Nothing is ever moved to empty a node. No consolidation, and no setting for it.
Limits on churn. One add per tick. The next add waits for a heartbeat after the last one landed plus 2 minutes (shc frappe/apps/portal/portal/autoscale.py:114-117). One add per pending entry. A node lives at least one cooldown. Base nodes rank first, so a bought server is used only when nothing else fits, and empties as replicas leave.
Limits on spend. “Most servers” and the licence check. When the size list is saved the portal shows the worst case per hour: most servers x the dearest allowed size. There is no money setting. The price is recorded per add (shc frappe/apps/portal/portal/autoscale.py:229-235) and stays a record.
Autoscaling off. The same document, autoscale: false. Pending replicas stay pending and visible. Nothing is bought and nothing is cordoned. A node the operator joins by hand unblocks them through the same pass.
Old clusters. The portal reads version 2 only. A cluster on an older build is shown “autoscale needs a cluster update” and gets no server. The portal ships first.
Credentials and trust. The cloud token and the per-provision SSH key stay in the portal (brief). The cluster holds no cloud credential. The cost: whoever controls the portal can create and delete servers in that account, so the token should be scoped to one project.
Money. The document is the same whether the portal buys on the customer’s own cloud account, as a paid feature, or sells its own capacity. Only the portal’s provider differs.
9. Who owns what
Section titled “9. Who owns what”| piece | owner | why |
|---|---|---|
| quota, percentage keys, default quota, the render of request and limit | ce | the resolver is ce (ce modules/app/resource_quota.go). Without it one node oversells |
| the engine, fit, base-first term, pending state, placement pass | ce | ce core/placement, ce modules/scheduler |
| the kernel settings, the node agent’s memory stop, GPU count and binding | ce | executors and the node agent are ce. One node needs them to stay alive |
| volume and scratch limits | shc keeps its limiter (shc modules/quota/disk_cap.go). ce modules/storage/quotas has no real caller (ce modules/storage/storageapi/types.go:227) and is deleted | one limiter |
deploy.disruption | ce key, read by shc’s drain | drain is shc (shc modules/cluster/drain.go) |
| building the capacity document | ce | ce modules/capacity builds today’s report |
| the scorer, failover, drain, separation, rebalance, cordon and uncordon | shc | they live there, and multi-node is the paid layer |
| the tenant budget and its sweep, licence cores | shc | the split doc names the quota module and the licence machinery as private (ce docs/internal/ce-ee/ce-ee-split.md:19, :48, :65) |
| the heartbeat and portal reachability | shc | shc modules/portal/wire |
| allowed sizes, size choice, buy, join, delete, caps, the cloud credential, banners | portal | only the portal holds a cloud account |
The split doc doesn’t settle the scheduler pieces: rg -i "autoscal|capacity|placement engine|scheduler" ce/docs/internal/ce-ee/ce-ee-split.md finds nothing. The table is argued from where the code sits. A ce node with no portal still places safely and shows pending. Buying servers is the paid part.
10. The twelve questions
Section titled “10. The twelve questions”- The quota. The size of the box for one replica: per service for CPU, memory, GPU and scratch, per volume for disk. Request and limit are percentages of it. It seeds from a flag, else the authored limit, else the authored reservation, else 1Gi, 1 CPU, 10Gi a volume, 2Gi scratch. The default can kill an app that needs more than 1Gi, in place, at its own limit, with the limit and its source printed. The other option is an uncapped service, which makes every neighbour’s request false.
- Scopes. The seven that exist (ce core/scope/scope.go:61). They carry percentages and
deploy.disruption, never quotas. Workload tiers beat place tiers, so node and location rows are fallbacks. The existing config machinery carries it. There is no service tier, so per-service values are flags. - What an operator types. Quota flags for five resources, request and limit flags for CPU and memory, eight percentage keys,
deploy.disruption, and the rules that exist. Four portal settings. Eight nouns against 11 or 14 Kubernetes names. - Placement. Filter as today, fit on requests for CPU, memory, GPU and disk, rank with base nodes first, tie to the lowest node id. Pending is a replica state in the assignment store with ask, per-node reasons and a class. The
roomentries in the capacity document are the autoscaler’s record. - Killing. The kernel contains a container at its memory limit. On an oversold node past its mark the node agent stops the replica furthest over its request, with the kernel as the backstop. CPU and disk never kill. System apps and replicas inside their request are never chosen. Nothing preempts.
- Disruption.
deploy.disruption:rolling(default) ornever, per deployment, at any workload scope. It governs drain, separation and rebalance. A forced drain stopsneverdeployments last. - Disk. Request and limit per volume and for scratch, all at 100% by default. Enforced by block size, zfs, btrfs, LINSTOR and xfs project quotas. Not enforced on ext4, plain directories, unquoted shares and incus scratch, and the output says so. Pressure is fixed in place by a person. Moves are free, re-attach or copy, and a copy happens only on an operator’s rule change or drain.
- GPU. A whole-device count per kind, reported by the node, counted in the fit, bound by id on the leader. Docker, podman and incus pass devices. Apple has none.
- Failure. Section 7. Each failure has one state, one retry owner and one unblock. A buy that doesn’t help is bounded by one add per pending entry and answered
bought_no_fit. - The autoscaler. The cluster publishes pending asks, node occupancy and disk shortfalls. The portal picks a size, buys one server per tick under a server cap and a licence check, and deletes only nodes the cluster drained empty. With the portal unreachable nothing is bought, cordoned or deleted. The credential stays in the portal. The document doesn’t depend on whose cloud account it is.
- Churn. No consolidation. A healthy replica moves only for an operator’s rule change or drain, and for the separation pass when a node joins (shc subsystems/separation_pass.go:26-31), which moves only replicas with no data or shared data. Rebalance runs only when asked (shc modules/rebalance/rebalance.go:1-5). Automatic movers pass the freeze gate and 6 a minute.
- The split. Section 9. What one node needs to stay alive is ce. Multi-node movers, tenant budgets and the heartbeat are shc. Buying is the portal.
11. Departures from what Clay said
Section titled “11. Departures from what Clay said”- RESOLVED 2026-11-11, in the design’s favour. “whatever the request is in the compose file that by default is your quota … and if that’s not specified then if there’s a limit there then by default that becomes the quota”. Clay has since ruled the other way himself: “the limit is the quota … if you only set request in your docker compose file it actually will treat it as the quota”. That is exactly what this says. The limit seeds first. An app with reservation 512M and limit 2G would get a 512M box and die at 512M while its packager allowed 2G. Reversing through the percentage (quota = request / request%) equals the request at 100% and saves nothing.
- “limits are what are used when it needs to kill something”. True for a container at its own limit. To save a node the order uses requests, because every replica there is under its limit.
- “anything on this note it’s gonna be at this percentage”. Supported, as a fallback. A tenant, environment, stack or deployment row beats it. If the node row won, a prod tenant at 100% would be oversold because of where it landed.
- “limits and request … For both CPU disc GPU And memory”. GPU has a quota only. Half a device is not something a runtime hands out. A block volume has request = limit = size, because the device is allocated whole.
- “set a limit or a request not as a percentage but as an explicit value in which case the quota for that thing is essentially ignored”. Ignored for that figure, not for the budget, which charges the larger of the two. And it is a per-service flag at deploy, not a scoped key, because the narrowest scope is a whole deployment.
- “if nothing then there’s just kind of a same default”. Kept, with its cost: the 1Gi default kills an app that needs more, in place, with the cause printed.
- “you’re not going to unnecessarily just kind of consolidate things all the time”. Narrowed to never, with no setting. A consolidation switch can be added later without changing the model.
- “it’s gonna do that one last”. Kept as an order inside a forced drain, not as a third value.
- “that’s when auto scaler would actually go create some resources”. True for a pending replica. A disk that is short on one node is reported and not bought for: the portal has no attach path.
- “unless I guess it’s temporary or a femoral disc then maybe you would” kill at the disk limit. Not done. The write fails. Stopping for scratch needs a usage read that overlay storage doesn’t give without a scan.
- “when it runs out of memory … so that the node itself doesn’t get taken down”. Kept, with the node agent choosing and the kernel as backstop, because the kernel alone ranks processes and gets multi-process apps wrong.
Departures from the SHC-1058 scheduler design
Section titled “Departures from the SHC-1058 scheduler design”The mechanics design settled seventeen open questions by agent on Clay’s standing delegation on 2026-09-30 and recorded them as reversible. This design departs from it in three places: its decisions S5 and S6, and the rule stated in its 3.3.
- Unknown capacity. Its 3.3 keeps a node whose
memory_bytesis NULL as a candidate, never read as zero, and the ce comment agrees that a node with no known capacity fits. Section 3 has such a node take no new bind, reasoncapacity_unknown. The old rule was free while placement requests were zeroed: nothing was being fitted, so letting the node through cost nothing. Once requests are counted, a node of unknown capacity that fits everything is a hole that silently oversells every time it is chosen. Batch 2 causes this reversal, not a preference. - Priority. Its 3.7 and S5 give four priorities, ordering reschedule after a node is lost, drain moves and pending retry. This design keeps only
system: pending retries system first and then oldest first, drain order comes fromdeploy.disruption, and low and high rank nothing (sections 3 and 14, item 2). That is this design narrowing a reversible agent decision, not an owner ruling, and what readslowandhightoday is still untraced. - Node-pressure eviction. Its 3.20 and S6 say no eviction: a pressured node takes no new work and the kernel’s OOM killer enforces memory limits. Section 4 has the node agent stop the replica furthest over its request on an oversold node, with the kernel as the backstop. Clay asked for that himself on 2026-11-11, “when it runs out of memory … so that the node itself doesn’t get taken down” (item 11 above), which reverses S6. The narrowing still holds: it is a restart in place and never a move, so it is not kubelet eviction. And his choice 7 put the memory request at 80%, so the stop is armed on every node rather than on none.
12. The decisions and how they were made
Section titled “12. The decisions and how they were made”All seven answered from the phone, 2026-11-11, 04:06Z to 04:45Z. He picked the recommendation every time. Choice 5 is the only provisional one: he did not know the terms when he answered it, so the explanation now in section 8 is owed back to him before it counts as settled.
| # | Pick | Settled |
|---|---|---|
| 1 | (a) limit 100% of the quota | yes |
| 2 | (a) CPU request 25% | yes |
| 3 | (a) 1Gi and 1 CPU, enforced | yes |
| 4 | (a) rolling or never, per deployment, operator only | yes |
| 5 | (b) base nodes only | provisional |
| 6 | (a) rename the tenant cap to budget.* | yes, and “no backwards compatibility please” |
| 7 | memory request 80% (rv_f09f4a1bbcc1a5c4) | yes |
Choice 2’s note went further than the number and became the bound in section 2.
- Default memory limit. (a) 100% of the quota: a node can’t be oversold at defaults, and an app living between 1.0 and 1.5 of its authored limit is now killed at that limit. (b) 150%, today’s value, raised from 1.0 on purpose (ce modules/app/resource_quota.go:208-212): no app changes behaviour, and a full node carries 50% more limits than memory, so the node-saving stop is live on every cluster. Recommend (a), with a list of services whose peak sat above their authored limit handed to a person first.
- Default CPU request. (a) 25%: 4 to 1 sharing, memory stays the reason servers are bought. (b) 50%: 2 to 1, about half the CPU density. (c) 100%: no sharing, and CPU limits written as loose ceilings buy servers. Recommend (a). CPU shortage slows and never kills.
- A service that authors nothing. (a) 1Gi and 1 CPU, enforced: fewer kill loops on unknown compose files, 7 such services per 8 GiB node. (b) 512M and 0.5 CPU: 14 per node, more kills. (c) refuse the install until a quota is typed: no surprise, and a compose file from outside the catalog doesn’t install unedited. Recommend (a). 12 of 114 catalog compose files author no limit: the eleven
external-*connectors ands3, all underapps/standalone(shc dev72ba2060,git grep -L 'limits:' -- 'apps/*compose.yaml'). They take this default with no new catalog gate. - The disruption setting. (a)
rollingornever, per deployment, set by the operator only. (b) a number per service, default 1, 0 for never, which an app can seed: finer, and it lets a package set policy. (c) (a) plus a time window: covers “only at night” and adds a clock grammar. Recommend (a). - Local durable data on bought servers. (a) allowed: that server is never released, and the document says why. (b) such replicas bind only to base nodes and are pending, class
rule, otherwise: every bought server can be returned. Recommend (b). A server that can never be released is spend with no exit. - Two things called quota. (a) rename the tenant cap
quota.*tobudget.*andshc quotatoshc budget: one meaning per word. (b) keep both: no rename, and “quota” means a service’s box in one place and a tenant’s cap in another. Recommend (a). The no-compat ruling makes the rename a delete and an add. - Default memory request. (a) 80%: a node fits about a quarter more than it holds, commitment lands near 1.25x, and a node absorbs one app peaking without killing anything. (b) 90%: commitment near 1.11x, the safest number that is still compression, and so small a gain that anyone chasing density overrides it and the real number ends up in a hundred deployfiles. (c) 100%: reserved equals committed, nothing is ever oversold and nothing is ever killed for memory it was promised, at the cost of stranded capacity on every node. Recommend (a). Answered (a), against the compression visualisation prepared for this decision.
The option texts above are what was put to him and are left unedited as the record. One of them has since gone stale by his own later answer: choice 1(a) says “a node can’t be oversold at defaults”, which was true while the memory request was still assumed to be 100%. Choice 7 set it to 80%, so a full node is oversold by a quarter at defaults and section 4’s stop is armed everywhere. Sections 2 and 4 carry the current statement.
13. Getting there
Section titled “13. Getting there”Each batch lands whole and deletes what it replaces.
- Pending, the document and the portal, together. The pending replica state, the placement pass, the version 2 document, the portal reading it and choosing a size, base-first ranking, the new meaning of empty, no cordon when off or silent, uncordon. Delete the install hold and replay, X409197CAPAWT, X409198CAPDRP, the failover word
pending, and the refusal of updates and scale-ups for room. The ce!3474 ledger check stays one batch longer and now produces a pendingroomreplica, so the portal is never blind. - Requests count. Delete the zeroing in
serviceReserve, the ledger steer,OverCapacity,ledgerShortandawaitLedger, with everyLedgerMemoryBytesreader: ce core/pod/scale.go, ce modules/fence/placement_capacity.go, ce modules/fence/explain.go, ce modules/fence/placement_inputs.go. Add the default quota and the bound-replica-keeps-its-node rule. Unknown capacity fits nowhere. Before this lands, every system app service gets an authored memory limit, so none meets the 1Gi default. - One key grammar. The eight percentage keys and the request and limit flags replace the four ratio keys. Delete scoped
deploy.cpu_quotaanddeploy.memory_quotaand the marker rung. Defaults move to choices 1, 2 and 7: memory 80 and 100, CPU 25 and 100. - Kill settings.
memory.oom.group, the score, the CPU request as a weight on every runtime, the node agent’s stop, the second exit cause. - Disk. Volume and scratch requests in the fit, default volume quota,
disk_short. Delete ce modules/storage/quotas. - GPU. Device report, count fit, id binding,
--gpu-quota. - Disruption.
deploy.disruptionreplacesx-shc-disruption.max_unavailable. Forced drain order. Drain through the limiter. One scorer for every caller. DeleteBinPackWeights.
Kept: the engine, the scope ladder, the atomic bind, the freeze gate, movability, both marks, selfclean, the heartbeat, the portal’s guards.
Catalog apps that author only limits. They need no edit. After batch 2 each service’s quota is its authored limit, its memory request is that limit and its CPU request a quarter of its CPU limit. They are counted for the first time. Nothing running is stopped or moved: a bound replica keeps its node across the re-render. Nodes whose requests now exceed capacity take no new bind, shc doctor names them, and the operator adds a node or lowers deploy.memory_request. After batch 3 with choice 1(a), a service that lived above its authored limit is killed at it, which is why the list goes to a person first.
14. What changed from the first answer
Section titled “14. What changed from the first answer”- “If memory’s request defaults to 100% of the quota, a node can’t be oversold”. The premise moved twice. It was false while the limit was 150%, today’s default: a node full at 7 GiB of requests carries 10.5 GiB of limits, so the limit default moves to 100%. Then Clay set the memory request to 80% on 2026-11-11, so a full node carries 1.25 x its allocatable in limits on purpose and CAN be oversold at defaults. The node-saving stop is armed everywhere rather than nowhere. That is the chosen trade, not an oversight.
- “whoever is furthest over its request goes first, lowest priority first”. The order by request stays. The kernel can’t apply it to multi-process replicas, so the node agent chooses and the kernel is the backstop. There are no priority classes beyond system.
- “One quota per service per resource … set at any scope”. Quotas aren’t scoped. Flag, app, built-in default. Scopes carry percentages only, as he said.
- “a workload scope beats a node scope”. Kept. The ladder already orders it so (ce core/scope/scope.go:61). The node row is a fallback.
- “The autoscaler’s answer to ‘disk is short on node X’ is ‘attach more storage to node X’”. The portal can’t attach storage. A node’s disk shortfall is reported for a person.
- “Copying data … only as a planned move someone asked for”. Kept and made exact: a rule change or a drain, cost printed first. There is no move verb.
- “move any time, only in a window, or never”. Two values, no window.
- “It publishes what’s missing … and what’s spare”. Kept and extended: what a new node must be, what an empty node loses, a class that says whether buying helps. Empty now counts volumes. A drained node can come back.
- “No consolidation by default”. Now never, with no setting.
- New since the first answer: pending as a stored state with no expiry, a default quota, GPU counting, bound replicas keeping their node through an update, and the portal choosing the size.