The storage constraint: what a move costs today, and what would make it cheap
Status: open design. Research only. Nothing here is built.
Lane 3 of the placement/rebalancing research. Research and design only. Nothing here is implemented; nothing here touches an estate.
Every claim is marked MEASURED (I ran it or read it off the source at the commit named), DERIVED (arithmetic on a measured number, with the arithmetic shown), INFERRED (a reading I could not close), or RULING-OWED (a decision that is the owner’s, not mine).
Bases: shc @ 80e0c4441, ce @ 3d2a0ee. Worktrees
/home/admin/selfhosted-cloud/reb2-storage (branch
docs/rebalancer-storage-lane3) and /home/admin/selfhosted-cloud/reb2-storage-ce
(detached at ce main).
0. The finding, in one page
Section titled “0. The finding, in one page”Placement is downstream of storage, and storage is downstream of a declaration that nobody has written.
SHC has built four separate storage mechanisms. Three of them have an empty declaration corpus, and the fourth is structurally undeclarable:
| mechanism | state of the code | declarations in the 99-entry catalogue |
|---|---|---|
x-shc-volumes, block-volume binding, fenced, 9 cloud providers | BUILT, wired end to end | 0 |
x-shc-scale.<svc>.backup, per-service replica window | BUILT, wired (daemon/backup_scaler.go) | 0 |
| shared filesystem (nfs / cephfs providers, link table, node provisioners) | BUILT | structurally 0: Volume.access_mode CHECK admits only `rwo |
hooks.pre_backup / post_backup, app quiesce bracket | BUILT, wired | 24 pre / 7 with a full bracket |
The consequence, and this is the number that matters:
MEASURED: by construction,
movetoday selectsbackup_restorefor every deployment on both estates, on every runtime. Therebindfast path has a real-world corpus of zero. Theephemeral_onlyfast path has a real-world corpus of five.
The move machinery is excellent and it is running in its slowest gear with the handbrake on, because the catalogue never told it anything about storage.
Recommendation (§6): do not build a new storage layer. Declare the one that
exists. x-shc-volumes on a small, argued subset of stateful apps (not all
74) is the cheapest thing that converts any move from a byte copy into an API
call. Shared filesystem is rejected on this substrate for a reason that is
arithmetic, not taste: on Hetzner, CephFS needs block volumes underneath it, so
it costs strictly more than block volumes and adds a cluster to operate.
1. (a) The cost of a move today, per class
Section titled “1. (a) The cost of a move today, per class”1.1 The three strategies, and which one fires
Section titled “1.1 The three strategies, and which one fires”ce/modules/clone/move_strategy.go::SelectMoveStrategy reduces a per-mount
classification to one of three strategies. MEASURED (source read):
any mount NotRebindable -> BACKUP_RESTORE (MOVE-6, fail-closed)no persistent mount && runtime==swarm -> EPHEMERAL_ONLYno persistent mount && runtime!=swarm && move -> BACKUP_RESTORE (MOVE-5)all mounts Rebindable && runtime==swarm -> REBINDany CrossProvider -> BACKUP_RESTORE (copy-then-swap)Two gates collapse this to a single answer on the live estates.
Gate 1: MOVE-5 kills every fast path on the compose runtime.
relocateBlocked := nodeChanging && !swarm. A node relocation is only a
placement re-render on swarm. MEASURED: on a compose-runtime node, every
move to another node is backup_restore, including whoami.
Gate 2: with no volume rows, every persistent mount classifies
NotRebindable. Trace through
ce/daemon/clone_move_adapters.go::ClassifyMountDecision for a mount with no
x-shc-volumes binding (i.e. all of them):
repo.AttachmentNodeFor(...)finds no row →volumeID="",currentNode="",location="",providerProven=false.providerfalls back to the deployment’s--storagestamp, defaulting to"local".targetStorageis empty (no--storageoverride) → the cross-provider arm is skipped.isSharedFSProvider("local")is false → the shared-FS arm is skipped. (And even if the operator stamped--storage nfs,!in.ProviderProvenreturns NotRebindable; the stamp is a claim, not evidence.)in.Location == ""→ the location-scoped block arm is skipped.- Final arm:
TargetNode == CurrentNode && CurrentNode != "";CurrentNodeis"", so this is false →MountClassNotRebindable.
MEASURED, deterministic, no live probe needed. With zero x-shc-volumes in
the catalogue there are zero volume rows, and with zero volume rows every
persistent mount on every app returns NotRebindable. The whole
runRebindMove orchestration, the explicit Rebind primitive, the lease, the
nine cloud volume providers, all correct, all unreachable.
1.2 The distribution across the catalogue
Section titled “1.2 The distribution across the catalogue”Method (reproducible; see §10): parsed all 99 apps/*/app.yaml +
compose.yaml (jinja stripped), classified every compose volume entry with a
Python port of ce/core/volume/resolve.go::DetectMountType, then applied
ce/daemon/clone_move_adapters.go::PersistentMountsFromCompose’s persistence
rule.
Cross-check: my classifier independently returns 74 of 99 stateful,
matching the number recorded from the earlier session and the
docs/rebalancer-fork count of apps declaring a volumes: block. Two
independent derivations agreeing is the reason I trust the rest of the table.
MEASURED: mount-type census over all 99 catalogue entries:
mount class (SHC’s own MountType) | count |
|---|---|
DATA (persistent; becomes <deployDir>/mounts/<n> or a swarm named volume) | 362 references / 211 distinct |
| FILE (regenerated at render) | 15 |
| RUNTIME (scratch) | 7 |
| SYSTEM (absolute host bind) | 7 |
| tmpfs / anonymous | 1 |
MEASURED: distinct DATA mounts per app:
| distinct DATA mounts | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 9 | 13 | 15 | 18 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| apps | 25 | 25 | 22 | 12 | 6 | 3 | 2 | 1 | 1 | 1 | 1 |
Heaviest: greenbone 18, wazuh 15, mailcow 13, matrix 9, pulp 6,
plane 6, gitlab 5, highlight 5, librechat 5.
MEASURED: the “cheap” 25 is not 25. Of the 25 entries with no DATA mount:
- 10 declare zero services: the nine
external-*adapters plusstandalone/s3. Nothing runs; they are plug-only stacks. They are not cheap to place, they are not placeable. - 8 are test fixtures (
fixtures/*,*/tests/apps/*-consumer). - 5 are real, placeable, stateless apps:
huly-nextcloud,nextcloud-picker,affine-rs,gitlab-huly,identity-broker, every one a single-service bridge or sidecar. - 2 remainder are fixture/consumer overlaps already counted.
MEASURED: the
ephemeral_onlyfast path, on a swarm estate, applies to 5 non-fixture apps. On a compose estate it applies to none.
1.3 What the heavy path actually costs
Section titled “1.3 What the heavy path actually costs”Phase order for Move, read off ce/modules/clone/service.go:537-686
(MEASURED):
preflightStagingSpace → backupPhase → stopPhase → measureSourceTrees → recoverPhase (ship + restore + bring-up) → verifyPhase → dataPhase → uninstallPhaseThe capture runs while the source is live (each app’s own quiesce bracket
runs inside it). The outage window is stop → recover completes.
Cost 1: the archive is written to the same disk as the data
Section titled “Cost 1: the archive is written to the same disk as the data”preflight_space.go: the operation stages a full archive under
FilesystemProvider.StagingDir() (= <StateHome>/staging) before anything
moves. MEASURED from the earlier binding-model analysis and confirmed in
this file: source and target must each hold dataset + archive simultaneously.
Combined with the MEASURED estate fact that no node on either estate has a
second disk (one 610.4 GB sda, sda1 ext4 at /, six nodes), this is a hard
ceiling: a deployment larger than roughly 45% of the node’s free disk cannot
be relocated at all. The gate refuses with X507632CLNSPC; margins are
estimate + estimate/10 + 64 MiB, doubled for a multi-member (--closure)
move because the per-member archives and the group .bkg coexist.
Cost 2: the wire format is base64 inside JSON
Section titled “Cost 2: the wire format is base64 inside JSON”ce/daemon/services_clone_remote.go::shipBakToPeer streams the archive to the
target as a sequence of clone.recover_chunk RPCs. Each chunk is
cloneRecoverChunkBytes = 2 MiB raw, base64-encoded into a JSON envelope,
POSTed synchronously, one round trip per chunk, capped by internal_rpc’s 4 MiB
body limit.
I reproduced exactly this shape (2 MiB read → base64 → json.Marshal → HTTP
POST → json.Unmarshal → base64 decode → WriteAt) against a loopback
server on this 48-core box with tmpfs on both ends (scratchpad/shipbench).
No network, no TLS, no disk seek, no tar, no restic. That makes the result a
ceiling, not an estimate.
MEASURED (5 runs, 512 MiB each):
| run | wire inflation | throughput |
|---|---|---|
| 1 | 1.33× | 69.8 MiB/s |
| 2 | 1.33× | 65.2 MiB/s |
| 3 | 1.33× | 68.8 MiB/s |
| 4 | 1.33× | 62.3 MiB/s |
5 (GOMAXPROCS=2) | 1.33× | 61.6 MiB/s |
Inflation is exactly 4/3 and deterministic. Throughput barely moves when cores are cut from 48 to 2; it is bound by the synchronous request-per-2-MiB round trip, not by CPU. That matters: it will not improve on bigger nodes, and it will degrade over a real mTLS link.
DERIVED: ceiling wall-time for the ship leg alone, at 65 MiB/s:
| dataset | ship leg (ceiling) |
|---|---|
| 1 GiB | 16 s |
| 8.44 GiB | 2 min 13 s |
| 100 GiB | 26 min |
| 250 GiB | 66 min |
The ship leg is one of four legs. Total bytes touched per move ≈ 3× the
dataset (read tree → write archive; read archive → ship; receive → write →
untar), before the two MeasureDeploymentTree walks the gates run.
Cost 3: the one real dataset size I could find
Section titled “Cost 3: the one real dataset size I could find”MEASURED (recorded in ce/modules/backup/phases.go, from a live capture):
nc on 2026-08-10 held maintenance:mode --on from 04:04:38 to 04:20:01
(15 min 23 s) around an 8.44 GB archive. That fix moved the un-quiesce
earlier so the backup window shrank; it does not shrink a move window,
because a move keeps the deployment down until the target is serving.
MEASURED (docs/proposals/2026-08-07-backup-schedules.md): openobserve
≈ 5 GB; timeouts.quiesce_hold defaults to 300 s, and a bracket that
overruns it escalates to a full stop-and-retake.
Cost 4: nine tenths of the catalogue’s byte cost is unmeasured, and I cannot measure it
Section titled “Cost 4: nine tenths of the catalogue’s byte cost is unmeasured, and I cannot measure it”I have two dataset sizes for 74 stateful apps. NAMED LIMIT: no estate
contact, so no du and no shc backup ls histogram. The honest ordinal
classification is below, and it is INFERRED from what each app stores, not
observed:
| class | apps (examples) | expected order |
|---|---|---|
| config-only | uptime-kuma, shlink, vaultwarden, kutt | MB |
| DB-and-index | most of the 74 | 100 MB – 10 GB |
| repo / registry / artifact | gitlab, gitea, pulp, minio, dependency-track | 10 – 500 GB |
| media | immich, jellyfin, peertube, nextcloud | 100 GB – TB |
What would close this, and it is one command per node: a du -sb over
<StateHome>/deployments/*/*/* on each node of both estates, which is exactly
what MeasureDeploymentTree already does. That is an estate action and is
therefore outside my mandate, flagged as owed work, not done.
1.4 Two defects the measurement surfaced
Section titled “1.4 Two defects the measurement surfaced”★★ D-1 (P1, latent): the data gate is blind on the swarm runtime.
dataPhase refuses to destroy a source unless the target’s deployment tree
holds ≥25% of the source’s bytes. MeasureDeploymentTree walks
app.DeploymentDir(...) (<StateHome>/deployments/<tenant>/<stack>/<env>)
and nothing else (MEASURED, ce/modules/clone/filesystem_local.go:443).
On the swarm runtime an unbound DATA mount is deliberately not rewritten
to a host bind; it stays a docker named volume
(ce/modules/app/install_runner.go:4088-4100), so its bytes live under
/var/lib/docker/volumes/, outside the tree the gate measures. Both sides
therefore measure small and similar, the ratio lands near 1.0, and the gate
passes with full marks, for a deployment whose real data it never looked at.
The gate’s own header names this class (“52 KB / 91 KB / 121 KB nightlies for
multi-gigabyte swarm stacks”) and it catches it only when the source tree is
large, i.e. on compose.
Mitigating (and a CORRECTION to the recorded ground truth): the named-volume
capture gap is CLOSED at ce HEAD. ce/modules/backup/namedvolume.go +
namedvolume_observed.go discover unbound named volumes, capture each under its
own post-resume pause bracket, refuse the whole backup on an unexplained empty
read (X500451SWMEMP), and record per-volume reads in the manifest. The
memory note “swarm backups are metadata only” is stale. So is the warning
string in SwarmNamedVolumeBackupWarnings (“its data will not travel with this
operation”), which still fires on every swarm move.
What is not closed is the gate. The capture may now be correct while the gate that licenses destroying the source still cannot tell.
PROPOSED fix (not implemented): give FilesystemProvider a second
measurement (the deployment’s named volumes, via the same SwarmVolumeCapturer
seam the capture already uses) and have measureSourceTrees / dataPhase sum
tree + named volumes. Uncertainty must stay a refusal, per the file’s own
disposition.
D-2 (P3, cosmetic-but-misleading): SwarmNamedVolumeBackupWarnings
contradicts namedvolume.go. One of them is wrong on every swarm move today.
2. (b) What would make the bytes cheap to move
Section titled “2. (b) What would make the bytes cheap to move”2.1 Candidate A: block volumes (x-shc-volumes)
Section titled “2.1 Candidate A: block volumes (x-shc-volumes)”What exists. Everything except the declaration. DaemonVolumeBinder.Bind
(MEASURED, ce/daemon/volume_binder.go:94-280) runs
EnsureVolume → Provision → relocate-if-attached-elsewhere → BumpLeaseAndAttach
(single monotonic lease writer) → provider.Attach → node-side EnsureMounted
(mkfs-if-empty, mount, fstab) → mark attached, and returns the host mountpoint
<deployDir>/mounts/<name>, byte-identical to the host-bind path, so
ApplyPrep, backup, restore and the compose render need no change.
RebindVolume is the explicit detach→lease-bump→attach→lease-gated-mount
primitive the move fast path already calls. Nine providers ship (hcloud,
aws-ebs, gcepd, azure-disk, cinder, dovolume, linode-volume, vultr-volume,
cloudstack-volume).
What a move becomes. stop → cloud detach → cloud attach → mount → re-render
→ start. Zero bytes on the network. The staging-disk ceiling of §1.3
disappears for the bound mounts. runRebindMove already rolls back
volume-by-volume on a failed target start.
What else it unlocks, already built:
ce/subsystems/impl/volume_failover.go: leader-only automatic failover-rebind. Off by default (storage.volumes.auto_failover), gated on a proof of old-node fencing (attachment lease expiry + ≥45 s dwell, or a STONITH confirm; the defaultNoopStonithAgentnever confirms), never crossesnode.labels["az"]. MEASURED: it fires for nothing today because zero apps declare a volume.ce/modules/app/swarm_placement.go::ApplyVolumePlacement: pins an RWO volume-backed service to its attachment node and clampsreplicas: 1. This is a placement input the scheduler would have to respect, not fight.
★★ The hard constraint nobody has costed. MEASURED (Hetzner docs, this session): a volume is 10 GB – 10 TB, single-attach, and a server takes at most 16 volumes.
That number is load-bearing and it kills the naive design:
- The catalogue declares 211 distinct DATA mounts. One block volume per
mount caps a node at 16 mounts.
greenbonealone declares 18 and would not fit on a node by itself. - DERIVED: bitspur
compose01runs 128 containers across dozens of deployments. Even at one volume per deployment (mount the volume once and put that deployment’s wholemounts/tree inside it) a node’s budget is 16 stateful deployments. The recorded promanager shape (31–33 deployments on one node) already exceeds it.
RULING-OWED R-3.1: block volumes are a per-deployment escape hatch for the few big stateful deployments, not a catalogue-wide storage layer. The design must pick the deployments, and the budget is 16 per node.
Money. MEASURED, with a caveat I will not paper over: third-party sources disagree on the current Hetzner volume price: €0.044/GB/month net (quoted as checked 2026-08-03) versus €0.057/GB/month (quoted as an April-2026 increase). RULING-OWED R-3.2: confirm against an actual invoice before any number is put in a plan. The shape of the bill at both ends:
| scenario | GB | €0.044 | €0.057 |
|---|---|---|---|
| 6 nodes × 1 × 100 GB deployment volume | 600 | €26/mo | €34/mo |
| 12 volumes × 100 GB (2 nodes at budget) | 1 200 | €53/mo | €68/mo |
| 96 volumes × 100 GB (6 nodes at the 16 cap) | 9 600 | €422/mo | €547/mo |
DERIVED: at €0.044, one 100 GB volume ≈ €4.40/month ≈ 10% of a cx53’s monthly cost. For the handful of deployments where a move currently means a 25-minute outage, that is cheap. At 96 volumes it is a second infrastructure bill, which is the argument for the budget in R-3.1.
Moving parts added: one x-shc-volumes block per declared app; a
storage.volumes.provider estate config; a hcloud connection (MEASURED
from the binding-model analysis: both estates already carry exactly one
volume-capable hetzner connection, so the substrate is present and
configuration-free); nothing new to operate. Backups, restore, render and the
sandbox are unchanged.
New failure mode, stated plainly: a bound RWO mount becomes a hard
placement constraint. ApplyVolumePlacement pins the service and clamps it to
one replica; the volume attaches only in its own Hetzner location. A scheduler
that ignores this produces Pending tasks, not errors.
2.2 Candidate B: a shared filesystem (NFS / CephFS)
Section titled “2.2 Candidate B: a shared filesystem (NFS / CephFS)”What exists. More than expected, and none of it usable.
- Providers:
ce/modules/storage/providers/{nfs,cephfs,local}; managed bring-up viashc storage nfs enable(real: leader-gated provider CRUD +/internal/storage/nfs/enable-serverandenable-clientdispatched over the cluster mTLS peer transport) andshc storage cephfs enable(records a provider row; the ceph cluster itself is operator-run, with a fullmodules/storage/cephcephadm/mon/mgr/mds/osd driver present). storageapi.ProviderType.IsShared()is a single total authority guarded by a source-parsing exhaustiveness test. Both move classifiers, the node-support probe and the restore-time--storageresolver all go through it.
Why it is dead code. Three independent blocks, each sufficient on its own:
- MEASURED: the volume table forbids the row.
ent/schema/volume.go:40:Check: "access_mode IN ('rwo', 'rox')".ValidateBindingsexplicitly refusesaccess: rwxwithX400328VOLACC, and says why: absorbingrwxintorwowould make swarm stamp a node pin and a replica clamp the author asked the opposite of. So no shared-FS mount can ever beproviderProven. - MEASURED:
ClassifyMountDecisiontherefore always answers NotRebindable on the shared-FS arm:if !in.ProviderProven { return MountClassNotRebindable }. Both shared-FS arms of both classifiers are unreachable. - MEASURED: nothing routes deployment bytes onto a provider anyway.
app.DeploymentDiris unconditionallyfilepath.Join(stateHome, "deployments", tenant, stack, environment)and panics on an emptystateHome.paths.MountsUnder(deployDir)is unconditional.storageapi.Provider.BasePath()has zero non-storageapi callers (verified by grep acrossmodules/,daemon/,cli/). The--storagestamp is, in the classifier’s own words, “a label over bytes that are on local disk”.
The --driver local --opt type=nfs observation is correct and does not
help. It is stock docker with no plugin. But SHC does not create docker
volumes for DATA mounts on compose at all; it rewrites them to host binds
under <deployDir>/mounts/. MEASURED: the only place ce reads
driver_opts.device is modules/backup/namedvolume.go:190, on the swarm
named-volume capture path. Adopting the NFS local-driver shape would mean
changing the compose renderer’s DATA-mount strategy, i.e. it is not a
one-line adoption of a stock feature.
Cost on this substrate: this is the decisive part.
- NFS: needs a server. On the current estates that is one of the six nodes,
on its own 610.4 GB
sda. DERIVED: this converts every stateful deployment’s storage into a dependency on one node, strictly worse availability than today, where a node failure loses only its own deployments. Hetzner offers no managed low-latency NFS in the cloud product; Storage Boxes are CIFS/SMB/SSHFS and are not a POSIX substrate a database will run on. - CephFS: needs OSD block devices. MEASURED: no node on either estate
has a second disk.
ce/modules/storage/ceph/osd.go::Addtakes adeviceargument and shellsceph orch daemon add osd <host>:<device>. To run Ceph here you would first buy Hetzner block volumes to feed the OSDs, then run 3× replication on top.
DERIVED, and it settles the fork: on Hetzner, a shared filesystem costs ≥3× the block-volume bill (replication factor) plus the block volumes themselves plus a Ceph cluster to operate. Shared FS is a strict superset of the cost of Candidate A and delivers a property (
rwx) that the catalogue does not ask for.
Which apps would a shared FS actually suit? Honestly: very few. The
catalogue is single-writer by construction. MEASURED: 65 of 99 apps declare
x-shc-scale, and every one of the 65 has a max: 1 somewhere. rwx buys
nothing for a service capped at one replica. The genuine rwx candidates are
blob trees behind a scale-out front (nextcloud files, immich uploads,
peertube media), and for all three, object storage (the already-shipped
s3 plug) is the upstream-supported answer, not a POSIX share. nextcloud’s
compose already carries a complete OBJECTSTORE_S3_* block behind
{% if integrations.s3 is defined %}.
2.3 Candidate C: put StateHome itself on a share (considered, rejected)
Section titled “2.3 Candidate C: put StateHome itself on a share (considered, rejected)”MEASURED: paths.resolveStateHome honours SHC_STATE_HOME, so an operator
could point the whole state home at an NFS mount and make every deployment
directory shared with no code change at all. It is the cheapest possible
implementation of “shared storage” and it is the wrong one:
StateHomealso holds the rqlite data, the vault, the move journal (ce/modules/clone/move_journal.go, deliberately a node-local file so it needs no migration), the staging dir and per-node fstab-managed mounts. Sharing it makes every node-local durability assumption false at once.- The move journal’s whole design premise (a durable node-local record written ahead of the act) is violated.
- Cost: near zero to try, catastrophic and hard to reverse when it fails.
REJECTED. Recorded because it is the first thing anyone will suggest and it needs a written reason, not a shrug.
2.4 Candidate D: app-native move (already in use, under-recognised)
Section titled “2.4 Candidate D: app-native move (already in use, under-recognised)”MEASURED: gitlab excludes gitlab-data, gitlab-config, gitlab-logs
from the mount snapshot and instead runs hooks.pre_backup: gitlab-backup-create (a native gitlab-backup dump into the gitlab-backups
mount) plus hooks.post_restore: gitlab-backup-restore which does
gitlab-ctl stop puma; gitlab-ctl stop sidekiq, replays, and waits for puma.
That is a per-app move path that is already declared, already exercised, and
sidesteps both the byte cost (a dump is far smaller than a gitaly tree) and the
consistency problem (the app’s own tool guarantees it). MEASURED: 28 apps
already declare backup.yaml exclude.mounts covering 60 of the 211 DATA
mounts (28%): caches, brokers, logs and, in gitlab’s case, the entire live
tree.
This is the cheapest lever in the whole lane and it needs no new mechanism. For every app whose upstream ships a dump/restore tool, the bytes that move are the dump, not the tree.
3. (c) Data gravity as a placement input
Section titled “3. (c) Data gravity as a placement input”3.1 SHC already has the graph
Section titled “3.1 SHC already has the graph”MEASURED. The integration table (ce/ent/schema/integration.go) records
(tenant, provider_deployment_id, provider_socket_name, consumer_deployment_id, consumer_plug_name, connection_values, status) with a UNIQUE index on
(tenant, consumer_deployment_id, consumer_plug_name). app.Service. GetConnectedStacks walks it in both directions and resolves each peer in its
own scope (a cross-tenant keycloak@master provider comes back under master,
not the consumer’s tenant).
The clone module already consumes it as a transitive closure: --closure runs
BFS to a cap of 8 stacks, and its own test fixture is exactly the gravity shape:
license-broker → keygen → keygen-db.
The data-gravity graph is not missing. It is built, persisted, directed, transitively walkable, scope-correct, and already spent by
move.
3.2 The gravity that matters is not app→app, it is app→db
Section titled “3.2 The gravity that matters is not app→app, it is app→db”MEASURED: SHC has no -db companion convention in code; grep for "-db"
across both repos returns one unrelated hit in the caprover adapter. A -db
companion is an operator-created deployment joined by an integration edge.
So the relationship is exactly and only the integration edge, which is the
thing that already exists.
3.3 Whether co-location is a soft score or a hard constraint depends on the fabric posture
Section titled “3.3 Whether co-location is a soft score or a hard constraint depends on the fabric posture”This is the part I did not expect and it changes the design.
MEASURED (ce/modules/app/fabric_render.go,
ce/modules/fabric/{posture,network}.go): cross-node integration on the
compose runtime is carried by the SHC fabric: a per-node local bridge plus
a WireGuard shc0 overlay, with the renderer writing pinned ipv4_address
per service and extra_hosts on the consumer side of every cross-node
integration, preserving the same alias strings swarm would have resolved via
DNS. And: ApplyFabricTarget returns its input unchanged on every posture
that is not shc, “dormancy is structural”. A cluster on posture none, on
swarm, or with no recorded fabric row renders byte-for-byte as if the file did
not exist.
DERIVED, three regimes:
| posture | cross-node app→db | gravity is |
|---|---|---|
shc | routed over shc0 + extra_hosts | a soft score (latency, failure domain) |
swarm | swarm overlay network | a soft score |
unset / none, compose runtime | not routed | a hard co-location constraint |
NAMED LIMIT: I cannot read either estate’s fabric posture row without estate contact. If posture is unset on a compose estate (which is the default and the zero value) then a scheduler that splits a consumer from its provider produces a deployment that comes up and cannot reach its database. That is not a latency regression, it is an outage.
RULING-OWED R-3.3: before any scheduler places a deployment on a node other than its provider’s, the fabric posture must be read and a non-
shc, non-swarmposture must make co-location a hard admission constraint, not a score.
3.4 What a gravity-aware scorer should read, and what it must not
Section titled “3.4 What a gravity-aware scorer should read, and what it must not”PROPOSED, in the order I would implement it:
- Hard constraint: bound RWO volume. If any of the deployment’s mounts is
a bound RWO block volume, the node is decided:
ApplyVolumePlacementwill pin it there anyway. The scorer must not pretend to choose. - Hard constraint: posture (R-3.3). On a non-routed posture, restrict the candidate set to the node holding every integration peer.
- Soft score: co-location with providers. Reward placing a consumer on
the node already running its transitive closure (
GetConnectedStacks, cap 8). Failure-domain and latency both point the same way. - Soft score: anti-gravity for the closure’s own replicas. Not a spread rule: two members of the same consistency group on one node share a failure domain, and the group backup/restore path treats them as one unit anyway. RULING-OWED R-3.4 whether availability or latency wins here; I do not think it can be decided without the owner’s HA posture.
What it must not read: the deployment’s --storage stamp. The classifier’s
own comment is the argument: nothing validates it at install, nothing routes
mounts by it, “it names where the operator ASKED the bytes to live, never where
they ARE”.
Dependency I could not verify: an earlier session recorded the integration table as having 0 rows on both estates. If that is still true, every soft gravity score is computed over an empty graph and silently degrades to “no opinion”. A scorer reading an empty graph must say so, not score 0.
4. (d) The background-worker trap and the standby declaration
Section titled “4. (d) The background-worker trap and the standby declaration”4.1 First, the good news: today’s move has no double-run risk
Section titled “4.1 First, the good news: today’s move has no double-run risk”MEASURED: the phase order is backup → stop → measure → recover. The source
is stopped before the target is recovered. runRebindMove calls
StopDeployment first (LEASE-1) for the same reason.
And on swarm, runEphemeralMove’s re-render inherits Docker’s default update
order, stop-first. MEASURED: no app in the catalogue declares
update_config, and grep finds no order: start-first anywhere in either repo.
Standby is not required by today’s move primitive. It is required by the zero-downtime move that does not exist yet. This corroborates
docs/rebalancer-safety§5.5 from an independent direction.
4.2 How many of the 99 would need a standby declaration?
Section titled “4.2 How many of the 99 would need a standby declaration?”The honest answer is every stateful app that runs any background writer, and the compose graph cannot tell you which those are.
I tried to derive it. Method: for each app, classify every compose service as
routed (has ports/expose), one-shot (restart: no|on-failure), datastore
(name/image matches a datastore vocabulary), or background writer.
MEASURED: the derivation fails in both directions:
It undercounts, badly. The worst offenders show zero background services:
| app | services in compose | background writers actually present |
|---|---|---|
gitlab | gitlab, valkey | puma, sidekiq, gitaly, gitlab-ctl cron, all inside one omnibus container |
wordpress | wordpress | wp-cron, fired by HTTP requests |
nextcloud | nextcloud, valkey, nextcloud-pgclient | cron.php background jobs |
matrix | synapse, element, … | synapse background tasks, media retention |
immich | immich-server, … | microservices, merged into the server image upstream |
The proof that these are real writers is in SHC’s own files: gitlab’s
gitlab-backup-restore.job.yaml runs gitlab-ctl stop puma; gitlab-ctl stop sidekiq; SHC already knows sidekiq is there; the compose graph does not.
It overcounts too. My heuristic flagged elastic, kvs, redpanda,
*-pgclient sidecars and jitsi-jicofo as workers. They are not; they are
datastores, backup sidecars and signalling services.
MEASURED CONCLUSION: standby cannot be derived. Any attempt produces a set that is wrong on the highest-value apps in the catalogue. The 25 apps my scan flagged with a named worker idiom (
dify,windmill,medusa,monica,pulp,mailcow,hyperswitch,keygen,mautic,frappe,journiv,plane,defectdojo×2,gitlab-runner,openreplay,huly, …) are a lower bound with false positives, not an answer.
The upper bound is the useful number: 74 stateful apps, of which 73 are non-fixture. Assume every one needs a declaration until its author says otherwise.
4.3 The declaration vocabulary already ships (twice) and is unused
Section titled “4.3 The declaration vocabulary already ships (twice) and is unused”MEASURED:
-
hooks.pre_backup/hooks.post_backupinapp.yaml. Census over all 99:post_install30,post_update30,pre_backup24,post_restore19,post_backup7,pre_update2.Only 7 apps have a full enter/exit bracket:
nextcloud,uptime-kuma,supabase,mail,mariadb,mongodb,mysql. The other 17pre_backuphooks are dumps, not quiesces:gitlab’s isgitlab-backup-create, and it has nopost_backupbecause there is nothing to undo.The 7 brackets are exactly the right shape:
nextcloud: enable-maintenance-mode / disable-maintenance-mode;mariadb / mysql / mongodb / supabase: quiesce / resume. -
x-shc-scale.<svc>.backup: a per-service replica clamp for the archive window, implemented ince/daemon/backup_scaler.goviadocker compose up -d --no-recreate --scale svc=N, with partial-window rollback. MEASURED: 0 of 99 apps declare it. The file’s own header says so: “No app declares a backup/restore window today (all are max-only), so this is a strict no-op in production.”MEASURED: 65 apps declare
x-shc-scale; all 65 usemax:only.
The standby mechanism is not missing. Two of them ship. The declaration corpus is 7 and 0. This is the same pattern as
x-shc-volumes, third instance in one lane.
4.4 ★★ The cautionary tale, and why “declared” is not “true”
Section titled “4.4 ★★ The cautionary tale, and why “declared” is not “true””apps/standalone/mariadb/quiesce.job.yaml is the single most instructive file
I read this session. MEASURED, from its own header:
The job declared a quiesce and never performed one, on any estate, ever. It
issued SET GLOBAL super_read_only = ON, a MySQL variable MariaDB has never
implemented (MDEV-9458, WON’T FIX), copy-pasted from apps/standalone/mysql
across an engine boundary nothing in the catalogue checked. The client stopped
at the first error, the preceding FLUSH TABLES WITH READ LOCK died with the
connection milliseconds later, and every mariadb archive ever taken was read off
a live, unfrozen server, stamped as quiesced.
It has since been fixed to FLUSH TABLES WITH READ LOCK; SET GLOBAL read_only = ON; UNLOCK TABLES, and it now proves itself on a second connection,
matching the gate value as a string (ON) or a number (1) because
MariaDB 12.3.2 and 11.8.8 spell the same boolean differently and @@read_only+0
silently reads a gated server as open.
RULING-OWED R-3.5: a standby declaration MUST carry its own verification, and the verification must run on a fresh connection / process, or the declaration is worse than its absence: it converts “we know we cannot move this safely” into “we believe we can”. A move that spends an unverified standby as permission to start a second writer is a data-loss bug with a green log line.
4.5 What happens to an app that declares none
Section titled “4.5 What happens to an app that declares none”PROPOSED, and it agrees with docs/rebalancer-safety §5.4:
An app with no verified standby is never selected for a zero-downtime
cutover. It is not refused a move; it takes today’s stop-first path, which is
correct and always has been. The selection record must name the reason
(NO_STANDBY(<app> declares no verified pre_standby)), because a scheduler that
silently declines to move the one deployment causing the imbalance is a
scheduler nobody trusts.
The corollary is the design’s actual shape: the standby declaration buys downtime, not correctness. Correctness is already bought by stop-first. That reframes the whole item from “blocking prerequisite” to “optimisation with a per-app opt-in”, which is a much better place for it to sit.
4.6 One more constraint on any standby design
Section titled “4.6 One more constraint on any standby design”MEASURED: timeouts.quiesce_hold defaults to 300 s, and a bracket that
overruns it stops the stack, retakes, and restarts it, unattended. Any standby
window that could exceed 5 minutes (i.e. any of the media/registry class in
§1.3) must either raise that timeout deliberately or accept that its “graceful”
standby degrades into a hard stop.
5. Constraints established, in one list
Section titled “5. Constraints established, in one list”Each of these is a hard number a placement design must satisfy.
- MEASURED: Zero of 99 apps declare
x-shc-volumes. Therefore every persistent mount classifiesNotRebindableandmoveselectsbackup_restorefor every deployment, on both estates, today. - MEASURED: MOVE-5: on the compose runtime, every cross-node move is
backup_restore, including stateless apps. - MEASURED: 74 of 99 catalogue entries are stateful (211 distinct DATA mounts). Only 5 non-fixture apps are both placeable and stateless; 10 entries run no container at all.
- MEASURED: Move wire format is base64-in-JSON, 2 MiB synchronous chunks: 1.33× inflation, ≤65–70 MiB/s ceiling on loopback, round-trip-bound (not CPU-bound, so bigger nodes do not help).
- MEASURED: A move stages a full archive on the same disk as the data.
No node on either estate has a second disk (610.4 GB
sda, ×6). A deployment above ~45% of free disk cannot be relocated at all. - MEASURED: Hetzner volumes: 10 GB–10 TB, single-attach, max 16 per server. A per-mount volume design is infeasible; a per-deployment design caps a node at 16 stateful deployments.
- MEASURED:
Volume.access_modeCHECK isIN ('rwo','rox');rwxis refused withX400328VOLACC. No shared filesystem can own a volume row, so both shared-FS classifier arms are unreachable. - MEASURED:
DeploymentDiris unconditional andstorageapi.Provider.BasePath()has zero callers. Nothing routes a deployment’s bytes onto a storage provider. - MEASURED: 65 of 99 apps declare
x-shc-scale, all withmax:only;x-shc-scale.<svc>.backupis wired and declared by 0. - MEASURED: Only 7 apps have a full
pre_backup/post_backupbracket. 17 more have apre_backupthat is a dump, not a quiesce. - MEASURED: Standby cannot be derived from compose: the four highest-value
stateful apps (
gitlab,wordpress,nextcloud,matrix) show zero background services and all run background writers. - MEASURED:
timeouts.quiesce_hold= 300 s, and overrun escalates to a hard stop. - DERIVED: On a non-
shc, non-swarmfabric posture, cross-node integration is unrouted, so co-location is a hard admission constraint, not a score. - MEASURED:
dataPhasemeasures only the deployment directory, so it is blind to swarm named volumes (defect D-1).
6. Recommendation
Section titled “6. Recommendation”Direction: declare x-shc-volumes on a budgeted subset. Reject shared
filesystem. Harvest the app-native move path first, because it is free.
Ordered by cost-to-value, with what each buys:
Step 0: free, do it regardless: measure the byte cost
Section titled “Step 0: free, do it regardless: measure the byte cost”One du -sb sweep over <StateHome>/deployments/*/*/* on all six nodes. It is
the same walk MeasureDeploymentTree performs. Without it, every sizing
decision below is a guess. This is estate work and is not mine to run.
Step 1: cheapest lever: app-native dump/restore, already declared
Section titled “Step 1: cheapest lever: app-native dump/restore, already declared”Cost: zero new mechanism. 28 apps already exclude 60 of 211 DATA mounts;
gitlab already excludes its entire live tree in favour of a native dump.
Extending that pattern to every app whose upstream ships a dump tool shrinks
the moved bytes by an order of magnitude for the apps where it works, with no
schema change, no cloud spend and no new failure mode.
Step 2: fix the gate before making moves more attractive (defect D-1)
Section titled “Step 2: fix the gate before making moves more attractive (defect D-1)”dataPhase must measure named volumes as well as the tree, or a swarm move can
destroy a source on the strength of a comparison that never saw the data. This
is a prerequisite, not a nice-to-have: everything below makes moves more
frequent.
Step 3: the storage direction: x-shc-volumes, budgeted
Section titled “Step 3: the storage direction: x-shc-volumes, budgeted”- Who declares one: the deployments that are both (a) large enough that the §1.3 byte cost is a real outage, and (b) inside the 16-per-node budget. Start from the Step-0 measurement, not from the catalogue.
- What it costs: €0.044–0.057/GB/month (R-3.2: confirm on the invoice); ~€4.40/month per 100 GB at the low quote. One compose block per app. Both estates already carry the volume-capable connection.
- What it buys: the move becomes stop → detach → attach → start, zero
bytes on the network, and the staging-disk ceiling disappears for the bound
mounts. It also arms
volume_failover.go(already built, already fence-proof-gated, already off by default) which is the only automatic node-failure story SHC has. - What it constrains: a bound RWO mount is a hard placement pin plus a single-replica clamp. The scheduler must treat it as an input, not a choice.
Step 4: do not build shared storage on this substrate
Section titled “Step 4: do not build shared storage on this substrate”On Hetzner, CephFS requires block volumes as OSD devices (no node has a second
disk), then multiplies them by the replication factor, then adds a Ceph cluster
to operate. NFS on a cluster node converts every stateful deployment into a
dependency on one machine. Neither buys a property the catalogue asks for:
all 65 apps that declare a scale cap declare max: 1, so rwx has no
consumer. For the three genuine multi-writer blob cases (nextcloud, immich,
peertube), the answer is the S3 plug that already ships.
This is a recommendation about the substrate SHC runs on today, not a permanent doctrine. On a substrate with a managed multi-attach filesystem (EFS, Filestore) the arithmetic changes and this should be revisited.
Step 5: standby as an opt-in optimisation, never a prerequisite
Section titled “Step 5: standby as an opt-in optimisation, never a prerequisite”Reuse the vocabulary that ships (hooks.pre_backup/post_backup,
x-shc-scale.<svc>.backup). Require a self-proving verification per R-3.5;
the mariadb job is the model, including its second-connection read-back and its
dual-spelling match. An app without one keeps today’s stop-first move, and the
selection record must say so by name.
7. Rejected alternatives, with their costs
Section titled “7. Rejected alternatives, with their costs”| alternative | why rejected | cost if pursued anyway |
|---|---|---|
| Shared filesystem (CephFS) | Needs OSD block devices; no node has a second disk. Costs block volumes × replication factor, plus a Ceph cluster. Delivers rwx, which 65/99 apps structurally cannot use. | ~3× the block-volume bill + a new operational surface + the access_mode schema change |
| Shared filesystem (NFS on a cluster node) | Converts every stateful deployment into a single-node dependency, strictly worse availability than today | Cheap to build, expensive the first time that node reboots |
SHC_STATE_HOME on a share | Shares the rqlite data, the vault, the node-local by design move journal and the staging dir. Violates the journal’s premise. | ~zero to try, catastrophic and hard to reverse |
| One block volume per DATA mount | 211 distinct mounts vs a 16-per-server cap. greenbone alone declares 18. | Infeasible; fails at the first dense node |
| Derive standby from the compose graph | MEASURED to fail on gitlab, wordpress, nextcloud, matrix, the four apps where being wrong costs the most | A confident wrong answer, which is the worst outcome available |
| Start-first cutover without standby | 65/99 apps are max: 1; two writers on one dataset. Today’s stop-first is correct. | Data loss, with a green health probe (the exact 2026-08-09 shape) |
Raise cloneRecoverChunkBytes | The transfer is round-trip-bound, not chunk-bound, and >3 MiB raw exceeds internal_rpc’s 4 MiB body cap | Every chunk starts failing, the cliff the streaming replaced |
8. What I could not do
Section titled “8. What I could not do”- No estate contact, so: no per-deployment byte sizes (I have exactly two:
nextcloud8.44 GB,openobserve≈5 GB), no read of either estate’s fabric posture row, no confirmation of whether theintegrationtable is still empty, no live cross-node ship measurement. - The ship benchmark is a loopback ceiling, not a field measurement. Real cross-node throughput is bounded above by 65–70 MiB/s and I do not know how far below.
- Hetzner volume pricing is contradictory across third-party sources (€0.044 vs €0.057/GB/month). Not resolved; do not put either in a plan without an invoice.
- The
x-shc-volumesdeclaration effort is unestimated. I know it is one compose block per app; I do not know how many of the 74 need a size chosen by someone who knows the workload.
9. Rulings owed
Section titled “9. Rulings owed”| # | question | why it blocks |
|---|---|---|
| R-3.1 | Are block volumes a budgeted per-deployment escape hatch (16/node) or something the catalogue declares broadly? | Decides whether §6 Step 3 is a shortlist or a campaign |
| R-3.2 | Confirm the Hetzner volume price on an actual invoice | €0.044 vs €0.057 is a 30% swing on every number in §2.1 |
| R-3.3 | Must a non-shc/non-swarm fabric posture make co-location a hard admission constraint? | Without it a scheduler can produce a deployment that cannot reach its database |
| R-3.4 | For a consistency group: co-locate (latency) or spread (failure domain)? | Directly contradictory scoring terms; not derivable from the code |
| R-3.5 | Must a standby declaration carry its own fresh-connection verification? | The mariadb precedent says an unverified declaration is worse than none |
| R-3.6 | Is defect D-1 (data gate blind on swarm) a P1 to fix before any rebalancer ships? | Everything in this lane makes moves more frequent |
10. Reproducing the measurements
Section titled “10. Reproducing the measurements”The catalogue censuses in §1.2, §4.2 and §4.3 are single-pass reads of
apps/*/{app.yaml,compose.yaml,backup.yaml} at shc 80e0c4441. Each is
restated here so the number can be re-derived without the scratch scripts.
Mount-type census / stateful count (§1.2). For every service’s volumes:
entry, split source:target[:mode]; then apply DetectMountType verbatim:
a source starting with temp/tmp is RUNTIME; a source starting with / is
SYSTEM (the source decides; the container-side target carries no authority);
a /tmp target on a relative source is RUNTIME; ./ or ../ is FILE; a
bare name that exists as a path inside the app dir is FILE; anything else is
DATA, i.e. persistent. Long-form entries with type: tmpfs are ephemeral.
Expect 362 DATA references / 211 distinct, 74 of 99 entries stateful.
Placeable-and-stateless count (§1.2). Of the 25 with no DATA mount,
subtract the entries whose services: map is empty (grep -c the nine
apps/standalone/external-* plus apps/standalone/s3, all declare zero
services) and the eight under apps/fixtures/ and */tests/apps/.
Hook census (§4.3). hooks: is in app.yaml, not backup.yaml;the
job bodies live in sibling *.job.yaml files. Counting jobs: keys in
backup.yaml gives the wrong answer (it finds one pre_backup, in
ciso-assistant); counting hooks: keys in app.yaml gives 24 / 7.
Scale census (§2.2, §4.3). grep -l x-shc-scale apps/*/*/compose.yaml
→ 65; every one matches max:\s*1; parsing each service’s x-shc-scale map
for a backup key → 0.
Ship-leg benchmark (§1.3). ~120 lines of Go: write an N-MiB file to tmpfs;
read it in 2 MiB chunks (cloneRecoverChunkBytes); for each chunk build
{"args":[],"kwargs":{"session":…,"offset":…,"chunk_b64":<base64>}},
json.Marshal, POST to a loopback net/http handler that json.Unmarshals,
base64-decodes and WriteAts. Time the loop. This is
ce/daemon/services_clone_remote.go::shipBakToPeer with the network and TLS
removed, so the result is an upper bound. Expect 1.33× wire inflation exactly
and 60–70 MiB/s, roughly unchanged at GOMAXPROCS=2.