The storage model (proposal)
Status: open design. Slices 1 and 2 are built (the store registry, attestations and the mounts root), and the 2026-08-14 models rulings rule the open halves. What is built after slice 2 is not recorded here.
Written 2026-08-02. Verified first-hand against ce at f222808 (the shc
go.mod pin, origin/main) in the clean worktree /var/tmp/ce-storemodel,
and against shc at d6b3cb0 (origin/main). Neither production estate was
contacted by this document. Live facts (empty registries, 24 deployments,
archives on the deployment disk) come from three earlier read-only audits and
are attributed as second-hand every time they are used.
The owner’s framing:
“Part of the problem with storage and its straddling things is initially storage actually just was a path on a node, and then later I added it to be a path on a shared storage provider. And then later I decided to add block storage. So it’s kind of like things being slapped on top of other things that weren’t architected correctly. So what I want you to do is think about how you would design something that supports all three, but from a first principles approach. Take some hints and thoughts about how Proxmox does this because I believe Proxmox supports shared storage as well as local, as well as block. But for shared storage, it would be nice if it’s actually literally the path mounts just like local is.”
This document consumes and must not contradict
the deployment binding model. That
document supplies durability (would losing these bytes be data loss) and the
bound / unbound / bound_to / anchor vocabulary. This document supplies
the other half (anchor, the set of nodes that can serve a mount’s bytes
without copying) and nothing else. Where the two disagree, the binding model
wins on deployment scope and this one wins on storage scope; the disagreements
are enumerated in what this changes in the binding
model.
The model
Section titled “The model”A store is a named place that holds bytes for deployments. It has one locator, a set of nodes it can be reached from, and a declaration of what it is allowed to hold. Local disks, NFS and CephFS shares, cloud block planes and archive repositories are all stores; they differ only in how the locator turns into bytes on a node.
Everything else in the model is derived from two per-mount questions the store answers:
- REACH: which nodes can serve these bytes without copying them.
- EXCLUSIVITY: may more than one node hold them open at once.
| case | driver | locator | reach | exclusivity | what SHC does |
|---|---|---|---|---|---|
| node-local disk | dir | an absolute path | the one node that owns the path | not applicable; nothing else can reach it | verifies the path is present, and mounted if declared so |
| shared filesystem | nfs / cephfs | an absolute path | every node that attests the path is mounted and writable | concurrent | mounts it, then treats it exactly as dir |
| cloud block plane | block | a provider plus region token (hcloud/fsn1) | every node in the region that carries the credential | one node at a time, per device | provisions a device, fences, attaches, mounts it at the same path a dir store would have used |
| archive repository | restic | a restic repository URL | every node carrying the credential | concurrent, restic-locked | hands the locator to restic and carves nothing |
Four tables carry it, and three of them already exist:
| table | today | under this model |
|---|---|---|
store | store, a registry of places, written with no probe of any kind | same physical table, six columns added, every capability column probe-written |
store_attestation | store_node, an operator assertion consumed as evidence | replaced: a row written by the node about itself, carrying observed_at |
volume | a provisioned block device | unchanged, except provider free text becomes store_id |
volume_attachment | the single-writer lease | unchanged, not touched |
Two things this model deliberately does not add: a per-mount row, and a fourth noun for backups. Both are explained below, and both are load-bearing.
Did the store-is-a-path hypothesis hold?
Section titled “Did the store-is-a-path hypothesis hold?”Mostly. It holds completely for the owner’s stated case (shared storage is literally a path mount, indistinguishable from local below the selector) and it breaks in exactly two places, neither of which needs a new concept.
Where it holds, and the proof is already in the tree. All three filesystem
provider implementations build the identical subtree under a base path:
deploymentPath → MkdirAll(0o750) → for _, sub := range []string{"mounts", "files", "app"}, byte-for-byte in modules/storage/providers/local/provider.go,
.../nfs/provider.go and .../cephfs/provider.go. Three types, one layout.
Proxmox reached the same answer at source level: PVE::Storage::NFSPlugin and
PVE::Storage::CephFSPlugin both declare use base qw(PVE::Storage::Plugin)
and override only check_config, activate_storage, status /
deactivate_storage and check_connection. They override nothing about
allocation, naming, listing, snapshotting or path computation; get_subdir() is
"$path/$subdir" in the base directory plugin and NFS inherits it whole. Mount
it at a path, then delegate to the local implementation.
Break 1: a block store has no path until a device is attached, and the joint
is narrower than it looks. DaemonVolumeBinder.hostMountpoint already
computes “the SAME <deployDir>/mounts/<name> path the host-bind branch
resolves to, so ApplyPrep runs byte-identically”. A block-backed mount and a
host bind resolve to the same string. So the path is not dynamic; it is a
pure function of (store, tenant, stack, environment, mount). What differs is
who makes that path have the right bytes before the container starts, and that
step already has a universal gate: PrepOp.RequireMounted
(core/volume/prep.go:122) refuses with X409034VOLNMT before any mkdir, seed
or chown. The gate is correct. It is wired to the wrong predicate: one setter
in the whole tree, modules/app/install_runner.go:3189, in the block binder.
So a block store is a dir store with one extra step before the path is real,
and the model expresses that as a driver, not as a kind.
Break 2: an archive repository is a namespace, not a path, and it never
can be. LocalResticRepoPath returns the empty string for s3:, b2:,
gs:, azure:, sftp:, rest:, swift: and rclone:
(modules/backup/restic.go), and the shipped default.yaml comment offers
s3:s3.amazonaws.com/mybucket/backups and b2:bucket-name:/backups as the
multi-node configuration. An off-box archive is the point of an off-box
archive, so excluding these is not available.
The formulation that survives both breaks. A store is a namespace that
resolves (deployment, mount, content) to a location, and for three of the
four drivers that location is a path. Nothing downstream ever needed the path;
it needed the resolver. storageapi.Provider.BasePath(), which has zero
production callers today, was already the right shape and was never called.
The two axes held. Reach is real and it is the whole of anchor.
Exclusivity is real, and Proxmox’s model corrects the brief’s placement of it:
a shared iSCSI LUN carrying an LVM volume group is shared 1 and serves many
concurrently-running guests, each on its own logical volume. One-at-a-time is a
property of a device, not of the store that mints devices. SHC agrees
already: volume_attachment.volume_id is UNIQUE (one attachment per volume)
while one block plane serves every deployment on the estate. So exclusivity
lives on the provisioned volume, where the lease already is.
Vocabulary
Section titled “Vocabulary”One word meaning three things is what produced this. Every surviving concept,
checked against the binding model’s reserved set: bind (addressing, owned by
the node addressing model, untouched),
binding → provisioned (binding model slice 8), and bound / unbound /
bound_to / anchor reserved for deployment scope only.
| word | means | collision check |
|---|---|---|
| store | a named place that holds bytes: one locator, a reach, a content set | the binding model’s Q1 already proposed store for rung 1. This model widens it to all four axes. --store is unused across ce’s flag registrations |
| driver | how a store’s locator becomes bytes on a node: dir, nfs, cephfs, block, restic | new word, no collision. Selects an implementation; nothing branches on it outside the store package |
| content | what a store may hold: data, archive | Proxmox’s word for the same idea |
| reach | the node set that can serve a store’s bytes without copying | this document’s word. It is a resolver, never a column; see anchor |
| attestation | one node’s own observation of one store, with observed_at | new word, no collision. Deliberately not “status”: a status is a fact, an attestation is a claim with a timestamp and a writer |
| volume | unchanged, a provisioned block device row | already the code’s word, and the binding model’s rung-2 word. Slice 8’s hasBinding → hasVolume rename makes it unambiguous |
| attachment | unchanged, which node currently holds a volume, and the lease generation | already the code’s word |
Two nouns the binding model reserved are retired before they are spent.
sharewas to be rung 3’s noun. Under this model a share is a store whose driver isnfsorcephfs. Keeping a separate noun would re-create the straddle: two words for one object, told apart by a string.repositorywas to be the fourth axis’s noun. Under this model an archive repository is a store whose content isarchive. The word survives as restic’s word for its own on-disk format, which is what it actually names.
--storage becomes --store <name>, and it is validated. Today the flag
takes an unvalidated free-text label; shc storage add cannot even produce a
value that is not local, because it hardcodes Type: "local"
(modules/storage/commands/add.go:78) and offers no --type. Under this model
--store names a row, is refused if the row does not exist, and selects the
store for every mount of the deployment that does not override it.
deployment.storage is RETIRED IN PLACE and deployment.store is added.
Migrations are append-only, so the column stays; no code reads or writes it.
There is an exact in-repo precedent with the reasoning already written down:
deployment.launcher, ent/schema/deployment.go:73-78: “RETIRED IN PLACE
(pre-1.0) … No code reads or writes the column; it stays in the schema only
because migrations are frozen (atlas.sum). Always NULL on new rows.” Reusing
the old column with a new meaning is the accretion pattern this document exists
to stop: one name, two meanings, told apart by when you read it.
x-shc-volumes becomes x-shc-mounts. The rename is free: zero of 73
catalogue apps declare x-shc-volumes (measured). Each entry carries
{store, size, access, disposable}; store is the per-mount override of the
deployment’s --store. The parser must stop being permissive: today an
unrecognised access token falls back to RWO, commented “the parser stays
permissive” (core/volume/binding.go), so access: rwx (a request for shared
storage) silently yields a single-writer block volume. Under this model an
unknown token is a parse error.
The three cases, end to end
Section titled “The three cases, end to end”The path formula is uniform and is the whole trick:
data <store.locator>/deployments/<tenant>/<stack>/<env>/mounts/<name>archive <store.locator>/backups/<tenant>/<stack>Set the seeded default store’s locator to {state_home} and every one of those
strings is byte-identical to what the system produces today: DeploymentDir,
MountsRoot, DaemonBackupPathsResolver.BackupsDir. The existing state-home
tree already is a correctly laid-out store. That is not a coincidence to
exploit; it is evidence the layout was always right and only the selector was
missing.
Render artifacts never move. compose.yaml, manifest.json, app/,
vars.yaml, runtime/, state.json stay node-local under
<state_home>/deployments/... unconditionally, forever. Two reasons, and the
second is the decisive one. They are regenerated from the manifest on every
install and recover, so nobody ever asks which nodes can serve them without
copying. And two nodes rendering one deployment into one shared directory
collide: the source’s rollback snapshot and the target’s fresh render resolve
to the same path. Only mounts/ names a store.
Case 1: a second disk on one node (rung 1)
Section titled “Case 1: a second disk on one node (rung 1)”shc store add fast --driver dir --path /mnt/nvme0 --node swarm01shc store status fastshc install gitlab --store fastrequire_mounted defaults true, because the locator is not under the node’s
own state home. swarm01’s attestor forks a deadline-bounded -d /mnt/nvme0,
reads /proc/mounts (the isMountPoint helper already exists at
modules/storage/doctor.go), write-probes a dotfile, writes a witness file, and
attests ok with the filesystem it found. Every other node attests absent,
correctly, and that is why this is one store with reach 1, not one store with
reach 3.
Install resolves /mnt/nvme0/deployments/acme/gitlab/prod/mounts/data.
ApplyPrep runs with RequireMounted true, so if /mnt/nvme0 failed to mount,
install refuses instead of seeding the image’s initial content onto the bare
mountpoint directory on the root filesystem, where it would be shadowed the
moment the disk came back. That is the single highest-value fix in this
document and it needs no new machinery: it binds an existing correct gate to
store.require_mounted instead of to binder != nil.
reach = {swarm01}, anchor = {swarm01}, singleton, so the deployment is
BOUND. shc move gitlab --node swarm02 finds the target outside the anchor and
takes backup+restore, exactly as today.
Today there is no way to do this at all: DeploymentDir and MountsRoot are
unconditional.
Case 2: a NAS or CephFS share (rung 3)
Section titled “Case 2: a NAS or CephFS share (rung 3)”shc store add nas --driver nfs --server 10.0.0.9 --export /srv/shc --path /mnt/nasshc store status nasshc install huly --store nasEach node in scope performs the client mount, the one thing the nfs driver
adds over dir, exactly as NFSPlugin::activate_storage mounts if the export
is absent from /proc/mounts and then calls SUPER::activate_storage. After
that moment the store is indistinguishable from case 1. Each node then attests:
mounted, writable, fsid, device 10.0.0.9:/srv/shc, witness contents,
observed_at.
The witness cross-check is the one thing this model adds that Proxmox does
not have. At store creation the leader writes a UUID to
<locator>/.shc-store-id. Each node reports what it reads there. All-equal
corroborates the shared claim; any disagreement marks the store as lying and
refuses placement. Proxmox never checks this, and “lying up” about shared is
its worst documented failure: migration skips the copy and the guest starts
against a same-named path holding nothing.
Data lands at /mnt/nas/deployments/acme/huly/prod/mounts/db, literally a path
mount, indistinguishable from local below the selector, which is what the owner
asked for.
reach = {compose01, swarm01, swarm02}, so anchor is that set, no singleton,
UNBOUND. shc move huly --node swarm01 finds the target inside the anchor,
stops the source, re-renders on the target, starts. Zero bytes copied. SHC
writes no placement constraint; a node.hostname== pin would destroy exactly
the relocatability the share was bought for.
If swarm02’s attestation goes stale or reports unmounted, it silently leaves
the anchor and a move to it is refused with the reason (“store nas last
observed 41m ago, TTL 5m”) rather than approved on a link row and warned about.
Case 3: a cloud block volume (rung 2)
Section titled “Case 3: a cloud block volume (rung 2)”shc store add hz-fsn1 --driver block --provider hcloud --location fsn1 --connection hetznershc install postgres --store hz-fsn1shared is schema-refused on driver=block, the way LvmThinPlugin::options()
simply omits shared so that lvmthin: x / shared 1 is rejected by the config
schema rather than believed. Attestation for a block store asks whether the node
can reach the plane’s control API with these credentials, not whether a path is
present.
An app declaring x-shc-mounts: {pgdata: {size: 10Gi, access: rwo}} installs:
the binder provisions, stamps volume.location = fsn1, BumpLeaseAndAttach
performs the single monotonic compare-and-set, the node mounts the device at
<state_home>/deployments/.../mounts/pgdata, and RequireMounted confirms it,
the same gate as cases 1 and 2, now reached because the store said so rather
than because the binder did.
reach(hz-fsn1) = {nodes in fsn1 carrying the credential}, but the attachment
is on one node, so anchor = {swarm01} until a move re-attaches. Because the
mount is exclusive and its reach is larger than one node, SHC must name
which node holds it: a node.hostname== constraint plus replicas: 1, derived
from the live attachment. That is ApplyVolumePlacement unchanged in mechanism
and changed in justification: it is no longer a block special case, it is the
one rule below applied to the one case that satisfies it.
shc move postgres --node swarm02 is pure token math against the persisted
location, with no cloud round-trip: detach, lease bump, attach, mount,
re-render. No archive staged.
The pin rule, stated once
Section titled “The pin rule, stated once”SHC emits a placement constraint if and only if the mount is exclusive and its anchor is larger than one node.
Block: exclusive, reach many, so the pin is forced: without it the scheduler could place the task where the device is not. Share: concurrent, so no pin. Local: the anchor is already a singleton, so a pin adds nothing. This satisfies “unpinned means unpinned” without interpretation: SHC never adds a constraint the operator did not ask for, because the only constraint it adds is the one physics already imposed.
ApplyVolumePlacement today emits replicas: 1 and the hostname constraint in
the same loop body, one mechanism doing two jobs. Splitting them is what makes
the rule testable.
A mixed deployment
Section titled “A mixed deployment”pgdata on hz-fsn1, cache on node:swarm01. anchor = {swarm01} ∩ {swarm01, swarm02} = {swarm01}; the cache silently un-relocates the whole
stack. Declaring cache: {disposable: true} drops it from the intersection and
restores anchor = {swarm01, swarm02}; it is re-carved empty on the target.
That is a declaration, not a heuristic, and it is the same question the binding
model’s Q7 asks from the durability side; see what this
changes.
An empty intersection is refused at install, not discovered at move time.
anchor and bound_to
Section titled “anchor and bound_to”The binding model defines the predicate; this model supplies one term of it.
reach(store) = { n : attestation(store, n).ok ∧ fresh } ∩ (store.nodes ?? all_nodes)scope(m) = m has a volume row with a location ? { n : n.location = that location } : m has a volume row without a location ? { the attachment node } : all_nodesanchor(m) = reach(store(m)) ∩ scope(m)bound_to(D) = { n : ∃ m ∈ critical(D) with anchor(m) = {n} }, unchanged from
the binding model, still a derived set, still nothing stored. BOUND is still
bound_to ≠ ∅; unbound is still the complement. BOUND is not a flag and
takes no code path; it is |anchor| = 1 at deployment scope, and it falls
out of an intersection.
Reach is a resolver, not a column, and the reason is azure-disk. The
tempting simplification is to give each block plane a row per region and read
reach off the store. It fails on the provider whose driver reports no location:
ce’s own comment names it: “local (or a block volume whose provider reports no
location: azure-disk, or an hcloud/EBS bind whose location did not stamp): only
the current holder node can serve the data”. That reach is per-volume and only
knowable post-attach, so no store column can express it. Three implementations
behind one signature; the call site sees one set intersection.
What this deletes. ClassifyMountDecision today has four arms selected by
provider name and row shape: cross-provider, shared filesystem, location-scoped
block, and local. Under this model three of the four collapse into targetNode ∈ anchor(m). The fourth (cross-provider) survives with a better name:
cross-store, the case where the operator asked to re-home onto a different
store, which is a copy by definition and can never be a rebind. That is one
comparison plus one genuinely different operation, replacing a switch on a
string.
One special case survives, and laundering it would be dishonest. A swarm
docker-managed named volume names no store. Its bytes are on whichever node the
task landed on, and SwarmServiceStates decodes {Name, Replicas, Image} with
no node field, so SHC cannot say which. anchor = ∅, which is what an
intersection over nothing gives you, and the binding model already names that
state UNANCHORED and calls it a defect state rather than a posture. This
model does not fix it; it makes it avoidable, because any mount that names a
store has a known anchor. That narrows the binding model’s Q4 considerably;
see below.
The fourth axis: archives are a content type
Section titled “The fourth axis: archives are a content type”backup.repo is not a fourth axis. It is the same three questions with a
different content type. Derived from the consumer side, and independently
corroborated by Proxmox, which has no “backup storage” concept at all: backup
is a content type on the same store object, laid out under dump/ beside
images/<VMID>/ by one hash in the base plugin, and selected by capability:
die "can't use storage '$storage' for backups - wrong content type\n" if (!$scfg->{content}->{backup}); in PVE/VZDump.pm.
Of the questions an archive target must answer, five are the questions a data
store already answers: where do the bytes live; which nodes can read them
(ce says this out loud in its own source: “The default backup.repo is a
NODE-LOCAL directory, so a backup row (raft-replicated, visible cluster-wide)
can reference snapshot bytes that exist only on the node that created it”,
modules/backup/export_peers.go); is there room; is it reachable now; and the
sixth, which is only expressible if both sides are the same kind of object.
So backup.repo becomes backup.store, a name resolved against the
registry and gated on archive ∈ content. The restic URL moves to the store’s
locator; the backup.aws_* / b2_* / azure_* / google_* keys move to the
store’s connection_name, a soft credential reference that
store.connection_name already carries and whose schema comment
already describes exactly this use. Four namespaces for “where bytes live”
become one identifier namespace with referential integrity.
The default is unchanged and deliberately so: the seeded node:<id> store
carries content = {data, archive} and locator = {state_home}, so archives
keep landing at {state_home}/backups/<tenant>/<stack>, byte-identical to
today. Both estates keep working through the upgrade untouched.
Recommendation: the archives-on-the-restore-disk exposure, on its own
Section titled “Recommendation: the archives-on-the-restore-disk exposure, on its own”This is live on both estates today, it is larger than the modelling question, and it should be ruled on separately from everything else in this document.
Second-hand from two independent live audits: backup.repo is null
(core/config/default.yaml:188, verified), neither ops repository overrides it
(verified by grep of both), so on both estates the restic repository sits on the
same /dev/sda1 as the deployments it would restore. Measured: promanager 4.4G
of deployments beside 629M of archives; bitspur 4.6G beside 1.1M. Every worker
node carries its own separate node-local repository as well.
Worse, the operator cannot ask. shc config get backup.repo returns
X400002VALINP key not found on both estates, because a null in
default.yaml is not a declared default and the cascade read treats “no value
anywhere and no declared default” as an unknown key
(modules/config/commands/get.go). That is a bug independent of this model
and should be fixed regardless of the ruling.
And there is a fifth same-device consumer nobody has counted. BackupsDir is
hardcoded to <state_home>/backups/<tenant>/<stack>
(daemon/backup_seam_adapters.go:588-590) and phases.go:210 uses it as the
per-run working directory, irrespective of backup.repo. So pointing
backup.repo at S3 does not move everything off the disk. The four others are
deployments/, the repository itself, restore/<id> staging, and tmp/clone/
for the move archive, the last deliberately kept off tmpfs.
Three recommendations, in order of what each buys:
- Make it visible, do not refuse it.
store.failure_domainis a derived opaque token (node:<id>:dev:<st_dev>for a path store,hcloud:fsn1for a plane,s3:<region>:<bucket>for an archive) written by the registering node and refreshed by the attestor, never typed by an operator. Then “my archives share a failure domain with the data they would restore” is a one-line predicate.shc store lsshows it; a doctor diagnosis raises “N deployments’ archives share a failure domain with their data”. Compare failure domains, never path strings; two paths on one disk is one exposure, and showing paths teaches operators to “fix” it by picking another directory. Refusing would break both estates on upgrade for no safety gain. - Fix
shc config get backup.repo. The operator cannot currently ask where their archives go. - Set it on both estates. Independent of this model: an archive store in a
different failure domain is the only real answer to the relocation ceiling,
and it is the thing the binding model’s own consolation depends on. Its
--discard-localescape and its slice-5 recoverability report are offering a copy that today dies with the original.
The same predicate pays a second time as the free-space preflight the move heavy
path completely lacks: there is no Statfs guard anywhere in modules/clone
or modules/backup, so an over-large move fails mid-flight after the archive
has consumed the disk, with the source already stopped (phase order is
backup → stop → recover → uninstall). Once archive can name a different store
the ceiling stops being structural; until then the preflight at least fails
before stopping the source.
The promise
Section titled “The promise”Shared storage means a deployment can be moved to another node without copying its data. It does not mean the deployment survives a dead node by itself.
That sentence belongs at the top of the operator documentation, because “shared storage” reads as high availability to every operator who has met it elsewhere, and that misread is how the recorded trilemma gets re-opened silently a year from now. SHC picks operability and reliability; ruling BUILD on rung 3 does not spend the third corner, provided the promise is stated this way, because every transition stays operator-initiated.
Three machines would be needed for the stronger promise, and this model builds none of them:
- A store-liveness authority that can declare a node dead. Attestation only decays to UNKNOWN and fails closed; it never adjudicates. The storage module registers exactly one cron, and it reaps expired attachment leases.
- A fencing authority for shares.
lease_generationfences a device; detach is the fence. A shared directory has no device to detach, so “the node looks dead, start it elsewhere” would be two writers on one directory with nothing between them. - An automatic actor.
StopDeploymentis deliberately “a normal StopStack, never a force-detach”, and the only presumed-dead route is an operator-invoked verb.
The single-writer story under swarm
Section titled “The single-writer story under swarm”A share is concurrent at the filesystem, which is precisely why nothing else stops two replicas of MariaDB writing one datadir. Three facts, all verified at the pin:
ApplyScaleExtensionclamps authoreddeploy.replicasdown tox-shc-scale.maxat render, and then strips the marker from the rendered compose (modules/app/scale_extension.go). Enforced once, then erased.ApplyVolumePlacementforcesreplicas: 1, but only for RWO block mounts, and zero of 73 catalogue apps declare a block mount. It never fires.Service.Scalevalidatesreq.Replicas == nil || *req.Replicas < 0 || *req.Replicas > 100and nothing else (modules/app/service.go:865-868, read directly), then dispatches straight todocker service scale.
So shc stack scale <stack> mariadb 3 succeeds today against an app declaring
max: 1. The owner’s non-bypassable ruling is currently unmet at HEAD, on
both runtimes, and it is unmet independently of storage.
The fix belongs at the replica-deciding seam, not in the storage model:
effective_max = min( authored x-shc-scale.max , storage_ceiling )storage_ceiling = 1 if any non-disposable mount is exclusive, else unboundedchecked in Service.Scale, re-reading the authored max from the packaged source
compose that survives at <deployDir>/app/compose.yaml, which ScaleWindowOps
already does, so this needs no schema change and no new file. 49 apps declare
x-shc-scale across 62 max: 1 lines (measured). Under this model most of
those become redundant rather than load-bearing, which is the right outcome: the
constraint should come from the shape of the data, not from every app author
remembering.
Storage does not make the catalogue clustered. Even on CephFS every mount
defaults to rwo. A share moves Postgres; it does not replicate it.
What this model cannot do
Section titled “What this model cannot do”- It does not survive a dead node automatically. A node dies and its BOUND deployments stay down until an operator moves them. Deliberate; see the three missing machines above.
- It cannot prove ongoing content identity on a shared store. The witness file proves every node saw the same tree at probe time. A split-brain re-export, or two SHC installs pointed at one export, can pass it. Proxmox documents the same boundary: “It is not advisable to use the same storage pool on different Proxmox VE clusters. Some storage operation need exclusive access”, because its locks are cluster-scoped and nothing spans two clusters. A dqlite-recorded claim means nothing to a second daemon writing the same directory.
- It does not rescue an unanchored swarm named volume. No store, no anchor,
data wherever the task landed, and
swarmNamedVolumeBackupWarningsalready says the capture gap exists. This model makes binding it to a store possible; it does not do it. - It does not model
vapps/.preservedDeploymentState = {mounts, vapps}(verified,modules/app/install_runner.go:2080), and the in-file comment is explicit that vApps are per-instance state nothing re-derives, and that removing the directory both strands every instance and silently disarmsshc protect. It is control-plane metadata, not app bytes, so it should not be a mount on a store, but under this model data relocates while render artifacts stay node-local, so a move must carry it. This is a real loose end and it is called out as an open question. - It cannot express capability that varies by image format. Proxmox needs a
third dimension because qcow2-on-NFS snapshots and raw-on-NFS does not
(
die "can't snapshot this image format" if $format !~ m/^(qcow2|qed)$/). SHC has no image formats (a mount is a directory or a block device) so a per-store method refined by the mount’s own filesystem is enough. If SHC ever gains an image format,snapshot_methodbecomes the wrong shape. That is the tripwire. - It adds no free-space preflight. It makes the archive target selectable and the same-domain condition computable; someone still has to write the check.
- It cannot detect two different NFS exports that resolve to one backing filesystem on the server. A path-containment refusal at store-add catches nested locators on one node, which Proxmox documents as a hazard (“Such an aliased storage configuration can lead to two different volume IDs pointing to the exact same disk image”) and does not detect. Beyond that, nothing.
- Store names are cluster-unique while a path store’s meaning is per-node,
which is why the seeded local stores are
node:<id>rather than one sharedlocal. On a three-node clustershc store lsshows three local rows. That is honest (if the bytes differ by node it is not one store) but it is more rows than an operator expects. - It does not adopt an existing non-SHC dataset in place. A store’s locator
is create-only and its layout is
<locator>/deployments/....
Migration and slices
Section titled “Migration and slices”Append-only throughout, and the tooling already enforces it: migrate/flatten
self-refuses unless I_UNDERSTAND_FLATTEN_BREAKS_LIVE_ESTATES=1, with the
comment that “this repo has learned twice that a documented-but-unenforced rule
is worse than none”. Current history is five .sql files plus atlas.sum.
The migration is unusually cheap because there is nothing to convert. Second-hand
from two live audits: shc storage ls is empty and shc volume ls is empty on
both estates, deployment.storage is NULL on all 24 deployments, backup.repo
is unset, and zero of 73 apps declare a block mount. All three rungs have zero
installed base. This is a forward migration, not a data conversion, and that
window will not stay open.
The DDL-mirror obligation is narrower than assumed. The shc storage and
volume tests run ent’s own client.Schema.Create, so those tables have no
mirror at all. The deployment mirror that does exist,
tests/unit/modules/tenant/service_test.go, is a four-column minimum-viable
subset (id, tenant_name, status, name) so adding deployment.store
does not redden it. Verify by running the shc unit suite before and after; do
not take this on trust.
Slice 1: the funnel. No schema. Output byte-identical.
Section titled “Slice 1: the funnel. No schema. Output byte-identical.”One accessor, MountsRoot(tenant, stack, environment), returning
paths.StateHome()-derived output for every deployment, exactly as today.
Measured at the pin: 26 non-test sites construct a mounts path, across
roughly 15 files. Route all of them. This slice is worth landing alone, because
a bug in it is invisible and the only proof available is diffing rendered paths
before and after.
Five sites need individual attention, not mechanical replacement:
modules/backup/manifest_inventory.go:91:Path: "mounts/" + nameleaks the layout into the backup manifest format. This is an archive compatibility change, not a path change.modules/backup/restore.go:1033, 1229, 1292: threefilepath.Walksites that each take the first directory namedmountsandSkipDir. These do not fail when a root moves; they silently restore the wrong tree. They must take the resolved root. This is the worst site in the program and it appears in no grep forDeploymentDir.shc/modules/supportbundle/certs.go:90: a hardcoded six-segment glob ending.../mounts/acme/*.json, which will return zero matches silently and quietly drop ACME key-presence reporting from every support bundle.
Three invariants survive unchanged because they compare basenames under a
directory they were handed: preservedDeploymentState = {mounts, vapps},
separatelyArchivedDirs = {mounts, app}, orphanEnvSkipNames = {mounts, files}
(all three verified at the pin). mounts/ stays a literal path segment in the
formula, so nothing about them changes.
If the program stops here: nothing changed. One accessor exists.
Slice 2: schema and seed. Zero bytes move.
Section titled “Slice 2: schema and seed. Zero bytes move.”Rename the ent type StorageProvider → Store, keeping the physical table via
the entsql.Annotation{Table: "store"} that already exists; the
logical rename is free, no DDL. Add driver, shared, nodes, content,
require_mounted, state, failure_domain. Add store_attestation; drop
store_node (empty on both estates). Add volume.store_id
replacing free-text provider (retired in place). Add deployment.store;
retire deployment.storage in place.
Seed, idempotently, at node self-register, the same place node.location is
already stamped: one node:<id> store per node, driver=dir,
locator={state_home}, shared=false, require_mounted=false,
content={data, archive}, nodes=[self], plus that node’s own attestation.
N stores, one per node, never one store with reach N; if the bytes differ
by node it is not one store, and modelling it as one is Proxmox’s documented
data-loss case. Backfill deployment.store for all 24 rows.
If the program stops here: shc store ls tells the truth for the first
time. Nothing else changes, and no byte has moved.
Slice 3: evidence and gates. Still zero bytes moved.
Section titled “Slice 3: evidence and gates. Still zero bytes moved.”- Node-written attestations on a node-local cron; every path touch fork-isolated
and deadline-bounded, per-store parallel. Reach reads attestations with a
freshness bound; stale decays to UNKNOWN and UNKNOWN fails closed.
nodeSupportsProvider(today a bareCOUNTover the link table, matched by provider type across all rows, so any NFS row anywhere satisfies a question about a different store) is rewired.X503085SHFSMT, a warning standing in for a gate, becomes unreachable and is deleted. store.require_mounteddrivesPrepOp.RequireMounted. Highest value per line in the whole program.--storage→--store, validated against a row.- The replica ceiling moves to
Service.Scale. backup.storeand content-type gating;failure_domain; the doctor diagnosis; theshc config get backup.repofix.- One capability
Methodenum with a package-level exhaustiveness assertion. There are three independently-maintained lists today and the drift is already a live bug:probeLVMreturnsInstant: truefor thin LVM whilesnapshots.Get("lvm")is not a registered method, so every LVM-thin host silently pays double-pass rsync for a capability it has. Adding CephFS to only one list would reproduce it exactly. The compile-time array-size assertion atsubsystems/impl/volume_failover.go:75is the house precedent for the technique.
If the program stops here: the system refuses honestly. One store per node, one archive target per node, everything visible, nothing relocatable that was not relocatable before.
Slice 4: a store may name a different locator. GATED. This slice moves bytes.
Section titled “Slice 4: a store may name a different locator. GATED. This slice moves bytes.”Rung 1 lands here. The gate has four conditions, all four asserted on the post-injection artifact rather than on a guard’s return value:
- Three decoy mutation tests, red before green. Seed two stores; plant a decoy
mounts/directory the walk reaches first; assert the restore took the right tree, assert the relocated deployment’s mounts classify persistent rather than falling through to host binds, and assert the support-bundle glob returns non-zero ACME matches for a non-default-store traefik. - The six single-root
DeploymentsDir()consumers enumerate across stores. require_mountedasserted at render for any non-default store.- Path-containment refusal at store-add.
The classifiers are kept, not deleted, and re-anchored.
PersistentMountsFromCompose computes mountsPrefix := filepath.Join(deployDir, "mounts") + separator and hostbind.go computes the same deployPrefix
inverted (both verified at the pin). Two designs proposed replacing both with a
SELECT over a per-mount table. That is rejected, for the reason the binding
model already gives: durability is derived, nothing stored. A derivation cannot
drift; a row is a claim that every mount-creating path (install, recover,
restore, both uninstall variants, vApp install and uninstall, migrate-from,
group restore) must remember to write. Miss one and the mount becomes invisible
to reach, invisible to the anchor, and invisible to backup capture while the
operator is told the backup succeeded. So the classifiers keep deriving; they
simply derive from a resolved root instead of an assumed one, and they must
be re-anchored in this slice, not after it.
Which root do they resolve? The one recorded in the deployment’s rendered state at render time, not a fresh lookup. A store row edited after install must not retroactively re-anchor a deployment whose bytes are already somewhere.
Slice 5: rung 3 goes live.
Section titled “Slice 5: rung 3 goes live.”--driver nfs|cephfs, the client mount, the witness cross-check,
shc store status. Binding-model decision-table cells 2 and 7 become reachable
for the first time.
Slice 6: rung 2 folds in.
Section titled “Slice 6: rung 2 folds in.”volume.store_id becomes the selector, volume.provider retires in place,
CanonicalVolumeProvider, which exists solely because a spelling drift “crept
into READ sites … and silently missed every persisted row”, becomes
deletable, and three of ClassifyMountDecision’s four arms collapse to
targetNode ∈ anchor(m). Last, because block is the only rung with a
working state machine (the bounded compare-and-set in
modules/storage/volume_repo.go, the attachment lease, the compile-time
ordering assertion) and zero users. Worst risk-to-reward to touch early.
What Proxmox got right, and what it got wrong
Section titled “What Proxmox got right, and what it got wrong”Sources: the PVE admin guide chapter chapter-pvesm.html and its asciidoc
source; the per-backend wiki pages; and upstream Perl at git HEAD of
pve-storage, pve-manager and qemu-server on git.proxmox.com.
Taken:
- A shared filesystem is a path, and after the mount it is the directory
plugin.
NFSPluginandCephFSPlugininherit 100% of layout and path computation fromPVE::Storage::Plugin. This is the owner’s stated preference and upstream proves it works. - Create-only identity fields.
DirPlugin::options()markspath => { fixed => 1 }; NFS markspath,server,exportfixed. Where a store is is create-only; what it is allowed to do is editable. nodesandsharedare separate fields and neither derives from the other.nodesis an operator restriction;sharedis a claim about content identity. The docs are explicit that they compose. Collapsing them into one reach number loses the restriction axis, which is the one operators use daily.- Content types, which is what absorbs the backup target. One store serves
guest disks and dumps by listing
content images,backup; selection is by capability. is_mountpoint. “Assume the given path is an externally managed mountpoint and consider the storage offline if it is not mounted.”- Late-bound store-qualified identifiers.
<STORAGE_ID>:<volume-name>; the absolute path is never stored and is recomputed at use time. Moving a store’s path rewrites zero guest configs. - Fork-isolated, deadline-bounded probes. Even a bare
-d $pathruns underrun_fork_with_timeoutwith a 2-second deadline;dfthe same; NFS reachability 10 seconds with a TCP-ping fallback. - DECLARE, DISCOVER and OBSERVE as separate verbs:
pvesm add,pvesm scan,pvesm status, none of them conflated with the config record. - Schema-level refusal where a flag is meaningless.
LvmThinPlugin::options()omitsshared, soshared 1on a thin pool is rejected rather than believed.
Deliberately not taken:
is_mountpointdefaults tono. “directory is expected to be a mount point but is not mounted” is one of the most-searched PVE errors, precisely because the safe behaviour is opt-in. This model inverts the default: any store whose locator is not the node’s own state home requires it.sharedas a pure declaration with no witness. Proxmox never verifies it, and lying up about it is its worst documented failure. This model adds the witness file and a leader-side cross-check.- Per-technology plugin proliferation. PVE 9 dropped GlusterFS outright as
unmaintained upstream, and the migration advice is to mount it by hand and add
it as a directory storage. The mount-it-yourself
dirpath is the one that survives, so this model keeps the driver count minimal and makesdirfully capable. - A serial synchronous status sweep. One hung NFS server wedging
pvestatdand greying out a whole cluster is the classic PVE outage. SHC’s attestor is per-store parallel and deadline-bounded from the start; retrofitting that after the first incident is not a plan. - No aliased-storage detection. Proxmox documents the hazard and does not detect it. SHC’s store rows are cheaper to create and its paths far likelier to nest, so a path-containment refusal at store-add is required.
- Mixed content with no quota. “If a storage runs full, all guests using volumes on that storage receive IO errors. This can cause file system inconsistencies and may corrupt your data.” Mixing archives with live data is Proxmox’s default arrangement and its only defence is that paragraph. This model adds the failure-domain diagnosis and a per-store threshold.
- Persisting no evidence. Proxmox probes synchronously at use time and memoises within one operation. That is simple and always current, and it is a genuine alternative to attestation; see the weakest point.
The weakest point, and what would overturn it
Section titled “The weakest point, and what would overturn it”The classifiers keep deriving, and they now derive from a root that must agree with the root the renderer used.
This is the recommendation’s load-bearing choice, and it is the one I am least
certain of. Rejecting a per-mount table keeps capture and durability
derivation-driven, which is the property that has kept the fleet’s data intact
through all the accretion: a derivation cannot drift because it re-reads the
truth on every call. But it moves the failure into a narrower place rather than
removing it: PersistentMountsFromCompose, hostbind.go and the three restore
walks must all resolve the same mounts root the renderer wrote to. If a
store row is edited, or if one path re-resolves live while another reads the
recorded value, the classifier silently mis-anchors and the failure is quiet in
exactly the direction that loses bytes.
The mitigation is in the design: record the resolved root in the deployment’s rendered state at install and have every classifier read that, never re-resolve, but the mitigation is itself a stored value that can drift from the filesystem, so it converts a two-party agreement into a two-party agreement with one honest witness. That is better, not solved.
What would overturn it: if per-mount store overrides turn out to be common
rather than exceptional (a stack with pgdata on a block plane and the rest on
a share is not exotic) then one recorded root per deployment stops being
enough, every classifier needs a per-mount root, and a per-mount record becomes
unavoidable. At that point the binding model’s “derive, never store” doctrine
has to be re-opened with the owner rather than worked around, and the answer is
probably a per-mount row plus a fail-closed drift detector that refuses the move
and refuses the backup-succeeded claim whenever the rows and the filesystem
disagree.
Two smaller weak points, named so they are not discovered later.
The freshness bound on attestation converts a correctness problem into an
availability problem, and I am inventing the number. A node that is up but whose
attestor is wedged silently leaves every anchor and stops being able to receive
moves. That fails closed, which is right, but it is a control-plane failure mode
caused by a monitoring loop rather than by storage, and it does not exist today.
Proxmox’s alternative (probe synchronously at use time, persist nothing) is
genuinely simpler, and every consumer of reach in SHC is a rare, synchronous,
operator-initiated operation, so it would be sufficient. I chose persisted
attestation for shc store status and the doctor diagnosis, which is a real
buy; I have not measured the trade, and if the cluster peer transport cannot
perform a bounded-deadline cross-node probe the synchronous option is not
available anyway. That is worth checking before slice 3 is written.
The witness cross-check proves agreement at probe time and nothing after.
Corrections to the record
Section titled “Corrections to the record”Every one of these contradicts a fact the task brief, one of the three candidate designs, or the panel treated as current. They are listed because each is load-bearing somewhere else.
- ★ The
--storageone-flag data-loss path is already CLOSED upstream. ce7f427cc(“fix(move): an unverifiable--storageclaim can no longer relocate a deployment onto an empty volume”, 2026-08-02) landed before this document was written and is in the current shc pin. A Rebindable or Shared verdict now requiresProviderProven(a persisted volume row whose provider column is non-empty) on both the live and the dead-node classifier. The deployment stamp is still read as a fallback name, and its comment now says so in full, but it can no longer buy a no-copy rebind. Every candidate design and two panel lenses ranked “delete the positive read of--storage” as the highest-priority first slice on the grounds that it is the only live loss path. It is not live. Sequence the program without it. - ★ The shared-filesystem arms of both move classifiers are dead code at
HEAD, by that same commit’s analysis: nothing in the product can write a
volume row for a shared filesystem; the binder writes rows only for block
backends, and the ent
Volumeschema constrainsaccess_modeto('rwo','rox')with the comment “RWX stays on the shared-FS (cephfs/nfs) path and is never a volume row”. A test asserts the schema half against a real database. The arms were kept deliberately as a tombstone pending the owner’s ruling. This model is what makes them live again, and it does so by putting the bytes on the share rather than by trusting a label. ClassifyMountDecisionhas FOUR arms, not three. Cross-provider is first, and it is not a variant of the other three; it is a re-home onto a different store, which is a copy by definition. All three candidate designs said three and proposed collapsing all of them.isSharedFSProvideris already a derived predicate, backed by a single map instorageapiand two AST guards: one that reddens if aProviderTypeis added without being classified, one that reddens on any string literal in the daemon package spelling a shared kind. Two designs proposed deleting aType == NFS || Type == CephFSexpression that upstream has already collapsed.ce/migrationsholds FIVE.sqlfiles, not four and not the “5” one audit reported before a later correction to four. A fifth (20260802033902_node_voter_slot.sql) landed 2026-08-02.- The three first-match
mountswalks inmodules/backup/restore.goare at lines 1033, 1229 and 1292 at the pin, not 974 / 1170 / 1233. They are production restore code, not tests, and they fail by picking the wrong tree rather than by failing. - A
deploymentcolumn addition does NOT redden the shc DDL mirror. The mirror is a four-column minimum-viable subset. One audit stated the opposite and it changes the cost of the schema slice. BackupsDiris a per-run working directory, not the archive location. It is hardcoded to<state_home>/backups/<tenant>/<stack>and used byphases.goirrespective ofbackup.repo, so a remotebackup.repostill leaves a same-disk consumer. One design flagged the hardcode but read it as “backup.repodoes not move everything”; the precise reading is that this is a fifth consumer nobody counted.x-shc-scalecounts: 49 apps, 62max: 1lines (measured atd6b3cb0). The brief said 48 apps and 62 declarations.bound_todoes not exist in code. Confirmed by grep. It is this document’s and the binding model’s proposed term, a design to be made, not an invariant to preserve. That is good news: nothing has to keep working.- The
access_modeschema comment says both halves. One design cited “so a future ‘rwx’ addition doesn’t force a Go-code change” as the schema anticipating a widened volume table. The same paragraph also says RWX “is never a volume row”. Both are in the same comment, and this model takes the second half: shared filesystems are stores, not volumes. - The relocation ceiling is not “~45% of the disk”. That number could not be confirmed. Cite the shape: a same-node relocation needs the dataset, the archive and the restored copy free at once (roughly one third); a cross-node one needs the dataset and the archive on the source (roughly one half). And there is no free-space guard at all, so the failure is mid-flight rather than a refusal.
What this changes in the binding model
Section titled “What this changes in the binding model”The deployment binding model is the consumer of everything here. Its open questions, one by one.
Q1: --storage names three questions; which is real? ANSWERED, and its
recommendation is superseded by the owner’s ruling. That analysis recommended
FINISH / FINISH / RETIRE, retiring rung 3 on the grounds that its unique buy
is high availability for stateful workloads, which the trilemma permanently gave
up, and that its non-unique buy never arrives because relocateBlocked := nodeChanging && !swarm sends every compose relocation to backup+restore anyway.
The owner ruled BUILD on rung 3. This model builds it and narrows the
promise to relocatability, which does not spend the trilemma’s third corner,
and it removes the second objection by construction, because a shared store is
the case where a compose relocation genuinely has nothing to copy. The three
rungs stop being three questions: one selector, --store <name>, names the
place, and the driver decides how the place becomes bytes.
Q1’s vocabulary table is changed. It proposed four nouns for four axes:
store, volume, share, repository. This model keeps two (store,
volume) and retires share and repository as entity names, because four
nouns for four kinds of one thing is the straddle in a new spelling. Q1’s
negative rule survives verbatim and is stronger here: a shared kind must never
be a legal value in a slot that means “which of this node’s disks”; under
this model that slot does not exist, because the slot names a store and the
store knows its own reach.
Q1’s amendment 1 is ANSWERED: backup.repo is a content type on a store,
not a fourth axis. Its amendment 2 is ACCEPTED: rung 1 must not silently
re-admit rung 3 through the back door, and the probe it demanded is exactly this
model’s attestation, made mandatory in slice 3 before any store may name a
non-default locator in slice 4.
Q4: should install auto-pin, or refuse? NARROWED. UNANCHORED arises when SHC cannot say which node holds a mount’s bytes. Any mount that names a store has a known anchor, so under this model the question survives only for the residue: docker-managed named volumes that name no store. That is a much smaller question with a much cheaper answer, and it removes the objection that auto-pinning makes swarm silently non-HA for stateful workloads; a stateful swarm service on a shared store needs no pin at all.
Q7: should there be a rebuildable durability class? OVERLAPS. This
model’s per-mount disposable: true declaration in x-shc-mounts is the same
question arriving from the storage side: a mount an operator is willing to lose
should not collapse the anchor intersection. Q7 asks whether losing it should
warn instead of refuse. The two should be ruled on together, and they should
be one declaration, not two; otherwise an app author declares a cache
throwaway for relocation and critical for removal, or the reverse.
Q2, Q3, Q5, Q6, Q8: UNTOUCHED. Whether the data-loss refusal is
force-able, where durability is declared, who wins between a binding pin and
an app’s own deploy.placement, whether “unbound but nowhere to run” warns or
refuses, and whether cluster repair’s sweep survives are all deployment-scope
questions this model neither answers nor complicates.
The decision table changes in one place. Cells 2 and 7 (an unbound deployment whose critical mounts sit on a shared filesystem) are recorded as unreachable, because the schema cannot express a shared data path. Slice 5 makes them reachable. That is the point of the exercise, and it should be re-verified against the table when slice 5 is planned.
Open questions: owner rulings
Section titled “Open questions: owner rulings”S1. Refuse a new install whose archive store shares a failure domain with its data store, or only warn? The recommendation is warn, because refusing breaks both estates on upgrade and the exposure predates the model. “Warn on existing, refuse on new” is a coherent third option and it is the difference between a diagnostic and a guarantee. No prior ruling was found.
S2. Does a share hold only mounts/, or the whole deployment directory?
This model says only mounts/, and stakes the render-collision argument on it.
The three dead provider implementations all lay out {mounts, files, app},
implying the whole tree was once intended. Deciding otherwise changes slice 1’s
blast radius from 26 sites to roughly 141 and puts the rendered compose on the
share.
S3. Where does vapps/ ride? It is preserved per-instance state that is
neither app bytes nor a render product. Under this model it stays node-local
while data relocates, so a move must carry it explicitly, and it arguably
belongs in the database rather than on disk. This is a genuine loose end.
S4. Persisted attestation, or synchronous probe at use time? Proxmox
persists nothing and re-probes. Every consumer of reach in SHC is
operator-initiated and synchronous, so the simpler option would work, at the
cost of shc store status and the doctor diagnosis. Persisting adds a freshness
constant nobody has data for and a new control-plane failure mode. Check
first whether the cluster peer transport can perform a bounded-deadline
cross-node probe; if it cannot, this question is closed by the transport.
S5. Is the x-shc-scale.max fix in scope here? It is a replica-path fix,
not a storage fix, and it is currently unmet. It could land before slice 1 and
be independently valuable.
References
Section titled “References”- The deployment binding model:
durability,bound_to, the decision table, Q1’s three-rung analysis - The node addressing model: owns
bind; this document does not touch that glossary - Proxmox VE storage:
https://pve.proxmox.com/pve-docs/chapter-pvesm.html,https://pve.proxmox.com/wiki/Storage pve-storageupstream:Plugin.pm,DirPlugin.pm,NFSPlugin.pm,CephFSPlugin.pm,LvmThinPlugin.pmathttps://git.proxmox.com/?p=pve-storage.gitpve-managerPVE/VZDump.pm,qemu-serverPVE/QemuMigrate.pm