Skip to content
SHC Docs

The storage model (proposal)

Status: open design. Slices 1 and 2 are built (the store registry, attestations and the mounts root), and the 2026-08-14 models rulings rule the open halves. What is built after slice 2 is not recorded here.

Written 2026-08-02. Verified first-hand against ce at f222808 (the shc go.mod pin, origin/main) in the clean worktree /var/tmp/ce-storemodel, and against shc at d6b3cb0 (origin/main). Neither production estate was contacted by this document. Live facts (empty registries, 24 deployments, archives on the deployment disk) come from three earlier read-only audits and are attributed as second-hand every time they are used.

The owner’s framing:

“Part of the problem with storage and its straddling things is initially storage actually just was a path on a node, and then later I added it to be a path on a shared storage provider. And then later I decided to add block storage. So it’s kind of like things being slapped on top of other things that weren’t architected correctly. So what I want you to do is think about how you would design something that supports all three, but from a first principles approach. Take some hints and thoughts about how Proxmox does this because I believe Proxmox supports shared storage as well as local, as well as block. But for shared storage, it would be nice if it’s actually literally the path mounts just like local is.”

This document consumes and must not contradict the deployment binding model. That document supplies durability (would losing these bytes be data loss) and the bound / unbound / bound_to / anchor vocabulary. This document supplies the other half (anchor, the set of nodes that can serve a mount’s bytes without copying) and nothing else. Where the two disagree, the binding model wins on deployment scope and this one wins on storage scope; the disagreements are enumerated in what this changes in the binding model.


A store is a named place that holds bytes for deployments. It has one locator, a set of nodes it can be reached from, and a declaration of what it is allowed to hold. Local disks, NFS and CephFS shares, cloud block planes and archive repositories are all stores; they differ only in how the locator turns into bytes on a node.

Everything else in the model is derived from two per-mount questions the store answers:

  • REACH: which nodes can serve these bytes without copying them.
  • EXCLUSIVITY: may more than one node hold them open at once.
casedriverlocatorreachexclusivitywhat SHC does
node-local diskdiran absolute paththe one node that owns the pathnot applicable; nothing else can reach itverifies the path is present, and mounted if declared so
shared filesystemnfs / cephfsan absolute pathevery node that attests the path is mounted and writableconcurrentmounts it, then treats it exactly as dir
cloud block planeblocka provider plus region token (hcloud/fsn1)every node in the region that carries the credentialone node at a time, per deviceprovisions a device, fences, attaches, mounts it at the same path a dir store would have used
archive repositoryrestica restic repository URLevery node carrying the credentialconcurrent, restic-lockedhands the locator to restic and carves nothing

Four tables carry it, and three of them already exist:

tabletodayunder this model
storestore, a registry of places, written with no probe of any kindsame physical table, six columns added, every capability column probe-written
store_attestationstore_node, an operator assertion consumed as evidencereplaced: a row written by the node about itself, carrying observed_at
volumea provisioned block deviceunchanged, except provider free text becomes store_id
volume_attachmentthe single-writer leaseunchanged, not touched

Two things this model deliberately does not add: a per-mount row, and a fourth noun for backups. Both are explained below, and both are load-bearing.


Mostly. It holds completely for the owner’s stated case (shared storage is literally a path mount, indistinguishable from local below the selector) and it breaks in exactly two places, neither of which needs a new concept.

Where it holds, and the proof is already in the tree. All three filesystem provider implementations build the identical subtree under a base path: deploymentPath → MkdirAll(0o750) → for _, sub := range []string{"mounts", "files", "app"}, byte-for-byte in modules/storage/providers/local/provider.go, .../nfs/provider.go and .../cephfs/provider.go. Three types, one layout. Proxmox reached the same answer at source level: PVE::Storage::NFSPlugin and PVE::Storage::CephFSPlugin both declare use base qw(PVE::Storage::Plugin) and override only check_config, activate_storage, status / deactivate_storage and check_connection. They override nothing about allocation, naming, listing, snapshotting or path computation; get_subdir() is "$path/$subdir" in the base directory plugin and NFS inherits it whole. Mount it at a path, then delegate to the local implementation.

Break 1: a block store has no path until a device is attached, and the joint is narrower than it looks. DaemonVolumeBinder.hostMountpoint already computes “the SAME <deployDir>/mounts/<name> path the host-bind branch resolves to, so ApplyPrep runs byte-identically”. A block-backed mount and a host bind resolve to the same string. So the path is not dynamic; it is a pure function of (store, tenant, stack, environment, mount). What differs is who makes that path have the right bytes before the container starts, and that step already has a universal gate: PrepOp.RequireMounted (core/volume/prep.go:122) refuses with X409034VOLNMT before any mkdir, seed or chown. The gate is correct. It is wired to the wrong predicate: one setter in the whole tree, modules/app/install_runner.go:3189, in the block binder. So a block store is a dir store with one extra step before the path is real, and the model expresses that as a driver, not as a kind.

Break 2: an archive repository is a namespace, not a path, and it never can be. LocalResticRepoPath returns the empty string for s3:, b2:, gs:, azure:, sftp:, rest:, swift: and rclone: (modules/backup/restic.go), and the shipped default.yaml comment offers s3:s3.amazonaws.com/mybucket/backups and b2:bucket-name:/backups as the multi-node configuration. An off-box archive is the point of an off-box archive, so excluding these is not available.

The formulation that survives both breaks. A store is a namespace that resolves (deployment, mount, content) to a location, and for three of the four drivers that location is a path. Nothing downstream ever needed the path; it needed the resolver. storageapi.Provider.BasePath(), which has zero production callers today, was already the right shape and was never called.

The two axes held. Reach is real and it is the whole of anchor. Exclusivity is real, and Proxmox’s model corrects the brief’s placement of it: a shared iSCSI LUN carrying an LVM volume group is shared 1 and serves many concurrently-running guests, each on its own logical volume. One-at-a-time is a property of a device, not of the store that mints devices. SHC agrees already: volume_attachment.volume_id is UNIQUE (one attachment per volume) while one block plane serves every deployment on the estate. So exclusivity lives on the provisioned volume, where the lease already is.


One word meaning three things is what produced this. Every surviving concept, checked against the binding model’s reserved set: bind (addressing, owned by the node addressing model, untouched), binding → provisioned (binding model slice 8), and bound / unbound / bound_to / anchor reserved for deployment scope only.

wordmeanscollision check
storea named place that holds bytes: one locator, a reach, a content setthe binding model’s Q1 already proposed store for rung 1. This model widens it to all four axes. --store is unused across ce’s flag registrations
driverhow a store’s locator becomes bytes on a node: dir, nfs, cephfs, block, resticnew word, no collision. Selects an implementation; nothing branches on it outside the store package
contentwhat a store may hold: data, archiveProxmox’s word for the same idea
reachthe node set that can serve a store’s bytes without copyingthis document’s word. It is a resolver, never a column; see anchor
attestationone node’s own observation of one store, with observed_atnew word, no collision. Deliberately not “status”: a status is a fact, an attestation is a claim with a timestamp and a writer
volumeunchanged, a provisioned block device rowalready the code’s word, and the binding model’s rung-2 word. Slice 8’s hasBinding → hasVolume rename makes it unambiguous
attachmentunchanged, which node currently holds a volume, and the lease generationalready the code’s word

Two nouns the binding model reserved are retired before they are spent.

  • share was to be rung 3’s noun. Under this model a share is a store whose driver is nfs or cephfs. Keeping a separate noun would re-create the straddle: two words for one object, told apart by a string.
  • repository was to be the fourth axis’s noun. Under this model an archive repository is a store whose content is archive. The word survives as restic’s word for its own on-disk format, which is what it actually names.

--storage becomes --store <name>, and it is validated. Today the flag takes an unvalidated free-text label; shc storage add cannot even produce a value that is not local, because it hardcodes Type: "local" (modules/storage/commands/add.go:78) and offers no --type. Under this model --store names a row, is refused if the row does not exist, and selects the store for every mount of the deployment that does not override it.

deployment.storage is RETIRED IN PLACE and deployment.store is added. Migrations are append-only, so the column stays; no code reads or writes it. There is an exact in-repo precedent with the reasoning already written down: deployment.launcher, ent/schema/deployment.go:73-78: “RETIRED IN PLACE (pre-1.0) … No code reads or writes the column; it stays in the schema only because migrations are frozen (atlas.sum). Always NULL on new rows.” Reusing the old column with a new meaning is the accretion pattern this document exists to stop: one name, two meanings, told apart by when you read it.

x-shc-volumes becomes x-shc-mounts. The rename is free: zero of 73 catalogue apps declare x-shc-volumes (measured). Each entry carries {store, size, access, disposable}; store is the per-mount override of the deployment’s --store. The parser must stop being permissive: today an unrecognised access token falls back to RWO, commented “the parser stays permissive” (core/volume/binding.go), so access: rwx (a request for shared storage) silently yields a single-writer block volume. Under this model an unknown token is a parse error.


The path formula is uniform and is the whole trick:

data <store.locator>/deployments/<tenant>/<stack>/<env>/mounts/<name>
archive <store.locator>/backups/<tenant>/<stack>

Set the seeded default store’s locator to {state_home} and every one of those strings is byte-identical to what the system produces today: DeploymentDir, MountsRoot, DaemonBackupPathsResolver.BackupsDir. The existing state-home tree already is a correctly laid-out store. That is not a coincidence to exploit; it is evidence the layout was always right and only the selector was missing.

Render artifacts never move. compose.yaml, manifest.json, app/, vars.yaml, runtime/, state.json stay node-local under <state_home>/deployments/... unconditionally, forever. Two reasons, and the second is the decisive one. They are regenerated from the manifest on every install and recover, so nobody ever asks which nodes can serve them without copying. And two nodes rendering one deployment into one shared directory collide: the source’s rollback snapshot and the target’s fresh render resolve to the same path. Only mounts/ names a store.

Case 1: a second disk on one node (rung 1)

Section titled “Case 1: a second disk on one node (rung 1)”
Terminal window
shc store add fast --driver dir --path /mnt/nvme0 --node swarm01
shc store status fast
shc install gitlab --store fast

require_mounted defaults true, because the locator is not under the node’s own state home. swarm01’s attestor forks a deadline-bounded -d /mnt/nvme0, reads /proc/mounts (the isMountPoint helper already exists at modules/storage/doctor.go), write-probes a dotfile, writes a witness file, and attests ok with the filesystem it found. Every other node attests absent, correctly, and that is why this is one store with reach 1, not one store with reach 3.

Install resolves /mnt/nvme0/deployments/acme/gitlab/prod/mounts/data. ApplyPrep runs with RequireMounted true, so if /mnt/nvme0 failed to mount, install refuses instead of seeding the image’s initial content onto the bare mountpoint directory on the root filesystem, where it would be shadowed the moment the disk came back. That is the single highest-value fix in this document and it needs no new machinery: it binds an existing correct gate to store.require_mounted instead of to binder != nil.

reach = {swarm01}, anchor = {swarm01}, singleton, so the deployment is BOUND. shc move gitlab --node swarm02 finds the target outside the anchor and takes backup+restore, exactly as today.

Today there is no way to do this at all: DeploymentDir and MountsRoot are unconditional.

Terminal window
shc store add nas --driver nfs --server 10.0.0.9 --export /srv/shc --path /mnt/nas
shc store status nas
shc install huly --store nas

Each node in scope performs the client mount, the one thing the nfs driver adds over dir, exactly as NFSPlugin::activate_storage mounts if the export is absent from /proc/mounts and then calls SUPER::activate_storage. After that moment the store is indistinguishable from case 1. Each node then attests: mounted, writable, fsid, device 10.0.0.9:/srv/shc, witness contents, observed_at.

The witness cross-check is the one thing this model adds that Proxmox does not have. At store creation the leader writes a UUID to <locator>/.shc-store-id. Each node reports what it reads there. All-equal corroborates the shared claim; any disagreement marks the store as lying and refuses placement. Proxmox never checks this, and “lying up” about shared is its worst documented failure: migration skips the copy and the guest starts against a same-named path holding nothing.

Data lands at /mnt/nas/deployments/acme/huly/prod/mounts/db, literally a path mount, indistinguishable from local below the selector, which is what the owner asked for.

reach = {compose01, swarm01, swarm02}, so anchor is that set, no singleton, UNBOUND. shc move huly --node swarm01 finds the target inside the anchor, stops the source, re-renders on the target, starts. Zero bytes copied. SHC writes no placement constraint; a node.hostname== pin would destroy exactly the relocatability the share was bought for.

If swarm02’s attestation goes stale or reports unmounted, it silently leaves the anchor and a move to it is refused with the reason (“store nas last observed 41m ago, TTL 5m”) rather than approved on a link row and warned about.

Terminal window
shc store add hz-fsn1 --driver block --provider hcloud --location fsn1 --connection hetzner
shc install postgres --store hz-fsn1

shared is schema-refused on driver=block, the way LvmThinPlugin::options() simply omits shared so that lvmthin: x / shared 1 is rejected by the config schema rather than believed. Attestation for a block store asks whether the node can reach the plane’s control API with these credentials, not whether a path is present.

An app declaring x-shc-mounts: {pgdata: {size: 10Gi, access: rwo}} installs: the binder provisions, stamps volume.location = fsn1, BumpLeaseAndAttach performs the single monotonic compare-and-set, the node mounts the device at <state_home>/deployments/.../mounts/pgdata, and RequireMounted confirms it, the same gate as cases 1 and 2, now reached because the store said so rather than because the binder did.

reach(hz-fsn1) = {nodes in fsn1 carrying the credential}, but the attachment is on one node, so anchor = {swarm01} until a move re-attaches. Because the mount is exclusive and its reach is larger than one node, SHC must name which node holds it: a node.hostname== constraint plus replicas: 1, derived from the live attachment. That is ApplyVolumePlacement unchanged in mechanism and changed in justification: it is no longer a block special case, it is the one rule below applied to the one case that satisfies it.

shc move postgres --node swarm02 is pure token math against the persisted location, with no cloud round-trip: detach, lease bump, attach, mount, re-render. No archive staged.

SHC emits a placement constraint if and only if the mount is exclusive and its anchor is larger than one node.

Block: exclusive, reach many, so the pin is forced: without it the scheduler could place the task where the device is not. Share: concurrent, so no pin. Local: the anchor is already a singleton, so a pin adds nothing. This satisfies “unpinned means unpinned” without interpretation: SHC never adds a constraint the operator did not ask for, because the only constraint it adds is the one physics already imposed.

ApplyVolumePlacement today emits replicas: 1 and the hostname constraint in the same loop body, one mechanism doing two jobs. Splitting them is what makes the rule testable.

pgdata on hz-fsn1, cache on node:swarm01. anchor = {swarm01} ∩ {swarm01, swarm02} = {swarm01}; the cache silently un-relocates the whole stack. Declaring cache: {disposable: true} drops it from the intersection and restores anchor = {swarm01, swarm02}; it is re-carved empty on the target. That is a declaration, not a heuristic, and it is the same question the binding model’s Q7 asks from the durability side; see what this changes.

An empty intersection is refused at install, not discovered at move time.


The binding model defines the predicate; this model supplies one term of it.

reach(store) = { n : attestation(store, n).ok ∧ fresh } ∩ (store.nodes ?? all_nodes)
scope(m) = m has a volume row with a location ? { n : n.location = that location }
: m has a volume row without a location ? { the attachment node }
: all_nodes
anchor(m) = reach(store(m)) ∩ scope(m)

bound_to(D) = { n : ∃ m ∈ critical(D) with anchor(m) = {n} }, unchanged from the binding model, still a derived set, still nothing stored. BOUND is still bound_to ≠ ∅; unbound is still the complement. BOUND is not a flag and takes no code path; it is |anchor| = 1 at deployment scope, and it falls out of an intersection.

Reach is a resolver, not a column, and the reason is azure-disk. The tempting simplification is to give each block plane a row per region and read reach off the store. It fails on the provider whose driver reports no location: ce’s own comment names it: “local (or a block volume whose provider reports no location: azure-disk, or an hcloud/EBS bind whose location did not stamp): only the current holder node can serve the data”. That reach is per-volume and only knowable post-attach, so no store column can express it. Three implementations behind one signature; the call site sees one set intersection.

What this deletes. ClassifyMountDecision today has four arms selected by provider name and row shape: cross-provider, shared filesystem, location-scoped block, and local. Under this model three of the four collapse into targetNode ∈ anchor(m). The fourth (cross-provider) survives with a better name: cross-store, the case where the operator asked to re-home onto a different store, which is a copy by definition and can never be a rebind. That is one comparison plus one genuinely different operation, replacing a switch on a string.

One special case survives, and laundering it would be dishonest. A swarm docker-managed named volume names no store. Its bytes are on whichever node the task landed on, and SwarmServiceStates decodes {Name, Replicas, Image} with no node field, so SHC cannot say which. anchor = ∅, which is what an intersection over nothing gives you, and the binding model already names that state UNANCHORED and calls it a defect state rather than a posture. This model does not fix it; it makes it avoidable, because any mount that names a store has a known anchor. That narrows the binding model’s Q4 considerably; see below.


The fourth axis: archives are a content type

Section titled “The fourth axis: archives are a content type”

backup.repo is not a fourth axis. It is the same three questions with a different content type. Derived from the consumer side, and independently corroborated by Proxmox, which has no “backup storage” concept at all: backup is a content type on the same store object, laid out under dump/ beside images/<VMID>/ by one hash in the base plugin, and selected by capability: die "can't use storage '$storage' for backups - wrong content type\n" if (!$scfg->{content}->{backup}); in PVE/VZDump.pm.

Of the questions an archive target must answer, five are the questions a data store already answers: where do the bytes live; which nodes can read them (ce says this out loud in its own source: “The default backup.repo is a NODE-LOCAL directory, so a backup row (raft-replicated, visible cluster-wide) can reference snapshot bytes that exist only on the node that created it”, modules/backup/export_peers.go); is there room; is it reachable now; and the sixth, which is only expressible if both sides are the same kind of object.

So backup.repo becomes backup.store, a name resolved against the registry and gated on archive ∈ content. The restic URL moves to the store’s locator; the backup.aws_* / b2_* / azure_* / google_* keys move to the store’s connection_name, a soft credential reference that store.connection_name already carries and whose schema comment already describes exactly this use. Four namespaces for “where bytes live” become one identifier namespace with referential integrity.

The default is unchanged and deliberately so: the seeded node:<id> store carries content = {data, archive} and locator = {state_home}, so archives keep landing at {state_home}/backups/<tenant>/<stack>, byte-identical to today. Both estates keep working through the upgrade untouched.

Recommendation: the archives-on-the-restore-disk exposure, on its own

Section titled “Recommendation: the archives-on-the-restore-disk exposure, on its own”

This is live on both estates today, it is larger than the modelling question, and it should be ruled on separately from everything else in this document.

Second-hand from two independent live audits: backup.repo is null (core/config/default.yaml:188, verified), neither ops repository overrides it (verified by grep of both), so on both estates the restic repository sits on the same /dev/sda1 as the deployments it would restore. Measured: promanager 4.4G of deployments beside 629M of archives; bitspur 4.6G beside 1.1M. Every worker node carries its own separate node-local repository as well.

Worse, the operator cannot ask. shc config get backup.repo returns X400002VALINP key not found on both estates, because a null in default.yaml is not a declared default and the cascade read treats “no value anywhere and no declared default” as an unknown key (modules/config/commands/get.go). That is a bug independent of this model and should be fixed regardless of the ruling.

And there is a fifth same-device consumer nobody has counted. BackupsDir is hardcoded to <state_home>/backups/<tenant>/<stack> (daemon/backup_seam_adapters.go:588-590) and phases.go:210 uses it as the per-run working directory, irrespective of backup.repo. So pointing backup.repo at S3 does not move everything off the disk. The four others are deployments/, the repository itself, restore/<id> staging, and tmp/clone/ for the move archive, the last deliberately kept off tmpfs.

Three recommendations, in order of what each buys:

  1. Make it visible, do not refuse it. store.failure_domain is a derived opaque token (node:<id>:dev:<st_dev> for a path store, hcloud:fsn1 for a plane, s3:<region>:<bucket> for an archive) written by the registering node and refreshed by the attestor, never typed by an operator. Then “my archives share a failure domain with the data they would restore” is a one-line predicate. shc store ls shows it; a doctor diagnosis raises “N deployments’ archives share a failure domain with their data”. Compare failure domains, never path strings; two paths on one disk is one exposure, and showing paths teaches operators to “fix” it by picking another directory. Refusing would break both estates on upgrade for no safety gain.
  2. Fix shc config get backup.repo. The operator cannot currently ask where their archives go.
  3. Set it on both estates. Independent of this model: an archive store in a different failure domain is the only real answer to the relocation ceiling, and it is the thing the binding model’s own consolation depends on. Its --discard-local escape and its slice-5 recoverability report are offering a copy that today dies with the original.

The same predicate pays a second time as the free-space preflight the move heavy path completely lacks: there is no Statfs guard anywhere in modules/clone or modules/backup, so an over-large move fails mid-flight after the archive has consumed the disk, with the source already stopped (phase order is backup → stop → recover → uninstall). Once archive can name a different store the ceiling stops being structural; until then the preflight at least fails before stopping the source.


Shared storage means a deployment can be moved to another node without copying its data. It does not mean the deployment survives a dead node by itself.

That sentence belongs at the top of the operator documentation, because “shared storage” reads as high availability to every operator who has met it elsewhere, and that misread is how the recorded trilemma gets re-opened silently a year from now. SHC picks operability and reliability; ruling BUILD on rung 3 does not spend the third corner, provided the promise is stated this way, because every transition stays operator-initiated.

Three machines would be needed for the stronger promise, and this model builds none of them:

  1. A store-liveness authority that can declare a node dead. Attestation only decays to UNKNOWN and fails closed; it never adjudicates. The storage module registers exactly one cron, and it reaps expired attachment leases.
  2. A fencing authority for shares. lease_generation fences a device; detach is the fence. A shared directory has no device to detach, so “the node looks dead, start it elsewhere” would be two writers on one directory with nothing between them.
  3. An automatic actor. StopDeployment is deliberately “a normal StopStack, never a force-detach”, and the only presumed-dead route is an operator-invoked verb.

A share is concurrent at the filesystem, which is precisely why nothing else stops two replicas of MariaDB writing one datadir. Three facts, all verified at the pin:

  • ApplyScaleExtension clamps authored deploy.replicas down to x-shc-scale.max at render, and then strips the marker from the rendered compose (modules/app/scale_extension.go). Enforced once, then erased.
  • ApplyVolumePlacement forces replicas: 1, but only for RWO block mounts, and zero of 73 catalogue apps declare a block mount. It never fires.
  • Service.Scale validates req.Replicas == nil || *req.Replicas < 0 || *req.Replicas > 100 and nothing else (modules/app/service.go:865-868, read directly), then dispatches straight to docker service scale.

So shc stack scale <stack> mariadb 3 succeeds today against an app declaring max: 1. The owner’s non-bypassable ruling is currently unmet at HEAD, on both runtimes, and it is unmet independently of storage.

The fix belongs at the replica-deciding seam, not in the storage model:

effective_max = min( authored x-shc-scale.max , storage_ceiling )
storage_ceiling = 1 if any non-disposable mount is exclusive, else unbounded

checked in Service.Scale, re-reading the authored max from the packaged source compose that survives at <deployDir>/app/compose.yaml, which ScaleWindowOps already does, so this needs no schema change and no new file. 49 apps declare x-shc-scale across 62 max: 1 lines (measured). Under this model most of those become redundant rather than load-bearing, which is the right outcome: the constraint should come from the shape of the data, not from every app author remembering.

Storage does not make the catalogue clustered. Even on CephFS every mount defaults to rwo. A share moves Postgres; it does not replicate it.


  • It does not survive a dead node automatically. A node dies and its BOUND deployments stay down until an operator moves them. Deliberate; see the three missing machines above.
  • It cannot prove ongoing content identity on a shared store. The witness file proves every node saw the same tree at probe time. A split-brain re-export, or two SHC installs pointed at one export, can pass it. Proxmox documents the same boundary: “It is not advisable to use the same storage pool on different Proxmox VE clusters. Some storage operation need exclusive access”, because its locks are cluster-scoped and nothing spans two clusters. A dqlite-recorded claim means nothing to a second daemon writing the same directory.
  • It does not rescue an unanchored swarm named volume. No store, no anchor, data wherever the task landed, and swarmNamedVolumeBackupWarnings already says the capture gap exists. This model makes binding it to a store possible; it does not do it.
  • It does not model vapps/. preservedDeploymentState = {mounts, vapps} (verified, modules/app/install_runner.go:2080), and the in-file comment is explicit that vApps are per-instance state nothing re-derives, and that removing the directory both strands every instance and silently disarms shc protect. It is control-plane metadata, not app bytes, so it should not be a mount on a store, but under this model data relocates while render artifacts stay node-local, so a move must carry it. This is a real loose end and it is called out as an open question.
  • It cannot express capability that varies by image format. Proxmox needs a third dimension because qcow2-on-NFS snapshots and raw-on-NFS does not (die "can't snapshot this image format" if $format !~ m/^(qcow2|qed)$/). SHC has no image formats (a mount is a directory or a block device) so a per-store method refined by the mount’s own filesystem is enough. If SHC ever gains an image format, snapshot_method becomes the wrong shape. That is the tripwire.
  • It adds no free-space preflight. It makes the archive target selectable and the same-domain condition computable; someone still has to write the check.
  • It cannot detect two different NFS exports that resolve to one backing filesystem on the server. A path-containment refusal at store-add catches nested locators on one node, which Proxmox documents as a hazard (“Such an aliased storage configuration can lead to two different volume IDs pointing to the exact same disk image”) and does not detect. Beyond that, nothing.
  • Store names are cluster-unique while a path store’s meaning is per-node, which is why the seeded local stores are node:<id> rather than one shared local. On a three-node cluster shc store ls shows three local rows. That is honest (if the bytes differ by node it is not one store) but it is more rows than an operator expects.
  • It does not adopt an existing non-SHC dataset in place. A store’s locator is create-only and its layout is <locator>/deployments/....

Append-only throughout, and the tooling already enforces it: migrate/flatten self-refuses unless I_UNDERSTAND_FLATTEN_BREAKS_LIVE_ESTATES=1, with the comment that “this repo has learned twice that a documented-but-unenforced rule is worse than none”. Current history is five .sql files plus atlas.sum.

The migration is unusually cheap because there is nothing to convert. Second-hand from two live audits: shc storage ls is empty and shc volume ls is empty on both estates, deployment.storage is NULL on all 24 deployments, backup.repo is unset, and zero of 73 apps declare a block mount. All three rungs have zero installed base. This is a forward migration, not a data conversion, and that window will not stay open.

The DDL-mirror obligation is narrower than assumed. The shc storage and volume tests run ent’s own client.Schema.Create, so those tables have no mirror at all. The deployment mirror that does exist, tests/unit/modules/tenant/service_test.go, is a four-column minimum-viable subset (id, tenant_name, status, name) so adding deployment.store does not redden it. Verify by running the shc unit suite before and after; do not take this on trust.

Slice 1: the funnel. No schema. Output byte-identical.

Section titled “Slice 1: the funnel. No schema. Output byte-identical.”

One accessor, MountsRoot(tenant, stack, environment), returning paths.StateHome()-derived output for every deployment, exactly as today. Measured at the pin: 26 non-test sites construct a mounts path, across roughly 15 files. Route all of them. This slice is worth landing alone, because a bug in it is invisible and the only proof available is diffing rendered paths before and after.

Five sites need individual attention, not mechanical replacement:

  • modules/backup/manifest_inventory.go:91: Path: "mounts/" + name leaks the layout into the backup manifest format. This is an archive compatibility change, not a path change.
  • modules/backup/restore.go:1033, 1229, 1292: three filepath.Walk sites that each take the first directory named mounts and SkipDir. These do not fail when a root moves; they silently restore the wrong tree. They must take the resolved root. This is the worst site in the program and it appears in no grep for DeploymentDir.
  • shc/modules/supportbundle/certs.go:90: a hardcoded six-segment glob ending .../mounts/acme/*.json, which will return zero matches silently and quietly drop ACME key-presence reporting from every support bundle.

Three invariants survive unchanged because they compare basenames under a directory they were handed: preservedDeploymentState = {mounts, vapps}, separatelyArchivedDirs = {mounts, app}, orphanEnvSkipNames = {mounts, files} (all three verified at the pin). mounts/ stays a literal path segment in the formula, so nothing about them changes.

If the program stops here: nothing changed. One accessor exists.

Slice 2: schema and seed. Zero bytes move.

Section titled “Slice 2: schema and seed. Zero bytes move.”

Rename the ent type StorageProvider → Store, keeping the physical table via the entsql.Annotation{Table: "store"} that already exists; the logical rename is free, no DDL. Add driver, shared, nodes, content, require_mounted, state, failure_domain. Add store_attestation; drop store_node (empty on both estates). Add volume.store_id replacing free-text provider (retired in place). Add deployment.store; retire deployment.storage in place.

Seed, idempotently, at node self-register, the same place node.location is already stamped: one node:<id> store per node, driver=dir, locator={state_home}, shared=false, require_mounted=false, content={data, archive}, nodes=[self], plus that node’s own attestation. N stores, one per node, never one store with reach N; if the bytes differ by node it is not one store, and modelling it as one is Proxmox’s documented data-loss case. Backfill deployment.store for all 24 rows.

If the program stops here: shc store ls tells the truth for the first time. Nothing else changes, and no byte has moved.

Slice 3: evidence and gates. Still zero bytes moved.

Section titled “Slice 3: evidence and gates. Still zero bytes moved.”
  • Node-written attestations on a node-local cron; every path touch fork-isolated and deadline-bounded, per-store parallel. Reach reads attestations with a freshness bound; stale decays to UNKNOWN and UNKNOWN fails closed. nodeSupportsProvider (today a bare COUNT over the link table, matched by provider type across all rows, so any NFS row anywhere satisfies a question about a different store) is rewired. X503085SHFSMT, a warning standing in for a gate, becomes unreachable and is deleted.
  • store.require_mounted drives PrepOp.RequireMounted. Highest value per line in the whole program.
  • --storage → --store, validated against a row.
  • The replica ceiling moves to Service.Scale.
  • backup.store and content-type gating; failure_domain; the doctor diagnosis; the shc config get backup.repo fix.
  • One capability Method enum with a package-level exhaustiveness assertion. There are three independently-maintained lists today and the drift is already a live bug: probeLVM returns Instant: true for thin LVM while snapshots.Get("lvm") is not a registered method, so every LVM-thin host silently pays double-pass rsync for a capability it has. Adding CephFS to only one list would reproduce it exactly. The compile-time array-size assertion at subsystems/impl/volume_failover.go:75 is the house precedent for the technique.

If the program stops here: the system refuses honestly. One store per node, one archive target per node, everything visible, nothing relocatable that was not relocatable before.

Slice 4: a store may name a different locator. GATED. This slice moves bytes.

Section titled “Slice 4: a store may name a different locator. GATED. This slice moves bytes.”

Rung 1 lands here. The gate has four conditions, all four asserted on the post-injection artifact rather than on a guard’s return value:

  1. Three decoy mutation tests, red before green. Seed two stores; plant a decoy mounts/ directory the walk reaches first; assert the restore took the right tree, assert the relocated deployment’s mounts classify persistent rather than falling through to host binds, and assert the support-bundle glob returns non-zero ACME matches for a non-default-store traefik.
  2. The six single-root DeploymentsDir() consumers enumerate across stores.
  3. require_mounted asserted at render for any non-default store.
  4. Path-containment refusal at store-add.

The classifiers are kept, not deleted, and re-anchored. PersistentMountsFromCompose computes mountsPrefix := filepath.Join(deployDir, "mounts") + separator and hostbind.go computes the same deployPrefix inverted (both verified at the pin). Two designs proposed replacing both with a SELECT over a per-mount table. That is rejected, for the reason the binding model already gives: durability is derived, nothing stored. A derivation cannot drift; a row is a claim that every mount-creating path (install, recover, restore, both uninstall variants, vApp install and uninstall, migrate-from, group restore) must remember to write. Miss one and the mount becomes invisible to reach, invisible to the anchor, and invisible to backup capture while the operator is told the backup succeeded. So the classifiers keep deriving; they simply derive from a resolved root instead of an assumed one, and they must be re-anchored in this slice, not after it.

Which root do they resolve? The one recorded in the deployment’s rendered state at render time, not a fresh lookup. A store row edited after install must not retroactively re-anchor a deployment whose bytes are already somewhere.

--driver nfs|cephfs, the client mount, the witness cross-check, shc store status. Binding-model decision-table cells 2 and 7 become reachable for the first time.

volume.store_id becomes the selector, volume.provider retires in place, CanonicalVolumeProvider, which exists solely because a spelling drift “crept into READ sites … and silently missed every persisted row”, becomes deletable, and three of ClassifyMountDecision’s four arms collapse to targetNode ∈ anchor(m). Last, because block is the only rung with a working state machine (the bounded compare-and-set in modules/storage/volume_repo.go, the attachment lease, the compile-time ordering assertion) and zero users. Worst risk-to-reward to touch early.


What Proxmox got right, and what it got wrong

Section titled “What Proxmox got right, and what it got wrong”

Sources: the PVE admin guide chapter chapter-pvesm.html and its asciidoc source; the per-backend wiki pages; and upstream Perl at git HEAD of pve-storage, pve-manager and qemu-server on git.proxmox.com.

Taken:

  • A shared filesystem is a path, and after the mount it is the directory plugin. NFSPlugin and CephFSPlugin inherit 100% of layout and path computation from PVE::Storage::Plugin. This is the owner’s stated preference and upstream proves it works.
  • Create-only identity fields. DirPlugin::options() marks path => { fixed => 1 }; NFS marks path, server, export fixed. Where a store is is create-only; what it is allowed to do is editable.
  • nodes and shared are separate fields and neither derives from the other. nodes is an operator restriction; shared is a claim about content identity. The docs are explicit that they compose. Collapsing them into one reach number loses the restriction axis, which is the one operators use daily.
  • Content types, which is what absorbs the backup target. One store serves guest disks and dumps by listing content images,backup; selection is by capability.
  • is_mountpoint. “Assume the given path is an externally managed mountpoint and consider the storage offline if it is not mounted.”
  • Late-bound store-qualified identifiers. <STORAGE_ID>:<volume-name>; the absolute path is never stored and is recomputed at use time. Moving a store’s path rewrites zero guest configs.
  • Fork-isolated, deadline-bounded probes. Even a bare -d $path runs under run_fork_with_timeout with a 2-second deadline; df the same; NFS reachability 10 seconds with a TCP-ping fallback.
  • DECLARE, DISCOVER and OBSERVE as separate verbs: pvesm add, pvesm scan, pvesm status, none of them conflated with the config record.
  • Schema-level refusal where a flag is meaningless. LvmThinPlugin::options() omits shared, so shared 1 on a thin pool is rejected rather than believed.

Deliberately not taken:

  • is_mountpoint defaults to no. “directory is expected to be a mount point but is not mounted” is one of the most-searched PVE errors, precisely because the safe behaviour is opt-in. This model inverts the default: any store whose locator is not the node’s own state home requires it.
  • shared as a pure declaration with no witness. Proxmox never verifies it, and lying up about it is its worst documented failure. This model adds the witness file and a leader-side cross-check.
  • Per-technology plugin proliferation. PVE 9 dropped GlusterFS outright as unmaintained upstream, and the migration advice is to mount it by hand and add it as a directory storage. The mount-it-yourself dir path is the one that survives, so this model keeps the driver count minimal and makes dir fully capable.
  • A serial synchronous status sweep. One hung NFS server wedging pvestatd and greying out a whole cluster is the classic PVE outage. SHC’s attestor is per-store parallel and deadline-bounded from the start; retrofitting that after the first incident is not a plan.
  • No aliased-storage detection. Proxmox documents the hazard and does not detect it. SHC’s store rows are cheaper to create and its paths far likelier to nest, so a path-containment refusal at store-add is required.
  • Mixed content with no quota. “If a storage runs full, all guests using volumes on that storage receive IO errors. This can cause file system inconsistencies and may corrupt your data.” Mixing archives with live data is Proxmox’s default arrangement and its only defence is that paragraph. This model adds the failure-domain diagnosis and a per-store threshold.
  • Persisting no evidence. Proxmox probes synchronously at use time and memoises within one operation. That is simple and always current, and it is a genuine alternative to attestation; see the weakest point.

The weakest point, and what would overturn it

Section titled “The weakest point, and what would overturn it”

The classifiers keep deriving, and they now derive from a root that must agree with the root the renderer used.

This is the recommendation’s load-bearing choice, and it is the one I am least certain of. Rejecting a per-mount table keeps capture and durability derivation-driven, which is the property that has kept the fleet’s data intact through all the accretion: a derivation cannot drift because it re-reads the truth on every call. But it moves the failure into a narrower place rather than removing it: PersistentMountsFromCompose, hostbind.go and the three restore walks must all resolve the same mounts root the renderer wrote to. If a store row is edited, or if one path re-resolves live while another reads the recorded value, the classifier silently mis-anchors and the failure is quiet in exactly the direction that loses bytes.

The mitigation is in the design: record the resolved root in the deployment’s rendered state at install and have every classifier read that, never re-resolve, but the mitigation is itself a stored value that can drift from the filesystem, so it converts a two-party agreement into a two-party agreement with one honest witness. That is better, not solved.

What would overturn it: if per-mount store overrides turn out to be common rather than exceptional (a stack with pgdata on a block plane and the rest on a share is not exotic) then one recorded root per deployment stops being enough, every classifier needs a per-mount root, and a per-mount record becomes unavoidable. At that point the binding model’s “derive, never store” doctrine has to be re-opened with the owner rather than worked around, and the answer is probably a per-mount row plus a fail-closed drift detector that refuses the move and refuses the backup-succeeded claim whenever the rows and the filesystem disagree.

Two smaller weak points, named so they are not discovered later.

The freshness bound on attestation converts a correctness problem into an availability problem, and I am inventing the number. A node that is up but whose attestor is wedged silently leaves every anchor and stops being able to receive moves. That fails closed, which is right, but it is a control-plane failure mode caused by a monitoring loop rather than by storage, and it does not exist today. Proxmox’s alternative (probe synchronously at use time, persist nothing) is genuinely simpler, and every consumer of reach in SHC is a rare, synchronous, operator-initiated operation, so it would be sufficient. I chose persisted attestation for shc store status and the doctor diagnosis, which is a real buy; I have not measured the trade, and if the cluster peer transport cannot perform a bounded-deadline cross-node probe the synchronous option is not available anyway. That is worth checking before slice 3 is written.

The witness cross-check proves agreement at probe time and nothing after.


Every one of these contradicts a fact the task brief, one of the three candidate designs, or the panel treated as current. They are listed because each is load-bearing somewhere else.

  1. ★ The --storage one-flag data-loss path is already CLOSED upstream. ce 7f427cc (“fix(move): an unverifiable --storage claim can no longer relocate a deployment onto an empty volume”, 2026-08-02) landed before this document was written and is in the current shc pin. A Rebindable or Shared verdict now requires ProviderProven (a persisted volume row whose provider column is non-empty) on both the live and the dead-node classifier. The deployment stamp is still read as a fallback name, and its comment now says so in full, but it can no longer buy a no-copy rebind. Every candidate design and two panel lenses ranked “delete the positive read of --storage” as the highest-priority first slice on the grounds that it is the only live loss path. It is not live. Sequence the program without it.
  2. ★ The shared-filesystem arms of both move classifiers are dead code at HEAD, by that same commit’s analysis: nothing in the product can write a volume row for a shared filesystem; the binder writes rows only for block backends, and the ent Volume schema constrains access_mode to ('rwo','rox') with the comment “RWX stays on the shared-FS (cephfs/nfs) path and is never a volume row”. A test asserts the schema half against a real database. The arms were kept deliberately as a tombstone pending the owner’s ruling. This model is what makes them live again, and it does so by putting the bytes on the share rather than by trusting a label.
  3. ClassifyMountDecision has FOUR arms, not three. Cross-provider is first, and it is not a variant of the other three; it is a re-home onto a different store, which is a copy by definition. All three candidate designs said three and proposed collapsing all of them.
  4. isSharedFSProvider is already a derived predicate, backed by a single map in storageapi and two AST guards: one that reddens if a ProviderType is added without being classified, one that reddens on any string literal in the daemon package spelling a shared kind. Two designs proposed deleting a Type == NFS || Type == CephFS expression that upstream has already collapsed.
  5. ce/migrations holds FIVE .sql files, not four and not the “5” one audit reported before a later correction to four. A fifth (20260802033902_node_voter_slot.sql) landed 2026-08-02.
  6. The three first-match mounts walks in modules/backup/restore.go are at lines 1033, 1229 and 1292 at the pin, not 974 / 1170 / 1233. They are production restore code, not tests, and they fail by picking the wrong tree rather than by failing.
  7. A deployment column addition does NOT redden the shc DDL mirror. The mirror is a four-column minimum-viable subset. One audit stated the opposite and it changes the cost of the schema slice.
  8. BackupsDir is a per-run working directory, not the archive location. It is hardcoded to <state_home>/backups/<tenant>/<stack> and used by phases.go irrespective of backup.repo, so a remote backup.repo still leaves a same-disk consumer. One design flagged the hardcode but read it as “backup.repo does not move everything”; the precise reading is that this is a fifth consumer nobody counted.
  9. x-shc-scale counts: 49 apps, 62 max: 1 lines (measured at d6b3cb0). The brief said 48 apps and 62 declarations.
  10. bound_to does not exist in code. Confirmed by grep. It is this document’s and the binding model’s proposed term, a design to be made, not an invariant to preserve. That is good news: nothing has to keep working.
  11. The access_mode schema comment says both halves. One design cited “so a future ‘rwx’ addition doesn’t force a Go-code change” as the schema anticipating a widened volume table. The same paragraph also says RWX “is never a volume row”. Both are in the same comment, and this model takes the second half: shared filesystems are stores, not volumes.
  12. The relocation ceiling is not “~45% of the disk”. That number could not be confirmed. Cite the shape: a same-node relocation needs the dataset, the archive and the restored copy free at once (roughly one third); a cross-node one needs the dataset and the archive on the source (roughly one half). And there is no free-space guard at all, so the failure is mid-flight rather than a refusal.

The deployment binding model is the consumer of everything here. Its open questions, one by one.

Q1: --storage names three questions; which is real? ANSWERED, and its recommendation is superseded by the owner’s ruling. That analysis recommended FINISH / FINISH / RETIRE, retiring rung 3 on the grounds that its unique buy is high availability for stateful workloads, which the trilemma permanently gave up, and that its non-unique buy never arrives because relocateBlocked := nodeChanging && !swarm sends every compose relocation to backup+restore anyway. The owner ruled BUILD on rung 3. This model builds it and narrows the promise to relocatability, which does not spend the trilemma’s third corner, and it removes the second objection by construction, because a shared store is the case where a compose relocation genuinely has nothing to copy. The three rungs stop being three questions: one selector, --store <name>, names the place, and the driver decides how the place becomes bytes.

Q1’s vocabulary table is changed. It proposed four nouns for four axes: store, volume, share, repository. This model keeps two (store, volume) and retires share and repository as entity names, because four nouns for four kinds of one thing is the straddle in a new spelling. Q1’s negative rule survives verbatim and is stronger here: a shared kind must never be a legal value in a slot that means “which of this node’s disks”; under this model that slot does not exist, because the slot names a store and the store knows its own reach.

Q1’s amendment 1 is ANSWERED: backup.repo is a content type on a store, not a fourth axis. Its amendment 2 is ACCEPTED: rung 1 must not silently re-admit rung 3 through the back door, and the probe it demanded is exactly this model’s attestation, made mandatory in slice 3 before any store may name a non-default locator in slice 4.

Q4: should install auto-pin, or refuse? NARROWED. UNANCHORED arises when SHC cannot say which node holds a mount’s bytes. Any mount that names a store has a known anchor, so under this model the question survives only for the residue: docker-managed named volumes that name no store. That is a much smaller question with a much cheaper answer, and it removes the objection that auto-pinning makes swarm silently non-HA for stateful workloads; a stateful swarm service on a shared store needs no pin at all.

Q7: should there be a rebuildable durability class? OVERLAPS. This model’s per-mount disposable: true declaration in x-shc-mounts is the same question arriving from the storage side: a mount an operator is willing to lose should not collapse the anchor intersection. Q7 asks whether losing it should warn instead of refuse. The two should be ruled on together, and they should be one declaration, not two; otherwise an app author declares a cache throwaway for relocation and critical for removal, or the reverse.

Q2, Q3, Q5, Q6, Q8: UNTOUCHED. Whether the data-loss refusal is force-able, where durability is declared, who wins between a binding pin and an app’s own deploy.placement, whether “unbound but nowhere to run” warns or refuses, and whether cluster repair’s sweep survives are all deployment-scope questions this model neither answers nor complicates.

The decision table changes in one place. Cells 2 and 7 (an unbound deployment whose critical mounts sit on a shared filesystem) are recorded as unreachable, because the schema cannot express a shared data path. Slice 5 makes them reachable. That is the point of the exercise, and it should be re-verified against the table when slice 5 is planned.


S1. Refuse a new install whose archive store shares a failure domain with its data store, or only warn? The recommendation is warn, because refusing breaks both estates on upgrade and the exposure predates the model. “Warn on existing, refuse on new” is a coherent third option and it is the difference between a diagnostic and a guarantee. No prior ruling was found.

S2. Does a share hold only mounts/, or the whole deployment directory? This model says only mounts/, and stakes the render-collision argument on it. The three dead provider implementations all lay out {mounts, files, app}, implying the whole tree was once intended. Deciding otherwise changes slice 1’s blast radius from 26 sites to roughly 141 and puts the rendered compose on the share.

S3. Where does vapps/ ride? It is preserved per-instance state that is neither app bytes nor a render product. Under this model it stays node-local while data relocates, so a move must carry it explicitly, and it arguably belongs in the database rather than on disk. This is a genuine loose end.

S4. Persisted attestation, or synchronous probe at use time? Proxmox persists nothing and re-probes. Every consumer of reach in SHC is operator-initiated and synchronous, so the simpler option would work, at the cost of shc store status and the doctor diagnosis. Persisting adds a freshness constant nobody has data for and a new control-plane failure mode. Check first whether the cluster peer transport can perform a bounded-deadline cross-node probe; if it cannot, this question is closed by the transport.

S5. Is the x-shc-scale.max fix in scope here? It is a replica-path fix, not a storage fix, and it is currently unmet. It could land before slice 1 and be independently valuable.


  • The deployment binding model: durability, bound_to, the decision table, Q1’s three-rung analysis
  • The node addressing model: owns bind; this document does not touch that glossary
  • Proxmox VE storage: https://pve.proxmox.com/pve-docs/chapter-pvesm.html, https://pve.proxmox.com/wiki/Storage
  • pve-storage upstream: Plugin.pm, DirPlugin.pm, NFSPlugin.pm, CephFSPlugin.pm, LvmThinPlugin.pm at https://git.proxmox.com/?p=pve-storage.git
  • pve-manager PVE/VZDump.pm, qemu-server PVE/QemuMigrate.pm