Storage module
Layer 5. Objects. Objects
The storage module owns the storage provider inventory, the managed-provider control plane (NFS and CephFS singletons), and the capability matrix (quotas, snapshots, filesystem detection) that higher-level features read.
NOT IMPLEMENTED: providers do not hold deployment state. A deployment’s compose file, DATA mounts, generated files, and app definition are written under the node’s own state home:
paths.DeploymentsDir(), which is always<state_home>/deployments/.... No provider is consulted for that path.
storageapi.ProviderdeclaresBasePath()andCreateDeploymentFolder(), andlocal,nfsandcephfseach implement them, but nothing in either repo calls either method outside a unit test. Registering an NFS or CephFS provider mounts shared storage and records it in the inventory; it does not move any deployment onto it.Whether to make provider-held deployment state real or to retire the interface is an open question; see the deployment binding model proposal, Q1 and slice 6.
Provider kinds
Section titled “Provider kinds”| Kind | Cardinality | Shared | Managed | Base path |
|---|---|---|---|---|
local | many | no | no | /var/lib/shc (or configured) |
nfs | 1 per cluster | yes | yes | /mnt/shc/nfs |
cephfs | 1 per cluster | yes | yes | /mnt/shc/cephfs |
local is the only kind you can instantiate multiple times; each entry points at an existing mount on specific nodes and is registered as external (managed = false). NFS and CephFS are singletons with reserved names; the enable/disable lifecycle is the only way to create or remove them.
Module layout
Section titled “Module layout”Paths are relative to the pinned ce module.
modules/storage/├── register.go # module registration + lifecycle wiring├── service.go # StorageService (business logic)├── router.go # method_router endpoints at /api/method/shc.storage.*├── models.go # the store and store_node table constants├── interfaces.go # the seams the service consumes├── shim.go # registers local / NFS / CephFS providers, persists│ # the store rows├── shared_manager.go # SharedManager: cluster-singleton bookkeeping (NFS)├── shared_manager_cephfs.go # the CephFS half of the same lifecycle├── leader_forward.go # leader-aware dispatch for the shared-storage mutations├── connbridge.go # consumes a connection's vault-backed volume credentials├── crons.go # scheduled storage jobs├── doctor.go # `shc doctor` checks for storage providers├── volume_service.go # `shc volume ls/rm/rebind`├── volume_repo.go # block-volume + attachment rows├── volume_stonith.go # default (unconfigured) STONITH agent├── volume_rebind_gate.go # the volume-rebind edition gate├── storageapi/ # Provider interface, error codes, provider names├── volumeapi/ # block-volume registry, STONITH seam, types├── quotas/ # quota enforcement (xattr, XFS project, ZFS, scan)├── snapshots/ # snapshot engines (btrfs, zfs, rsync)├── hostrun/ # host command execution helper├── providers/ # one package per backend│ ├── local/ # local FS; also owns filesystem + capability detection│ ├── nfs/ # NFS│ ├── cephfs/ # CephFS│ ├── volume/ # shared block-volume plumbing│ ├── cloud/ # blank-import aggregator for the cloud backends│ └── awsebs/ azuredisk/ cinder/ cloudstackvol/ dovolume/ gcepd/│ hcloud/ linodevolume/ vultrvolume/├── ceph/ # Ceph cluster primitives (internal plumbing)│ ├── cmd.go # RunCeph / RunSystemctl subprocess helpers│ ├── config.go # CephClusterConfig struct│ ├── fsid.go # cluster fsid resolution│ ├── health.go # CephHealthMonitor│ ├── osd.go # CephOSDManager│ ├── mon.go # CephMonitorManager│ ├── mds.go # CephMDSManager│ ├── cephadm.go # cephadm wrapper (containerised daemons, SSH orch)│ ├── bootstrap.go # multi-node bootstrap (leader side)│ ├── peer.go # peer enrollment (SSH key + labels over mesh)│ ├── client.go # mon resolution / cephx client material│ ├── leave.go # node-leave teardown│ ├── platform_guard.go # host-platform preconditions│ ├── util.go # shared helpers│ └── manager.go # CephFSManager (orchestrator)├── nfs/ # NFS exports + cleanup│ ├── exports.go│ └── cleanup.go└── commands/ # CLI entrypoints ├── main.go # `shc storage` ├── add.go # `shc storage add` ├── list.go # `shc storage ls` ├── show.go # `shc storage show` ├── delete.go # `shc storage rm` ├── link.go # `shc storage link` + `shc storage unlink` ├── managed.go # `shc storage {nfs,cephfs} {enable|disable|status|join|leave}` └── daemon_client.go # daemon transport for the aboveThe ceph/ subpackage is deliberately a wrapper around real Ceph primitives. Files in it are named after the Ceph daemon they manage and call ceph / systemctl directly via ceph/cmd.go. Callers outside the subpackage must go through CephFSManager; the service and router never reach for ceph subprocesses.
Naming convention: ceph vs cephfs
Section titled “Naming convention: ceph vs cephfs”The two words are not interchangeable:
| Word | Meaning in SHC | Where it appears |
|---|---|---|
ceph | Low-level cluster primitives (OSD, MON, MDS, MGR) | modules/storage/ceph/, subprocess args, systemd units, xattrs, config defaults |
cephfs | The SHC-managed shared filesystem product | Provider kind, CLI root (shc storage cephfs), API routes (shc.storage.cephfs.*), request/response Go structs, mount path |
The orchestrator class is CephFSManager (the product) even though it composes CephOSDManager, CephMonitorManager, CephMDSManager, and CephHealthMonitor (the primitives). Anything user-facing is cephfs; anything talking to the actual cluster is ceph.
API surface
Section titled “API surface”All endpoints are POST, registered on method_router, and live at /api/method/:
| Endpoint | Purpose |
|---|---|
shc.storage.add | Register external local provider |
shc.storage.list | List providers (filterable by type/node) |
shc.storage.get | Show one provider |
shc.storage.delete | Remove provider (force opt-in) |
shc.storage.link | Attach external provider to a node |
shc.storage.unlink | Detach external provider from a node |
shc.storage.nfs.enable | Bring up managed NFS |
shc.storage.nfs.disable | Tear down managed NFS |
shc.storage.nfs.status | NFS server + mount state |
shc.storage.nfs.join | Pin NFS server role to a node |
shc.storage.nfs.leave | Release NFS server role |
shc.storage.cephfs.enable | Bring up managed CephFS (multi-node bootstrap) |
shc.storage.cephfs.disable | Tear down managed CephFS |
shc.storage.cephfs.status | Cluster/MDS/OSD health |
shc.storage.cephfs.join | Contribute an OSD |
shc.storage.cephfs.leave | Remove an OSD (optionally forced) |
shc.storage.pool.ls | Every storage pool and how full it is |
All endpoints are guarded by RequirePermissions(...) with platform:manage or stack:read and scoped to the tenant context.
How full volumes and pools are
Section titled “How full volumes and pools are”Every block provider (LINSTOR, Incus and each cloud’s) mounts a volume’s filesystem root at
<state home>/block-volumes/<volume id> on the node holding it. Each node runs the node-local cron
storage_usage once a minute: it statfs’s the roots the lease loop says it holds, its own state home, and,
where its ceph CLI answers, the SHC-managed Ceph cluster and its OSDs. It upserts one kv row of its own
(namespace storage.usage, keyed by node id, expiring after five minutes). Nothing measures on a request.
shc.storage.volume.lsandshc volume lscarryused_bytes,available_bytesandusage_measured_atper volume (the table prints USED and USE%;-o jsoncarries the bytes). All three are absent for a volume no node measured in the last five minutes.- The stack read’s
disk_bytesis the sum of its volumes’used_bytes, absent when none was measured, and the metrics rollup publishes it asshc_stack_disk_usage_bytes. shc.storage.pool.lsandshc storage pool ls(platform admin) list four kinds of pool:linstor(each storage pool on each node, from the controller of every volume connection, memoised for 30 s),ceph(the cluster and its OSDs),runtime(the pool each node’s runtime makes instance roots on, the row placement already keeps) andlocal(each node’s state home filesystem). A pool’stotal_bytes, the table’s SIZE, is its size as df prints it, a filesystem’s root reserve included. A local pool’sused_bytesandfree_bytesare df’s Used and Avail; a runtime pool’s used is its size less its free bytes, since placement’s row keeps no more, so on ext4 it counts the root reserve as used. USE% is used over used plus free, rounded up, as df’s Use% is.shc doctor:storage.volume-usageandstorage.pool-usageboth warn at 80% and fail at 95%. A volume’s percent is used over used plus available, as df counts it.storage.pool-usagealso names a pool source it could not read.
Managed-provider lifecycle
Section titled “Managed-provider lifecycle”Both NFS and CephFS follow the same lifecycle shape:
stateDiagram-v2 [*] --> absent absent --> enabled: enable enabled --> absent: disable enabled --> enabled: join/leave (capacity or role changes) enabled --> degraded: node failure degraded --> enabled: rebalance / re-elect- NFS:
enableinstalls kernel NFS on a single voter node, exports/mnt/shc/nfs, and mounts it on every other node.joinpins the server role to a specific node;leavereleases that pin and re-elects a server. - CephFS:
enablebootstraps a Ceph cluster (MON/MGR on voters, MDS on voters), creates a filesystem, and mounts it at/mnt/shc/cephfs.join --deviceadds a local OSD;leave --osd-iddrains and removes one.disabletears that Ceph cluster down.
Automated multi-node CephFS enable is implemented via cephadm: the leader bootstraps the cluster (ceph/bootstrap.go, ceph/cephadm.go), distributes the cephadm SSH key + placement labels to peers over the mesh (ceph/peer.go, daemon/services_ceph_peer.go), and brings up MON/MGR/MDS placement before creating the filesystem and mounting /mnt/shc/cephfs. The whole path is unit-tested through the injected Runner, but it carries VALIDATION CAVEAT comments for a reason: the real bootstrap flags, SSH-over-mesh plumbing, and kernel mounts are only truly exercised on live nodes, and no rig scenario runs them yet; treat it as unvalidated until one is green on a ≥3-node cell.
Node-leave integration
Section titled “Node-leave integration”LeaveLocalNode in modules/storage/ceph/leave.go tears down this node’s Ceph footprint when the node leaves: drain local OSDs, remove the local mon, stop the local mds, and stop the local ceph targets. Failures are warnings on LocalLeaveResult.
Capability matrix
Section titled “Capability matrix”The local provider (providers/local/provider.go) maps on-disk filesystem types to SHC capability flags:
| Filesystem | Quota method | Snapshot method |
|---|---|---|
| XFS | project quota | rsync fallback |
| Btrfs | subvolume quota | btrfs snapshots |
| ZFS | dataset quota | zfs snapshots |
| CephFS | xattr quota | cephfs snapshots |
| NFS | none (scan only) | rsync fallback |
| ext4/other | scan only | rsync fallback |
Provider rows carry quota_method and snapshot_method so StackService can decide at install time whether a deployment can satisfy its quota/snapshot requirements or must fall back to scan/rsync.
What --storage does today
Section titled “What --storage does today”shc install --storage <name> records the name. It does not place anything.
The value is written to the deployment.storage column and to the storage
field of the deployment’s manifest.json sidecar. ReadManifest does not
read a state.json beside it. Two things read
it back:
-
The clone / move classifiers (
daemon/clone_move_adapters.go,daemon/clone_dead_move_adapters.go) read it as the backend behind any persistent host-bind mount that carries no block-volume row; an empty value reads aslocal. The label is not proof of a provider. Both carry aProviderProvenfact, set only inside the branch that read aVolumerow AND got a non-empty provider column off it, and a shared-filesystem verdict (Rebindablelive,DeadMountSharedon the dead-node path) requires it. Live, an unprovennfs/cephfslabel degrades toNotRebindable, which forces the whole deployment onto backup+restore and so carries the bytes. On the dead-node path a label with no row at all isDeadMountLost, refused unless--discard-localnames the loss; a label whose row names no provider instead falls through to the block/topology rules, which fail closed toDeadMountUnreachable. Each case names the unverifiable claim in the plan warnings, so the slower answer reads as deliberate. Note the consequence of the paragraph below: a shared label always mismatches the resolved block provider, so the binder never writes a row for one, and at HEAD those shared arms are therefore unreachable in production. -
BuildVolumeBinder(daemon/volume_binder.go) returns nil (the host-bind path) when a non-empty--storagenames something other than the resolved block-volume provider. Only one of its four construction sites can see the flag. The daemon-served install threads it (modules/app/install_async.go→app.ResolveVolumeBinder→ theVolumeBinderProviderseam the daemon fills at boot). The other three pass a hardcoded empty string:shc install(daemon/services_seams_stack.go, whose own comment says--storage“isn’t plumbed onto StackInstallRequest yet”), the restore-time provision attacher (daemon/services_backup_config.go), andshc volume rm|rebind(daemon/volume_ops.go).shc install,shc updateandshc uninstallgo through shcd. With shcd unreachable they refuse withX503738DMNREQand change nothing.shc updatenever reaches the resolve at all, becauseinstall_async.go:139is the seam’s ONLY call site in either repo, so it has no binder to decline with in the first place. No comment states a reason for the update path’s omission; treat it as unstated, not as a decision.The decline is also conditional on there being a block-volume backend to decline: with
storage.volumes.providerunset and no volume-capable connection, the binder is nil for every install and--storageselects nothing. And the resolved provider is always a block backend, so annfsorcephfslabel mismatches it and the deploy stays on the host bind with noVolumerow written, the fact the move classifiers were rebuilt around.
Nothing else consults it. In particular the deployment’s files land in
<state_home>/deployments/... either way.
Omitting --storage leaves the column empty, which the classifiers read
as local. There is no tenant, environment or cluster default.
ResolveDefaultStorage and StorageConfig.default_storage are not in
the tree. --storage records a name. It does not place the deployment.
Whether that name becomes a placement input is
the deployment binding model proposal,
Q1.
See Also
Section titled “See Also”- Storage Commands: user-facing CLI
- File Locations: on-disk layout
- Data Model: deployment state schema
- Node Architecture: storage-provider matrix per node
- Backup System: backs up DATA mounts from the node’s state home
- Deployment binding model, the
--storagegap on this page